Archive for the ‘Linux’ Category

Enable printing from Windows and Macs remotely through Linux Print server – Share Brother Printer DCP-1610W with Linux CUPS and Samba Windows Share

Thursday, March 23rd, 2017

Brother-Printer-model-DCP-1610W-printer-scanner-and-copier-in-one-picture
I've recently bought a new Printer model Brother DCP 1610W and as in my home I have already a small Linux router and a web server where this blog and a couple of other websites runs and I need multiple PC / notebook / mobile phone enabled people to print on the Printer easily pretty much like a Printing server for a Small Office environment.

To do that of course I needed it configured to be accessible remotely for print via LAN and Wireless network. The task is not a complex one and printing remotely over the network is a standard thing many company organizations / universities and univerities does for quite some time and hence nowadays most printers are network connect ready so you just have to place them inside your home or corporate network and use the time to configure them via their web configuration interface or even some have their own embedded wifi adapter, as well as many printers nowdays can even be ready to print directly by just connecting the Printer to the Wi-Fi network and installing its drivers on a Win host.

Anyhow the most common way for both home printer configurations and corporate I'm aware of still is to Share the printer via Windows Server or Win Server Domain so anyone connected to the Network to be able to Add the printer via Winblows.

In the case i'm going to describe below my home the Wi-Fi router is connected to an 5 Port Network Switch (HUB) which on its hand is connected to the Linux router which serves multiple things (a Linux router, a hosting server (web server and a database server hosted, a mail server, traffic proxy server, a firewall and a NAT router), I decided to Share the printer to Wi-Fi connected and LAN clients directly switched via an UTP cable to the switch by using the good old Linux Samba Sharing server.

I did not actually do that for a really long time hence before I started I did some quick research to get an idea on the general steps to partake to succeed in Sharing the Printer over the network of this Debian's Wiki SystemPrinting Guide was mostly helpful.

 

1. Downloading and Installing necessery Brother Printer deb packages
 

A small remark to make here is my Linux server is running Debian GNU / Linux and hence this article is giving details on how Printer can be Shared on Debian though a minor adaptation of the article should make it possible to install also on any RHEL / CentOS / SuSE etc. Redhat based RPM Linux distribution.)

First step to do is to download Brother printer vendor provided drivers as of moment of writting this article they're here

To download the drivers get the proper links and use wget or curl to download all the necessery .deb archives in lets say in /root/brother-printer-drivers e.g. before that create the folder with:
 

root@linux:/root# mkdir /root/brother-printer-drivers

Also it might be helpful for those who need some other Brother Printer Linux driver complete list of Brother Printer all Linux drivers as of time of writting this post is found on this URL here

Next you need to install following Brother printer driver deb packages brscan-skey brscan4 dcp1610wcupswrapper dcp1610wlpr

root@linux:/root# cd brother-printer-drivers
root@linux:/root/brother-printer-drivers# dpkg -i –force-all brscan-skey-0.2.4-1.amd64.deb

root@linux:/root# dpkg -i –force-all brscan4-0.4.4-1.amd64.deb

root@linux:/root# dpkg -i –force-all dcp1610wcupswrapper-3.0.1-1.i386.deb

root@linux:/root# dpkg -i –force-all dcp1610wlpr-3.0.1-1.i386.deb

root@linux/root# cd  ../


Once installed dpkg -l should show like so:
 

root@linux:/root# dpkg -l |grep -i brother
ii  brscan-skey                                0.2.4-1                      Brother Linux scanner S-KEY tool
ii  brscan4                                    0.4.4-1                      Brother Scanner Driver
ii  dcp1610wcupswrapper                        3.0.1-1                      Brother DCP-1610W CUPS wrapper driver
ii  dcp1610wlpr                                3.0.1-1                      Brother DCP-1610W LPR driver

Brother's vendor provided packages will install drivers under /opt/brother
 

root@linux:/root# ls -al /opt/brother/
общо 16
drwxr-xr-x 4 root root 4096 яну 26 13:58 ./
drwxr-xr-x 3 root root 4096 яну 26 13:55 ../
drwxr-xr-x 3 root root 4096 яну 26 13:58 Printers/
drwxr-xr-x 4 root root 4096 яну 26 13:58 scanner/

 

2. Installing CUPS Printing Service and related Filters and Postscript packages necessery for PDF processing on CUPS server side

 

root@linux:/root#  apt-get install –yes cups cups-client cups-common cups-pdf cups-ppdc foomatic-db foomatic-db-engine foomatic-filters foomatic-filters-ppds openprinting-ppds lpr hp-ppd hpijs cups-pdf ghostscript-cups

Your printing should work normally without cups-pdf and ghostscript-cups packages installed but I install them just in case if PDF processing is problematic you can skip that.

It is also useful to install sane and sane-utils packages if you're going to use the brother's scanner capabilities.

root@linux:/root# apt-get install –yes sane sane-utils

Note that considering that all packages installed fine and the CUPS service is running, this should have set a proper printer into /etc/printcap a short database used to describe printers. printcap file is being used by UNIX's spooling system and allows you to dynamic addition and deletion of printers, for Linux / *Nix hosts which have more than one printer connected and added in CUPs records for the various printer goes there.
With a single Brother DCP-1610W Printer like my case is you should have records similar to these:

root@linux:~/brother-printer-drivers# cat /etc/printcap
DCP1610W:\
        :mx=0:\
        :sd=/var/spool/lpd/DCP1610W:\
        :sh:\
        :lp=/dev/usb/lp0:\
        :if=/opt/brother/Printers/DCP1610W/lpd/filter_DCP1610W:

 

 

3. Adding a Printer in CUPS the easy way through CUPS Printing System Web Interface

 

CUPS has a nice web interface for setting up and administering printers and print queues.

Below is a selfexplanatory screenshot of Add Printer screen 

add-a-new-printer-cups-web-admin-interface-screenshot-in-a-firefox-browser  .

 

Use your favourite browser (Firefox, Opera, Chromium, lynx, elinks – yes the great news is console / terminal browsers are also supported well by cups web iface) to display interface and add a printer via the Administration screen. If you are asked for a username and password see here.

cups-web-admin-interface-accessed-in-browser-listing-brother-dcp1610w-printer-screenshot

There are three sections. The first is for local printers; that is, printers which are usually attached to the machine you are using. These are very often printers using a USB connection but can be parallel or serial port printers.

Adding a USB printer is a common occurance and one should automatically be detected as a local printer and a URI (Unified Resource Indicator) for its connection displayed on the next page.
 

The Other Network Printers section requires you to specify the destination for the remote print queue/printer, which could be on the local network or many kilometres away. AppSocket is almost always available on a network printer and other devices and requires only the IP address of the printer and a port number. An Internet Printing Protocol (IPP) URI is the preferrred choice for connecting to another CUPS server because it is CUPS' native protocol. ipp14 is the ipp backend from CUPS 1.4 and Debian-specific. It is provided because some devices do not work with the current ipp backend, which has a stricter adherence to the IPP standard. A Line Printer Daemon (LPD) URI could be considered if the remote printing service does not support IPP satisfactorily or at all. As before, when a remote print queue is doing the filtering choose RAW as the Make/Manufacturer.

 

4. Printer Status and Control testing whether CUPS printing is up and running


 

Once cups is installed and hopefully up and running you should see the cups process up and running to check it do:

root@linux:/root# /etc/init.d/cups status; ps axuwwf|grep -i cups|grep -v grep
Status of Common Unix Printing System: cupsd is running.
root      2815  0.0  0.0  75364  2912 ?        Ss   Mar17   0:00 /usr/sbin/cupsd -C /etc/cups/cupsd.conf

To get some further testing you can also use lpstat command and should get ouput similar to belows:
 

root@linux:/root# lpstat -t
scheduler is running
no system default destination
device for DCP1610W: usb://Brother/DCP-1610W%20series
DCP1610W accepting requests since Fri Mar 17 23:03:37 2017
printer DCP1610W disabled since Fri Mar 17 23:03:37 2017 –
        Unplugged or turned off

At the moment of issuing above command it shows printer is disabled because of moment of execution the printer was turned off for a while cause I was not needing it you should get usually an output of enabled and ready to print.

lpstat is also about to report whether a queue is accepting jobs and what is yet to be printed you can do

 

 

5. Install and Configure Samba Sharing Server on the Linux server


 

You can setup CUPS to allow Windows machines to print to a CUPS server using an http address.

First, install the samba package. When you are asked to use WINS, say yes.

root@linux:/root#  apt-get install samba

Next you might want to set setup your /etc/cups/cupsd.conf file by default CUPS would listen to LPD's port 631 if you don't have a strong firewall isolating you from the Internet on port 631 you might want to change that port to another one lets say to Port 49651.


I personally prefer keep the default port 631 and do use a robust firewall. If you want to change it modify config to something like:

# Our private connection we listen to Listen *:49651 # Allow machines on local network to use printers <Location /printers> Order allow,deny Allow 192.168.0.* Allow 192.168.1.* </Location>

If you like to filter access to CUPs daemon to receive Printing requests to be originating only from the local network place in smb.conf also something with your private network ranges:

# Allow machines on local network to use printers

<Location /printers>
Order allow,deny
Allow 192.168.0.*
Allow 192.168.1.*
Allow 192.168.2.*
</Location>
<Location />
  # Allow remote administration…
  Order allow,deny
##  Allow all
Allow 192.168.0.*
Allow 192.168.1.*
Allow 192.168.2.*
</Location>
<Location />
  # Allow remote administration…
  Order allow,deny
##  Allow all
Allow 192.168.0.*
Allow 192.168.1.*
Allow 192.168.2.*
</Location>
<Location /admin>
  # Allow remote administration…
  Order allow,deny
##  Allow all
Allow 192.168.0.*
Allow 192.168.1.*
Allow 192.168.2.*
</Location>

 

This will listen on port 49651 from any network. You may use some other port number besides 631. Note that the dynamic and/or private ports as specified by the IANA are in the range 49152 through 65535. Also, this will only allow computers from the local network to print to the CUPS printers.
 

6. Use CUPS Printing server to print over the network directly

 

 

Next you need to restart the CUPS daemon once again as it will be used for samba printing
 


# service cups restart


Now on each Windows machine, Choose that you want to install a network printer and that you want to install a printer on the Internet or home/office network. The URL you will use should be smth like:
 


http://<cups_server_hostname>:49651/printers/DCP1610W

 

Lastly, select the Brother downloaded from Internet or the one that's available on the Install CD, for any other vendor printer if it is lets say HP Printer or Canon to install use the respective provided driver or as a last resort use the Generic section driver labeled MS Publisher Color Printer.

 

 

7. Configure Samba to Share CUPS network enabled printer


I've done a minor changes in default installed /etc/samba/smb.conf to make the printer accessible from The Samba server here is the main things to consider changing:
 

# Change this to the workgroup/NT-domain name your Samba server will part of
   workgroup = WORKGROUP

#   security = user
security = share

[printers]
   comment = PC Freak Printer
   browseable = yes
   path = /var/spool/samba
   printable = yes
   guest ok = yes
   read only = yes
   create mask = 0700

# Windows clients look for this share name as a source of downloadable
# printer drivers
[print$]
   comment = Printer Drivers
   path = /var/lib/samba/printers
   browseable = yes
   read only = yes
   guest ok = yes


Next restart Samba server to make the new setting take affect:
 

root@linux:/# /etc/init.d/samba restart
Stopping Samba daemons: nmbd smbd.
Starting Samba daemons: nmbd smbd.
root@linux:/# ps axu|grep -E "smb|nmb"
root     21887  0.0  0.0 169588  1904 ?        Ss   16:53   0:00 /usr/sbin/nmbd -D
root     21892  0.0  0.0 197560  3272 ?        Ss   16:53   0:00 /usr/sbin/smbd -D
root     21894  0.0  0.0 197560  1564 ?        S    16:53   0:00 /usr/sbin/smbd -D
root     21899  0.0  0.0 112368   840 pts/6    S+   16:53   0:00 grep -E smb|nmb

root@linux:/#

Complete current smb.conf configuration I use to make the Brother Printer DCP 1610W accesible via network share is here

This section needs updating as you can setup print server via samba print sharing just by uploading drivers.

When printing to windows printers in an NT domain using SMB the Device URI should use similar to:

 

smb://username:password@domain/server/printername

 

This allows Samba to authenticate against a domain controller for acces to the printer queue.

In my case as you can see in below smb.conf configuration I've configured Samba security = share which will allow anyone to access the samba server without authentication so you can omit  username:password@ part

One good way to determine the printername  (in case you are not sure of) is to use smbclient command line tool. computername refers to the name of the machine that shares the printer:

 

smbclient -L copmputername


computername is the name of the samba server machine or its IP address


E.g.
 

hipo@linux:~$ smbclient -L //192.168.0.1/
Enter Attitude's password:
Domain=[WORKGROUP] OS=[Unix] Server=[Samba 3.5.6]

        Sharename       Type      Comment
        ———       —-      ——-
        print$          Disk      Printer Drivers
        IPC$            IPC       IPC Service (pcfreak server)
        DCP1610W        Printer   DCP1610W
Domain=[WORKGROUP] OS=[Unix] Server=[Samba 3.5.6]

        Server               Comment
        ———            ——-
        PCFREAK              pcfreak server

        Workgroup            Master
        ———            ——-
        WORKGROUP            WORKGROUP


Check the ouput for entries of Type "Printer":


The resulting (Linux / Mac OS) Samba Share access URL from the output above would be

smb://192.168.0.1/DCP1610W


 

 

8. Adding Printer to your Windows machines to enable actual Remote Samba Sharing printing

Assuming you already know the Printer share name, here is what I needed to do to have the Printer Added on each of Windows Desktop PCs and Notebooks

 

Control Panel -> Devices and Printers -> Add a printer -> (Add a Network wireless or bluetooth printer)

Then instead of Searching the printer to click on:

The printer that I wasn't listed

add-samba-network-share-brother-dcp-1610w-printer-to-windows-7-machine-no-printer-found-from-add-printer

Then type in the URL or IP (as in my case) leading to the printer as you see in below screenshot:

 

9. Printer Samba Sharing Using Macintosh notebook as the Client and Debian as the Server

 

1. Assuming you have cups to set up the printer on Debian as described above.

2. On the Mac (OS X 10.4+) start Print and Fax from System Preferences. Use the + button to add a printer.

3. Look first in the "Default" tab. If the automagic printer-sharing has worked, and your Mac is connected to the local network properly, then the Debian-based printer should already be visible in the list.

Just select it and use the recommended print driver. If you face problems you can try to play with
Gutenprint Printer drivers to make it printing.

4. If your printer is not visible in the Default tab, then try adding it on the "IP" tab.

Pick IPP as the protocol, give the plain IP address of the server in the address box (in my case that's 192.168.0.1, and in the Queue box put
"printers/DCP1610W

Put whatever helps you identify the printer in the Name and Location boxes (fields), and choose a printer driver than matches Brother DCP1610W or with another printer installed whatever you used to set up the printer on Debian .
Finally Pray that God help you to make it work and press the Add button. If you prayed honestly and repenting for your sins perhaps you will have mercy and it will work, of course if not try to research online on how to fix it further by God's grace.

Note that making printing work on Mac is a little bit of tricky and it might cause you some extra effort / nerves to complete.

 

10. Some other Useful maintanance commands you might need in future CUPS Printer queue jobs maintance

 

For displaying or setting print queue options and defaults:

lpoptions -p <print_queue_name> -l

Stopping and starting print queues. Acceptance and rejection of jobs sent to a destination:
 

cupsdisable <print_queue_name>
cupsenable <print_queue_name>
cupsaccept <print_queue_name>
cupsreject <print_queue_name>


To Cancel all jobs on a destination and additionally delete job data files:

cancel -a <print_queue_name>
cancel -a -x <print_queue_name>

That's all folks, Thanks God the printer should be working. Enjoy!

Share this on

Show directory structure bash script on Linux howto – See hierarchical directory tree structure one liner shell script

Friday, February 24th, 2017

show-directory-structure-see-hierarchical-directory-tree-structure-on-linux-with-tree-command-and-with-bash-shell-scripts

If you have Sys Adminned Linux or *Nix OS like, whether for some shell scripting purpose or just for sake of keeping a backup you should have definitely come
into some need to list a tree of a directories content in a hierarchical order.

The most obvious way to do that on Linux is by simply using:

1.  "tree" command (not installed by default on most Linux distributions so in order to have it on Deb / Debian based Linux do:
 

# apt-get install –yes tree


On Fedora / CentOS Redhat Linux (RHEL) etc. install with:

# yum –yes install tree

 

By the way for those that needs tree on FreeBSD / BSD UNIX, tree is also available on that platform you can install it with:
 

pkg_add -vr tree


Then simply check man tree to get idea on how to use it, the easiest way to use the command tree once package is installed is to run tree inside directory of choice, i.e.

 

$ cd /somedir
$ tree -a

.
├── acpi
│   ├── events
│   │   └── powerbtn-acpi-support
│   └── powerbtn-acpi-support.sh
├── adduser.conf
├── adjtime
├── aliases
├── alternatives
│   ├── ABORT.7.gz -> /usr/share/postgresql/9.5/man/man7/ABORT.7.gz
│   ├── aclocal -> /usr/bin/aclocal-1.11
│   ├── aclocal.1.gz -> /usr/share/man/man1/aclocal-1.11.1.gz
│   ├── ALTER_AGGREGATE.7.gz -> /usr/share/postgresql/9.5/man/man7/ALTER_AGGREGATE.7.gz
│   ├── ALTER_COLLATION.7.gz -> /usr/share/postgresql/9.5/man/man7/ALTER_COLLATION.7.gz
│   ├── ALTER_CONVERSION.7.gz -> /usr/share/postgresql/9.5/man/man7/ALTER_CONVERSION.7.gz
 

 

To get a list of only directories with tree use:
 

$ tree -d /

 

  │   ├── bin
│   │   ├── boot
│   │   │   └── grub
│   │   │       └── locale
│   │   ├── disk
│   │   │   ├── Books
│   │   │   │   ├── 200 E-BOOKS
│   │   │   │   │   ├── McGraw-Hill – Windows Server 2003
│   │   │   │   │   ├── Oreilly.Access.Cookbook.2nd.Edition-LiB
│   │   │   │   │   ├── Oreilly.ActionScript.Cookbook.eBook-LiB
│   │   │   │   │   ├── OReilly.ActionScript.The.Definative.Guide.WinAll.Retail-EAT
│   │   │   │   │   ├── Oreilly.Active.Directory.2nd.Edition.eBook-LiB
│   │   │   │   │   ├── Oreilly.Active.Directory.Cookbook.eBook-LiB
│   │   │   │   │   ├── Oreilly.ADO.Dot.NET.Cookbook.eBook-LiB
│   │   │   │   │   ├── Oreilly.Amazon.Hacks.eBook-LiB
│   │   │   │   │   ├── OREILLY.ANT.THE.DEFINITIVE.GUIDE-JGT
│   │   │   │   │   ├── Oreilly.Apache.Cookbook.eBook-LiB
│   │   │   │   │   ├── Oreilly.AppleScript.The.Definitive.Guide.eBook-LiB
│   │   │   │   │   ├── Oreilly.ASP.Dot.NET.In.A.Nutshell.2nd.Edition.eBook-LiB
│   │   │   │   │   ├── OReilly.Better.Faster.Lighter.Java.Jun.2004.eBook-DDU
│   │   │   │   │   ├── Oreilly.BLAST.eBook-LiB
│   │   │   │   │   ├── OReilly.BSD.Hacks.May.2004.eBook-DDU
│   │   │   │   │   ├── Oreilly.Building.Embedded.Linux.Systems.eBook-LiB

If you have a colorful terminal and you like colors for readability the -C option is quite handy

 

$ tree -C /

 

tree-command-linux-hierarchical-structure-directory-tree
 

To list the directory tree with permissions included use tree cmd like so:

 

$ tree -L 2 -p /usr

/usr/
├── [drwxr-xr-x]  bin
│   ├── [-rwxr-xr-x]  [
│   ├── [lrwxrwxrwx]  2to3 -> 2to3-2.6
│   ├── [-rwxr-xr-x]  2to3-2.6
│   ├── [-rwxr-xr-x]  411toppm
│   ├── [-rwxr-xr-x]  7z
│   ├── [-rwxr-xr-x]  7za
│   ├── [-rwxr-xr-x]  a2p
│   ├── [-rwxr-xr-x]  ab
│   ├── [-rwxr-xr-x]  ac
│   ├── [lrwxrwxrwx]  aclocal -> /etc/alternatives/aclocal
│   ├── [-rwxr-xr-x]  aclocal-1.11
│   ├── [-rwxr-xr-x]  acpi


Another truly handy option of tree is to list the directory structure index with included file sizes information

 

$ tree -L 2 -sh /bin

/bin
├── [903K]  bash
├── [147K]  bsd-csh
├── [ 30K]  bunzip2
├── [681K]  busybox
├── [ 30K]  bzcat
├── [   6]  bzcmp -> bzdiff
├── [2.1K]  bzdiff
├── [   6]  bzegrep -> bzgrep
├── [4.8K]  bzexe
├── [   6]  bzfgrep -> bzgrep
├── [3.6K]  bzgrep
├── [ 30K]  bzip2
├── [ 14K]  bzip2recover
├── [   6]  bzless -> bzmore
├── [1.3K]  bzmore
├── [ 51K]  cat
├── [ 59K]  chgrp
├── [ 55K]  chmod
├── [ 63K]  chown
├── [ 10K]  chvt
├── [127K]  cp
├── [134K]  cpio
├── [  21]  csh -> /etc/alternatives/csh
├── [104K]  dash


To list a directory tree of a search pattern, lets say all files with .conf extensions use:
 

$ tree -P *.conf

/etc/ca-certificates.conf [error opening dir]
/etc/dante.conf [error opening dir]
/etc/debconf.conf [error opening dir]
/etc/deluser.conf [error opening dir]
/etc/discover-modprobe.conf [error opening dir]
/etc/fuse.conf [error opening dir]
/etc/gai.conf [error opening dir]
/etc/gpm.conf [error opening dir]
/etc/gssapi_mech.conf [error opening dir]
/etc/hdparm.conf [error opening dir]
/etc/host.conf [error opening dir]
/etc/idmapd.conf [error opening dir]
/etc/inetd.conf [error opening dir]
/etc/insserv.conf [error opening dir]
/etc/irssi.conf [error opening dir]
/etc/kernel-img.conf [error opening dir]
/etc/ld.so.conf [error opening dir]
/etc/libao.conf [error opening dir]
/etc/libaudit.conf [error opening dir]
/etc/logrotate.conf [error opening dir]
/etc/memcached.conf [error opening dir]
/etc/mke2fs.conf [error opening dir]
/etc/mongodb.conf [error opening dir]
/etc/mtools.conf [error opening dir]
/etc/multitail.conf [error opening dir]
/etc/nsswitch.conf [error opening dir]
/etc/ntp.conf [error opening dir]
/etc/ocamlfind.conf [error opening dir]

 

 

tree -I option does exclude all petterns you don't want tree to list

Here are few other tree useful options:

  • tree -u /path/to/file – displays the users owning the files
  • tree -g /path/to/file – display the groups owning the files
  • tree -a /path/to/file – display the hidden files/folders
  • tree -d /path/to/file – display only the directories in the hierarchy


However there might be some cases where you have to support a Linux server or you just have to write a script for a non-root user and you might not have the permissins to install the tree command to make your life confortable. If that's the case then you can still use a couple of command line tools and tricks (assuming you have permissions) to list a log a directory / files and subdirectories tree structure in a hierarchical tree like command order

2. Print a list of all sub-directories and files within a directory tree

To print all directories within any path of choise on a server use
 

$ find /path/ -type d -print

 

To print all files within a root filesystem hierarchically with find command

Another way to do it in a more beautiful output is by using find in conjunction with awk
 

$ find . -type d -print 2>/dev/null|awk '!/\.$/ {for (i=1;i<NF;i++){d=length($i);if ( d < 5  && i != 1 )d=5;printf("%"d"s","|")}print "—"$NF}'  FS='/'

 

|—bashscripts
|          |—not-mine
|          |—various
|          |—output
|          |—examples
|          |—fun
|          |—educational
|          |—backdoor-cgi
|          |—fork-bombs
|          |—tmp
|          |    |—old
|          |—bullshits
|—packages
|       |—ucspi-ssl-0.70.2
|       |               |—package
|       |               |—compile
|       |               |      |—rts-tmp
|       |               |—command
|       |               |—src
|—bin
|—package
|      |—host
|      |    |—superscript.com
|      |    |              |—command
|—mnt
|    |—tmpfs
|    |—disk
|    |—flash_drive
|    |—ramfs
|    |—cdrom

3. Get a list of the directories on filesystem structure with one-liner ls + sed script
 

$ ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//–/g' -e 's/^/ /' -e 's/-/|/'

 

 |-bin
 |-boot
 |-dev
 |—net
 |—pts
 |-downloads
 |—autorespond-2.0.5
 |—–debian
 |—deb-packages
 |—–IP-Country-2.28
 |——-bin
 |——-blib
 |———arch
 |———–auto

….
 

4. Print all files within root filesystem and issue any command on each of the files
 

ls -R1 / |    while read l; do case $l in *:) d=${l%:};; "") d=;; *) echo "$d/$l";; esac; done


Above command just prints all the found files with full-path if you want to check the file size or file type you just check echo command with any command you need to execute on each of the listed file

5. Get a list hierarchical directory Linux tree with bash shell scripts: Assuming that the server where you need to have a list of the directory filesystem structure in a tree fashion has bash you could use this little script called tree.sh to do the job or for a full filesystem hierarchical tree like directory structure use fulltree.sh

Share this on

Install and make Apache + PHP to work with PosgreSQL database server on Debian Linux and set up server Web Posgre interface Pgpadmin howto

Wednesday, June 15th, 2016

make-apache-php-work-with-postgresql-pgsql-and-install-postgresql-db-web-admin-interface

In previous article I've wrote on how to install postgresql on Debian Linux using the deb repository this was necessery to import some PostGres DBs, however this was not enough to run the posgresql php based website aimed as connection from Apache / PHP module to PostGre was failing after a bit of investigation and a check in phpinfo(); I've realized the module PHP module for postgres pgsql.so was missing, here is what I did in order to install it:
 

debian:~# apt-get install php5-pgsql phppgadmin libapache2-mod-auth-pgsql 

PHP sessions enable configuration

As it is common a common problem with PHP applications written to use PostGres is to loose sessions and by default PHP does not have configured sessions.save_path it is a very good practice to directly enable it in /etc/php5/apache2/php.ini open the file in text editor:
 

debian:~# vim /etc/php5/apache2/php.ini


Find the commented directive line:
 

;session.save_path = “/tmp”


and uncomment it, i.e.:
 

session.save_path = “/tmp”


Quit saving vim with the usual :wq!

The 3 modules provides pgsql.so for PHP and mod_auth_pgsql.so for Apache2, the 3rd packae phpgadmin provides a Web administration interface for installed PostgreSQL servers Databases, for those experienced with MySQL Database its the same as PHPMyAdmin.

 

 Here is quick configuration for use of PostgreAdmin interface:

By default PHPPGADMIN installation process configure the Apache2 server' /etc/phppgadmin/apache.conf  to use  /etc/apache2/conf.d/phppgadmin


Here is the default my server package instaleld  file content:

 

Alias /phppgadmin /usr/share/phppgadmin

<Directory /usr/share/phppgadmin>

DirectoryIndex index.php
AllowOverride None

order deny,allow
deny from all
allow from 127.0.0.0/255.0.0.0 ::1/128
# allow from all

<IfModule mod_php5.c>
  php_flag magic_quotes_gpc Off
  php_flag track_vars On
  #php_value include_path .
</IfModule>
<IfModule !mod_php5.c>
  <IfModule mod_actions.c>
    <IfModule mod_cgi.c>
      AddType application/x-httpd-php .php
      Action application/x-httpd-php /cgi-bin/php
    </IfModule>
    <IfModule mod_cgid.c>
      AddType application/x-httpd-php .php
      Action application/x-httpd-php /cgi-bin/php
    </IfModule>
  </IfModule>
</IfModule>

</Directory>

It is generally a good practice to change the default Alias location of phppgadmin, so edit the file and change it to something like:
 

Alias /phppostgresgadmin /usr/share/phppgadmin

 

  • Then phpPgAdmin is available at http://servername.com/phppostgresadmin (only from localhost, however in my case I wanted to be able to access it also from other hosts so allowed PostgresGadmin from every hosts, to do so, I've commented in above config

 

# allow from 127.0.0.0/255.0.0.0 ::1/128

 

and uncommented #allow from all line, e.g.:
 

allow from all


Also another thing here is in your VirtualHost whenever you plan to access the PHPPGADMIN is to include in config ( in my case this is the file /etc/apache2/sites-enabled/000-default before (</VirtualHost> end line) following Alias:
 

Alias /phpposgreadmin /usr/share/phppgadmin


Then to access PostGreSQL PHP Admin interface in Firefox / Chrome open URL:

 

http://your-default-domain.com/phpposgreadmin

phpPgAdmin-postgresql-php-web-interface-debian-linux-screenshot
 

 

Configure access to a remote PostgreSQL Server

With PhpPgAdmin, you can manage many PostgreSQL servers locally (on the localhost) or on remote hosts.

First, you have to make sure that the distant PostgreSQL server can handle your request, that you can connect to it. You can do this by modifying the /etc/postgresql/9.5/main/filepg_hba.conf and adding a line like:

# PhpPgAdmin server access host all db_admin xx.xx.xx.xx 255.255.255.255 md5

Then, you need to add your distant PostgreSQL server into the config file for PhpPgAdmin. This file is  /etc/phppgadmin/config.inc.php the default postgresql port is 5432, however you might have configured it already to use some different port if you're not sure about the port number the postgresql is listening check it out:

 

debian:~# grep -i port /etc/postgresql/*/main/postgresql.conf
etc/postgresql/9.5/main/postgresql.conf:port = 5433                # (change requires restart)
/etc/postgresql/9.5/main/postgresql.conf:                    # supported by the operating system:
/etc/postgresql/9.5/main/postgresql.conf:                    # supported by the operating system:
/etc/postgresql/9.5/main/postgresql.conf:# ERROR REPORTING AND LOGGING


To login to phppgadmin interface there is no root administrator user such as in PHP so you will need to priorly create some user and later use it for connection from Postgres Web interface.

To create from console new user in postgres:
 

debian:~# su – postgres
posgres@debian:~$ psql template1
posgres@debian:~$ psql -d template1 -U postgres

 

Welcome to psql 9.5, the PostgreSQL interactive terminal. Type: \copyright for distribution terms \h for help with SQL commands \? for help on internal slash commands \g or terminate with semicolon to execute query \q to quit template1=#

template1=# CREATE USER MyNewUser WITH PASSWORD 'myPassword';


To add a new database to postgres from shell:

template1=# CREATE DATABASE NewDatabase;
 

template1=# GRANT ALL PRIVILEGES ON DATABASE NewDatabase to MyNewUser;

 

template1=# q

Last command instructs it to quit if you want to get more info about possible commands do type instead of q ? for general help or for database / table commands only h
If you need to connect to NewDatabase just to test first it works in console before trying it from postgrepgadmin

 

 

 

 

 

posgres@debian:~$ psql -d NewDatabase -U MyNewUser

 

 

 

 

Share this on

Install postgresql on Debian Squeeze / How to install PostGreSQL on Obsolete Debian installation

Friday, June 10th, 2016

how-to-install-postgresql-on-obsolete-old-debian-squeeze-tutorial

If you're in position like me to be running an old version of Debian (Squeeze) and you need to install PostgreSQL you will notice that the Debian 6.0 standard repositories are no longer active and apt-get update && apt-get upgrade are returning errors, thus because this Debian release is already too old and even the LTS repositories are inactive it is impossible to install postgresql with the usual.

To get around the situation first thing I did was to try to add followin Debian  repositories. to /etc/apt/sources.list
 

deb http://ftp.debian.net/debian-backports squeeze-backports-sloppy main
deb http://archive.debian.org/debian-archive/debian/ squeeze main contrib non-free
deb http://archive.debian.org/debian-archive/debian/ squeeze-lts main contrib non-free

After adding it I continued getting missing package errors while trying:
 

# apt-get update && apt-get install postgresql postgresql-client
….
…..

 

E: Some index files failed to download. They have been ignored, or old ones used instead.


Thus I googled a bit and I found the following PostgreSQL instructions working Debian 7.0 Wheeze and decided to try it 1 in 1 just changing the repository package wheezy word with squeeze
in original tutorial postgre's deb repositories are:

 

deb http://apt.postgresql.org/pub/repos/apt/ wheezy-pgdg main


I've only changed that one with:

 

deb http://apt.postgresql.org/pub/repos/apt/ squeeze-pgdg main

 

I guess though this worked for Debian Squeeze installing current versions such as Debian 8.0 Jessis and newer wouldn't be a prolem if you just change the debian version keyword witht he distribution for which you need the postgresql package


Here is all the consequential steps I took to make the PostgreSQL 9.5 running on my old and unsupported Debian 6.0 Squeeze

Create /etc/apt/sources.list.d/pgdg.list. The distributions are called codename-pgdg. In the example, replace wheezy with the actual distribution you are using:

# vim /etc/apt/sources.list.d/pgdg.list

 

deb http://apt.postgresql.org/pub/repos/apt/ squeeze-pgdg main

debian:~# apt-get –yes install wget ca-certificates debian:~# wget –quiet -O – https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add – debian:~# apt-get update debian:~# apt-get upgrade debian:~# apt-get –yes install postgresql-9.5 pgadmin3

Next step is to connect to PostGreSQL and create database user and a database # su – postgres $ psql

Create a new database user and a database:

postgres=# CREATE USER mypguser WITH PASSWORD 'mypguserpass'; postgres=# CREATE DATABASE mypgdatabase OWNER mypguser;

 

or

# createuser mypguser #from regular shell # createdb -O mypguser mypgdatabase

Quit from the database

postgres=# q

Connect as user mypguser to new database

# su – mypguser $ psql mypgdatabase

or

# psql -d mypgdatabase -U mypguser

If you get errors like:

psql: FATAL: Ident authentication failed for user "mypguser"

edit pg_hba.conf in /etc/postgresql/9.5.Y/main/pg_hba.conf

 

local all all trust # replace ident or peer with trust

reload postgresql

/etc/init.d/postgresql reload …

 


To make sure that PostGreSQL is running on the system check the following processes are present on the server:

 

 

 

root@pcfreak:/var/www/images# ps axu|grep -i post postgres 9893 0.0 0.0 318696 16172 ? S 15:20 0:00 /usr/lib/postgresql/9.5/bin/postgres -D /var/lib/postgresql/9.5/main -c config_file=/etc/postgresql/9.5/main/postgresql.conf postgres 9895 0.0 0.0 318696 1768 ? Ss 15:20 0:00 postgres: checkpointer process postgres 9896 0.0 0.0 318696 2700 ? Ss 15:20 0:00 postgres: writer process postgres 9897 0.0 0.0 318696 1708 ? Ss 15:20 0:00 postgres: wal writer process postgres 9898 0.0 0.0 319132 2564 ? Ss 15:20 0:00 postgres: autovacuum launcher process postgres 9899 0.0 0.0 173680 1652 ? Ss 15:20 0:00 postgres: stats collector process root 14117 0.0 0.0 112404 924 pts/1 S+ 16:09 0:00 grep -i post

 

 


Well that's all folks now you will have the postgresql running on its default port 5433:

 

debian:/etc/postgresql/9.5/main# grep -i port postgresql.conf
port = 5433 # (change requires restart)
# supported by the operating system:
# supported by the operating system:
# ERROR REPORTING AND LOGGING # %r = remote host and port

 

 


Well that's it folks thanks The Lord Jesus Christ grace by the prayers of John The Baptist and Saint Sergij Radonezhki it works 🙂

 

 

Share this on

How to force logrorate process logs / Make logrotate changes take effect immediately

Sunday, April 10th, 2016

how-to-force-logrorate-to-process-logs-make-logrorate-changes-take-effect-immediately-log-rotate-300x299

Dealing with logrorate as admins we need to change or add new log-rorate configurations (on most Linux distributions configs are living uder
/etc/logrotate.d/
 

logrotate uses crontab to work. It's scheduled work, not as daemon, so usually no need to reload its configuration.
When the crontab executes logrotate, it will use your new config file automatically.

Most of the logrotate setups I've seen on various distros runs out of the /etc/cron.daily

$ ls -l /etc/cron.daily/logrotate
-rwxr-xr-x 1 root root 180 May 18  2014 /etc/cron.daily/logrotate

Here is content of cron job scheduled script:

$ cat /etc/cron.daily/logrorate

#!/bin/sh /usr/sbin/logrotate /etc/logrotate.conf EXITVALUE=$? if [ $EXITVALUE != 0 ]; then /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]" fi exit 0

Configurations change to lograte configs takes effect on next crontab run,
but what If you need to test your config you can also execute logrotate
on your own with below ommand:

 

logrotate -vf /etc/logrotate.conf 

If you encounter some issues with just modified or newly added logrorate script to check out the status of last logrorate executing bunch of log-rotate scripts run on Debian  / Ubuntu etc. deb based Linux:

cat /var/lib/logrotate/status

Or on RHEL, Fedora, CentOS Linux


cat /var/lib/logrotate.status

logrotate state -- version 2

 

"/var/log/syslog" 2016-4-9
"/var/log/dpkg.log" 2016-4-1
"/var/log/unattended-upgrades/unattended-upgrades.log" 2012-9-20
"/var/log/unattended-upgrades/unattended-upgrades-shutdown.log" 2013-5-17
"/var/log/apache2/mailadmin.pc-freak.net-access.log" 2012-9-19
"/var/log/snort/portscan.log" 2012-9-12
"/var/log/apt/term.log" 2016-4-1
"/var/log/squid/access.log" 2015-3-21
"/var/log/mysql/mysql-slow.log" 2016-4-9
"/var/log/debug" 2016-4-3
"/var/log/mysql.log" 2016-4-9
"/var/log/squid/store.log" 2015-3-21
"/var/log/apache2/mailadmin.pc-freak.net-error.log" 2012-9-19
"/var/log/daemon.log" 2016-4-3
"/var/log/munin/munin-update.log" 2016-4-9
"/var/log/unattended-upgrades/unattended-upgrades*.log" 2013-5-16
"/var/log/razor-agent.log" 2015-2-19
"/var/log/btmp" 2016-4-1
"/var/log/squid/*.log" 2014-11-24
"/var/log/munin/munin-graph.log" 2016-4-9
"/var/log/mysql/mysql.log" 2012-9-12
"/var/log/munin/munin-html.log" 2016-4-9
"/var/log/clamav/freshclam.log" 2016-4-3
"/var/log/munin/munin-node.log" 2016-1-23
"/var/log/mail.info" 2016-4-3
"/var/log/apache2/other_vhosts_access.log" 2016-4-3
"/var/log/exim4/rejectlog" 2012-9-12
"/var/log/squid/cache.log" 2015-3-21
"/var/log/messages" 2016-4-3
"/var/log/stunnel4/stunnel.log" 2012-9-19
"/var/log/apache2/php_error.log" 2012-10-21
"/var/log/ConsoleKit/history" 2016-4-1
"/var/log/rsnapshot.log" 2013-4-15
"/var/log/iptraf/*.log" 2012-9-12
"/var/log/snort/alert" 2012-10-17
"/var/log/privoxy/logfile" 2016-4-3
"/var/log/auth.log" 2016-4-3
"/var/log/postgresql/postgresql-8.4-main.log" 2012-10-21
"/var/log/apt/history.log" 2016-4-1
"/var/log/pm-powersave.log" 2012-11-1
"/var/log/proftpd/proftpd.log" 2016-4-3
"/var/log/proftpd/xferlog" 2016-4-1
"/var/log/zabbix-agent/zabbix_agentd.log" 2016-3-25
"/var/log/alternatives.log" 2016-4-7
"/var/log/mail.log" 2016-4-3
"/var/log/kern.log" 2016-4-3
"/var/log/privoxy/errorfile" 2013-5-28
"/var/log/aptitude" 2015-5-6
"/var/log/apache2/access.log" 2016-4-3
"/var/log/wtmp" 2016-4-1
"/var/log/pm-suspend.log" 2012-9-20
"/var/log/snort/portscan2.log" 2012-9-12
"/var/log/mail.warn" 2016-4-3
"/var/log/bacula/log" 2013-5-1
"/var/log/lpr.log" 2012-12-12
"/var/log/mail.err" 2016-4-3
"/var/log/tor/log" 2016-4-9
"/var/log/fail2ban.log" 2016-4-3
"/var/log/exim4/paniclog" 2012-9-12
"/var/log/tinyproxy/tinyproxy.log" 2015-3-25
"/var/log/munin/munin-limits.log" 2016-4-9
"/var/log/proftpd/controls.log" 2012-9-19
"/var/log/proftpd/xferreport" 2012-9-19
"/var/spool/qscan/qmail-queue.log" 2013-5-15
"/var/log/user.log" 2016-4-3
"/var/log/apache2/error.log" 2016-4-3
"/var/log/exim4/mainlog" 2012-10-16
"/var/log/privoxy/jarfile" 2013-5-28
"/var/log/cron.log" 2016-4-3
"/var/log/clamav/clamav.log" 2016-4-3

 

The timestamp date next to each of the rotated service log is when the respective log was last rorated

It is also a handy thing to rorate only a certain service log, lets say clamav-server, mysql-server, apache2 and nginx
 


logrorate /etc/logrorate.d/clamav-server
logrorate /etc/logrorate.d/mysql-server
logrotate /etc/logrotate.d/nginx

Share this on

Check linux install date / How do I find out how long a Linux server OS was installed?

Wednesday, March 30th, 2016

linux-check-install-date-howto-commands-on-debian-and-fedora-tux_the_linux_penguin_by_hello

To find out the Linux install date, there is no one single solution according to the Linux distribution type and version, there are some common ways to get the Linux OS install age.
Perhaps the most popular way to get the OS installation date and time is to check out when the root filesystem ( / ) was created, this can be done with tune2fs command

 

server:~# tune2fs -l /dev/sda1 | grep 'Filesystem created:'
Filesystem created:       Thu Sep  6 21:44:22 2012

 

server:~# ls -alct /|tail -1|awk '{print $6, $7, $8}'
sep 6 2012

 

root home directory is created at install time
 

 

server:~# ls -alct /root

 

root@server:~# ls -lAhF /etc/hostname
-rw-r–r– 1 root root 8 sep  6  2012 /etc/hostname

 

For Debian / Ubuntu and other deb based distributions the /var/log/installer directory is being created during OS install, so on Debian the best way to check the Linux OS creation date is with:
 

root@server:~# ls -ld /var/log/installer
drwxr-xr-x 3 root root 4096 sep  6  2012 /var/log/installer/
root@server:~# ls -ld /lost+found
drwx—— 2 root root 16384 sep  6  2012 /lost+found/

 

On Red Hat / Fedora / CentOS, redhat based Linuces , you can use:

 

rpm -qi basesystem | grep "Install Date"

 

basesystem is the package containing basic Linux binaries many of which should not change, however in some cases if there are some security updates package might change so it is also good to check the root filesystem creation time and compare whether these two match.

Share this on

Migrate Webserver and SQL data from old SATA Hard drive to SSD to boost websites performance / Installing new SSD KINGSTON 120GB hard disk on Linux

Monday, March 28th, 2016

ssd-linux-migrate-webserver-and-mysql-from-old-SATA-to-SSD-Kingston-Hard-drive-to-boost-performance-installing-new-SSD-on-Debian-linux
Blog and websites hosted on a server were giving bad performance lately and the old SATA Hard Disk on the Lenovo Edge server seemed to be overloaded from In/Out operations and thus slowing down the websites opeining time as well as SQL queries (especially the ones from Related Posts WordPress plugin was quite slow. Sometimes my blog site opening times were up to 8-10 seconds.

To deal with the issue I obviously needed a better speed of I/O of hard drive thus as I've never used SSD hard drives so far,  I decided to buy a new SSD (Solid State Drive) KINGSTON SV300S37A120G, 605ABBF2, max UDMA/133  hard disk.
Mounting the hard disk physically on the computer tower case wasn't a big deal as there are no rotating elements of the SSD it doesn't really matter how it is mounted main thing is that it is being hooked up somewhere to the case.

I was not sure whether the SSD HDD is supported by my Debian GNU / Linux so I had see whether Linux Operating System has properly detected your hard disk use dmesg

1. Check if SSD Hard drive is supported in Linux

 

linux:~# dmesg|grep -i kingston
[    1.182734] ata5.00: ATA-8: KINGSTON SV300S37A120G, 605ABBF2, max UDMA/133
[    1.203825] scsi 4:0:0:0: Direct-Access     ATA      KINGSTON SV300S3 605A PQ: 0 ANSI: 5

 

linux:~# dmesg|grep -i sdb
[    1.207819] sd 4:0:0:0: [sdb] 234441648 512-byte logical blocks: (120 GB/111 GiB)
[    1.207847] sd 4:0:0:0: [sdb] Write Protect is off
[    1.207848] sd 4:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[    1.207860] sd 4:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.207928]  sdb: unknown partition table
[    1.208319] sd 4:0:0:0: [sdb] Attached SCSI disk

 

Well great news as you see from above output obviously the Kingston SSD HDD was detected by the kernel.
I've also inspected whether the proper dimensions of hard drive (all 120 Gigabytes are being detected by the OS):

 

linux:~# fdisk -l /dev/sdb
Disk /dev/sdb: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders, total 234441648 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Even better as the proper HDD sizing was detected by Linux kernel.
Next thing to do was of course to create ext4 filesystem on the SSD HDD.
I wanted to give 2 separate partitions for my Webserver Websites DocumentRoot directories which all lay under the standard Apache location inside /var/www as well as MySQL data folder which is also under the standard Debian based Linuces – /var/lib/mysql as the SQL data directory was just 3.3 GB size, I've decided to reserve 20GB gigabytes for the MySQL and another 100 GB for my PHP / CSS / JS / HTML and other data files /var/www.
 

2. Create SSD partitions with cfdisk

Hence I needed to create:

1. SSD partition of 100GB
2. SSD partition of 20GB

I have cfdisk installed and I believe, the easiest way to create the partitions is using interactive partitioner as CFDISK instead of fdisk: so in order to make the proper partitions I've ran

 

linux:~# cfdisk /dev/sdb


I' will skip explainig details on how to use CFDISK as it is pretty standard – display or manipulate disk partition table tool.
Just press on NEW button (moving with arrow keys buttons) and choose the 2 partitions size 100000 and 20000 MB (one thing to note here is that you have to choose between Primary and  Logical creation of partitions, as my SSD is a secondary drive and I already have a ) and then press the
WRITE button to save all the partition changes.

!!! Be very careful here as you might break up your other disks data make sure you're really modifying the SSD Hard Drive and not your other /dev/sda or other attached external Hard drive or ATA / SATA disk.
Press the WRITE button only once you're absolutely sure, you do it at your own (always create backup of your other data and don't blame me if something goes wrong) …

Once created the two partitions will look like in the screenshot below:
creating-linux-partitions-with-cfdisk-linux-partitioning-tool.png

 


3. Create ext4 filesystem 100 and 20 GB partitions

Next thing to do before the two partitions are ready to mount under Webserver's files documentroot /var/www and /var/lib/mysql is to create ext4 filesystem, though some might prefer to stick to ext3 or reiserfs partition, I would recommend you use ext4 for the reason ext4 according to my quick research is said to perform much better with SSD Hard Drives.

The tool to create the ext4 filesystems is mkfs4.ext4 it is provided by debian package e2fsprogs I have it already installed on my server, if you don't have it just go on and install it with:
 

linux:~# apt-get install –yes e2fsprogs

 

To create the two ext4 partitions run:
 

linux:~# mkfs4.ext4 /dev/sdb5

 

linux:~# mfs4.ext4 /dev/sdb6


Here the EXT4 filesystem on partition that is supposed to be 100 Gigabytes will take 2, 3 minutes as the dimensions of partition are a bit bigger, so if you don't want to get boring go grab a coffee, once the partitions are ready you can evaluate whether everyhing is properly created with fdisk you should get output like the one below

 

linux:~# fdisk -l /dev/sdb
Disk /dev/sdb: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders, total 234441648 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1              63   234441647   117220792+   5  Extended
/dev/sdb5             126    39070079    19534977   83  Linux
/dev/sdb6        39070143   234441647    97685752+  83  Linux

 

4. Mount newly created SSD partitions under /var/www and /var/lib/mysql

Before I mounted /var/www and /var/lib/mysql in order to be able to mount under the already existing directories I had to:

1. Stop Apache and MySQL server
2. Move Mysql and Apache Documentroot and Data directories to -bak
3. Create new empty /var/www and /var/lib/mysql direcotries
4. Copy backpups ( /var/www-bak and /var/lib/mysql-bak ) to the newly mounted ext4 SSD partitions

To achieve that I had to issue following commands:
 

linux:~# /etc/init.d/apache2 stop
linux:~# /etc/init.d/mysql stop

linux:~# mv /var/www /var/www-bak
linux:~# mv /var/lib/mysql /var/lib/mysql-bak

linux:~# mkdir /var/www
linux:~# mkdir /var/lib/mysql
linux:~# chown -R mysql:mysql /var/lib/mysql


Then to manually mount the SSD partitions:
 

linux:~# mount  /dev/sdb5 /var/lib/mysql
linux:~# mount /dev/sdb6 /var/www


To check that the folders are mount into the SSD drive, ran mount cmd:

 

linux:~# mount
/dev/sda1 on / type ext3 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
/dev/sdc1 on /backups type ext4 (rw)
/dev/sdb5 on /var/lib/mysql type ext4 (rw,relatime,discard,data=ordered))
/dev/sdb6 on /var/www type ext4 (rw,relatime,discard,data=ordered))

 

That's great now the filesystem mounts fine, however as it an SSD drive and SSD drives are being famous for having a number of limited writes on disk before the drive lifetime is over it is a good idea to increase a bit the lifetime of the SSD by mounting the SSD partitions with noatime and errors=remount-ro (in order to not log file access times to filesystem table and to remount the FS read only in case of some physical errors of the drive).

5. Configure SSD partitions to boot every time Linux reboots

Now great, the filesystems gets mounted fine so next thing to do is to make it automatically mount every time the Linux OS boots up, this on GNU / Linux is done through /etc/fstab, for my 2 ext4 partitions this is the content to add at the end of /etc/fstab:

 

/dev/sdb5               /var/lib/mysql      ext4        noatime,errors=remount-ro       0       1
/dev/sdb6               /var/www        ext4    noatime,errors=remount-ro       0       1

 

quickest way to add it without a text editor is to echo to the end of file:
 

linux:~# cp -rpf /etc/fstab /etc/fstab.bak_25_03_2016
linux:~# echo ' /dev/sdb5               /var/lib/mysql      ext4        noatime,errors=remount-ro,discard       0       1' >> /etc/fstab
linux:~# echo ' /dev/sdb6               /var/www        ext4    noatime,errors=remount-ro,discard       0       1 ' >> /etc/fstab


Then mount again all the filesystems including the 2 new created SSD (100 and 20 GB) partitions:
 

linux:~# umount /var/www
linux:~# umount /var/lib/mysql
linux:~# mount -a


To assure properly mounted with noatime and remount-ro on errors options:


linux:~# mount | grep -i sdb
/dev/sdb5 on /var/lib/mysql type ext4 (rw,noatime,errors=remount-ro)
/dev/sdb6 on /var/www type ext4 (rw,noatime,errors=remount-ro)

 

It is also a good idea to check a statistics of disk free command:
 

linux:~# df -h|grep -i sdb
/dev/sdb5         19G  0G    19G  0% /var/lib/mysql
/dev/sdb6         92G   0G    92G  0% /var/www


6. Copy all Webserver and SQL data from backupped directories to new SSD mounted

Last but not least is to copy all original content files from /var/www-bak and /var/lib/mysql-bak to the new freshly  created SSD partitions, though copying the files can be made with normal linux copy command (cp),
I personally prefer rsync because rsync is much quicker and more efficient in copying large amount of files in my case this were 48 Gigabytes.

To copy files from with rsync:

 

linux:~# rsync -av –log-file /var/log/backup.log  /var/www-bak /var/www
linux:~# rsync -av –log-file /var/log/backup.log  /var/lib/mysql-bak /var/lib/mysql


Then ofcourse, finally to restore my websites normal operation I had to bring up the Apache Webservers and MySQL service

 

linux:~# /etc/init.d/apache2 start
linux:~# /etc/init.d/mysql start


7. Optimizing SSD performance with periodic trim (discard of unused blocks on a mounted filesystem)

As I digged deeper into how to even further optimize SSD drive performance I learned about the cleaning action TRIM of the partitions for a long term performance proper operation, to understand it better think about trimming like Windows degrament operatin.
 

NAME
fstrim – discard unused blocks on a mounted filesystem

SYNOPSIS
fstrim [-o offset] [-l length] [-m minimum-free-extent] [-v] mountpoint

DESCRIPTION
fstrim is used on a mounted filesystem to discard (or "trim") blocks which are not in use by the filesystem. This is useful for
solid-state drives (SSDs) and thinly-provisioned storage.

By default, fstrim will discard all unused blocks in the filesystem. Options may be used to modify this behavior based on range or
size, as explained below.


Trimming is really necessery, otherwise SSD become very slow after some time. All modern SSD's support TRIM, but older SSD's from before 2010 usually don't.
Thus for an older SSD you'll want to check this on the website of the manufacturer.

As I mentioned earlier TRIM is not supported by all SSD drives, to check whether TRIM is supported by SSD:

linux:~# hdparm -I /dev/sdb|grep -i -E 'trim|discard'
                  *          Data set Management TRIM supported (limit 1 block)

It's easiest to let the system perform an automatic TRIM. That can be done in several ways.

The quickest way for trimming is to place into /etc/rc.local trim  commands, in my case it was the following commands:

 

fstrim -v /var/lib/mysql
fstrim -v /var/www

To add it I've used my favourite vim text editor.
Adding commands to rc.local will make SSD trimming be executed at boot time so this will reduce a bit the downtime during the trim with some time so perhaps for those like me which are running a crually important websites a better

An alternative way is to schedule a daily cron job to do just place a new job in /etc/cron.daily/trim e.g.:
 

linux:~# vim /etc/cron.daily/trim

 

#!/bin/sh
fstrim -v /var/lib/mysql
fstrim -v /var/www

linux:~# chmod +x /etc/cron.daily/trim

However the best way to enable automatic trimming to SSD  is to just add the discard parameter to /etc/fstab I've already done that earlier in this article.

Not really surprising the increase of websites opening (page load times) were decreased dramatically web page loading waiting time fall down 2 to 2.5 times, so the moral of story for me is always when possible from now on to use SSD in order to have superb websites opening times.

To sum it up what was achieved with moving my data into SSD Drive, before moving websites and SQL data to SSD drive the websites were opening for 6 to 10 seconds now sites open in 2 to 4.5 seconds which is below 5 seconds (the normal waiting time for a user to see your website).
By the way it should be not a news forfor people that are into Search Engine Optimization but might be for some of unexperienced new Admins and Webmasters that, all that all page opening times that  exceeds 5 secs is considered to be a slow website (and therefore perhaps not worthy to read).
The high load page times >5 secs makes the website also less interesting not only for end users but also for search engines (Google / Yahoo / Bing / Baidoo etc.) will is said to crawl it less if website is slow.
Search Engines are said to Index much better and crawl more frequently into more responsive websites.
Hence implementing SSD to a server and decreasing the page load time should bring up my visitors stats a bit too.

Well that's all for today, hope you enjoyed 🙂

Share this on

Remove string line from file on Linux and BSD – Delete entire line with string from file

Tuesday, March 15th, 2016

linux-remove-lines-containing-string-with-sed

If you're already used too using grep -v "sometring" filename to print everything from a file without the certain grepped string output and you want to do the same to delete lines based on strings without having to output the grepped string to a file and then overwritting the original file:
 

grep -v 'whatever' filename > filename1
mv filename1 filename


A much better way to delete an whole line containing a string match from a file is to use sed
sed
should be the tool of choice especially if you're scripting because sed is especially made for such batch edittings.

Here is how to do delete an entire line based on a given string:

 

sed –in-place '/some string to search and delete/d' myfilename


It might be a good idea to also create backups just to make sure something doesn't get deleted incidently to do use:

sed –in-place=.bak '/some string to search and delete/d' myfilename

If you need to wipe out an exact string from all files within a folder you might use a for loop or perl (some good examples check my previous article here)

In short to use bash's for loop here is how to backup and remove all lines with a string match within all files within a Linux directory:

 

for f in *.txt; do sed –in-place '/some string/d'
"$f"; done
find -name '*.txt' -exec sed –in-place=.bak '/some
string/d' "{}" ';'

 

BTW SED is really rich editor and some people got so much into it that there is even a sed written text (console) version of arkanoid 🙂

sed-text-editor-written-arkanoid-game-linux-bsd

If you want to break the ice and get some fun in your boring sysadmin life get sed arkanoid code from here.
I have it installed under pc-freak.net free ASCII Games entertainment service, so if you want to give it a try just login and give a try.

Enjoy 🙂

Share this on

Tools to scan a Linux / Unix Web server for Malware and Rootkits / Lynis and ISPProtect – clean Joomla / WordPress and other CMS for malware and malicious scripts and trojan codes

Monday, March 14th, 2016

Linux-BSD-Unix-Rootkit-Malware-XSS-Injection-spammer-scripts-clean-howto-manual

If you have been hacked or have been suspicious that someone has broken up in some of the shared web hosting servers you happent o manage you already probably have tried the server with rkhuter, chroot and unhide tools which gives a general guidance where a server has been compromised

However with the evolution of hacking tools out there and the boom of Web security XSS / CSS / Database injections and PHP scripts vulnerability catching an intruder especially spammers has been becoming more and more hard to achieve.

Just lately a mail server of mine's load avarage increased about 10 times, and the CPU's and HDD I/O load jump over the sky.
I started evaluating the situation to find out what exactly went wrong with the machine, starting with a hardware analysis tools and a physical check up whether all was fine with the hardware Disks / Ram etc. just to find out the machine's hardware was working perfect.
I've also thoroughfully investigated on Logs of Apache, MySQL, TinyProxy and Tor server and bind DNS and DJBDns  which were happily living there for quite some time but didn't found anything strange.

Not on a last place I investigated TOP processes (with top command) and iostat  and realized the CPU high burst lays in exessive Input / Output of Hard Drive. Checking the Qmail Mail server logs and the queue with qmail-qstat was a real surprise for me as on the queue there were about 9800 emails hanging unsent, most of which were obviously a spam, so I realized someone was heavily spamming through the server and started more thoroughfully investigating ending up to a WordPress Blog temp folder (writtable by all system users) which was existing under a Joomla directory infrastructure, so I guess someone got hacked through the Joomla and uploaded the malicious php spammer script to the WordPress blog. I've instantly stopped and first chmod 000 to stop being execuded and after examing deleted view73.php, javascript92.php and index8239.php which were full of PHP values with binary encoded values and one was full of encoded strings which after being decoding were actually the recepient's spammed emails.
BTW, the view*.php javascript*.php and index*.php files were owned by www-data (the user with which Apache was owned), so obviously someone got hacked through some vulnerable joomla or wordpress script (as joomla there was quite obscure version 1.5 – where currently Joomla is at version branch 3.5), hence my guess is the spamming script was uploaded through Joomla XSS vulnerability).

As I was unsure wheteher the scripts were not also mirrored under other subdirectories of Joomla or WP Blog I had to scan further to check whether there are no other scripts infected with malware or trojan spammer codes, webshells, rootkits etc.
And after some investigation, I've actually caught the 3 scripts being mirrored under other webside folders with other numbering on filename view34.php javascript72.php, index8123.php  etc..

I've used 2 tools to scan and catch malware the trojan scripts and make sure no common rootkit is installed on the server.

1. Lynis (to check for rootkits)
2. ISPProtect (Proprietary but superb Website malware scanner with a free trial)

1. Lynis – Universal security auditing tool and rootkit scanner

Lynis is actually the well known rkhunter, I've used earlier to check servers BSD and Linux servers for rootkits.
To have up-to-date version of Lynis, I've installed it from source:
 

cd /tmp
wget https://cisofy.com/files/lynis-2.1.1.tar.gz
tar xvfz lynis-2.1.1.tar.gz
mv lynis /usr/local/
ln -s /usr/local/lynis/lynis /usr/local/bin/lynis

 


Then to scan the server for rootkits, first I had to update its malware definition database with:
 

lynis update info


Then to actually scan the system:
 

lynis audit system


Plenty of things will be scanned but you will be asked on a multiple times whether you would like to conduct different kind fo system services and log files, loadable kernel module rootkits and  common places to check for installed rootkits or server placed backdoors. That's pretty annoying as you will have to press Enter on a multiple times.

lynis-asking-to-scan-for-rootkits-backdoors-and-malware-your-linux-freebsd-netbsd-unix-server

Once scan is over you will get a System Scan Summary like in below screenshot:

lynis-scanned-server-for-rootkit-summer-results-linux-check-for-backdoors-tool

Lynis suggests also a very good things that might be tampered to make the system more secure, so using some of its output when I have time I'll work out on hardening all servers.

To prevent further incidents and keep an eye on servers I've deployed Lynis scan via cron job once a month on all servers, I've placed under a root cronjob on every first dae of month in following command:

 

 

server:~# crontab -u root -e
0 3 1 * * /usr/local/bin/lynis –quick 2>&1 | mail -s "lynis output of my server" admin-mail@my-domain.com)

 

2. ISPProtect – Website malware scanner

ISPProtect is a malware scanner for web servers, I've used it to scan all installed  CMS systems like WordPress, Joomla, Drupal etc.
ISPProtect is great for PHP / Pyhon / Perl and other CMS based frameworks.
ISPProtect contains 3 scanning engines: a signature based malware scanner, a heuristic malware scanner, and a scanner to show the installation directories of outdated CMS systems.
Unfortunately it is not free software, but I personally used the FREE TRIAL option  which can be used without registration to test it or clean an infected system.
I first webserver first locally for the infected site and then globally for all the other shared hosting websites.

As I wanted to check also rest of hosted websites, I've run ISPProtect over the all bunch of installed websites.
Pre-requirement of ISPProtect is to have a working PHP Cli and Clamav Anti-Virus installed on the server thus on RHEL (RPM) based servers make sure you have it installed if not:
 

server:~# yum -y install php

server:~# yum -y install clamav


Debian based Linux servers web hosting  admins that doesn't have php-cli installed should run:
 

server:~# apt-get install php5-cli

server:~# apt-get install clamav


Installing ISPProtect from source is with:

mkdir -p /usr/local/ispprotect
chown -R root:root /usr/local/ispprotect
chmod -R 750 /usr/local/ispprotect
cd /usr/local/ispprotect
wget http://www.ispprotect.com/download/ispp_scan.tar.gz
tar xzf ispp_scan.tar.gz
rm -f ispp_scan.tar.gz
ln -s /usr/local/ispprotect/ispp_scan /usr/local/bin/ispp_scan

 

To initiate scan with ISPProtect just invoke it:
 

server:~# /usr/local/bin/ispp_scan

 

ispprotect-scan-websites-for-malware-and-infected-with-backdoors-or-spamming-software-source-code-files

I've used it as a trial

Please enter scan key:  trial
Please enter path to scan: /var/www

You will be shown the scan progress, be patient because on a multiple shared hosting servers with few hundred of websites.
The tool will take really, really long so you might need to leave it for 1 hr or even more depending on how many source files / CSS / Javascript etc. needs to be scanned.

Once scan is completed scan and infections found logs will be stored under /usr/local/ispprotect, under separate files for different Website Engines and CMSes:

After the scan is completed, you will find the results also in the following files:
 

Malware => /usr/local/ispprotect/found_malware_20161401174626.txt
Wordpress => /usr/local/ispprotect/software_wordpress_20161401174626.txt
Joomla => /usr/local/ispprotect/software_joomla_20161401174626.txt
Drupal => /usr/local/ispprotect/software_drupal_20161401174626.txt
Mediawiki => /usr/local/ispprotect/software_mediawiki_20161401174626.txt
Contao => /usr/local/ispprotect/software_contao_20161401174626.txt
Magentocommerce => /usr/local/ispprotect/software_magentocommerce_20161401174626.txt
Woltlab Burning Board => /usr/local/ispprotect/software_woltlab_burning_board_20161401174626.txt
Cms Made Simple => /usr/local/ispprotect/software_cms_made_simple_20161401174626.txt
Phpmyadmin => /usr/local/ispprotect/software_phpmyadmin_20161401174626.txt
Typo3 => /usr/local/ispprotect/software_typo3_20161401174626.txt
Roundcube => /usr/local/ispprotect/software_roundcube_20161401174626.txt


ISPProtect is really good in results is definitely the best malicious scripts / trojan / trojan / webshell / backdoor / spammer (hacking) scripts tool available so if your company could afford it you better buy a license and settle a periodic cron job scan of all your servers, like lets say:

 

server:~# crontab -u root -e
0 3  1 * *   /usr/local/ispprotect/ispp_scan –update && /usr/local/ispprotect/ispp_scan –path=/var/www –email-results=admin-email@your-domain.com –non-interactive –scan-key=AAA-BBB-CCC-DDD


Unfortunately ispprotect is quite expensive so I guess most small and middle sized shared hosting companies will be unable to afford it.
But even for a one time run this tools worths the try and will save you an hours if not days of system investigations.
I'll be glad to hear from readers if aware of any available free software alternatives to ISPProtect. The only one I am aware is Linux Malware Detect (LMD).
I've used LMD in the past but as of time of writting this article it doesn't seems working any more so I guess the tool is currently unsupported / obsolete.

 

Share this on