Posts Tagged ‘shells’

Nginx increase security by putting websites into Linux jails howto

Monday, August 27th, 2018


If you're sysadmining a large numbers of shared hosted websites which use Nginx Webserver to interpret PHP scripts and serve HTML, Javascript, CSS … whatever data.

You realize the high amount of risk that comes with a possible successful security breach / hack into a server by a malicious cracker. Compromising Nginx Webserver by an intruder automatically would mean that not only all users web data will get compromised, but the attacker would get an immediate access to other data such as Email or SQL (if the server is running multiple services).

Nowadays it is not so common thing to have a multiple shared websites on the same server together with other services, but historically there are many legacy servers / webservers left which host some 50 or 100+ websites.

Of course the best thing to do is to isolate each and every website into a separate Virtual Container however as this is a lot of work and small and mid-sized companies refuse to spend money on mostly anything this might be not an option for you.

Considering that this might be your case and you're running Nginx either as a Load Balancing, Reverse Proxy server etc. , even though Nginx is considered to be among the most secure webservers out there, there is absolutely no gurantee it would not get hacked and the server wouldn't get rooted by a script kiddie freak that just got in darknet some 0day exploit.

To minimize the impact of a possible Webserver hack it is a good idea to place all websites into Linux Jails.


For those who hear about Linux Jail for a first time,
chroot() jail is a way to isolate a process / processes and its forked children from the rest of the *nix system. It should / could be used only for UNIX processes that aren't running as root (administrator user), because of the fact the superuser could break out (escape) the jail pretty easily.

Jailing processes is a concept that is pretty old that was first time introduced in UNIX version 7 back in the distant year 1979, and it was first implemented into BSD Operating System ver. 4.2 by Bill Joy (a notorious computer scientist and co-founder of Sun Microsystems). Its original use for the creation of so called HoneyPot – a computer security mechanism set to detect, deflect, or, in some manner, counteract attempts at unauthorized use of information systems that appears completely legimit service or part of website whose only goal is to track, isolate, and monitor intruders, a very similar to police string operations (baiting) of the suspect. It is pretty much like а bait set to collect the fish (which in this  case is the possible cracker).


BSD Jails nowadays became very popular as iPhones environment where applications are deployed are inside a customly created chroot jail, the principle is exactly the same as in Linux.

But anyways enough talk, let's create a new jail and deploy set of system binaries for our Nginx installation, here is the things you will need:

1. You need to have set a directory where a copy of /bin/ls /bin/bash /bin/,  /bin/cat … /usr/bin binaries /lib and other base system Linux system binaries copy will reside.


server:~# mkdir -p /usr/local/chroot/nginx


2. You need to create the isolated environment backbone structure /etc/ , /dev, /var/, /usr/, /lib64/ (in case if deploying on 64 bit architecture Operating System).


server:~# export DIR_N=/usr/local/chroot/nginx;
server:~# mkdir -p $DIR_N/etc
server:~# mkdir -p $DIR_N/dev
server:~# mkdir -p $DIR_N/var
server:~# mkdir -p $DIR_N/usr
server:~# mkdir -p $DIR_N/usr/local/nginx
server:~# mkdir -p $DIR_N/tmp
server:~# chmod 1777 $DIR_N/tmp
server:~# mkdir -p $DIR_N/var/tmp
server:~# chmod 1777 $DIR_N/var/tmp
server:~# mkdir -p $DIR_N/lib64
server:~# mkdir -p $DIR_N/usr/local/


3. Create required device files for the new chroot environment


server:~# /bin/mknod -m 0666 $D/dev/null c 1 3
server:~# /bin/mknod -m 0666 $D/dev/random c 1 8
server:~# /bin/mknod -m 0444 $D/dev/urandom c 1 9


mknod COMMAND is used instead of the usual /bin/touch command to create block or character special files.

Once create the permissions of /usr/local/chroot/nginx/{dev/null, dev/random, dev/urandom} have to be look like so:


server:~# ls -l /usr/local/chroot/nginx/dev/{null,random,urandom}
crw-rw-rw- 1 root root 1, 3 Aug 17 09:13 /dev/null
crw-rw-rw- 1 root root 1, 8 Aug 17 09:13 /dev/random
crw-rw-rw- 1 root root 1, 9 Aug 17 09:13 /dev/urandom


4. Install nginx files into the chroot directory (copy all files of current nginx installation into the jail)

If your NGINX webserver installation was installed from source to keep it latest
and is installed in lets say, directory location /usr/local/nginx you have to copy /usr/local/nginx to /usr/local/chroot/nginx/usr/local/nginx, i.e:


server:~# /bin/cp -varf /usr/local/nginx/* /usr/local/chroot/nginx/usr/local/nginx


5. Copy necessery Linux system libraries to newly created jail

NGINX webserver is compiled to depend on various libraries from Linux system root e.g. /lib/* and /lib64/* therefore in order to the server work inside the chroot-ed environment you need to transfer this libraries to the jail folder /usr/local/chroot/nginx

If you are curious to find out which libraries exactly is nginx binary dependent on run:

server:~# ldd /usr/local/nginx/usr/local/nginx/sbin/nginx (0x00007ffe3e952000) => /lib/x86_64-linux-gnu/ (0x00007f2b4762c000) => /lib/x86_64-linux-gnu/ (0x00007f2b473f4000) => /lib/x86_64-linux-gnu/ (0x00007f2b47181000) => /usr/local/lib/ (0x00007f2b46ddf000) => /lib/x86_64-linux-gnu/ (0x00007f2b46bc5000) => /lib/x86_64-linux-gnu/ (0x00007f2b46826000)
        /lib64/ (0x00007f2b47849000) => /lib/x86_64-linux-gnu/ (0x00007f2b46622000)

The best way is to copy only the libraries in the list from ldd command for best security, like so:


server: ~# cp -rpf /lib/x86_64-linux-gnu/ /usr/local/chroot/nginx/lib/*
server: ~# cp -rpf library chroot_location



However if you're in a hurry (not a recommended practice) and you don't care for maximum security anyways (you don't worry the jail could be exploited from some of the many lib files not used by nginx and you don't  about HDD space), you can also copy whole /lib into the jail, like so:


server: ~# cp -rpf /lib/ /usr/local/chroot/nginx/usr/local/nginx/lib


NOTE! Once again copy whole /lib directory is a very bad practice but for a time pushing activities sometimes you can do it …

6. Copy /etc/ some base files and , prelink.conf.d directories to jail environment


server:~# cp -rfv /etc/{group,prelink.cache,services,adjtime,shells,gshadow,shadow,hosts.deny,localtime,nsswitch.conf,nscd.conf,prelink.conf,protocols,hosts,passwd,,,resolv.conf,host.conf}  \


server:~# cp -avr /etc/{,prelink.conf.d} /usr/local/chroot/nginx/nginx/etc

7. Copy HTML, CSS, Javascript websites data from the root directory to the chrooted nginx environment


server:~# nice -n 10 cp -rpf /usr/local/websites/ /usr/local/chroot/nginx/usr/local/

This could be really long if the websites are multiple gigabytes and million of files, but anyways the nice command should reduce a little bit the load on the server it is best practice to set some kind of temporary server maintenance page to show on the websites index in order to prevent the accessing server clients to not have interrupts (that's especially the case on older 7200 / 7400 RPM non-SSD HDDs.)


8. Stop old Nginx server outside of Chroot environment and start the new one inside the jail

a) Stop old nginx server

Either stop the old nginx using it start / stop / restart script inside /etc/init.d/nginx (if you have such installed) or directly kill the running webserver with:


server:~# killall -9 nginx


b) Test the chrooted nginx installation is correct and ready to run inside the chroot environment


server:~# /usr/sbin/chroot /usr/local/chroot/nginx /usr/local/nginx/nginx/sbin/nginx -t
server:~# /usr/sbin/chroot /usr/local/chroot/nginx /usr/local/nginx/nginx/sbin/nginx


c) Restart the chrooted nginx webserver – when necessery later


server:~# /usr/sbin/chroot /nginx /usr/local/chroot/nginx/sbin/nginx -s reload


d) Edit the chrooted nginx conf

If you need to edit nginx configuration, be aware that the chrooted NGINX will read its configuration from /usr/local/chroot/nginx/nginx/etc/conf/nginx.conf (i'm saying that if you by mistake forget and try to edit the old config that is usually under /usr/local/nginx/conf/nginx.conf



Using GNU screen for opening multiple shells within a single shell / Handy way to keep constant SSH connections to servers from 1 single shell

Wednesday, October 31st, 2012

Handay way to keep ssh connections persistent after ssh logout / Using GNU screen for opening multiple shells within a shell

As GNU / Linux user I use screen window manager to manage multiple SSH connections (all over one ssh connection) to a host over the last maybe 10 years. Though screen is generally popular it is still possible some novice sysadmin did not use it or (hope not) never heard of it. For those who don't use GNU screen still, give it a try; launch it within a (system bash, csh etc.) shell and then inside the main screen window launch multiple screen internal sessions (by pressing simultaneously) keys CTRL + ALT + c .

Each CTRL + ALT + c makes screen open a new "Virtual Window (pty)"  inside itself, the multiple screen sub-instances are kept in memory of main screen program loaded in memory. In a way Virtual Windows of screen in logic are very Similar to Apache's Webserver Apache Parent and Child processes.

Anyways to test screen type in console:

pcfreak:~$ screen

And press enter twice, it will launch under screen a new instance of your current logged in shell (if you logged in bash will open bash if zsh – zsh etc.)

Afterwards, you can open multiple Virtual Screens as I've mentioned with pressing CTRL + ALT + c.
Moving between all open screen sessions is done with simultaneous press of:

CTRL + ALT + pTo move to previous screen Virtual Window Shell Session
CTRL + ALT + n
– To move to next screen Virtual Window Session

The most useful from all screen functionality is DETACH. You can detach (like save state of) curreng GNU Screen active sesion by  pressing together:

CTRL + a + d – Detach current active GNU Screen session

Screen supports detaching multiple sessions (whether 2 or more screen sessions run with identical user credentials).

An example use of multiple detached screen sessions would be if you login via SSH with a certain user lets say user myuser and later detach by pressing CTRL + a + d after which open new session, you will get in shell message similar to:

[detached from 1549.pts-11.pcfreak]

The msg indicates new screen session is detached. Onwards run screen once again, for sake of test typing in same shell once again:

pcfreal:~$ screen

After screen loads its second session press again CTRL + a + d – to detach second active session, again you will get msg:

[detached from 15691.pts-0.pcfreak]

Next on you can use screen to list all active window sessions by issuing:

pcfreak:~$ screen -list
There are screens on:
        1549.pts-11.pcfreak     (10/27/2012 09:45:58 PM)        (Detached)
        15691.pts-0.pcfreak     (10/24/2012 02:50:06 PM)        (Detached)
2 Sockets in /var/run/screen/S-hipo.

To attach to detached active GNU screen session, use:

pcfreak:~$ screen -r PID_OF_SESSION

For example to attach to 1 listed screen session 1549:

pcfreak:~$  screen -r 1549

To attach to second one 15691:

pcfreak:~$ screen -r 15691

The -r  switch stands for re-attach and second part of PID name like in above example pts-11.pcfreak pts-0.pcfreak is just indicating the hostname where screen was detached as well as the pty (pseudo tty number assigned to detached session), the time included shows the exact time in which main screen session was started for instance for screen 1549 it is 10/27/2012 09:45:58 PM.

The 2 Sockets in /var/run/screen/S-hipo displays the directory location of the screen socket, on each screen user startup a separte directory is created in /var/run/screen, the attach detach of screens is done via using a UNIX socket (fifo named pipe):

pcfreak :~$ ls
1549.pts-11.pcfreak|  15691.pts-0.pcfreak|  byobu.reload-required*
pcfreak:~$ cd /var/run/screenS-hipo
pcfreak:/var/run/screen/S-hipo$ file 1549.pts-11.pcfreak
1549.pts-11.pcfreak: fifo (named pipe)
The use case of screen are really up to your imagination, however for running programs which require you to have a permanent interactive terminal, like lets say: Midnight Commander (mc), iptraf, tcpdump, irssi (IRC chat client), ettercap, sniffit 🙂 screen is really *precious sysadmin tool
screen is great for PuTTY users who dislike that putty doesn't by default support tab-bed multple SSH logins (you might like to check my previous post where I explain about Putty Connection manager which supports tabbed ssh)
Actually, i'm so accustomed to SCREEN, that it is quite hard to imagine the times when it was not. I guess sysadmin life was much difficult back then 🙂

Many people who still remember irc clients like BitchX and epic and the IRC times should remember, how well known and frequent people used to detach those progs or even detach eggdrops with specific TCL scripts inside separate screen sessions.

The most useful use of screen of course is to open multiple SSH sessions to different server nodes and keep permanently logged in on hosts by detaching the screen session.

I can think of 3 main advantages of using ssh inside single screen session:

1. At any time you can login to just one server instead of (for exmpl. 10 servers), and use this one server as a reference through which you can "stuntly" check statuses of all 10 hosts with no need to login 10 times via SSH or with a Putty client (if logging from Windows)
 2. If you're using unstable often interrupted lets say modem (dial up) line to connect to the Internet and you need continuation of previously interrupted SSH ssh login due to interrupted connection

 3. You can save a lot of time and effort of typing passwords multiple times at ssh login prompts 

Of course there are disadvantages too;

From security point of view it is a weak practice to keep logged in to multiple servers via SSH from one single screen session. If someone sniffs user password with which screen is started and attach to the screen session he will suddenly be granted to access to 10 more servers! 
Anyhow for  lazy people who believe to maintain high security policies, e.g.:

 a. do not login to SSH sessions from Windows hosts
 b. use some kind of UNIX / Linux / BSD based OS
 c. login from a host used only by a single person
etc. etc. ,  keeping screen detached with multiple sshs might save you a lot of time; this is especially if you have to login 10 times to the servers a day changing location – lets say if you use (notebook and travel a lot).

GNU screen also understands some commands, which can configure the Shell Prompt of it as long as color gamma of main and sub-screen (virtual) sessions. To have a screen shell prompt outline and blue color gamma as in the picture in beginning of my post you can download and use my .screenrc into your ~/.screenrc i.e.:

pcfreak:/~$ cd ~
pcfreak:/~$ wget -q
pcfreak:/~:$ mv .my-screenrc .screenrc
Another good screenrc configured to make your screen sessions more user friendly is here.

Below I include a screenshot on how screen sessions looks like, whenever above "user friendly screenrc" is used:

Screen custom screenrc in gnome terminal Debian Squeeze Linux screenshot
 It might be helpful to mention, there are newer piece of soft with richer screen functionality one of those newer virtual window managers (alternative to GNU screen) is byobu. I've tested byobu myself and I feel
Sometimes, when a screen session is interrupted, because screen running host is restarted or shutdown, dead screen sockets remain.

pcfreak:~$ screen -list
25542.pts-28.pcfreak (Dead ???)

1636.pts-21.pcfreak (Attached)

In case you see some screens, like this you should use screen -wipe to cleanse socket pointing to already non-existing screen:

pcfreak:~$ screen -wipe

Screen has plenty of other command shortcuts, all of them are starting with a key combination of CTRL + a + )some kbd letter)

CTRL + a + aDoes switch between first and last screen open windows

CTRL + a + HTurns screen log on for active screen session

Ctrl + a + mTurns (on/off) screen monitoring for activity of a screen shell (useful if you left kernel, openoffice or some huge app to compile

CTRL +a + _Turns monitoring and reports outside of screen session if a running shell inside screen is not active for more than 30 seconds

CTRL +a + shift + SIs very handy as it splits the screen between all logged in active screen sessions (Use control CTRL + a + tab to switch between splitted windows)

how to GNU Screen split windows Linux gnome-terminal shot

Ctrl + a + x locks the screen, in the same fashion as Screen Lock is done inside a GUI environment GNOME, KDE etc. Once pressed it can be unlocked after you type in your user pass. This is very handy if you have to go to toilet and you don't want your colleague to snuff in your console 🙂

It is also possible to switch between screen sub-virtual windows using:

CTRL + a + (number starting from 0), e.g.:

CTRL + a + 0
CTRL + a + 1
CTRL + a + …
CTRL + a + 9

There are plenty of other helpful functionalities which you might want to look in the manual (man screen) – check in the manual section DEFAULT KEY BINDINGS section

P. S.  – Some of screen keybindings, does not work in gnome-terminal and konsole and other terminal clients which already had a key bindings set to CTRL + a + whatever key. If that's the case you can change screen assigned keybindings through .screenrc

If you google around you will find a dozen of tricks you can do with screen, since my only goal of this article