Posts Tagged ‘hipo’

Howto Pass SSH traffic through a Secured Corporate Proxy server with corkscrew, using sshd as a standalone proxy service with no proxy installed on remote Linux server or VPS

Tuesday, November 19th, 2019

howto pass ssh traffic through proxy to remote server use remote machine as a proxy for connecting to the Internet

Working in the big bad corporate world (being employed in  any of the Fortune 500) companies, especially in an IT delivery company is a nasty thing in terms of User Personal Data Privacy because usually when employeed in any of a corporation, the company ships you with a personal Computer with some kind of pre-installed OS (most often this is Windows) and the computer is not a standalone one but joined in Active Directory (AD) belonging to Windows Domain and centrally administered by whoever.

As part of the default deplyed configuration in this pre-installed OS and software is that part or all your network traffic and files is being monitored in some kind of manner as your pre-installed Windows or Linux notebook given by the Corporation is having a set of standard software running in the background, and even though you have Windows Administrator there are many things you have zero control or even if you have changed it once the Domain Policy is triggered your custom made changes / Installed Programs that happen to be against the company policy are being automatically deleted, any registry changes made are being rewinded etc. Sometimes even by trying to manually clean up your PC from the corporate crapware,  you might breaks access to the corporate DMZ firewalled network. A common way to secure their employee PC data large companies have a Network seperation, your PC when not connected to the Corporate VPN is having a certain IP configuration and once connected to the Demilitarized Zone VPN those configuration changes and the PC have access to internal company infrastructure servers / router / switches / firewalls / SANs etc. Access to corporate Infrastructure is handled via crypted VPN clinet such as Cisco AnyConnect Secure Mobility Client which is perhaps one of the most used ones out there.

Part of the common software installed to Monitor your PC for threats / viruses / trojans among which is MCafee / EMET (Enhandced Mitigation Experience Toolkit) the PC is often prebundled with some kind of anti-malware (crapware) :). But the tip of the iceberg on user surveillance where most of surveillance happens is the default installed proxy on the PC which usually does keep track of all your remote accessed HTTP Website URLs accessed in plain text – traffic flowing on Port 80 and crypted one on standard (SSL) Port 443. This Web Traffic is handled by the Central Corporate proxy that is being deployed via some kind of Domain policy, every time the Computer joins the Windows domain. 

This of course is a terrible thing for your Browsing security and together with the good security practice to run your browser in Incognito mode, which makes all your browsing activity such as access URLs History or Saved Cookies data to be cleared up on a Browser close it is important to make sure you run your own personal traffic via a separate browser which you will use only for your own concern browsing such as Accessing your Bank Money Accounts to check your Monthly Sallary / Purchase things online via Amazon.com / Ebay.com, whether all of the rest traffic company related is trafficed via the default set corporate central proxy.
This is relatively easy sometimes in companies, where security is not of a top concern but in corporations with tightened security accessing remote proxy, or accessing even common daily news and Public Email websites or social media sites  Gmail.com / Twitter / Youtube will be filtered so the only way to reach them will be via some kind of Proxy and often this proxy is the only way out to the Free world from the corporate jail.

Here is where the good old SSH comes as a saving grace as it turns out SSH traffic could be trafficed over a proxy. In below article I will give you a short insight on how Proxy through SSH could be achieved to Secure your dailty web traffic and use SSH to reach your own server on the Internet as well as how you can copy securely data via SSH through corporate Proxy. 
 

1. How to view your corporate used (default) proxy / Check Proxy.pac file definitions

 

To get an idea what is the used proxy on your Corporate PC (as most corporate employee given notebooks are running some kind of M$ Windows)  you can go to:

Windows Control Panel -> Internet Options -> Connections -> Lan Settings


internet-properties-microsoft-windows-screenshot

Under the field Proxy server (check out the Proxy configured Address and Port number )

local-area-network-lan-settings-screenshot-windows-1
 

Having that as browsers venerate the so-called Proxy.pac file, to be rawly aware on some general Company Proxy configured definitions you can access in a browser the proxy itself fething the proxy.pac file for example.

 

http://your-corporate-firewall-rpoxy-url:8080/proxy.pac

 

This is helpful as some companies Proxies have some proxy rules that reveal some things about its Internet architecture and even some have some badly configured proxy.pac files which could be used to fool the proxy under some circumstances 🙂
 

2. Few of the reasons corporations proxy all their employee's work PC web traffic

 

The corporate proxying of traffic has a number of goals, some of which are good hearted and others are for mostly spying on the users.

 

1. Protect Corporate Employees from malicious Viruses / Trojans Horses / Malware / Badware / Whatever ware – EXCELLENT
2. Prevent users from acessing a set of sources that due to the corporate policy are considered harmful (e.g. certain addresses 
of information or disinformation of competitors, any Internet source that might preach against the corporation, hacking ralated websites etc.) – NOT GOOD (for the employee / user) and GOOD for the company
3.Spy on the users activity and be able to have evidence against the employee in case he decided to do anything harmful to the company evidences from proxy could even later be used in court if some kind of corpoate infringment occurs due to misbehave of the employee. – PERFECT FOR COMPANY and Complete breach of User privacy and IMHO totally against European Union privacy legislation such as GDRP
4. In companies that are into the field of Aritificial Intelligence / Users behavior could even be used to advance Self-learning bots and mechanisms – NASTY ! YAECKES

 

3. Run SSH Socks proxy to remote SSHd server running on common SSL 443 port

 

Luckily sysadmins who were ordered the big bosses to sniff on your Web behaviour and preferences could be outsmarted with some hacks.

To protect your Browsing behaviours and Secure your privacy perhaps the best option is to use the Old but gold practice o Securing your Networkf traffic using SSH Over Proxy and SSH Dynamic tunnel as a Proxy as explained in my previous article here.

how-to-use-sshd-server-as-a-proxy-without-a-real-proxy-ssh-socks5_proxy_linux
 

In short the quest way to have your free of charge SOCKS  Remote proxy to your Home based Linux installed OS server / VPN with a Public Internet address is to use ssh as so:

 

ssh -D 3128 UserName@IP-of-Remote-SSHD-Host -p 443

 

This will start the SOCKS Proxy tunnel from Corporate Work PC to your Own Home brew server.

For some convenience it is useful to set up an .alias (for cygwin) / linux users in .bashrc file:

 

alias proxy='ssh -D 3128 UserName@IP-of-Remote-SSHD-Host -p 443';

 

To start using the Proxy from browser, I use a plugin called FoxyProxy in Chrome and Firefox browsers
set-up to connect to localhost – 127.0.0.1:3128 for All Protocols as a SOCKs v5 Proxy.

The sshd Socks proxy can be used for multiple others for example, using it you can also pass on traffic from Mail client such as Thunderbird to your Email server if you're behind a firewall prohibiting access to the common POP3 port 110 or IMAP port TCP 143. 

4. How to access SSH through Proxy using jumphost SSH hop


If you're like me and you have on your Home Linux machine only one Internet address and you have already setupped an SSL enabled service (lets say Webmail) to listen to that Public Internet IP and you don't have the possibility to run another instance of /usr/bin/sshd on port 443 via configuration or manually one time by issuing:

 

/usr/sbin/sshd -p 443

 

Then you can use another ssh another Linux server as a jump host to your own home Linux sshd server. This can be done even by purchasing a cheap VPS server for lets say 3 dollars month etc. or even better if you have a friend with another Linux home server, you can ask him to run you sshd on TCP port 443 and add you an ssh account.
Once you have the second Linux machine as JumpHost to reach out to your own machine use:

 

ssh -J Your-User@Your-jump-host.com:443 hipo@your-home-server.com -v

 

To easify this a bit long line it is handy to use some kind of alias like:

 

alias sshhome='ssh -J Your-User@Your-jump-host.com:443 hipo@your-home-server.com -v'

 

The advantage here is just by issuing this sshd tunnel and keeping it open in a terminal or setting it up as Plink Putty tunnel you have all your Web Traffic Secured
between your Work Corporate PC and your Home Brew Server, keeping the curious eyes of your Company Security Officers from your own Web traffic, hence
separating the corporate privacy from your own personal privacy. Using the just established own SSH Proxy Tunnel to home for your non-work stuff browsing habits
from the corporate systems which are accessed by switching with a button click in FoxyProxy to default proxy settings.
 

5. How to get around paranoid corporate setup where only remote access to Corporate proxy on TCP Port 80 and TCP 443 is available in Browser only

 

Using straight ssh and to create Proxy will work in most of the cases but it requires SSH access to your remote SSH running server / VPS on TCP Port 22, however under some Fort-Nox like financial involved institutions and companies for the sake of tightened security, it is common that all Outbound TCP Ports are prohibited except TCP Port 80 and SSL 443 as prior said, so what can you do then to get around this badful firewall and access the Internet via your own server Proxy? 
The hack to run SSH server either on tcp port 80 or tcp port 443 on remote Host and use 443 / 80 to acess SSHD should work, but then even for the most paranoid corporations the ones who are PCI Compliant – PCI stands for (Payment Card Industry), e.g. works with Debit and Credit Card data etc, accessing even 80 or 443  ports with something like telnet client or netcat will be impossible. 
Once connected to the corporate VPN,  this 2 two ports firewall exceptions will be only accessible via the Corporate Proxy server defined in a Web Browser (Firefox / IE / Chrome etc.) as prior explained in article.

The remedy here is to use a 3rd party tools such as httptunnel or corkscrew that  are able to TUNNEL SSH TRAFFIC VIA CORPORATE PROXY SERVER and access your own resource out of the DMZ.

Both httptunnel and corkscrew are installable both on most Linux distros or for Windows users via CygWin for those who use MobaXterm.

Just to give you better idea on what corkscrew and (hts) httptunnel does, here is Debian packages descriptions.

# apt-cache show​ corkscrew
" corkscrew is a simple tool to tunnel TCP connections through an HTTP
 proxy supporting the CONNECT method. It reads stdin and writes to
 stdout during the connection, just like netcat.
 .
 It can be used for instance to connect to an SSH server running on
 a remote 443 port through a strict HTTPS proxy.
"

 

# apt-cache show httptunnel|grep -i description -A 7
Description-en: Tunnels a data stream in HTTP requests
 Creates a bidirectional virtual data stream tunnelled in
 HTTP requests. The requests can be sent via a HTTP proxy
 if so desired.
 .
 This can be useful for users behind restrictive firewalls. If WWW
 access is allowed through a HTTP proxy, it's possible to use
 httptunnel and, say, telnet or PPP to connect to a computer

Description-md5: ed96b7d53407ae311a6c5ef2eb229c3f
Homepage: http://www.nocrew.org/software/httptunnel.html
Tag: implemented-in::c, interface::commandline, interface::daemon,
 network::client, network::server, network::vpn, protocol::http,
 role::program, suite::gnu, use::routing
Section: net
Priority: optional
Filename: pool/main/h/httptunnel/httptunnel_3.3+dfsg-4_amd64.deb

Windows cygwin users can install the tools with:
 

apt-cyg install –yes corkscrew httptunnel


Linux users respectively with:

apt-get install –yes corkscrew httptunnel

or 

yum install -y corkscrew httptunnel

 

You will then need to have the following configuration in your user home directory $HOME/.ssh/config file
 

Host host-addrs-of-remote-home-ssh-server.com
ProxyCommand /usr/bin/corkscrew your-corporate-firewall-rpoxy-url 8080 %h %p

 

howto-transfer-ssh-traffic-over-proxy

Picture Copyright by Daniel Haxx

The best picture on how ssh traffic is proxied is the one found on Daniel Haxx's website which is a great quick tutorial which originally helped to get the idea of how corkscrew works in proxying traffic I warmly recommend you take a quick look at his SSH Through or over Proxy article.

Host-addrs-of-remote-home-ssh-server.com could be also and IP if you don't have your own domain name in case if using via some cheap VPN Linux server with SSH, or alternatively
if you don't want to spend money on buying domain for SSH server (assuming you don't have such yet) you can use Dyn DNS or NoIP.

Another thing is to setup the proper http_proxy / https_proxy / ftp_proxy variable exports in $HOME/.bashrc in my setup I have the following:
 

export ftp_proxy="http://your-corporate-firewall-rpoxy-url:8080"
export https_proxy="https://your-corporate-firewall-rpoxy-url:8080"
export http_proxy="http://your-corporate-firewall-rpoxy-url:8080"
export HTTP_PROXY="http://your-corporate-firewall-rpoxy-url:8080"
export HTTPS_PROXY="http://your-corporate-firewall-rpoxy-url:8080"


 

6. How to Transfer Files / Data via SSH Protocol through  Proxy with SCP and SFTP


Next logical question is how to Transfer your own personal encrypted files (that contains no corporate sensitive information) between your Work laptop and home brew Linux ssh server or cheap VPN.

It took me quite a lot of try-outs until finally I got it how Secure Copy (scp) command can be used toto transfer files between my Work Computer and my Home brew server using JumpHost, here is how:
 

scp -o 'ProxyJump Username@Jumpt-Host-or-IP.com:443' ~/file-or-files-to-copy* Username@home-ssh-server.com:/path/where/to/copy/files


I love using sftp (Secure FTP) command Linux client to copy files and rarely use scp so I have a lot of try-outs to connect interacitvely via the Corporate Proxy server over a Jump-Host:443 to my Destination home machine, 

 

I've tried using netcat as it was pointed in many articles online, like so to traffic my sftp traffic via my localhost binded SSH Socks proxy on :3128 together with netcat as shown in article prior example, using following line:
 

sftp -oProxyCommand='/bin/nc -X connect -x 127.0.0.1:3128 %h %p' Username@home-ssh-server.com 22

 

Also tried proxy connect like this:

 

sftp -o ProxyCommand="proxy-connect -h localhost -p 3128 %h %p" Username@home-ssh-server.com

 

Moreover, tried to use the ssh  command (-s) argument capability to invoke SSH protocol subsystem feature which is used to facilitiate use of SSH secure transport for other application
 

ssh -v -J hipo@Jump-Host:443 -s sftp root@home-ssh-server.com -v

open failed: administratively prohibited: open failed

 

Finally decided to give a try to the same options arguments as in scp and thanks God it worked and I can even access via the Corporate Proxy through the Jump Host SSH interactively via Secure FTP 🙂

!! THE FINAL WORKING SFTP THROUGH PROXY VIA SSH JUMPHOST !!
 

sftp -o 'ProxyJump Username@Jumpt-Host-or-IP.com:443' Username@home-ssh-server.com


To save time from typing this long line every time, I've setup the following alias to ~/.bashrc
 

alias sftphome='sftp -o 'ProxyJump Username@Jumpt-Host-or-IP.com:443' Username@home-ssh-server.com'

 

Conclusion

Of course using own Proxy via your Home brew SSH Machine as well as transferring your data securely from your Work PC (notebook) to Home does not completely make you Surveillance free, as the Corporate Windows installed OS image is perhaps prebundled with its own integrated Keylogger as well as the Windows Domain administrators have certainly access to connect to your PC and run various commands, so this kind of Security is just an attempt to make company has less control and know less on your browsing habits and the best solution where possible to secure your privacy and separate your Personal Space form Work space by using a second computer (if having the ability to work from home) with a KVM Switch device and switch over your Work PC and Home PC via it or in some cases (where companies) allows it, setup something like VNC server (TightVNC / RealVNC) on work PC and leave it all time running in office and connect remotely with vncviewer from your own controlled secured computer.

In article I've explained shortly common scenario found in corporate Work computers proxy setup, designed to Surveil all your move, mentioned few common softwares running by default to protect from Viruses and aimed to Protect user from malicious hacking tools, explained how to view your work notebook configured Proxy, shortly mentioned on Proxy.pac and hinted how to view proxy.pac config as well as gave few of the reasons why all web traffic is being routed over central proxy.

That's all folks, Enjoy the Freedom to be less surveilled !

Virtualbox Shared folder set up on Linux between Host and Guest OS – Set up Virtualbox shared folder to Copy files from PC Host to Guest

Wednesday, September 12th, 2018

mount-shares-between-host-OS-and-guest-virtual-machine-howto-virtualbox-vbox-logo

How to set-up Virtualbox shared folder to Copy files from PC Host  and Guest Virtualized OS?

Running VirtualBox Host is an easy thing to set-up across all Operating Systems.  Once you have it sooner or later you will need to copy files from the VM Host OS (that in my case is GNU / Linux) to the virtualized Guest operating system (again in my case that's again another Linux ISO running indide the Virtual Machine).

Below are steps to follow To use Virtualbox Shared Folder functionality to copy files between VBox and your Desktop / server Linux install.

1. Install Virtualbox Guest Additions CD Image ISO

I've explained how to add the Guest Additions CD image thoroughfully in my previous article Howto enable Copy / Paste Virtualbox betwen Linux guest and Host OS
Anyways I'll repeat myself below for sake of clarity:

To do so use Oracle VBox menus (on the booted virtualized OS VBox window):

 

Devices -> Insert Guest additions CD Image

 

Mount the ISO inside the Linux Virtual Machine:

root@debian:~# mount /media/cdrom1/
 

If the mount fails and there are no files inside the mount point it might be because the virtualbox-dkms and virtualbox-guest-dkms packages might be missing on the Host OS.

To install them (on Debian GNU / Linux) assuming that you're using virtualbox default distro packages /etc/apt/sources.list :
 

apt-get install –yes -qq virtualbox-dkms virtualbox-guest-dkms


and run:

 

root@debian:/media/cdrom1# cd /media/cdrom1; sh VBoxLinuxAdditions.run


2. Create directory for Shared Folder that will be used to access Host / OS files from the Guest Virtualized OS
 

root@debian:~# mkdir /mnt/shared_folder

 

3. Map from VBox program interface Shared folder settings and Mount /mnt/shared_folder location

virtualbox-virtual-machine-devices-shared-folders-shared-folder-settings-linux-screenshot

 

Devices -> Shared Folder -> Shared Folder settings -> Transient Folders (click blue folder add small button right)

 

From Transient Folders add whatever directory you want to be shared from your local notebook / PC to the VM.

virtualbox-devices-Shared-Folder-Add-Shared-Folder-add-share-linux-screenshotDepending on whether you would like to mount the shared folder only for reading files (choose Read Only) to make it a permanent shared folder (and not just for the one session of current running Virtual Machine until its killed use Make Permanent) or check Auto-Mount tick if you want the shared_folder mapping to be mounted on every VM boot.

Once the shared_folder directory location is set-up from GUI menu click OK and in order for the settings to take effect, you'll need to restart the VM Guest with Linux (use halt command from terminal) or Power Off the Machine via the VBox menus.

To mount use command like:

mount -t vboxsf name_of_folder_linked_from_vbox  /mnt/name_folder_guest_os/


mount-vboxsf-shared-folder-mnt-shared-linux-guest-screenshot

In my case I wanted to share home folder /home so the command I used is:

root@debian:~# mount -t vboxsf  shared_folder /mnt/shared_folder


If everything is fine your Host OS file content from /home will be visible (for read and write if you Mapped it so) 
under /mnt/shared_folder …

And as Turtles Ninja used to heavily say Cowabunga !!! 🙂
You have it mounted and ready for file share between Desktop -> Virtualized OS.

 

Bear in mind that above mount command has to run as root (superuser) to succeed.

You now could copy files from your Host OS (running the Virtual Machine) and the Guest OS (Virtualized OS) using /mnt/shared_folder mount point without problems.

The example is if you want to share files between VirtualBox installed Linux and the Guest (Desktop / server) OS, however at many cases mounting your Host OS directory for root users might be not very practical but, instead you might prefer to do the mount for specific non admin user, for example I prefer to do the shared folder mount with my pointed non-root username hipo.

Here is how to do above VM shared_folder mount for non-root user:

First you need to know the exact UID / GID (User ID / Group ID) of user, you can get that with id command:

 

hipo@linux:~$  id
uid=1000(hipo) gid=1000(hipo) groups=1000(hipo),24(cdrom),25(floppy),29(audio),30(dip),44(video),46(plugdev),108(netdev),114(bluetooth),115(lpadmin),119(scanner)

 

As you see UID / GID in my case are 1000 / 1000

hipo@linux:~$ sudo mount -t vboxsf -o rw,uid=1000,gid=1000 shared_folder /mnt/shared_folder

 

mount-virtual-box-shared_folder-with-non-administration-permissions-non-root-permissions-id-and-mount-command-screenshot-linux


4. Mounting configured shared_folder to automatically mount into the Guest OS Linux on every boot

a) Configuring shared_folder auto-mount using /etc/rc.local

If you need the shared_folder to automatically mount next-time you boot the virtual machine quickest way is to add the mount command to /etc/rc.local (on Debian 8 and Debian 9 and newer Ubuntu Linuxes rc.local is missing by default to enable it to work like it worked before read follow my previous article ).

b) Configuring auto-mount for shared_folder through /etc/fstab

The more professional way to auto-mount on emulated OS VM boot time,  you could add the vboxsf mount definitions to /etc/fstab with your favourite text-editor mcedit, nano, joe etc. … (for me that's vim).

Syntax of /etc/fstab is as follows:
 

<Device> <Mount Point> <Type> <Options> <Dump> <Pass>

root@linux:~# vim /etc/fstab

 

shared_folder /mnt/shared_folder                                vboxsf rw,uid=1000,gid=1000 0 0

Note that you will want to change 1000 / 1000, id / gid with the ones of the non-admin user you would like to add to mount it for.

A quick way to add it to /etc/fstab with a shell one-liner is with command
 

root@linux:~# echo 'shared_folder /mnt/shared_folder                                vboxsf rw,uid=1000,gid=1000 0  0' >> /etc/fstab

An alternative way to add a user to have permissions for vboxsf file system (without specifying the long -o uid=1000,gid=1000 options is to simply add the username in question to group vboxsf like so:

c) Adding non super user username to vboxsf group

root@linux:~# usermod -G vboxsf hipo
root@linux:~# grep -i vboxsf /etc/group
vboxsf:x:999:hipo

 

hipo@linux:~$ sudo mount -t vboxsf  shared_folder /mnt/shared_folder

 

without the extra arguments and the options to pass to /etc/fstab (for eventual requirement to auto mount the shared_folder) would be more simple e.g.:

 

echo 'shared_folder /mnt/shared_folder                                vboxsf ' >> /etc/fstab

 

One note to make here is if the uesr is added to vboxsf the line for /etc/fstab to auto mount to mount for root user and non-root will be identical.

Then you can get the /etc/fstab auto-mount configured tested by running:

c) Checking auto-mount is working

hipo@linux:~# mount -a
hipo@linux:~# mount |grep -i vboxsf
shared_folder on /mnt/shared_folder type vboxsf (rw,nodev,relatime)


5. What if you end up with mounting failed errors ? – What might be causing the mounting failed Protocol error (a few things to check to solve)


In case of troubles with the mount you might get an error like:

hipo@linux:~# mount -t vboxsf  share_folder /mnt/shared_folder

/sbin/mount.vboxsf: mounting failed with the error: Protocol error


This error might be caused because of Insert Guest Additions CD Image might be not properly enabled and installed using the ISO provided VBoxLinuxAdditions.sh shell script.
Other common reason you might get this error if you have mistyped the Folder name: given in Shared Folders -> Folder Path -> Add Share for example I have given shared_folder as a Map name but as you can see in above mount -t vboxsf, I've mistyped share_folder instead of the correct one shared_folder inserted.
In some VBox releases this error was caused by bugs in the Virtual Machine.
 

virtualbox-virtual-machine-shared-folder-transient-folder-add-folder-linux-VM-guest-linux

One useful tip is to be able to check whether a Virtualbox Virtual Machine has a configured shared_folder (if you're logging to manage the machine on remote server – nomatter whether you have logged in with VNC / Teamviewer / Citrix etc. or via SSH session.

To do so use VBoxControl as of time of writting usually located on most distributions under (/usr/bin/VBoxControl)
 

 

hipo@linux:~# VBoxControl sharedfolder list -automount
Oracle VM VirtualBox Guest Additions Command Line Management Interface Version 5.2.18
(C) 2008-2018 Oracle Corporation
All rights reserved.

 

Auto-mounted Shared Folder mappings (0):

No Shared Folders available.

You can use VBoxControl command to get set and list a number of settings on the VBox VM, here is an useful example with it where you get information about numerous VBox info values:

 

root@linux:~# VBoxControl guestproperty enumerate
Oracle VM VirtualBox Guest Additions Command Line Management Interface Version 5.2.18
(C) 2008-2018 Oracle Corporation
All rights reserved.

 

Name: /VirtualBox/GuestInfo/OS/Product, value: Linux, timestamp: 1536681633430852000, flags: <NULL>
Name: /VirtualBox/GuestInfo/Net/0/V4/IP, value: 10.0.2.15, timestamp: 1536681633438717000, flags: <NULL>
Name: /VirtualBox/HostInfo/GUI/LanguageID, value: en_US, timestamp: 1536697521395621000, flags: RDONLYGUEST
Name: /VirtualBox/GuestInfo/Net/0/MAC, value: 08002762FA1C, timestamp: 1536681633442120000, flags: <NULL>
Name: /VirtualBox/GuestInfo/OS/ServicePack, value: <NULL>, timestamp: 1536681633431259000, flags: <NULL>
Name: /VirtualBox/HostInfo/VBoxVerExt, value: 5.2.18, timestamp: 1536681619002646000, flags: TRANSIENT, RDONLYGUEST
Name: /VirtualBox/GuestInfo/Net/0/V4/Netmask, value: 255.255.255.0, timestamp: 1536681633440157000, flags: <NULL>
Name: /VirtualBox/GuestInfo/OS/Version, value: #1 SMP Debian 4.9.110-3+deb9u2 (2018-08-13), timestamp: 1536681633431125000, flags: <NULL>
Name: /VirtualBox/GuestAdd/VersionExt, value: 5.2.18, timestamp: 1536681633431582000, flags: <NULL>
Name: /VirtualBox/GuestAdd/Revision, value: 124319, timestamp: 1536681633432515000, flags: <NULL>
Name: /VirtualBox/HostGuest/SysprepExec, value: <NULL>, timestamp: 1536681619002355000, flags: TRANSIENT, RDONLYGUEST
Name: /VirtualBox/GuestInfo/OS/LoggedInUsers, value: 1, timestamp: 1536681673447293000, flags: TRANSIENT, TRANSRESET
Name: /VirtualBox/GuestInfo/Net/0/Status, value: Up, timestamp: 1536681633443911000, flags: <NULL>
Name: /VirtualBox/GuestInfo/Net/0/Name, value: enp0s3, timestamp: 1536681633445302000, flags: <NULL>
Name: /VirtualBox/HostGuest/SysprepArgs, value: <NULL>, timestamp: 1536681619002387000, flags: TRANSIENT, RDONLYGUEST
Name: /VirtualBox/GuestAdd/Version, value: 5.2.18, timestamp: 1536681633431419000, flags: <NULL>
Name: /VirtualBox/HostInfo/VBoxRev, value: 124319, timestamp: 1536681619002668000, flags: TRANSIENT, RDONLYGUEST
Name: /VirtualBox/GuestInfo/Net/0/V4/Broadcast, value: 10.0.2.255, timestamp: 1536681633439531000, flags: <NULL>
Name: /VirtualBox/HostInfo/VBoxVer, value: 5.2.18, timestamp: 1536681619002613000, flags: TRANSIENT, RDONLYGUEST
Name: /VirtualBox/GuestInfo/OS/LoggedInUsersList, value: hipo, timestamp: 1536681673446498000, flags: TRANSIENT, TRANSRESET
Name: /VirtualBox/GuestInfo/Net/Count, value: 1, timestamp: 1536698949773993000, flags: <NULL>
Name: /VirtualBox/GuestInfo/OS/Release, value: 4.9.0-7-amd64, timestamp: 1536681633431001000, flags: <NULL>
Name: /VirtualBox/GuestInfo/OS/NoLoggedInUsers, value: false, timestamp: 1536681673447965000, flags: TRANSIENT, TRANSRESET
Name: /VirtualBox/GuestAdd/HostVerLastChecked, value: 5.2.18, timestamp: 1536681702832389000, flags: <NULL>

Hope you enjoyed ! Have phun! 🙂

How to turn keyboard backlight on GNU / Linux, keyboard no backlight solution

Friday, October 20th, 2017

how-to-make-CM_Storm_Devastator-keyboard_backlight-work-on-linux-enabled-disable-keyboard-glowing-gnu-linux

If you're a GNU / Linux user and you happen to buy a backlighted keyboard, some nice new laptop whose keyboard supports the more and more modern keyboard growing or if you happen to install a GNU / Linux for a Gamer friend no matter the Linux distribution, you might encounter sometimes  problem even in major Linux distributions Debian / Ubuntu / Mint / Fedora with keyboard backlight not working.

Lets say you buy a Devastator II backlighted keyboard or any other modern keyboard you plug it into the Linux machine and there is no nice blinking light coming out of the keyboard, all the joy is gone yes I know. The free software coolness would have been even more grandiose if your keyboard was shiny and glowing in color / colors 🙂

But wait, there is hope for your joy to be made complete.

To make the keyboard backlight switch on Just issue commands:

 

xmodmap -e 'add mod3 = Screen_Lock'

 

# Turn on the keyboard bright lamps
xset led on

# Turns off the keyboard bright lamps
xset led off


If you want to make the keyboard backlight be enabled permanent the easiest solution is to

– add the 3 command lines to /etc/rc.local

E.g. to do so open /etc/rc.local and before exit 0 command just add the lines:

 

vim /etc/rc.local

 

xmodmap -e 'add mod3 = Screen_Lock'

# Turn on the keyboard bright lamps
xset led on

# Turns off the keyboard bright lamps
xset led off


If you prefer to have the keyboard colorful backlight enable and disabled from X environment on lets say GNOME , here is how to make yourself an icon that enabled and disables the colors.

That's handy because at day time it is a kind of meaningless for the keyboard to glow.

Here is the shell script:

#!/bin/bash
sleep 1
xset led 3
xmodmap -e 'add mod3 = Scroll_Lock'


I saved it as /home/hipo/scripts/backlight.sh

(don't forget to make it executable!, to do so run):

 

chmod +x /home/hipo/scripts/backlight.sh


Then create  the .desktop file at /etc/xdg/autostart/backlight.desktop so that it runs the new shell script, like so:

[Desktop Entry]
Type=Application
Name=Devastator Backlight
Exec=/home/hipo/scripts/backlight.sh
Icon=system-run
X-GNOME-Autostart-enabled=true

Share SCREEN terminal session in Linux / Screen share between two or more users howto

Wednesday, October 11th, 2017

share-screen-terminal-session-in-linux-share-linux-unix-shell-between-two-or-more-users

 

1. Short Intro to Screen command and what is Shared Screen Session

Do you have friends who want to learn some GNU / Linux or BSD basics remotely? Do you have people willing to share a terminal session together for educational purposes within a different network? Do you just want to have some fun and show off yourself between two or more users?

If the answer to the questions is yes, then continue on reading, otherwise if you're already aware how this is being done, just ignore this article and do something more joyful.

So let me start.

Some long time ago when I was starting to be a Free Software user and dedicated enthusiast, I've been given by a friend an interesting freeshell hosting access and I stumbled upon / observed an interesting phenomenon, multiple users like 5 or 10 were connected simultaneously to the same shell sharing their command line.

I can't remember what kind of shell I happen to be sharing with the other logged in users with the same account, was that bash / csh / zsh or another one but it doesn't matter, it was really cool to find out multiple users could be standing together on GNU / Linux and *BSD with the same account and use the regular shell for chatting or teaching each others  new Linux / Unix commands e.g. being able to type in shell simultaneously.

The multiple shared shell session was possible thanks to the screen command

For those who hear about screen for a first time, here is the package description:

 

# apt-cache show screen|grep -i desc -A 1
Description-en: terminal multiplexer with VT100/ANSI terminal emulation
 GNU Screen is a terminal multiplexer that runs several separate "screens" on

Description-md5: 2d86b86ed6058a04c540802e49312f40
Homepage: https://savannah.gnu.org/projects/screen
root@jericho:/usr/local/src/pure-python-otr# apt-cache show screen|grep -i desc -A 2
Description-en: terminal multiplexer with VT100/ANSI terminal emulation
 GNU Screen is a terminal multiplexer that runs several separate "screens" on
 a single physical character-based terminal. Each virtual terminal emulates a


Description-md5: 2d86b86ed6058a04c540802e49312f40
Homepage: https://savannah.gnu.org/projects/screen
Tag: hardware::input:keyboard, implemented-in::c, interface::text-mode,


There is plenty of things to use screen for as it provides you a way to open Virtual Terminals into a single ssh or physical console TTY login session and I've been in love with screen command since day 1 I found out about it.

To start using screen just invoke it into a shell and enter a screen command combinations that make various stuff for you.

 

2. Some of the most useful Daily Screen Key Combinations for the Sys Admin


To do use the various screen options, use the escape sequence (CTRL + Some Word), following by the command. For a full list of all of the available commands, run man screen, however
for the sake of interest below short listing shows some of most useful screen key combination invoked commands:

 

 

Ctrl-a a Passes a Ctrl-a through to the terminal session running within screen.
Ctrl-a c Create a new Virtual shell screen session within screen
Ctrl-a d Detaches from a screen session.
Ctrl-a f Toggle flow control mode (enable/disable Ctrl-Q and Ctrl-S pass through).
Ctrl-a k Detaches from and kills (terminates) the screen session.
Ctrl-a q Passes a Ctrl-q through to the terminal session running within screen (or use Ctrl-a f to toggle whether screen captures flow control characters).
Ctrl-a s Passes a Ctrl-s through to the terminal session running within screen (or use Ctrl-a f to toggle whether screen captures flow control characters).
Ctrl-a :kill Also detaches from and kills (terminates) the screen session.
Ctrl-a :multiuser on Make the screen session a multi-user session (so other users can attach).
Ctrl-a :acladd USER Allow the user specified (USER) to connect to a multi-user screen session.
Ctrl-a p Move around multiple opened Virtual terminals in screen (Move to previous)
Ctrl-a n Move backwards in multiple opened screen sessions under single shell connection


I have to underline strongly for me personally, I'm using the most

 

CTRL + A + D (to detach session),

CTRL + A + C to open new session within screen (I tend to open multiple sessions for multiple ssh connections with this),

CTRL + A + P, CTRL +  A + N – I use this twoto move around all my open screen Virtual sessions.
 

3. HOW TO ACTUALLY SHARE TERMINAL SESSION BETWEEN MULTIPLE USERS?


3.1 Configuring Shared Sessions so other users can connect

You need to  have a single user account on a Linux or Unix like server lets say that might be the /etc/passwd, /etc/shadow, /etc/group account screen and you have to give the password to all users to be participating into the shared screen shell session.

E.g. create new system account screen

root@jericho:~# adduser screen
Adding user `screen' …
Adding new group `screen' (1001) …
Adding new user `screen' (1001) with group `screen' …
The home directory `/home/screen' already exists.  Not copying from `/etc/skel'.
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Changing the user information for screen
Enter the new value, or press ENTER for the default
    Full Name []: Screen user to give users shared access to /bin/bash
    Room Number []:
    Work Phone []:
    Home Phone []:
    Other []:
Is the information correct? [Y/n] Y

Now distribute the user / pass pair around all users who are to be sharing the same virtual bash session via screen and instruct each of them to run:

hipo@jericho:~$  screen -d -m -S shared-session
hipo@jericho:~$

hipo@jericho:~$ screen -list

There is a screen on:
    4095.shared-session    (10.10.2017 20:22:22)    (Detached)
1 Socket in /run/screen/S-hipo.


3.2. Attaching to just created session
 

Simply login with as many users you need with SSH to the remote server and instruct them to run the following command to re-attach to the just created new session by you:

hipo@jericho:~$ screeen -x

That's all folks now everyone can type in simultaneously and enjoy the joys of the screen shared session.

If for some reasons more than one session is created by the simultaneously logged in users either as an exercise or by mistake i.e.:

hipo@jericho:~$ screen -list

There are screens on:
    4880.screen-session    (10.10.2017 20:30:09)    (Detached)
    4865.another-session    (10.10.2017 20:29:58)    (Detached)
    4847.hey-man    (10.10.2017 20:29:49)    (Detached)
    4831.another-session1    (10.10.2017 20:29:45)    (Detached)
4 Sockets in /run/screen/S-hipo.

You have to instruct everyone to connect actually to the exact session we need, as screen -x will ask them to what session they like to connect.

In that case to connect to screen-session, each user has to run with their account:

hipo@jericho:~$ screen -x shared-session


If under some circumstances it happened that there is more than one opened shared screen virtual session, for example screen -list returns:

 

hipo@jericho:~$ screen -list
There are screens on:
    5065.screen-session    (10.10.2017 20:33:20)    (Detached)
    4095.screen-session    (10.10.2017 20:30:08)    (Detached)

All users have to connect to the exact screen-session created name and ID, like so:

hipo@jericho:~$ screen -x 4095.screen-session
 


Here is the meaning of used options

 

-d option instructs screen to detach,
-m makes it multiuser session so other users can attach
-S argument is just to give the screen session a name
-list Sesssion gives the screen-session ID

Once you're over with screen session (e.g. all users that are learning and you show them stuff and ask them to test by themselves and have completed, scheduled tasks), to kill it just press CTRL + A + K
 

4. Share screen /bin/bash shell session with another user

Sharing screen session between different users is even more useful to the shared session of one user as you might have a *nix server with many users who might attach to your opened session directly, instead of being beforehand instructed to connect with a single user.

That's perfect also for educational purposes if you want to learn some Linux to a class of people, as you can use their ordinary accounts and show them stuff on a Linux / BSD  machine.

Assuming that you follow and created already screen-session with screen cmd

hipo@jericho:~$ screen -list
There is a screen on:
        5560.screen-session      (10.10.2017 20:41:06)   (Multi, attached)
1 Socket in /run/screen/S-hipo.

hipo@jericho:~$

Next attach to the session

bunny@jericho:~$ screen -r shared-session
bunny@jericho:~$ Ctrl-a :multiuser on
bunny@jericho:~$ Ctrl-a :acladd user2
bunny@jericho:~$ screen -x UserNameHere/shared-session
 

Here are 2 screenshots on what should happen if you had done above command combinations correctly:

screen-share-session-to-multi-users-screenshot-multiuser-on-on-gnome-terminal2

screen-share-session-to-multi-users-screenshot-multiuser-on-on-gnome-terminal3

In order to be able to share screen Virtual terminal ( VTY ) sessions between separate (different) logged in users, you have to have screen command be suid (SUID bit for screen is disabled in most Linux distributions for security reasons).

Without making SUID the screen binary file, you are to get the error:

hipo@jericho:/home/hipo$ screen -x hipo/shared

Must run suid root for multiuser support.

If you are absolutely sure you know what you're doing here is how to make screen command sticky bit:

 

root@jericho:/home/hipo# which screen
root@jericho:/home/hipo# /usr/bin/screen
root@jericho:/home/hipo# root@jericho:/home/hipo# root@jericho:/home/hipo# chmod u+s $(which screen)
chmod 755 /var/run/screen
root@jericho:/home/hipo# rm -fr /var/run/screen/*
exit

How much memory users uses in GNU / Linux and FreeBSD – Commands and Scripts to find user memory usage on Linux

Tuesday, February 17th, 2015

 

how-much-memory-users-use-in-gnu-linux-freebsd-command-to-find-and-show-ascending-descending-usage-of-system-memory-tux-memory-logo

 


If you have to administrate a heterogenous network with Linux and FreeBSD or other UNIX like OSes you should sooner or later need for scripting purposes to have a way to list how much memory separate users take up on your system. Listing memory usage per user is very helpful for admins who manager free-shells or for companies where you have developers, developing software directly on the server via ssh. Being able to check which process eats up most memory is essential for every UNIX / Linux sysadmin, because often we as admins setup (daemons) on servers and we forgot about their existence, just to remember they exist 2 years later and see the server is crashing because of memory exhaustion. Tracking server bottlenecks where RAM memory and Swapping is the bottleneck is among the main swiss amry knives of admins. Checking which user occupies all server memory is among the routine tasks we're forced to do as admins, but because nowdays servers have a lot of memory and we put on servers often much more memory than ever will be used many admins forget to routinely track users / daemons memory consumption or even many probably doesn't know how.  Probably all are aware of the easiest wy to get list of all users memory in console non interactively with free command, e.g.:
 

free -m
             total       used       free     shared    buffers     cached
Mem:         32236      26226       6010          0        983       8430
-/+ buffers/cache:      16812      15424
Swap:        62959        234      62725

 

but unfortunately free command only shows overall situation with memory and doesn't divide memory usage by user

Thus probably to track memory users the only known way for most pepole is to (interactively) use good old top command or if you like modern (colorful) visualization with htop:

debian:~# top

 

linux-check_memory_usage_by_logged-in-user-with-top-process-command-gnu-linux-freebsd-screenshot

Once top runs interactive press 'm' to get ordered list of processes which occupy most system memory on Linux server.Top process use status statistics will refresh by default every '3.0' seconds to change that behavior to '1' second press  s and type '1.0'. To get Sort by Memory Use in htop also press 'm'
 

[root@mail-server ~]# htop


htop_show_users_memory_usage_order_ascending-gnu-linux-screenshot

 

However if you need to be involved in scripting and setting as a cron job tasks to be performed in case if high memroy consumption by a service you will need to use few lines of code. Below are few examples on how Linux user memory usage can be shown with ps cmd.

Probably the most universal way to see memory usage by users on Debian / Ubuntu / CentOS / RHEL and BSDs (FreeBSD / NetBSD) is with below one liner:

 

server:~# ps hax -o rss,user | awk '{a[$2]+=$1;}END{for(i in a)print i” “int(a[i]/1024+0.5);}' | sort -rnk2
daemon 0
debian-tor 63
dnscache 1
dnslog 0
hipo 21
messagebus 1
mysql 268
ntp 2
privoxy 1
proftpd 1
qmaill 0
qmailq 0
qmailr 0
qmails 0
qscand 291
root 94
shellinabox 1
snmp 1
statd 1
vpopmail 80
www-data 6765

 

Output is in MBs

Below is output from machine where this blog is running, the system runs ( Apache + PHP + MySQL Webserver + Qmail Mail server and Tor) on Debian GNU / Linux.

 To get more human readable (but obscure to type – useful for scripting) output list of which user takes how much memory use on deb / rpm etc. based Linux :

 

server:~# echo "USER                 RSS      PROCS" ; echo "——————– ——– —–" ; \
ps hax -o rss,user | awk '{rss[$2]+=$1;procs[$2]+=1;}END{for(user in rss) printf “%-20s %8.0f %5.0f\n”, user, rss[user]/1024, procs[user];}' | sort -rnk2

 

USER                 RSS      PROCS
——————– ——– —–
www-data                 6918   100
qscand                    291     2
mysql                     273     1
root                       95   120
vpopmail                   81     4
debian-tor                 63     1
hipo                       21    15
ntp                         2     1
statd                       1     1
snmp                        1     1
shellinabox                 1     2
proftpd                     1     1
privoxy                     1     1
messagebus                  1     1
dnscache                    1     1
qmails                      0     2
qmailr                      0     1
qmailq                      0     2
qmaill                      0     4
dnslog                      0     1
daemon                      0     2

 

It is possible to get the list of memory usage listed in percentage proportion, with a tiny for bash loop and some awk + process list command
 

TOTAL=$(free | awk '/Mem:/ { print $2 }')
for USER in $(ps haux | awk '{print $1}' | sort -u)
do
    ps hux -U $USER | awk -v user=$USER -v total=$TOTAL '{ sum += $6 } END { printf "%s %.2f\n", user, sum / total * 100; }'
done

107 1.34
115 2.10
119 1.34
daemon 1.32
dnscache 1.34
dnslog 1.32
hipo 1.59
mysql 4.79
ntp 1.34
privoxy 1.33
proftpd 1.32
qmaill 1.33
qmailq 1.33
qmailr 1.32
qmails 1.33
qscand 4.98
root 1.33
snmp 1.33
statd 1.33
vpopmail 2.35
www-data 86.48

Also a raw script which can be easily extended to give you some custom information on memory use by user list_memory_use_by_user.sh is here.
You can also want to debug further how much memory a certain users (lets say user mysql and my username hipo) is allocating, this can easily be achieved ps like so:
 

root@pcfreak:~# ps -o size,pid,user,command -u mysql –sort -size
 SIZE   PID USER     COMMAND
796924 14857 mysql   /usr/sbin/mysqld –basedir=/usr –datadir=/var/lib/mysql –plugin-dir=/usr/lib/mysql/plugin –user=mysql –pid-file=/var/run/mysqld/mysqld.pid –socket=/var/run/mysqld/mysqld.sock –port=3306

 

root@pcfreak~# ps -o size,pid,user,command -u hipo –sort -size|less
 SIZE   PID USER     COMMAND
13408 19063 hipo     irssi
 3168 19020 hipo     SCREEN
 2940  2490 hipo     -bash
 1844 19021 hipo     /bin/bash
 1844 19028 hipo     /bin/bash
 1844 19035 hipo     /bin/bash
 1844 19042 hipo     /bin/bash
 1844 19491 hipo     /bin/bash
 1844 22952 hipo     /bin/bash
  744  2487 hipo     sshd: hipo@pts/0
  744  2516 hipo     sshd: hipo@notty
  524  2519 hipo     screen -r
  412  2518 hipo     /usr/lib/openssh/sftp-server

You see from below output user running with www-data (this is Apache Webserver user in Debian) is eating 86.48% of overall system memory and MySQL server user is using only 4.79% of available memory

Output is shown in Megabytes per username memory usage, and user memory usage is ordered (stepping-down / descentive) from top to bottom

Getting more thoroughful and easier to read reporting without beeing a 31337 bash coder you can install and use on Linux smem – memory reporting tool .

SMEM can provide you with following memory info:

  • system overview listing
  • listings by process, mapping, user
  • filtering by process, mapping, or user
  • configurable columns from multiple data sources
  • configurable output units and percentages
  • configurable headers and totals
  • reading live data from /proc
  • reading data snapshots from directory mirrors or compressed tarballs
  • lightweight capture tool for embedded systems
  • built-in chart generation


Installing smem on Debian 6 / 7 / Ubuntu 14.04 / Turnkey Linux etc. servers is done with standard:

 

debian:~# apt-get install –yes smem
….

 

 

To install smem on CentOS 6 / 7:

 

[root@centos ~ ]# yum -y install smem
….


On Slackware and other Linux-es where smem is not available as a package you can install it easily from binary archive with:

 

cd /tmp/
wget http://www.selenic.com/smem/download/smem-1.3.tar.gz
tar xvf smem-1.3.tar.gz
sudo cp /tmp/smem-1.3/smem /usr/local/bin/
sudo chmod +x /usr/local/bin/smem

 


Two most common smem uses are:

 

root@mail:~# smem -u
User     Count     Swap      USS      PSS      RSS
dnslog       1       44       48       54      148
qmaill       4      232      124      145      464
hipo        11    13552     8596     9171    13160
qscand       2     4500   295336   295602   297508
root       188   217312  4521080  4568699  7712776

 

Below command shows (-u – Report memory usage by user, -t – show totals, -k – show unix suffixes)

root@mail:~# smem -u -t -k
User     Count     Swap      USS      PSS      RSS
dnslog       1    44.0K    48.0K    54.0K   148.0K
qmaill       4   232.0K   124.0K   145.0K   464.0K
hipo        11    13.2M     8.4M     9.0M    12.9M
qscand       2     4.4M   288.4M   288.7M   290.5M
root       188   212.2M     4.3G     4.4G     7.4G
—————————————————
           206   230.1M     4.6G     4.6G     7.7G


To get users memory use by percentage with smem:
 

root@mail:~# smem -u -p
User     Count     Swap      USS      PSS      RSS
dnslog       1    0.00%    0.00%    0.00%    0.00%
qmaill       4    0.00%    0.00%    0.00%    0.01%
hipo        11    0.17%    0.11%    0.11%    0.16%
qscand       2    0.05%    3.63%    3.63%    3.66%
root       194    2.64%   56.18%   56.77%   95.56%

It is also useful sometimes when you want to debug system overloads caused by external hardware drivers loaded into kernel causing issues to get list of system wide memory use sorted by user

 

 root@mail:~# smem -w -p
Area                           Used      Cache   Noncache
firmware/hardware             0.00%      0.00%      0.00%
kernel image                  0.00%      0.00%      0.00%
kernel dynamic memory        38.30%     36.01%      2.28%
userspace memory             60.50%      0.98%     59.53%
free memory                   1.20%      1.20%      0.00%


smem is very nice as if you're running it on a Desktop Linux system with Xserver installed you can see also graphical output of memory use by application:
 

root@desktop-pc:~# smem –bar pid -c "pss uss"


smem_graphical_representation-of-which-user-application-is-consuming-most-memory-gnu-linux-kde-screenshot-smem-command-line-tool

smem can even generate graphical pie charts to visualize better memory use
 

root@desktop-pc:~# smem -P '^k' –pie=name

 

generate-graphical-staticstics-linux-memory-use-by-pie-chart

If there is a high percentage shown in firmware/hardware this means some buggy module is loaded in kernel eating up memory, to fix it debug further and remove the problematic module.
userspace memory actually shows the percantage of memory out of all server available RAM that is being consumed by applications (non kernel and other system processes which make the system move). You see in above example the kernel itself is consuming about 40% of system overall available memory. 

We all know the SWAP field stands for hard disk drive used as a memory when system is out, but there are 3 fields which smem will report which will be probably unclear for most here is also explanation on what USS / PSS / RSS means?

RSS is the Resident Set Size and is used to show how much memory is allocated to that process and is in RAM. It does not include memory that is swapped out. It does include memory from shared libraries as long as the pages from those libraries are actually in memory. It does include all stack and heap memory too.

There is also PSS (proportional set size). This is a newer measure which tracks the shared memory as a proportion used by the current process. So if there were two processes using the same shared library from before.

USS stands for Unique set size, USS is just the unshared page count, i.e. memory returned when process is killed 

PSS = Proportional set size, (PSS),  is a more meaningful representation of the amount of memory used by libraries and applications in a virtual memory system.  
Because large portions of physical memory are typically shared among multiple applications, the standard measure of memory usage known as resident set size (RSS) will significantly overestimate memory usage. The parameter PSS instead measures each application’s “fair share” of each shared area to give a realistic measure. For most admins checking out the output from RSS (output) should be enough, it will indicate which user and therefore which daemon is eating up all your memory and will help you to catch problematic services which are cause your server to run out of RAM and start swapping to disk.

Fun with Apache / Nginx Webserver log – Visualize webserver access log in real time

Friday, July 18th, 2014

visualize-graphically-web-server-access-log-logstalgia-nginx-apache-log-visualize-in-gnu-linux-and-windows
If you're working in a hosting company and looking for a graphical way to Visualize access to your Linux webservers – (Apache, Nginx, Lighttpd) you will be happy to learn about Logstalgia's existence. Logstalgia is very useful if you need to convince your Boss / company clients that the webservers are exceeding the CPU / Memory hardware limits physically servers can handle. Even if you don't have to convince anyone of anything logstalgia is cool to run if you want to impress a friend and show off your 1337 4Dm!N Sk!11Z 🙂 Nostalgia is much more pleasent way to keep an eye on your Webserver log files in real time better than (tail -f)

The graphical output of nostalgia is a pong-like battle game between webserver and never ending chain of web requests.

This is the official website description of Logstalgia:
 

Logstalgia is a website traffic visualization that replays web-server access logs as a pong-like battle between the web server and an never ending torrent of requests. Requests appear as colored balls (the same color as the host) which travel across the screen to arrive at the requested location. Successful requests are hit by the paddle while unsuccessful ones (eg 404 – File Not Found) are missed and pass through. The paths of requests are summarized within the available space by identifying common path prefixes. Related paths are grouped together under headings. For instance, by default paths ending in png, gif or jpg are grouped under the heading Images. Paths that don’t match any of the specified groups are lumped together under a Miscellaneous section.


To install Logstalgia on Debian / Ubuntu Linux there is a native package, so to install it run the usual:

apt-get --yes install logstalgia

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  logstalgia
0 upgraded, 1 newly installed, 0 to remove and 4 not upgraded.
Need to get 161 kB of archives.
After this operation, 1,102 kB of additional disk space will be used.
Get:1 http://mirrors.kernel.org/debian/ stable/main logstalgia amd64 1.0.0-1+b1 [161 kB]
Fetched 161 kB in 2s (73.9 kB/s)
Selecting previously deselected package logstalgia.
(Reading database ... 338532 files and directories currently installed.)
Unpacking logstalgia (from .../logstalgia_1.0.0-1+b1_amd64.deb) ...
Processing triggers for man-db ...
Setting up logstalgia (1.0.0-1+b1) ...


Logstalgia is easily installable from source code on non-Debian Linux distributions too, to install it on any non-debian Linux distrubution do:

cd /usr/local/src/ wget https://logstalgia.googlecode.com/files/logstalgia-1.0.5.tar.gz
 

–2014-07-18 13:53:23–  https://logstalgia.googlecode.com/files/logstalgia-1.0.3.tar.gz
Resolving logstalgia.googlecode.com… 74.125.206.82, 2a00:1450:400c:c04::52
Connecting to logstalgia.googlecode.com|74.125.206.82|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: 841822 (822K) [application/x-gzip]
Saving to: `logstalgia-1.0.3.tar.gz'

100%[=================================>] 841,822     1.25M/s   in 0.6s

2014-07-18 13:53:24 (1.25 MB/s) – `logstalgia-1.0.3.tar.gz' saved [841822/841822]

Untar the archive with:
 

tar -zxvf logstalgia-1.0.5.tar.gz

Compile and install it:

cd logstalgia
./configure
make
make install

 

How to use LogStalgia?

Syntax is pretty straight forward just pass the Nginx / Apache

Process Debian Linux Apache logs:

logstalgia /var/log/apache2/access.log


Process CentoS, Redhat etc. RPM based logs:

logstalgia /var/log/httpd/access.log
To process webserver log in real time with logstalgia:

tail -f /var/log/httpd/access_log | logstalgia -

To make logstalgia visualize log output you will need to have access to server physical console screen. As physical access is not possible on most dedicated servers – already colocated in some Datacenter. You can also use a local Linux PC / notebook installed with nostalgia to process webserver access logs remotely like so:

logstalgia-visualize-your-apache-nginx-lighttpd-logs-graphically-in-x-and-console-locally-and-remotely

ssh hipo@pc-freak.net tail -f /var/log/apache2/access.log | logstalgia --sync

Note! If you get an empty output from logstalgia, this is because of permission issues, in this example my user hipo is added in www-data Apache group – if you want to add your user to have access like me, issue on remote ssh server):
 

addgroup hipo www-data


Alterantively you can login with ssh with root, e.g. ssh root@pc-freak.net

If you're having a GNOME / KDE X environment on the Linux machine from which you're ssh-ing Logstalgia will visualize Webserver access.log requests inside a new X Window otherwise if you're on a Linux with just a console with no Xserver graphics it will visualize graphically web log statistics using console svgalib .

 

If you're planning to save output from nostalgia visualization screen for later use – lets say you have to present to your CEO statistics about all your servers  Webservers logs you can save nostalgia produced video in .ppm (netpbm) format.

Whether you have physical console access to the server:

logstalgia -1280x720 --output-ppm-stream output.ppm /var/log/httpd/access.log

Or if you just a have a PC with Linux and you want to save visualized content of access.log remotely:

ssh hipo@pc-freak.net tail -f /var/log/nginx/pc-freak-access.log | logstalgia -1280x720 --output-ppm-stream --sync output.ppm

 

ssh user@server1.cyberciti.biz tail -f /var/log/nginx/www.cyberciti.biz_access.log | logstalgia -1280x720 --output-ppm-stream --sync output.ppm

To make produced .ppm later usable you can use ffmpeg to convert to .mp4:

ffmpeg -y -r 60 -f image2pipe -vcodec ppm -i output.ppm -vcodec libx264 -preset ultrafast -pix_fmt yuv420p -crf 1 -threads 0 -bf 0 nginx.server.log.mp4

Then to play the videos use any video player, I usually use vlc and mplayer.

For complete info on Nostalgia – website access log visualizercheck home page on googlecode

If you're lazy to install Logstalgia, here is Youtube video made from its output:

Enjoy 🙂

Linux: Understanding uptime command Load Avarage statistics / When load avarage is high?

Wednesday, May 8th, 2013

GNU / Linux load avarage explained load avarage from top command

There is probably no Linux system administrator who, don't have idea about system  Load Avarage. Most of admins however does have some brought idea about what kind of load avarage is critical but doesn't have good understanding on the 3 digits returned as a load avarage i.e. – load average: 2.47, 2.27, 2.02 shown in above ascii graphs ( generated by tload command).
 

What is Load Avarage ?

  •  The number of blocking processes in the run queue averaged over a certain time period.

A blocking process is a process that is waiting for something to continue. Typically, a process is waiting to use:
 

  •  CPU Time,  Disk Input / Output oper. or Network I / O

Thus logically the higher the Load Avarage, the more processes has to wait for access to CPU, HDD and Network I/O.

The most two common commands used where load avarage appear are;
 

w – who

and

uptime

mx:/home/hipo# w
 11:07:56 up 513 days,  1:04,  1 user,  load average: 1.92, 1.95, 1.84
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
hipo     pts/0    pc-freak.net     Thu19    0.00s  0.06s  0.02s sshd: hipo [priv]

mail:/home/hipo# uptime
 11:03:59 up 513 days,  1:00,  2 users,  load average: 2.11, 1.91, 1.81

Other common place to check load avarage is in top cmd:

mail:/home/hipo# top

Linux top command load avarage showing server system load 3 digits of load avarage explained

a) Optimum machine use – Load Avarage 1 

So what does load avarage: 1.74, 1.90, 1.83 really means? The 3 digits are showing system load avarage over the last 1, 5 and 15 minutes time. Meaning;

 

– before 1 minute system had a load of 1.74
– 5 minutes before it was 1.90
– and 15 minutes back 1.83

Usually Load Avarage of more than 1 is considered critical. If a system is working with a load avarage of 1 this means the system is working capacity. In best cases in terms of optimizing processes on server with hardware it is good the system is working in load of 0.70 or 0.80. Whether a traffic the machine gets is planned in most cases a load avarage of exactly 1 means machine hardware is properly utilized. However whether the load avarage is hitting over 1 this usually means you have to think about moving server to new hardware. It is general rule of thumb that if system load is exceeeding 0.70 it is time to migrate to better hardware.

b) Load avarage on Multi-core / Multiprocessor servers

Load avarage of 4 on 4 CPU cores server hardware is optimum one. Each core / CPU on machine should get maximum of load avarage 1. Load avarage of 1 means CPU is utilized in 100%. Load avarage of 4 on 4 CPU server hardware means all 4 processors are working in their maximum power of 100%. For people who have multi processor server the best way to show utilization is by running htop. There all 4 CPUs will show idle of 0%.

Hence rule to calculate normal load avarage for server is;

1 Load Avarage per CPU. Therefore for 24 CPU Intel Xeon hardware. Load Avarage under 1*24.00 = 24.00 is considered normal. On such a server whether load avarage jumps to 50.00 / 70.00 or above server becomes totally irresponsive and it is very likely to hang because of over-heating. Even if it continues working it will work extremely slow and even simply operations like ssh to it will become hardly possible and sometimes even access via ssh will be not possible.

Therefore Rule of Thumb for calculating which load avarage is okay for a server is;

Number of CPU / Cores should not exceed digit returned in Load Avarage stats

c) Critical – Load avarage >5 – A sure sign for unresponsive or soon to hang server

On Computers with just 1 CPU, load avarage of 5 is sure sign running services will lag brutally and server will become inaccessible. For multicore / multiprocessor servers big troubles can be expected, whether load avarage is about  1/2 of the maximum number of of Load Avarage; (for 8 CPU Multicore hardware). A load avarage of 8 + ( 1/2 * 8 ) = 12 is sure sign system is stoned and running services inaccessible.

d) load avarage: 1.74, 1.90, 1.83 – Is 1 / 5 or 15 minutes LA numbers more important to consider?

All are important however 5 and 15 minutes load avg. give better indication on what's happening with machine as current load can peak for just a second to a higher number, being misleading.

To get number of CPU / Cores use cmd;

mail:/home/hipo# grep 'model name' /proc/cpuinfo | wc -l
24

For more precise info on CPU type and model use;

mail:/home/hipo# cat /proc/cpuinfo

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           E5645  @ 2.40GHz
stepping        : 2
cpu MHz         : 2400.094
cache size      : 12288 KB
physical id     : 0
siblings        : 12
core id         : 0
cpu cores       : 6
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida arat tpr_shadow vnmi flexpriority ept vpid
bogomips        : 4800.18
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:
……
………
 

If you hit abnormal high load avarage, it is useful to check in top process list what is top process / processes causing highest system load. It is useful to run ps with following arguments

mail:/home/hipo# ps axuwwf

Look in STAT column. Processes in STAT have 3 states;

  • R – Running
  • S – Sleeping
  • D – Waiting for something

Usually processes with status of D – are ones causing problems. If you get D STAT-ed processes check further what's wrong with them and fix it. If there are none,  simply, number of clients using machine pop-ed up meaning you need to quickly move to better hardware host.

e) Getting notified via email whether load avarage exceeds certain value

A good way to get notified or do certain action like restarting Apache WebServer or other common process causing high loads is through monit. Monit is very usefukl for notifying on high load avarages or even better for Restarting processes imposing high loads.

You can always use a few liners shell script to mail to email or SMS2Email mailbox similar to this tiny shell script to restart apache on high load.There is also a Ruby lang tool – Scout to monitor and report high load avarages

How to disable Debian GNU / Linux (Squeeze) Apache 2 version reporting to improve security – Hide Apache server version

Friday, October 5th, 2012

Debian GNU / Linux's Apache default behavior is to report Apache server name, version and Linux distribution version and codename.
This is shown as a minor security leak in many Security Scanner (audit) software like Nessus. It reveals a vital information which could help malicious attacker later to use exploit if the version number in question is vulnerable.

The quickest way to check, either Apache versioning and distro info is disabled is with telnet:

hipo@noah:~/Desktop$ telnet pc-freak.net 80
Trying 83.228.93.76…
Connected to pc-freak.net.
Escape character is '^]'.
HEAD / HTTP/1.0

Connection closed by foreign host.
hipo@noah:~/Desktop$ telnet pc-freak.net 80
Trying 83.228.93.76…
Connected to pc-freak.net.
Escape character is '^]'.
HEAD / HTTP/1.0

HTTP/1.1 200 OK
Date: Fri, 05 Oct 2012 10:48:36 GMT
Server: Apache/2.2.16 (Debian)
X-Powered-By: PHP/5.3.3-7+squeeze14
Vary: Accept-Encoding
Connection: close
Content-Type: text/html

Disabling this Distro version codename and version number reporting on Debian is done by changing in file /etc/apache2/conf.d/security directives:


ServerTokens OS
ServerSignature On

to


ServerSignature Off
ServerTokens ProductOnly

Here important note to make is if you try adding:

ServerSignature Off and ServerTokens Prod straight in Debian general config /etc/apache2/apache2.conf, but did not change the settings set for the vars through /etc/apache2/conf.d/security; settings from /etc/apache2/conf.d/security will overwrite ServerSignature / ServerTokens settings set in /etc/apache2/apache2.conf

I tried this myself (forgotting about /etc/apache2/conf.d/security) and adding both variables straight in apache2.conf. After Apache restart Apache version number and type of distribution continued be returned by the WebServer.
I thought something specific changed in Debian Squeeze – Apache/2.2.16 so this two variables are probably not working so I did a quick research online seing other people complaining also unable to disable Apache ver and Linux distro version and looking for a reason why. Well anyways if you happen to also ponder, why ServerSignature Off and ServerTokens ProductOnly does not take effect keep in mind it is due to overwritten settings via /etc/apache2/conf.d/security, changing the values there and restarting Apache and you're done 🙂

To make sure 100% Apache is no longer returning exact version number and host installed distro type, use telnet again:

hipo@noah:~/Desktop$ telnet pc-freak.net 80
Trying 83.228.93.76…
Connected to pc-freak.net.
Escape character is '^]'.
HEAD / HTTP/1.0

Connection closed by foreign host.

Enabling talkd (Console Chat) between logged in users on FreeBSD and other BSDs

Sunday, June 10th, 2012

Talk between two useres on  FreeBSD 7.2 screenshot, console peer to peer interactive talk program UNIX, Linux, BSD

Those who are in familiar with older UNIXes, UNIX BSD derivatives and GNU Linux should certainly remember the times, when we hackers used to talk to each other using talk service.

Those who don't know what talk command is it is a simple console / ssh utility to talk to another logged in users.

Talk is very similar to write and mesg one liner messasing utilities available for *nixes, the difference is it is intendted to provide interactive chat between the two logged in users. People who came to know UNIX or free software in older times most likely don't know talk, however I still remember how precious this tool was for communication back in the day.

I believe still it can be useful so I dediced to install ot on one FreeBSD host.

In order to have the talk service running on BSD it is necessery to have /usr/libexec/ntalkd installed on the system this however is installed by default with standard BSD OS installs, so no need for any external ports install to run it.

talk doesn't have it's own init script to start is not written to run as it own service but in order to run it is is necessery to enable it via inetd

Enabling it is done by;;;

1 — Editting /etc/inetd.conf

Inside the conf the line::

#ntalk dgram udp wait tty:tty /usr/libexec/ntalkd ntalkd

should be uncommented e.g, become ;;;

ntalk dgram udp wait tty:tty /usr/libexec/ntalkd ntalkd

2 — Restart inetd

freebsd# /etc/rc.d/inetd restart
Stopping inetd.
Starting inetd.

talk is planned to be used for peer to peer conversations over SSH so in a way it is the GRANDFATHER 🙂 of IRC, ICQ and Skype;;;

Here is an example on how talk is used ,, Let's say there are three logged in users

pcfreak# w
12:39PM up 3 days, 16:25, 3 users, load averages: 1.12, 0.91, 0.71
USER TTY FROM LOGIN@ IDLE WHAT
testuser p0 192.168.0.7 10:50AM - bash
hipo p3 192.168.0.8 12:23PM - w
root p4 :ttyp2:S.0 12:24PM - vim /usr/local/www/dat

I'm logged in with my username hipo and I would like to talk to testuser ;;;;

pcfreak% tty
/dev/ttyp3

You see I'm logged in on /dev/ttyp3 (this is the specific naming on BSDs) on Linux equivalent is /dev/tty3So to talk the other user testuser;;;;;-

$ talk testuser ttyp0
[No connection yet]
[Waiting for your party to respond]

The testuser logged in via SSH will then get a message ||;

Message from Talk_Daemon@pcfreak at 12:44 on 2012/06/10 ...
talk: connection requested by hipo@localhost
talk: respond with: talk hipo@localhost

To enter a talk session then the logged in testuser has to type:

$ talk hipo@localhost

 

AEWAN – a nice advanced GNU / Linux console ASCII art text editor

Saturday, May 19th, 2012

I'm a guy fascinated by ASCII art, since the very early days I saw a piece of this awesome digital art.

As time passed and computers went to be used mostly  graphics resolution, ASCII art loose its huge popularity from the early DOS and BBS (internet primordial days).

However, this kind  of art is still higly valued by true computer geeks.
In that manner of thoughts, lately I'm researching widely on ASCII art tools and ASCII art open source tools available for Linux.
Last time I check what is available for 'ASCII job' was before 5 years time. Recently I decided to review once again and see if there are new software for doing ascii manipulations on Linux and this is how this article got born.

My attention was caught by aewan (ASCII-art Editor Without A Name), while searching for ASCII keyword description packages with:

apt-cache search ascii

Aewan project official website is on sourceforge check it out here

Here is the complete description of the Debian package:

hipo@noah:~$ apt-cache show aewan|grep -i description -A 5
Description: ASCII-art Editor Without A Name
aewan is an ASCII art editor with support for multiple layers that can be
edited individually, colors, rectangular copy and paste, and intelligent
horizontal and vertical flipping (converts '\' to '/', etc). It produces
both stand-alone art files and an easy-to-parse format for integration
into your terminal applications.

I installed it to give it a try:

noah:~# apt-get --yes install aewan
Selecting previously deselected package aewan.
(Reading database ... 388522 files and directories currently installed.)
Unpacking aewan (from .../aewan_1.0.01-3_amd64.deb) ...
Processing triggers for man-db ...
Setting up aewan (1.0.01-3) ...

aewan package provides three executable binaries:

noah:~# dpkg -L aewan|grep -i /bin/ /usr/bin/aecat
/usr/bin/aewan
/usr/bin/aemakeflic

1. aewan binary is the ascii-art editor itself

2. aecat is utility to display an aewan documents (aewan format saved files)3. aemakeflictool to produce an animation from an aewan document

Next I ran it in plain console tty  to check how it is like:

hipo@noah:~$ aewan

Below are screenshots to give you an idea how powerful aewan ASCII art editor is:

AEWAN ASCII art editor entry information screen Debian GNU / Linux shot

Aewan immediate entry screen after start up

Aewan ASCII art editor Linux showing the major functionality of aewan on Debian GNU / Linux Squeeze

Aewan ASCII art editor – all of the supported tool functions

As you can see from the shot the editor is very feature rich. I was stunned to find out it even supports layers (in ASCII!!) (w0w!). 
It even has a Layers Manager (like GIMP) 🙂

To create my first ASCII art I used the:

New

menu.

This however didn't immediately show the prompt, where I can type  the ascii characters to draw my picture. In order to be able to draw inside the editor, its necessary to open at least one layer, through using the menu:

Add Layer (defaults)

then the interactive ASCII art editor appeared.

While an ASCII art is created with the editor you can select the color of the input characters by using Drawing Color menu seen in the above screenshot.

aewan drawing color choose color Linux shot

I've played few minutes and created a sample ascii art, just to test the color and editor "look & feel", my conclusions are the editor chars drawing is awesome.

Aewan ascii art produced on my Debian GNU / Linux host

All the commands available via menus are also accessible via a shortcut key combinations:

Aewan Linux Ascii art editor quick key shortcut commands

aewan controls are just great and definitely over-shadows every other text editor I used to draw an ASCII art so far.
Once saved the ASCII art, are by default saved in a plain gzipped ascii text. You can therefore simply zcat the the saves;
Don't expect zcat to show you the ascii as they're displayed in aewan, zcat-ing it will instead  display just the stored meta data; the meta data is interpreted and displayed properly only with aecat command.

aewan aecat displaying properly previously saved ascii art picture

I've checked online for rpm builds too and such are available, so installing on Fedora, CentOS, SuSE etc. is up to downloading the right distro / hardware architecture rpm package and running:

# rpm -ivh aewan*.rpm

On the official website, there are also instructions to compile from source, Slackware users and users of other distros which doesn't have a package build should compile manually with the usual:

$ tar -zxf aewan-1.0.01.tar.gz
$ cd aewan-1.0.01
$ ./configure
$ make
$ su -c "make install"

For those inrested to make animations with aemakeflic you need to first save a multiple layers of pictures. The idea of creating ASCII art video is pretty much like the old school way to make animation "draw every scene" and movie it. Once all different scene layers of the ASCII art animation are prepared one could use  aemakeflic to export all the ASCII layers as common video.

aemakeflic has the ability to export the ASCII animation in a runnable shell script to display the animation. The other way aemakeflic can be used is to produce a picture in kind of text format showing the video whether seen with  less cmd.
Making ASCII animation takes a lot of time and effort. Since i'm too lazy and I lack the time I haven't tested this functionality. Anyways I've seen some ascii videos on telnet  to remote hosts (some past time); therefore I guess they were made using aewan and later animated with aemakeflic.

I will close this post with a nice colorful ASCII art, made with aewan (picture is taken from the project page):

Aewan Flipping Selection Screenshot