access Archives - ☩ Walking in Light with Christ - Faith, Computing, Diary ☩ Walking in Light with Christ

Posts Tagged ‘access’

Linux: Howto Disable logging for all VirtualHosts on Apache and NGINX Webservers one liner

Wednesday, July 1st, 2020

disable-apache-nginx-logging-for-all-virtualhosts
Did you happen to administer Apache Webservers or NGINX webservers whose logs start to grow so rapidly that are flooding the disk too quickly?
Well this happens sometimes and it also happens that sometimes you just want to stop logging especially, to offload disk writting.

There is an easy way to disable logging for requests and errors (access_log and error_log usually residing under /var/log/httpd or /var/log/nginx ) for all configured Virtual Domains with a short one liner, here is how.

Before you start Create backup of /etc/apache2/sites-enabled / or /etc/nginx to be able to revert back to original config.

# cp -rpf /etc/apache2/sites-enabled/ ~/

# cp -rpf /etc/nginx/ ~/

1. Disable Logging for All Virtual Domains configured for Apache Webserver

First lets print what the command will do to make sure we don't mess something

# find /home/hipo/sites-enabled/* -exec echo sed -i 's/#*[Cc]ustom[Ll]og/#CustomLog/g' {} \;

You will get some output like

find /home/hipo//sites-enabled/* -exec echo sed -i 's/#*[Cc]ustom[Ll]og/#CustomLog/g' {} \;

find /etc/apache2/sites-enabled/* -exec sed -i 's/#*[Cc]ustom[Ll]og/#CustomLog/g' {} \;
find /etc/apache2/sites-enabled/* -exec sed -i 's/#*[Ee]rror[Ll]og/#ErrorLog/g' {} \;

2. Disable Logging for All configured Virtual Domains for NGINX Webserver

find /etc/nginx/sites-enabled/* -exec sed -i 's/#*access_log/#access_log/g' {} \;
find /etc/nginx/sites-enabled/* -exec sed -i 's/#*error_log/#error_log/g' {} \;

f course above substituations that will comment out with '#' occurances from file configs of only default set access_log and error_log / access.log, error.log
for machines where there is no certain convention on file naming and there are multiple domains in custom produced named log files this won't work.

This one liner was inspired from a friend's daily Martin Petrov. Martin blogged initially about this nice tip for those reading Cyrillic check out mpetrov.net, so. Thanks Marto ! 🙂

Tags: about, access, ALL, and, apache, apache webserver, apache webservers, apache2, are, BACK, blogged, check, command, Comment, config, configured, convention, course, create, custom
Posted in Everyday Life, Nginx, Web and CMS | No Comments »

Report haproxy node switch script useful for Zabbix or other monitoring

Tuesday, June 9th, 2020

For those who administer corosync clustered haproxy and needs to build monitoring in case if the main configured Haproxy node in the cluster is changed, I've developed a small script to be integrated with zabbix-agent installed to report to a central zabbix server via a zabbix proxy.
The script is very simple it assumed DC1 variable is the default used haproxy node and DC2 and DC3 are 2 backup nodes. The script is made to use crm_mon which is not installed by default on each server by default so if you'll be using it you'll have to install it first, but anyways the script can easily be adapted to use pcs cmd instead.

Below is the bash shell script:

UserParameter=active.dc,f=0; for i in $(sudo /usr/sbin/crm_mon -n -1|grep -i 'Node ' |awk '{ print $2 }'); do ((f++)); DC[$f]="$i"; done; \
DC=$(sudo /usr/sbin/crm_mon -n -1 | grep 'Current DC' | awk '{ print $1 " " $2 " " $3}' | awk '{ print $3 }'); \
if [ “$DC” == “${DC[1]}” ]; then echo “1 Default DC Switched to ${DC[1]}”; elif [ “$DC” == “${DC[2]}” ]; then \
echo "2 Default DC Switched to ${DC[2]}”; elif [ “$DC” == “${DC[3]}” ]; then echo “3 Default DC: ${DC[3]}"; fi

To configure it with zabbix monitoring it can be configured via UserParameterScript.

The way I configured it in Zabbix is as so:

1. Create the userpameter_active_node.conf
Below script is 3 nodes Haproxy cluster

# cat > /etc/zabbix/zabbix_agentd.d/userparameter_active_node.conf
UserParameter=active.dc,f=0; for i in $(sudo /usr/sbin/crm_mon -n -1|grep -i 'Node ' |awk '{ print $2 }'); do ((f++)); DC[$f]="$i"; done; \
DC=$(sudo /usr/sbin/crm_mon -n -1 | grep 'Current DC' | awk '{ print $1 " " $2 " " $3}' | awk '{ print $3 }'); \
if [ “$DC” == “${DC[1]}” ]; then echo “1 Default DC Switched to ${DC[1]}”; elif [ “$DC” == “${DC[2]}” ]; then \
echo "2 Default DC Switched to ${DC[2]}”; elif [ “$DC” == “${DC[3]}” ]; then echo “3 Default DC: ${DC[3]}"; fi

Once pasted to save the file press CTRL + D

The version of the script with 2 nodes slightly improved is like so:

UserParameter=active.dc,f=0; for i in $(sudo /usr/sbin/crm_mon -n -1|grep -i 'Node ' |awk '{ print $2 }' | sed -e 's#:##g'); do DC_ARRAY[$f]=”$i”; ((f++)); done; GET_CURR_DC=$(sudo /usr/sbin/crm_mon -n -1 | grep ‘Current DC’ | awk ‘{ print $1 ” ” $2 ” ” $3}’ | awk ‘{ print $3 }’); if [ “$GET_CURR_DC” == “${DC_ARRAY[0]}” ]; then echo “1 Default DC ${DC_ARRAY[0]}”; fi; if [ “$GET_CURR_DC” == “${DC_ARRAY[1]}” ]; then echo “2 Default Current DC Switched to ${DC_ARRAY[1]} Please check “; fi; if [ -z “$GET_CURR_DC” ] || [ -z “$DC_ARRAY[1]” ]; then printf "Error something might be wrong with HAProxy Cluster on $HOSTNAME "; fi;

The haproxy_active_DC_zabbix.sh script with a bit of more comments as explanations is available here
2. Configure access for /usr/sbin/crm_mon for zabbix user in sudoers

# vim /etc/sudoers
zabbix ALL=NOPASSWD: /usr/sbin/crm_mon

3. Configure in Zabbix for active.dc key Trigger and Item

Tags: access, ALL, and, Anyways, are, available, awk, bash shell, bash shell script, Below, bit, case, cat, Central, check, Cluster, cmd, Comments, conf, configure
Posted in Linux, Monitoring, Zabbix | No Comments »

Howto Pass SSH traffic through a Secured Corporate Proxy server with corkscrew, using sshd as a standalone proxy service with no proxy installed on remote Linux server or VPS

Tuesday, November 19th, 2019

howto pass ssh traffic through proxy to remote server use remote machine as a proxy for connecting to the Internet

Working in the big bad corporate world (being employed in any of the Fortune 500) companies, especially in an IT delivery company is a nasty thing in terms of User Personal Data Privacy because usually when employeed in any of a corporation, the company ships you with a personal Computer with some kind of pre-installed OS (most often this is Windows) and the computer is not a standalone one but joined in Active Directory (AD) belonging to Windows Domain and centrally administered by whoever.

As part of the default deplyed configuration in this pre-installed OS and software is that part or all your network traffic and files is being monitored in some kind of manner as your pre-installed Windows or Linux notebook given by the Corporation is having a set of standard software running in the background, and even though you have Windows Administrator there are many things you have zero control or even if you have changed it once the Domain Policy is triggered your custom made changes / Installed Programs that happen to be against the company policy are being automatically deleted, any registry changes made are being rewinded etc. Sometimes even by trying to manually clean up your PC from the corporate crapware, you might breaks access to the corporate DMZ firewalled network. A common way to secure their employee PC data large companies have a Network seperation, your PC when not connected to the Corporate VPN is having a certain IP configuration and once connected to the Demilitarized Zone VPN those configuration changes and the PC have access to internal company infrastructure servers / router / switches / firewalls / SANs etc. Access to corporate Infrastructure is handled via crypted VPN clinet such as Cisco AnyConnect Secure Mobility Client which is perhaps one of the most used ones out there.

Part of the common software installed to Monitor your PC for threats / viruses / trojans among which is MCafee / EMET (Enhandced Mitigation Experience Toolkit) the PC is often prebundled with some kind of anti-malware (crapware) :). But the tip of the iceberg on user surveillance where most of surveillance happens is the default installed proxy on the PC which usually does keep track of all your remote accessed HTTP Website URLs accessed in plain text – traffic flowing on Port 80 and crypted one on standard (SSL) Port 443. This Web Traffic is handled by the Central Corporate proxy that is being deployed via some kind of Domain policy, every time the Computer joins the Windows domain.

This of course is a terrible thing for your Browsing security and together with the good security practice to run your browser in Incognito mode, which makes all your browsing activity such as access URLs History or Saved Cookies data to be cleared up on a Browser close it is important to make sure you run your own personal traffic via a separate browser which you will use only for your own concern browsing such as Accessing your Bank Money Accounts to check your Monthly Sallary / Purchase things online via Amazon.com / Ebay.com, whether all of the rest traffic company related is trafficed via the default set corporate central proxy.
This is relatively easy sometimes in companies, where security is not of a top concern but in corporations with tightened security accessing remote proxy, or accessing even common daily news and Public Email websites or social media sites Gmail.com / Twitter / Youtube will be filtered so the only way to reach them will be via some kind of Proxy and often this proxy is the only way out to the Free world from the corporate jail.

Here is where the good old SSH comes as a saving grace as it turns out SSH traffic could be trafficed over a proxy. In below article I will give you a short insight on how Proxy through SSH could be achieved to Secure your dailty web traffic and use SSH to reach your own server on the Internet as well as how you can copy securely data via SSH through corporate Proxy.

1. How to view your corporate used (default) proxy / Check Proxy.pac file definitions

To get an idea what is the used proxy on your Corporate PC (as most corporate employee given notebooks are running some kind of M$ Windows) you can go to:

Windows Control Panel -> Internet Options -> Connections -> Lan Settings

internet-properties-microsoft-windows-screenshot

Under the field Proxy server (check out the Proxy configured Address and Port number )

local-area-network-lan-settings-screenshot-windows-1

Having that as browsers venerate the so-called Proxy.pac file, to be rawly aware on some general Company Proxy configured definitions you can access in a browser the proxy itself fething the proxy.pac file for example.

http://your-corporate-firewall-rpoxy-url:8080/proxy.pac

This is helpful as some companies Proxies have some proxy rules that reveal some things about its Internet architecture and even some have some badly configured proxy.pac files which could be used to fool the proxy under some circumstances 🙂

2. Few of the reasons corporations proxy all their employee's work PC web traffic

The corporate proxying of traffic has a number of goals, some of which are good hearted and others are for mostly spying on the users.

1. Protect Corporate Employees from malicious Viruses / Trojans Horses / Malware / Badware / Whatever ware – EXCELLENT
2. Prevent users from acessing a set of sources that due to the corporate policy are considered harmful (e.g. certain addresses
of information or disinformation of competitors, any Internet source that might preach against the corporation, hacking ralated websites etc.) – NOT GOOD (for the employee / user) and GOOD for the company
3.Spy on the users activity and be able to have evidence against the employee in case he decided to do anything harmful to the company evidences from proxy could even later be used in court if some kind of corpoate infringment occurs due to misbehave of the employee. – PERFECT FOR COMPANY and Complete breach of User privacy and IMHO totally against European Union privacy legislation such as GDRP
4. In companies that are into the field of Aritificial Intelligence / Users behavior could even be used to advance Self-learning bots and mechanisms – NASTY ! YAECKES
…

3. Run SSH Socks proxy to remote SSHd server running on common SSL 443 port

Luckily sysadmins who were ordered the big bosses to sniff on your Web behaviour and preferences could be outsmarted with some hacks.

To protect your Browsing behaviours and Secure your privacy perhaps the best option is to use the Old but gold practice o Securing your Networkf traffic using SSH Over Proxy and SSH Dynamic tunnel as a Proxy as explained in my previous article here.

how-to-use-sshd-server-as-a-proxy-without-a-real-proxy-ssh-socks5_proxy_linux

In short the quest way to have your free of charge SOCKS Remote proxy to your Home based Linux installed OS server / VPN with a Public Internet address is to use ssh as so:

ssh -D 3128 UserName@IP-of-Remote-SSHD-Host -p 443

This will start the SOCKS Proxy tunnel from Corporate Work PC to your Own Home brew server.

For some convenience it is useful to set up an .alias (for cygwin) / linux users in .bashrc file:

alias proxy='ssh -D 3128 UserName@IP-of-Remote-SSHD-Host -p 443';

To start using the Proxy from browser, I use a plugin called FoxyProxy in Chrome and Firefox browsers
set-up to connect to localhost – 127.0.0.1:3128 for All Protocols as a SOCKs v5 Proxy.

The sshd Socks proxy can be used for multiple others for example, using it you can also pass on traffic from Mail client such as Thunderbird to your Email server if you're behind a firewall prohibiting access to the common POP3 port 110 or IMAP port TCP 143.

4. How to access SSH through Proxy using jumphost SSH hop

If you're like me and you have on your Home Linux machine only one Internet address and you have already setupped an SSL enabled service (lets say Webmail) to listen to that Public Internet IP and you don't have the possibility to run another instance of /usr/bin/sshd on port 443 via configuration or manually one time by issuing:

/usr/sbin/sshd -p 443

Then you can use another ssh another Linux server as a jump host to your own home Linux sshd server. This can be done even by purchasing a cheap VPS server for lets say 3 dollars month etc. or even better if you have a friend with another Linux home server, you can ask him to run you sshd on TCP port 443 and add you an ssh account.
Once you have the second Linux machine as JumpHost to reach out to your own machine use:

ssh -J Your-User@Your-jump-host.com:443 hipo@your-home-server.com -v

To easify this a bit long line it is handy to use some kind of alias like:

alias sshhome='ssh -J Your-User@Your-jump-host.com:443 hipo@your-home-server.com -v'

The advantage here is just by issuing this sshd tunnel and keeping it open in a terminal or setting it up as Plink Putty tunnel you have all your Web Traffic Secured
between your Work Corporate PC and your Home Brew Server, keeping the curious eyes of your Company Security Officers from your own Web traffic, hence
separating the corporate privacy from your own personal privacy. Using the just established own SSH Proxy Tunnel to home for your non-work stuff browsing habits
from the corporate systems which are accessed by switching with a button click in FoxyProxy to default proxy settings.

5. How to get around paranoid corporate setup where only remote access to Corporate proxy on TCP Port 80 and TCP 443 is available in Browser only

Using straight ssh and to create Proxy will work in most of the cases but it requires SSH access to your remote SSH running server / VPS on TCP Port 22, however under some Fort-Nox like financial involved institutions and companies for the sake of tightened security, it is common that all Outbound TCP Ports are prohibited except TCP Port 80 and SSL 443 as prior said, so what can you do then to get around this badful firewall and access the Internet via your own server Proxy?
The hack to run SSH server either on tcp port 80 or tcp port 443 on remote Host and use 443 / 80 to acess SSHD should work, but then even for the most paranoid corporations the ones who are PCI Compliant – PCI stands for (Payment Card Industry), e.g. works with Debit and Credit Card data etc, accessing even 80 or 443 ports with something like telnet client or netcat will be impossible.
Once connected to the corporate VPN, this 2 two ports firewall exceptions will be only accessible via the Corporate Proxy server defined in a Web Browser (Firefox / IE / Chrome etc.) as prior explained in article.

The remedy here is to use a 3rd party tools such as httptunnel or corkscrew that are able to TUNNEL SSH TRAFFIC VIA CORPORATE PROXY SERVER and access your own resource out of the DMZ.

Both httptunnel and corkscrew are installable both on most Linux distros or for Windows users via CygWin for those who use MobaXterm.

Just to give you better idea on what corkscrew and (hts) httptunnel does, here is Debian packages descriptions.

# apt-cache show corkscrew
" corkscrew is a simple tool to tunnel TCP connections through an HTTP
proxy supporting the CONNECT method. It reads stdin and writes to
stdout during the connection, just like netcat.
.
It can be used for instance to connect to an SSH server running on
a remote 443 port through a strict HTTPS proxy.
"

# apt-cache show httptunnel|grep -i description -A 7
Description-en: Tunnels a data stream in HTTP requests
Creates a bidirectional virtual data stream tunnelled in
HTTP requests. The requests can be sent via a HTTP proxy
if so desired.
.
This can be useful for users behind restrictive firewalls. If WWW
access is allowed through a HTTP proxy, it's possible to use
httptunnel and, say, telnet or PPP to connect to a computer
—
Description-md5: ed96b7d53407ae311a6c5ef2eb229c3f
Homepage: http://www.nocrew.org/software/httptunnel.html
Tag: implemented-in::c, interface::commandline, interface::daemon,
network::client, network::server, network::vpn, protocol::http,
role::program, suite::gnu, use::routing
Section: net
Priority: optional
Filename: pool/main/h/httptunnel/httptunnel_3.3+dfsg-4_amd64.deb

Windows cygwin users can install the tools with:

apt-cyg install –yes corkscrew httptunnel

Linux users respectively with:

apt-get install –yes corkscrew httptunnel

yum install -y corkscrew httptunnel

You will then need to have the following configuration in your user home directory $HOME/.ssh/config file

Host host-addrs-of-remote-home-ssh-server.com
ProxyCommand /usr/bin/corkscrew your-corporate-firewall-rpoxy-url 8080 %h %p

howto-transfer-ssh-traffic-over-proxy

Picture Copyright by Daniel Haxx

The best picture on how ssh traffic is proxied is the one found on Daniel Haxx's website which is a great quick tutorial which originally helped to get the idea of how corkscrew works in proxying traffic I warmly recommend you take a quick look at his SSH Through or over Proxy article.

Host-addrs-of-remote-home-ssh-server.com could be also and IP if you don't have your own domain name in case if using via some cheap VPN Linux server with SSH, or alternatively
if you don't want to spend money on buying domain for SSH server (assuming you don't have such yet) you can use Dyn DNS or NoIP.

Another thing is to setup the proper http_proxy / https_proxy / ftp_proxy variable exports in $HOME/.bashrc in my setup I have the following:

export ftp_proxy="http://your-corporate-firewall-rpoxy-url:8080"
export https_proxy="https://your-corporate-firewall-rpoxy-url:8080"
export http_proxy="http://your-corporate-firewall-rpoxy-url:8080"
export HTTP_PROXY="http://your-corporate-firewall-rpoxy-url:8080"
export HTTPS_PROXY="http://your-corporate-firewall-rpoxy-url:8080"

6. How to Transfer Files / Data via SSH Protocol through Proxy with SCP and SFTP

Next logical question is how to Transfer your own personal encrypted files (that contains no corporate sensitive information) between your Work laptop and home brew Linux ssh server or cheap VPN.

It took me quite a lot of try-outs until finally I got it how Secure Copy (scp) command can be used toto transfer files between my Work Computer and my Home brew server using JumpHost, here is how:

scp -o 'ProxyJump Username@Jumpt-Host-or-IP.com:443' ~/file-or-files-to-copy* Username@home-ssh-server.com:/path/where/to/copy/files

I love using sftp (Secure FTP) command Linux client to copy files and rarely use scp so I have a lot of try-outs to connect interacitvely via the Corporate Proxy server over a Jump-Host:443 to my Destination home machine,

I've tried using netcat as it was pointed in many articles online, like so to traffic my sftp traffic via my localhost binded SSH Socks proxy on :3128 together with netcat as shown in article prior example, using following line:

sftp -oProxyCommand='/bin/nc -X connect -x 127.0.0.1:3128 %h %p' Username@home-ssh-server.com 22

Also tried proxy connect like this:

sftp -o ProxyCommand="proxy-connect -h localhost -p 3128 %h %p" Username@home-ssh-server.com

Moreover, tried to use the ssh command (-s) argument capability to invoke SSH protocol subsystem feature which is used to facilitiate use of SSH secure transport for other application

ssh -v -J hipo@Jump-Host:443 -s sftp root@home-ssh-server.com -v
…
open failed: administratively prohibited: open failed

Finally decided to give a try to the same options arguments as in scp and thanks God it worked and I can even access via the Corporate Proxy through the Jump Host SSH interactively via Secure FTP 🙂

!! THE FINAL WORKING SFTP THROUGH PROXY VIA SSH JUMPHOST !!

sftp -o 'ProxyJump Username@Jumpt-Host-or-IP.com:443' Username@home-ssh-server.com

To save time from typing this long line every time, I've setup the following alias to ~/.bashrc

alias sftphome='sftp -o 'ProxyJump Username@Jumpt-Host-or-IP.com:443' Username@home-ssh-server.com'

Conclusion

Of course using own Proxy via your Home brew SSH Machine as well as transferring your data securely from your Work PC (notebook) to Home does not completely make you Surveillance free, as the Corporate Windows installed OS image is perhaps prebundled with its own integrated Keylogger as well as the Windows Domain administrators have certainly access to connect to your PC and run various commands, so this kind of Security is just an attempt to make company has less control and know less on your browsing habits and the best solution where possible to secure your privacy and separate your Personal Space form Work space by using a second computer (if having the ability to work from home) with a KVM Switch device and switch over your Work PC and Home PC via it or in some cases (where companies) allows it, setup something like VNC server (TightVNC / RealVNC) on work PC and leave it all time running in office and connect remotely with vncviewer from your own controlled secured computer.

In article I've explained shortly common scenario found in corporate Work computers proxy setup, designed to Surveil all your move, mentioned few common softwares running by default to protect from Viruses and aimed to Protect user from malicious hacking tools, explained how to view your work notebook configured Proxy, shortly mentioned on Proxy.pac and hinted how to view proxy.pac config as well as gave few of the reasons why all web traffic is being routed over central proxy.

That's all folks, Enjoy the Freedom to be less surveilled !

Tags: access, alias, amd64, cygwin, Debian, fedora, get around corporate firewalls, good, hipo, host, How to, kind, linux users, Luckily, network traffic, pc-freak, pcfreak, personal privacy, ports, proxy socks ssh, Socks Proxy, ssh over proxy, ssh proxy without proxy, sshd, tcp, TightVNC, using
Posted in Company onboarding basics, Computer Security, Hacks, Various | No Comments »

Windows 7 how to configure Separator delimiter Divider between Pinned and Running Opened programs on Taskbar Classic theme pane

Saturday, October 12th, 2019

I've started to work as a External Consultant for Equens Worldline, who lately merged with Atos – a French multinational information technology.

As the company is mostly into the financial business and as any company that is involved with OLTP Online transaction processing , an information systems that typically facilitate and manage transaction-oriented applications, e.g. in a mortal language takes care for a numerous companies Credit / Debit card online payments.

Of course the technologies used both on a Server / Data Center level and for Employee personal used are a bit old fashioned. In that relation, most of the computers, I've seen used by Worldline – Atos including my work PC notebook is still running a custom version of Windows 7.

Once I had the Computer ready with the OS from the HelpDesk colleague, then it was ready to install, unfortunately, I didn't have administrator account on the Lenovo L serie laptop so I couldn't install applications. To get around this I've used PortableApps – the world's most popular portable software solution allowing you to take your favorite software with you with no need for any Installation and on Win OS computers, where you have limited access.
PortableApps is fully open source and free platform, it works from any synced cloud folder (DropBox, Google Drive, Box, etc), from your local PC on an internal or external drive, or on any portable storage device (USB flash drive, memory card, portable hard drive, etc). Because the applicatons are installed without being really binded to the Win OS, once the PortableApps ported Application is installed it can be easily moved by simply copying between PCs.

portableapps_windows_list-of-installable-programs-screenshot
PortableApps.com is currently used by millions of users all over the world and is the Largest Portable full collection of open source and freeware software, compatible commercial software, and partners in hardware industry for Microsoft Windows.

As I do often with the Windows fresh installed Windows machines I have to work on, install the great set of tools prior described in must have software on any fresh installed Windows OS as this is essential if you're a sysadmin or Advanced user for the daily work with the WinBlows

OS Graphical Interface simplicity is a crucial thing to me, cause if I have to work with Windows and spendan 8 hours each day 5 days a week in front of the Computer Display with this Complex unnecessery coloring (that Windows 7 / Windows 10 default Theme introduced), together with the visual aids on the Window Minimize and Maximize, puts a useless additional eye strain and besides that "fries" the brain (as the brain has to process all this mambo-jambo "virtual" reality graphics turning all down and up and mixing up the gamma of unreal colors for nothing …

So far so good I've set the needed software from above list and continued further to Change my Windows start menu to look like Windows 2000. Start menu and Theme To be the Classic Theme.

After setting up the Windows Classic Theme (which by the way is unfortunately, no longer available in Windows 10), I love to set my Windows Background picture to be a Sold Black Color, I do this as Black color is the less exhausting for the eyes – as one views the desktop picture thousand times and this is extra unnecessery work for the brain.

However next, unpleasant thing faced is even though now my Windows Start and Windows borders looked old-fashioned in the Windows 2000 Server, "wooden" style, the order of my (Open) Running applicaitons on the start menu bar were not isolated but were mixing, with the rest of my Pinned Program Starter icons

! Notice the Windows Media player icon being in the middle between the Lync, Firefox, Chrome and Internet Explorer line … !

I spend some time to rewind some memories on how I did make the pinned program to distinguish itself (separate) from the rest of the Opened Runnign Windows, and had in mind that I used some kind of separator added custom Toolbar, but the bad thing was I couldn't remember exactly how so I spend some good amount of time to Google around.

And found few articles on the topic proposing to add a new Custom empty launcher program of cmd.exe and change the icon of the empty launcher to a | graphical string
I've tried a lot of this approach but it didn't work and after some experimentations, finally I figured it out. The way to do it is to add a New Toolbar that is Pointing to a Directory with a list of Programs whose icons you want to have listed with icon launchers next to Start menu.

Here is step by step what I've done to achieve configuration of Hard Separator Delimiter between my Program Launchers and Running Applications.

1. Create a New Folder somewhere under your User Home directory, lets say:
C:\Users\Username\TaskbarPrograms

2. Add a shortcut or copy the programs starters (icons) to the TaskBar Programs

3. Unlock the Windows 7 Taskbar
E.g. Click over on any of the empty space on the taskbar after Windows Start menu button and untick Lock the Taskbar

4. Remove any Pinned programs (you have placed there earlier)

windows-7-unpin-program-from-taskbar

5. Again On an empty place on the Taskbar press the last button Mouse (Rightest button) to open Taskbar properties menu and
Go to ToolBars menu and Choose (New Toolbar)

new-toolbar-screenshot-windows-os-7

6. As a location for the new toolbar place C:\Users\Username\TaskbarPrograms

add-custom-Quick-Launch-Toolbar-Images-screenshot

The result should be that all the programs in the selected folder will appear in the Windows pane

8. Now go over the empty space of the Windows start menu bar
and untick Show Text / Show Title

9. Hold over the Taskbar Programs holder and drag and drop it constantly holding the left mouse button
and place it over the default separator Holder (for clarity marked in below screenshot as Marked over here).

10. Move the default separator that will appear on the left close to the list of shown applications
(shown in screenshot)

11. Open any of the applications (e.g. Firefox, Chrome, MobaXterm, Opera, RDP, VLC) the result should be a fine ordered icons. Hooray !!!

12. To make more space for the Opened Active (Running) Windows, a good hint is to stretch the start menu pane field – this is done by holding a right mouse button on the edge of the gray line, like shown in below screenshot.

Note that, you will need to remove any icons that are left over binded to the default separator (holder).

13. Finally on the Taskbar (empty space) menu with the Mouse, click again right mouse button and tick on the Lock the Taskbar

That's it now your Windows 7 start menu bar will look pretty much like Linux's GNOME 2.x and GNOME Mate, and if you're a a long years Linux Desktop user you will feel at home even on Windows 7.

To further tweak the Taskbar a very useful Program is Windows 7+ Taskbar Tweaker

"7+ Taskbar Tweaker allows you configure various aspects of the Windows taskbar which can't be tweaked via registry or taskbar properties. The tweaker is designed for Windows 7, Windows 8, Windows 8.1, and Windows 10.

windows-7-taskbart-tweaker-combine-grouped-buttons

I install it mostly because one single feature Combined Grouped Buttons which makes similar Windows to group together. Just having a number of Windows of the same type floating around the Taskbar is really irritating and becomes really confusing once multiple windows.

For example if you have opened 10 Windows of Explorer they will be group under the same window title pointer.

To make the Windows even more bizarre and feel more like a Linux powered Box go on and install MobaXterm as a substitute for Linux's gnome-terminal as well as the rest of things such as CygWin / GNUWin32 / GVim etc.

Also some good start up list of software I do install on new Windows Desktop PCs is described in my previous article Some standard software programs to install on Windows to make your Windows feel more like a Linux / Unix Desktop host

This time I did not needed that but on other locations another Windows useful hint I use every now and then is to set some custom programs (Lets say a PortableApps program) to Automatically run – described earlier in my earlier article how to add startup Windows 7 / 8 program run on startup.

P.S. I've tried to setup the same Classic Theme and Start menu button with the same order on Windows 10 OS, but even though a similar interface could be achieved, I've had problems with setting up the proper Running application Ordering, due to a different behavior of the Delimiter / Separators in Windows 10.

For me this Custom Windows interface and Window ordering works pretty well it saves me a lot of the eye strain and brain overcalculation for nothing.

After all, Lets not forget that God is in the Simplicity ! 🙂

Tags: access, Account, administrator, administrator account, after, again, aids, ALL, amount, and, another, ANY, application, applications, are, around, article, available, background, being
Posted in Everyday Life, Various, Windows | 1 Comment »

Create Linux Debian based small personal home Postfix Mail Server with RainLoop Webmail interface in less than 2 hours time

Saturday, October 5th, 2019

Why run an Own Home or Dedicated Own server nowadays is a must in the Digital Age ?

Having own home based email server should be the target (most basic step) of any person that respects his Email privacy as most of the Public accessible Email accounts such as Yahoo Mail! and Gmail are collecting personal flavour info and flavour data.

Thus to mitigate the possible Personal Security problems of using any of the multiple Public Email services it is rather easy nowdays to configure quickly a own based home Email server, Hire Cheap VPS for Own Email Service or even hire a Dedicated server – the cheapeast dedis nowadays are starting from 40 / 50 EUROs per server.

To protect your Email privacy to maximum and keep your email data owned by you hence in this small article, I'll explain how to setup a Postfix based Simple own hosted Email server in about 2 hours of time, but before proceeding with the actual install instructions, I'll say a bit more on the downsides of using public mail gmail.com / yahoo.com / yandex.ru etc.

– The Downsides of using Public free hosted email address such as Google / Yahoo / Yandex / Mail.com etc.

If you happen to have @gmail.com account then you're more or less surveilled by automated scripts software that prepares your virtual dossier – those of us who lived in the Communism of USSR, knew that each of us had a good personal record kept in policy no matter of his function in society, now the situation is no different besides the fact that your Corporate owned data is often sold to National Governments could leak out of its owner and a lot of very personal data could end up in the hands of some malicious blackmailers or spammers, that could have a very complete records of your interest, age, profession, marrital status, position in society, music preference, daily habits, something definitely most people want to avoid.

– Why using Android Phone with Gmail is terrible for your privacy / myaccount.google.com and what Google knows about you

To get further idea about the great expand of Surveillance, if you're an Android Mobile Phone User like me I recommend you login with your Android Email account to https://myaccount.google.com and find out what Google knows about you or better said what from what it knew is willing to show you.
Android OS phones collect a lot of data connected to your Mail use if you stay logged in in Browser (because you forgot to logout and left the new tab opened) as well as track you about the overall Google Search Engine made searches or Youtube watched and preferred genre videos – hence know about overall activity on the Net. And even worse perhaps there is much more details that are automatically shared to Google as you're continuously logged in to Android Google Play …

To illustrate what I mean in terms of How much we're surveilled nowadays, in one of my Gmail accounts I've been almost stunned seeing that, some Google Data Analytics scripts based on the Android Synchronized pictures has created a full featured nice funny Music video of "my life", showing me pictures of my marriage ceremonly, places, where I visit such as some Monasteries, close friends and a tons of very private data that in normal circumstances, they should never know about.

Earlier I've blogged how to Install and Configure a Postfix server with Dovecot and MySQL with a configuration setup that worked on Debian 7 and prior Debian servers which used to be among the only setups that can keep mail accounts inside the database that worked on the net and for more professional Email server to be serving a small / midsized company multiple email boxes perhaps it is still a good one to follow, however as above suggested initial config is a bit overcomplicated (and maybe outdated) for Linux novice users / sysadmins and for own small home servers with only few Mail accounts it might be not needed at all.
I've decided to write how to setup much more quickly and easily Working Internet Mail server with MailBoxes directly owned by each of the UNIX created local accounts.

1. Install prerequired packages for Email setup and set hostname

Update Debian Linux distribution to the latest:

apt-get –yes update && apt-get –yes upgrade

apt-get install curl net-tools bash-completion wget lsof nano mailutils bsdmailx

In this article I assume you already have your own Linux server.

To make the hosts file ber read before any local running DNS server in case if you're running bind or any other local caching DNS server add following lines to /etc/host.conf

vim /etc/host.conf
order hosts,bind
multi on

If you already haven't configured the machine hostname on Linux OS install time, next step is to set the machine hostname to the Fully Qualified Domain Name (FQDN) that will be used by mail server.

hostnamectl set-hostname mail.your-mail-domain.com

If you don't plan to regularly change the IP address of Mail server It is a good idea to add the IP -> Domain relation to /etc/hosts doing so should not waste DNS cycles to resolve the hostname but every time the hostname will be locally taken from /etc/hosts.

echo "86.78.10.1 your-mail-domain.com mail.your-mail-domain.com" >> /etc/hosts

getent ahosts mail.your-mail-domain.com
86.78.10.1    STREAM mail.your-mail-domain.com
86.78.10.1    DGRAM
86.78.10.1    RAW

Also check whether mail.your-mail-domain.com is pingable

ping mail.your-mail-domain.com

You should get a properly delivered ping packets with no losses.

2. Install Postfix Mail Server Debian package

Debian Postfix package provides a MTA (Message Transfer Agent). MTA is a pr

apt-get install –yes postfix

Next a ncurses interface will prompt you the usual Postfix type configuration where we have to choose Internet Site and type in the mail name your-mail-domain.com (note that here you have to type the domain without the mail. part) .

3. Setup Postfix Mail server configuration main.cf

Before starting make backup of main.cf

cp -rpf /etc/postfix/main.cf{,.backup}

vim /etc/postfix/main.cf

# See /usr/share/postfix/main.cf.dist for a commented, more complete version
smtpd_banner = $myhostname ESMTP
biff = no
# appending .domain is the MUA's job.
append_dot_mydomain = no
readme_directory = no
# See http://www.postfix.org/COMPATIBILITY_README.html — default to 2 on
# fresh installs.
compatibility_level = 2
# TLS parameters
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
myhostname = mail.your-mail-domain.com
mydomain = your-mail-domain.com
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
#myorigin = /etc/mailname
myorigin = $mydomain
mydestination = $myhostname, $mydomain, localhost.$mydomain, localhost
relayhost =
mynetworks = 127.0.0.0/8, 192.168.1.0/24 86.78.10.0/24
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
#inet_protocols = all
inet_protocols = ipv4
home_mailbox = Maildir/
# SMTP-Auth settings
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous
smtpd_sasl_local_domain = $myhostname
smtpd_recipient_restrictions = permit_mynetworks,permit_auth_destination,permit_sasl_authenticated,reject

Replace the myhostname, mydomain and mynetworks variables to match your own configurations.

If for example your Mail Host IP is 86.78.10.1 then to add only this IP to mynetworks 86.78.10.1/32, if you have a range of IPs such as a 254 IP's range set the proper mask 86.78.10.1/24 etc.

Next you better test the configuration to make sure you haven't syntax errors with:

postconf -n

If no syntax errors found the command should dump main.cf postfix set variables.

Once config is fine to load it and test the status of postfix run:

systemctl restart postfix
systemctl status postfix

systemctl-postfix-status-on-debian-linux-screenshot

Then lets check the listener and try out telnet connect.

netstat -tulpan|grep -i 25
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN 17715/master

telnet localhost 25
Trying 127.0.0.1…
Connected to localhost.
Escape character is '^]'.
220 mail.your-mail-domain.com ESMTP

4. Testing email sent is working on Postfix Mail Server

First lets test postfix is able to deliver to local email addresses by sending mail to mail root user mailbox

echo "This is a test mail body"| mail -s "Simple test mail" root

Everything should be fine and mail should be the last delivered to root MailBox to view it:

cat /root/Maildir/new/$(ls -1 /root/Maildir/new/|tail -1)
Return-Path: <root@localhost>
X-Original-To: root@localhost
Delivered-To: root@localhost
Received: by mx.adzone.pro (Postfix, from userid 0)
id 9C4FA7E1952; Thu, 3 Oct 2019 22:19:59 +0300 (EEST)
Subject: Simple test mail
To: <root@localhost>
X-Mailer: mail (GNU Mailutils 3.5)
Message-Id: <20191003191959.9C4FA7E1952@mail.your-mail-domain.com>
Date: Thu, 3 Oct 2019 22:19:59 +0300 (EEST)
From: root <root@localhost>

This is a test mail body

by default mail command will try to read email content from /var/mail/root file which will be empty file, because Postfix main.cf home_mailbox = Maildir/ emails served by MTA will be delivered to /$USER/Maildir/ seperated in 3 directories (cur – current, mail ends here once red, new – new incoming mail, tmp – temporary directory used by postfix delivery programs /usr/lib/postfix/sbin/master and qmgr.

# ls -al /$USER/Maildir
total 20
drwx—— 5 root root 4096 Sep 30 20:26 .
drwx—— 9 root root 4096 Oct 3 22:37 ..
drwx—— 2 root root 4096 Sep 30 20:26 cur
drwx—— 2 root root 4096 Oct 3 22:19 new
drwx—— 2 root root 4096 Oct 3 22:19 tmp

Hence to set for mail command to read its mails from /$USER/Maildir you need to add:

export MAIL=$HOME/Maildir

to end of ~/.bashrc red for user after very user login.

echo 'export MAIL=$HOME/Maildir' >> /root/.bashrc

or if you want to make the mail command to read from /$USER/Maildir system wide (e.g. for all users).

You can add a simple script mailenv.sh to /etc/profile.d/

echo 'export MAIL=$HOME/Maildir' ' >> /etc/profile.d/mailenv.sh

If mail is send successfully you should also get an empty mail queue:

# mailq
Mail queue is empty

To get further info on how email got delivered its delivery status:

tail -f /var/log/mail.log

Once we're sure everything is fine with mail delivery, next step is to enable the Server to be accessed via IMAP2 protocol in plain text or crypted connected via Dovecot.

5. Install and configure Dovecot IMAP daemon

Dovecot is server binded daemon, responsible to deliver messages to local recipients mailboxes in IMAP protocol. IMAP protocol runs by default on port 143 and port 993 (SSL) – used for secure communication between the client side and the server. IMAP (Internet Message Acess Protocol) is used by email clients to retrieve email messages from a mail server over a TCP/IP connection.
IMAP was designed with the goal of permitting complete management of an email box by multiple email clients, therefore clients generally leave messages on the server until the user explicitly deletes them.

The IMAP protocol also uses synchronization in order to assure that a copy of each message is saved on the server and allows users to create multiple directories on the server and move mails to this directories in order to sort the e-mails.

POP3 proto (listening on port 110) which is older in time, differs from IMAP as protocol won’t allow users to create multiple directories on the server to sort your mail. You only have the inbox folder to manage mail.
But enought talk, lets kick in and install dovecot.

apt install –yes dovecot-core dovecot-imapd
…

Once installed to make dovecot listen on 143 and 993 on all network interfaces edit /etc/dovecot/dovecot.conf find commented line #listen = *, :: and uncomment it

vim /etc/dovecot/dovecot.conf
#listen = *, ::
listen = *, ::
….

Next edit /etc/dovecot/conf.d/10-auth.conf to disable obligatory SSL / TSL authentication for connecting client programs

disable_plaintext_auth and auth_mechanisms vars should look like so

disable_plaintext_auth = no
auth_mechanisms = plain login

Then, change the default mail store format from Mbox to Maildir in /etc/dovecot/conf.d/10-mail.conf comment mail_location set directive and change it to mail_location = maildir:~/Maildir

vim /etc/dovecot/conf.d/10-mail.conf
…

## mail_location = mbox:~/mail:INBOX=/var/mail/%u
mail_location = maildir:~/Maildir

…

Finally edit /etc/dovecot/conf.d/10-master.conf

Search for smtp-auth block of code and change to look as following:

# Postfix smtp-auth
unix_listener /var/spool/postfix/private/auth {
mode = 0666
user = postfix
group = postfix
}

6. Restart dovecot and test if configuration works as expected

systemctl restart dovecot.service
systemctl status dovecot.service

netstat -tlpn

7. Create new local email user and test mail delivery and IMAP dovecot connectivity

Now as postfix and dovecot are running fine to have a functional email to test SMTP delivery, we need create a new working email address, that means a simple local UNIX account with adduser / useradd *nix commands as usual.

adduser support

telnet localhost 25

The commands sent after connection with Enters at the end of each line and delay of 1/2 secs are as so:

EHLO localhost
MAIL FROM: root
RCPT TO: support
DATA
Subject: Sample Subject
MAIL BODY
.

The last email in /home/support/Maildir/new/ should be the one we just sent.

cat /home/support/Maildir/new/$(ls -1 /home/support/Maildir/new/|tail -1)
Return-Path: <root@your-mail-domain.com>
X-Original-To: support
Delivered-To: support@your-mail-domain.com
Received: from localhost (localhost [127.0.0.1])
by mx.adzone.pro (Postfix) with ESMTP id CB5727E1117
for <support>; Fri, 4 Oct 2019 11:50:24 +0300 (EEST)
Subject: Sample Subject
Message-Id: <20191004085038.CB5727E1117@mail.your-mail-domain.com>
Date: Fri, 4 Oct 2019 11:50:24 +0300 (EEST)
From: root@your-mail-domain.com
MAIL BODY
This is a test email

As you see mail delivered as expected, next step is to check we can properly login to IMAP server provided by dovecot.
Thunderbird / Outlook Express / Sylpheed and rest of IMAP protocol supporting clients could be emulated also with a simple netcat / telnet connection and few commands, below is how:

nc localhost 143
x1 LOGIN support user_password
x2 LIST "" "*"
x3 SELECT Inbox
x4 LOGOUT

netcat-connect-and-talk-to-imap-protocol-howto-screenshot-debian-linux

8. Install and Configure RainLoop (PHP) WebMail

RainLoop is Simple, Modern and Fast Web-Based Email client, that supports IMAP over it can communicate to just tested Dovecot IMAP. The pros of RainLoop are the modest system requirements (no database backend required) and the mega simple installation and update.

RainLoop is written in PHP and uses Apache Webserver to serve content, as well has a dependency on few php modules, hence you have to install below .deb packs.

apt install –yes apache2 php7.0 libapache2-mod-php7.0 php7.0-curl php7.0-xml

Once above is installed and HTTP server is running on the backend the most simple way to install is to use the default Apache configured DocumentRoot directory which is /var/www/html to make the RainLoop a default opening Platform for the WebServer delete the default page /var/www/html/index.html, change directory to it and install via the PHP installer, e.g.

rm -f /var/www/html/index.html
cd /var/www/html
curl -sL https://repository.rainloop.net/installer.php | php

rainloop-installer-on-debian-linux-shot

Once RainLoop Webmail Installer is completed to login to Admin use the Host or IP address URL in browser, where it is installed for example:

http://192.168.0.102/?admin

However it is much better to set the webmail to be living on a seperate domain VirtualHost URL address , i. e. (depending on the serving backend Webserver Apache / Nginx etc.) – you can use mail.your-mail-domain.com

Default Rainloop user and pass to login is:

User: admin
Password: 12345

N.B. Please do not forget to change the default password to something more secure right after logging to prevent some malicious users to break in.

rainloop-php-easy-webmail-linux-user-login-screen

rainloop-php-webmail-client-linux-admin-panel-screenshot

rainloop-php-webmaill-linux-installable-packages-screenshot

Here is also how Rainloop normal email Send / Receive interface looks like (simplistic and intuitive)

rainmail-mailing-interface-gnu-linux-screenshot

If you want to redirect all Email for a certain user lets say support to another local existing mail address lets say to marketing

echo "support: marketing" >> /etc/aliases
newaliases

The default alias binding in /etc/aliases is for postmaster to deliver all received mails to root mailbox, once added the new alias the newaliases command is used initializes ealias database.

Here is how /etc/aliases looks after above new alias is added

cat /etc/aliases
# See man 5 aliases for format
postmaster: root
support: marketing

What is left is to configure properly is the Domain name DNS records if you will be using an external VirtualHost subdomain that is to be done from your domain registrer (Godaddy, Domain.Com, NameCheap etc.).

We're done with our setup and you can now happily use your Webmail or favourite IMAP client to send / receive mails to recipients. Note that because mail accounts email is to be kept locally on the server as a plain text files it is crucial to secure at max SSH access to the server or even better disable /bin/sh login with usermod command for all used Email addresses.

Here is how to disable interactive SSH / SFTP login to the support local existing UNIX account.

usermod -s /bin/false support

To check the change you can use finger command (assuming you have it installed on the system if not install it with apt install finger)

finger support
Login: support Name: Support
Directory: /home/support Shell: /bin/false
Never logged in.
No mail.
No Plan.

That's all folks, our goal to have an independent (self-owned) Email server from scratch to better Email account privacy is fulfilled.
Now you can be at peace that your email correspondence with friends / relatives / customers will be at your hands only and you and only you will have a sorted view of your Mail stored passwords and personal data over the years.

Enjoy ! 🙂

Tags: about, access, Account, addresses, Android Google Play, configured, Connected, connectivity, Create Linux Debian, email addresses, information, logout, Restart, running, scripts, small article, support, tmp, Update Debian Linux, version
Posted in Linux, Linux and FreeBSD Desktop, Postfix | No Comments »

Scanning ports with netcat “nc” command on Linux and UNIX / Checking for firewall filtering between source and destination with nc

Friday, September 6th, 2019

Netcat ( nc ) is one of that tools, that is well known in the hacker (script kiddie) communities, but little underestimated in the sysadmin world, due to the fact nmap (network mapper) – the network exploratoin and security auditing tool has become like the standard penetration testing TCP / UDP port tool.

nc is feature-rich network debugging and investigation tool with tons of built-in capabilities for reading from and writing to network connections using TCP or UDP.

Its Plethora of features includes port listening, port scanning & Transferring files due to which it is often used by Hackers and PenTesters as Backdoor. Netcat was written by a guy we know as the Hobbit <hobbit@avian.org>.

For a start-up and middle sized companies if nmap is missing on server usually it is okay to install it without risking to open a huge security hole, however in Corporate world, due to security policies often nmap is not found on the servers but netcat (nc) is present on the servers so you have to learn, if you haven't so to use netcat for the usual IP range port scans, if you're so used to nmap.

There are different implementations of Netcat, whether historically netcat was UNIX (BSD) program with a latest release of March 1996. The Linux version of NC is GNU Netcat (official source here) and is POSIX compatible. The other netcat in Free Software OS-es is OpenBSD's netcat whose ported version is also used in FreeBSD. Mac OS X also comes with default prebundled netcat on its Mac OS X from OS X version (10.13) onwards, on older OS X-es it is installable via MacPorts package repo, even FreeDOS has a port of it called NTOOL.

The (Swiss Army Knife of Embedded Linux) busybox includes a default leightweight version of netcat and Solaris has the OpenBSD netcat version bundled.

A cryptography enabled version fork exists that supports that supports integrated transport encryption capabilities called Cryptcat.

The Nmap suite also has included rewritten version of GNU Netcat named Ncat, featuring new possibilities such as "Connection Brokering", TCP/UDP Redirection, SOCKS4 client and server support, ability to "Chain" Ncat processes, HTTP CONNECT proxying (and proxy chaining), SSL connect/listen support and IP address/connection filtering. Just like Nmap, Ncat is cross-platform.

In this small article I'll very briefly explain on basic netcat – known as the TCP Army knife tool port scanning for an IP range of UDP / TCP ports.

1. Scanning for TCP opened / filtered ports remote Linux / Windows server

Everyone knows scanning of a port is possible with a simple telnet request towards the host, e.g.:

telnet SMTP.EMAIL-HOST.COM 25

The most basic netcat use that does the same is achiavable with:

$ nc SMTP.EMAIL-HOST.COM 25
220 jeremiah ESMTP Exim 4.92 Thu, 05 Sep 2019 20:39:41 +0300

Beside scanning the remote port, using netcat interactively as pointing in above example, if connecting to HTTP Web services, you can request remote side to return a webpage by sending a false referer, source host and headers, this is also easy doable with curl / wget and lynx but doing it with netcat just like with telnet could be fun, here is for example how to request an INDEX page with spoofed HTTP headers.

nc Web-Host.COM 25
GET / HTTP/1.1
Host: spoofedhost.com
Referrer: mypage.com
User-Agent: my-spoofed-browser

2. Performing a standard HTTP request with netcat

To do so just pype the content with a standard bash integrated printf function with the included end of line (the unix one is \n but to be OS independent it is better to use r\n – the end of line complition character for Windows.

printf "GET /index.html HTTP/1.0\r\nHost: www.pc-freak.net\r\n\r\n" | nc www.pc-freak.net 80

3. Scanning a range of opened / filtered UDP ports

To scan for lets say opened remote system services on the very common important ports opened from UDP port 25 till, 1195 – more specifically for:

UDP Bind Port 53
Time protocol Port (37)
TFTP (69)
Kerberos (88)
NTP 123
Netbios (137,138,139)
SNMP (161)
LDAP 389
Microsoft-DS (Samba 445)
Route BGP (52)
LDAPS (639)
openvpn (1194)

nc -vzu 192.168.0.1 25 1195

UDP tests will show opened, if no some kind of firewall blocking, the -z flag is given to scan only for remote listening daemons without sending any data to them.

4. Port Scanning TCP listening ports with Netcat

As prior said using netcat to scan for remote opened HTTP Web Server on port 80 an FTP on Port 23 or a Socks Proxy or MySQL Database on 3306 / PostgreSQL DB on TCP 5432 is very rare case scenario.

Below is example to scan a Local network situated IP for TCP open ports from port 1 till 7000.

# nc -v -n -z -w 5 192.168.1.2 1-7000

           nc: connect to host.example.com 80 (tcp) failed: Connection refused
           nc: connect to host.example.com 20 (tcp) failed: Connection refused
           Connection to host.example.com port [tcp/ssh] succeeded!
           nc: connect to host.example.com 23 (tcp) failed: Connection refused

Be informed that scanning with netcat is much more slower, than nmap, so specifying smaller range of ports is always a good idea to reduce annoying waiting …

The -w flag is used to set a timeout to remote connection, usually on a local network situated machines the timeout could be low -w 1 but for machines across different Data Centers (let say one in Berlin and one in Seattle), use as a minimum -w 5.

If you expect remote service to be responsive (as it should always be), it is a nice idea to use netcat with a low timeout (-w) value of 1 below is example:

netcat -v -z -n -w 1 scanned-hosts 1-1023

5. Port scanning range of IP addresses with netcat

If you have used Nmap you know scanning for a network range is as simple as running something like nmap -sP -P0 192.168.0.* (to scan from IP range 1-255 map -sP -P0 192.168.0.1-150 (to scan from local IPs ending in 1-150) or giving the network mask of the scanned network, e.g. nmap -sF 192.168.0.1/24 – for more examples please check my previous article Checking port security on Linux with nmap (examples).

But what if nmap is not there and want to check a bunch 10 Splunk servers (software for searching, monitoring, and analyzing machine-generated big data, via a Web-style interface.), with netcat to find, whether the default Splunk connection port 9997 is opened or not:

for i in `seq 1 10`; do nc -z -w 5 -vv splunk0$i.server-domain.com 9997; done

6. Checking whether UDP port traffic is allowed to destination server

Assuring you have access on Source traffic (service) Host A and Host B (remote destination server where a daemon will be set-upped to listen on UDP port and no firewall in the middle Network router or no traffic control and filtering software HUB is preventing the sent UDP proto traffic, lets say an ntpd will be running on its standard 123 port there is done so:

– On host B (the remote machine which will be running ntpd and should be listening on port 123), run netcat to listen for connections

# nc -l -u -p 123
Listening on [0.0.0.0] (family 2, port 123)

Make sure there is no ntpd service actively running on the server, if so stop it with /etc/init.d/ntpd stop
and run above command. The command should run as superuser as UDP port 123 is from the so called low ports from 1-1024 and binding services on such requires root privileges.

– On Host A (UDP traffic send host

nc -uv remote-server-host 123

If the remote port is not reachable due to some kind of network filtering, you will get "connection refused".
An important note to make is on some newer Linux distributions netcat might be silently trying to connect by default using IPV6, bringing false positives of filtered ports due to that. Thus it is generally a good idea, to make sure you're connecting to IPV6

$ nc -uv -4 remote-server-host 123

Another note to make here is netcat's UDP connection takes 2-3 seconds, so make sure you wait at least 4-8 seconds for a very distant located hosts that are accessed over a multitude of routers.

7. Checking whether TCP port traffic allowed to DST remote server

To listen for TCP connections on a specified location (external Internet IP or hostname), it is analogous to listening for UDP connections.

Here is for example how to bind and listen for TCP connections on all available Interface IPs (localhost, eth0, eth1, eth2 etc.)

nc -lv 0.0.0.0 12345

Then on client host test the connection with

nc -vv 192.168.0.103 12345
Connection to 192.168.0.103 12345 port [tcp/*] succeeded!

8. Proxying traffic with netcat

Another famous hackers use of Netcat is its proxying possibility, to proxy anything towards a third party application with UNIX so any content returned be printed out on the listening nc spawned daemon like process.
For example one application is traffic SMTP (Mail traffic) with netcat, below is example of how to proxy traffic from Host B -> Host C (in that case the yandex current mail server mx.yandex.ru)

linux-srv:~# nc -l 12543 | nc mx.yandex.ru 25

Now go to Host A or any host that has TCP/IP protocol access to port 12543 on proxy-host Host B (linux-srv) and connect to it on 12543 with another netcat or telnet.

to make netcat keep connecting to yandex.ru MX (Mail Exchange) server you can run it in a small never ending bash shell while loop, like so:

linux-srv:~# while :; do nc -l 12543 | nc mx.yandex.ru 25; done

Below are screenshots of a connection handshake between Host B (linux-srv) proxy host and Host A (the end client connecting) and Host C (mx.yandex.ru).

host-B-running-as-a-proxy-daemon-towards-Host-C-yandex-mail-exchange-server

Host B netcat as a (Proxy)

Host-A-Linux-client-connection-handshake-to-proxy-server-with-netcat
that is possible in combination of UNIX and named pipes (for more on Named pipes check my previous article simple linux logging with named pipes), here is how to run a single netcat version to proxy any traffic in a similar way as the good old tinyproxy.

On Proxy host create the pipe and pass the incoming traffic towards google.com and write back any output received back in the named pipe.

# mkfifo backpipe
# nc -l 8080 0<backpipe | nc www.google.com 80 1>backpipe

Other useful netcat proxy set-up is to simulate a network connectivity failures.

For instance, if server:port on TCP 1080 is the normal host application would connect to, you can to set up a forward proxy from port 2080 with

nc -L server:1080 2080

then set-up and run the application to connect to localhost:2080 (nc proxy port)

/path/to/application_bin –server=localhost –port=2080

Now application is connected to localhost:2080, which is forwarded to server:1080 through netcat. To simulate a network connectivity failure, just kill the netcat proxy and check the logs of application_bin.

Using netcat as a bind shell (make any local program / process listen and deliver via nc)

netcat can be used to make any local program that can receive input and send output to a server, this use is perhaps little known by the junior sysadmin, but a favourite use of l337 h4x0rs who use it to spawn shells on remote servers or to make connect back shell. The option to do so is -e

-e – option spawns the executable with its input and output redirected via network socket.

One of the most famous use of binding a local OS program to listen and receive / send content is by
making netcat as a bind server for local /bin/bash shell.

Here is how

nc -l -p 4321 -e /bin/sh

If necessery specify the bind hostname after -l. Then from any client connect to 4321 (and if it is opened) you will gain a shell with the user with which above netcat command was run. Note that many modern distribution versions such as Debian / Fedora / SuSE Linux's netcat binary is compiled without the -e option (this works only when compiled with -DGAPING_SECURITY_HOLE), removal in this distros is because option is potentially opening a security hole on the system.

If you're interested further on few of the methods how modern hackers bind new backdoor shell or connect back shell, check out Spawning real tty shells article.

For more complex things you might want to check also socat (SOcket CAT) – multipurpose relay for bidirectional data transfer under Linux.
socat is a great Linux Linux / UNIX TCP port forwarder tool similar holding the same spirit and functionality of netcat plus many, many more.

On some of the many other UNIX operating systems that are lacking netcat or nc / netcat commands can't be invoked a similar utilitiesthat should be checked for and used instead are:

ncat, pnetcat, socat, sock, socket, sbd

To use nmap's ncat to spawn a shell for example that allows up to 3 connections and listens for connects only from 192.168.0.0/24 network on port 8081:

ncat –exec "/bin/bash" –max-conns 3 –allow 192.168.0.0/24 -l 8081 –keep-open

9. Copying files over network with netcat

Another good hack often used by hackers to copy files between 2 servers Server1 and Server2 who doesn't have any kind of FTP / SCP / SFTP / SSH / SVN / GIT or any kind of Web copy support service – i.e. servers only used as a Database systems that are behind a paranoid sysadmin firewall is copying files between two servers with netcat.

On Server2 (the Machine on which you want to store the file)

nc -lp 2323 > files-archive-to-copy.tar.gz

On server1 (the Machine from where file is copied) run:

nc -w 5 server2.example.com 2323 < files-archive-to-copy.tar.gz

Note that the downside of such transfers with netcat is data transferred is unencrypted so any one with even a simple network sniffer or packet analyzier such as iptraf or tcpdump could capture the file, so make sure the file doesn't contain sensitive data such as passwords.

Copying partition images like that is perhaps best way to get disk images from a big server onto a NAS (when you can't plug the NAS into the server).

10. Copying piped archived directory files with netcat

On computer A:

export ARIBTRARY_PORT=3232
nc -l $ARBITRARY_PORT | tar vzxf –

On Computer B:

tar vzcf – files_or_directories | nc computer_a $ARBITRARY_PORT

11. Creating a one page webserver with netcat and ncat

As netcat could listen to port and print content of a file, it can be set-up with a bit of bash shell scripting to serve
as a one page webserver, or even combined with some perl scripting and bash to create a multi-serve page webserver if needed.

To make netact serve a page to any connected client run in a screen / tmux session following code:

while true; do nc -l -p 80 -q 1 < somepage.html; done

Another interesting fun example if you have installed ncat (is a small web server that connects current time on server on connect).

ncat -lkp 8080 –sh-exec 'echo -ne "HTTP/1.0 200 OK\r\n\r\nThe date is "; date;'

12. Cloning Hard disk partitions with netcat

rsync is a common tool used to clone hard disk partitions over network. However if rsync is not installed on a server and netcat is there you can use it instead, lets say we want to clone /dev/sdb
from Server1 to Server2 assuming (Server1 has a configured working Local or Internet connection).

On Server2 run:

nc -l -p 4321 | dd of=/dev/sdb

Following on Server2 to start the Partition / HDD cloning process run

dd if=/dev/sdb | nc 192.168.0.88 4321

Where 192.168.0.88 is the IP address listen configured on Server2 (in case you don't know it, check the listening IP to access with /sbin/ifconfig).

Next you have to wait for some short or long time depending on the partiiton or Hard drive, number of files / directories and allocated disk / partition size.

To clone /dev/sda (a main partiiton) from Server1 to Server2 first requirement is that it is not mounted, thus to have it unmounted on a system assuming you have physical access to the host, you can boot some LiveCD Linux distribution such as Knoppix Live CD on Server1, manually set-up networking with ifconfig or grab an IP via DHCP from the central DHCP server and repeat above example.

Happy netcating 🙂

Tags: access, addresses, after, ALL, ALLOW, allowed, allows, checking, command, destination, due, firewall, linux windows, network, nmap, Scanning, script kiddie, security policies, tool, unix
Posted in Linux, Linux and FreeBSD Desktop, Networking | 1 Comment »

Howto Configure Linux shell Prompt / Setup custom Terminal show Prompt using default shell variables PS1, PS2, PS3, PS4

Tuesday, August 27th, 2019

System Console, Command Operation Console or Terminal is a Physical device for text (command) input from keyboard, getting the command output and monitoring the status of a shell or programs I/O operations generated traditionally with attached screen. With the development of Computers, physical consoles has become emulated and the input output is translated on the monitor usually via a data transfer protocol historically mostly over TCP/IP connection to remote IP with telnet or rsh, but due to security limitations Consoles are now accessed over data (encrypted) network protocols with SHA2 / MD5 cryptography algorithm enabled such as over SSH (Secure Shell) network protocol..
The ancestors of physical consoles which in the past were just a Terminal (Monitoring / Monitor device attached to a MainFrame system computer).

Mainframe-physical-terminal-monitor-Old-Computer

What is Physical Console
A classical TTY (TeleTYpewriter) device looked like so and served the purpose of being just a communication and display deivce, whether in reality the actual computing and storage tape devices were in a separate room and communicating to Terminal.

mainframe-super-computer-computing-tape-machine
TTYs are still present in modern UNIX like GNU / Linux distrubions OSes and the BSD berkley 4.4 code based FreeBSD / NetBSD / OpenBSD if you have installed the OS on a physical computer in FreeBSD and Solaris / SunOS there is also tty command. TTY utility in *nix writes the name of the terminal attached to standard input to standard output, in Linux there is a GNU remake of same program part called GNU tty of coreutils package (try man tty) for more.

The physical console is recognizable in Linux as it is indicated with other tree letters pts – (pseudo terminal device) standing for a terminal device which is emulated by an other program (example: xterm, screen, or ssh are such programs). A pts is the slave part of a pts is pseudo there is no separate binary program for it but it is dynamically allocated in memory.
PTS is also called Line consle in Cisco Switches / Router devices, VTY is the physical Serial Console connected on your Cisco device and the network connection emulation to network device is creates with a virtual console session VTL (Virtual Terminal Line). In freebsd the actual /dev/pts* /dev/tty* temporary devices on the OS are slightly different and have naming such as /dev/ttys001.
But the existence of tty and pts emulator is not enough for communicating interrupts to Kernel and UserLand binaries of the Linux / BSD OS, thus to send the commands on top of it is running a System Shell as CSH / TSH / TCSH or BASH which is usually the first program set to run after user logs in over ptty or pseudo tty virtual terminal.

linux-tty-terminal-explained-brief-intro-to-linux-device-drivers-20-638

Setting the Bash Prompt in Terminal / Console on GNU / Linux

Bash has system environments to control multiple of variables, which are usually visible with env command, one important variable to change in the past was for example USER / USERNAME which was red by IRC Chat clients such as BitchX / irssi and could be displayed publicly so if not changed to a separate value, one could have known your Linux login username by simple /whois query to the Nickname in question (if no inetd / xinetd service was running on the Linux box and usually inetd was not running).

Below is my custom set USER / USERNAME to separate

hipo@pcfreak:~$ env|grep USER
USERNAME=Attitude
USER=Attitude

There is plenty of variables to tune email such as MAIL store directory, terminal used TERM, EDITOR etc. but there are some
variables that are not visible with env query as they're not globally available for all users but just for the single user, to show this ones you need to use declare command instead, to get a full list of All Single and System Wide defined variables and functions type declare in the bash shell, for readability, below is last 10 returned results:

hipo@pcfreak:~$ declare | tail -10
{
    local quoted=${1//\'/\'\\\'\'};
    printf "'%s'" "$quoted"
}
quote_readline ()
{
    local quoted;
    _quote_readline_by_ref "$1" ret;
    printf %s "$ret"
}

PS1 is present there virtually on any modern Linux distribution and is installed through user home's directory $HOME/.bashrc , ~/.profile or .bash_profile or System Wide globally for all existing users in /etc/passwd (password database file) from /etc/bash.bashrc
In Debian / Ubuntu / Mint GNU / Linux this system variable is set in user home's .bashrc but in Fedora / RHEL Linux distro,
PS1 is configured from /home/username/.bash_profile to find out where PS1 is located for ur user:

cd ~
grep -Rli PS1 .bash*

Here is one more example:

hipo@pcfreak:~$ declare|grep -i PS1|head -1
PS1='\[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '

hipo@pcfreak:~$ grep PS1 /etc/bash.bashrc
[ -z “$PS1” ] && return
# but only if not SUDOing and have SUDO_PS1 set; then assume smart user.
if ! [ -n “${SUDO_USER}” -a -n “${SUDO_PS1}” ]; then
PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '

Getting current logged in user shell configured PS1 variable can be done with echo:

hipo@pcfreak:~$ echo $PS1
\[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\u@\h:\w\$

So lets observe a little bit the meaning of this obscure line of (code) instructions code which are understood by BASH when being red from PS1 var to do so, I'll give a list of meaning of main understood commands, each of which is defined with \.

The ${debian_chroot} shell variable is defined from /etc/bash.bashrc

Easiest way to change PS1 is to export the string you like with the arguments like so:

root@linux:/home/hipo# export PS1='My-Custom_Server-Name# '
My-Custom_Server-Name# echo $PS1
My-Custom_Server-Name#

\a : an ASCII bell character (07)
\d : the date in “Weekday Month Date” format (e.g., “Tue May 26”)
\D{format} : the format is passed to strftime(3) and the result is inserted into the prompt string; an empty format results in a locale-specific time representation. The braces are required
\e : an ASCII escape character (033)
\h : the hostname up to the first ‘.’
\H : the hostname
\j : the number of jobs currently managed by the shell
\l : the basename of the shell's terminal device name
\n : newline
\r : carriage return
\s : the name of the shell, the basename of $0 (the portion following the final slash)
\t : the current time in 24-hour HH:MM:SS format
\T : the current time in 12-hour HH:MM:SS format
\@ : the current time in 12-hour am/pm format
\A : the current time in 24-hour HH:MM format
\u : the username of the current user
\v : the version of bash (e.g., 2.00)
\V : the release of bash, version + patch level (e.g., 2.00.0)
\w : the current working directory, with $HOME abbreviated with a tilde
\W : the basename of the current working directory, with $HOME abbreviated with a tilde
\! : the history number of this command
\# : the command number of this command
\$ : if the effective UID is 0, a #, otherwise a $
\nnn : the character corresponding to the octal number nnn
\\ : a backslash
\[ : begin a sequence of non-printing characters, which could be used to embed a terminal control sequence into the prompt
\] : end a sequence of non-printing characters

The default's PS1 set prompt on Debian Linux is:

echo $PS1
\[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\u@\h:\w\$

As you can see \u (print username) \h (print hostname) and \W (basename of current working dir) or \w (print $HOME/current working dir)
are the most essential, the rest are bell character, escape character etc.

A very good way to make your life easier and learn the abbreviations / generate exactly the PS1 PROMPT you want to have is with Easy Bash PS1 Generator Web Utility
with which you can just click over buttons that are capable to produce all of the PS1 codes.

1. How to show current hour:minute:seconds / print full date in Prompt Shell (PS)

Here is an example with setting the Bash Shell prompt to include also the current time in format hour:minute:seconds (very useful if you're executing commands on a critical servers and you run commands in some kind of virtual terminal like screen or tmux.

root@pcfreak:~# PS1="\n\t \u@\h:\w# "
14:03:51 root@pcfreak:/home#

PS1-how-to-setup-date-time-hour-minutes-and-seconds-in-bash-shell-prompt

export PS1='\u@\H \D{%Y-%m-%d %H:%M;%S%z}] \W ] \$ '

Make superuser appear in RED color (adding PS1 prompt custom color for a User)

root@pcfreak:~$ PS1="\\[$(tput setaf 1)\\]\\u@\\h:\\w #\\[$(tput sgr0)\\]"

how-to-change-colors-in-bash-prompt-shell-on-linux-shell-environment

In above example the Shell Prompt Color changed is changed for administrator (root) to shebang symbol # in red, green, yellow and blue for the sake to show you how it is done, however this example can be adapted for any user on the system. Setting different coloring for users is very handy if you have to administer Mail Server service like Qmail or other Application that consists of multiple small ones of multiple daemons such as qmail + vpopmail + clamd + mysql etc. Under such circumstances, coloring each of the users in different color like in the example for debugging is very useful.

Coloring the PS1 system prompt on Linux to different color has been a standard practice in Linux Server environments running Redhat Enterprise Linux (RHEL) and SuSE Enterprise Linux and some Desktop distributions such as Mint Linux.

To make The Root prompt Red colored only for system super user (root) on any Linux distribution
, add the following to /etc/bashrc, e.g.

vim /etc/bashrc

# If id command returns zero, you've root access.
if [ $(id -u) -eq 0 ];
then # you are root, set red colour prompt
PS1="\\[$(tput setaf 1)\\]\\u@\\h:\\w #\\[$(tput sgr0)\\]"
else # normal
PS1="[\\u@\\h:\\w] $"
fi

2. How to make the prompt of a System user appear Green

Add to ~/.bashrc following line

PS1="\\[$(tput setaf 2)\\]\\u@\\h:\\w #\\[$(tput sgr0)\\]"

3. Print New line, username@hostname, base PTY, shell level, history (number), newline and full working directory $PWD

export PS1='\n[\u@\h \l:$SHLVL:\!]\n$PWD\$ '

4. Showing the numbert of jobs the shell is currently managing.

This is useful if you run and switch with fg / bg (foreground / background) commands
to switch between jobs and forget some old job.

export PS1='\u@\H \D{%Y-%m-%d %H:%M;%S%z}] \W \$]'

Multi Lines Prompt / Make very colorful Shell prompt full of stats info

PS1="\n\[\033[35m\]\$(/bin/date)\n\[\033[32m\]\w\n\[\033[1;31m\]\u@\h: \[\033[1;34m\]\$(/usr/bin/tty | /bin/sed -e ‘s:/dev/::’): \[\033[1;36m\]\$(/bin/ls -1 | /usr/bin/wc -l | /bin/sed ‘s: ::g’) files \[\033[1;33m\]\$(/bin/ls -lah | /bin/grep -m 1 total | /bin/sed ‘s/total //’)b\[\033[0m\] -> \[\033[0m\]"

prompt-show-how-many-files-and-virtual-pts-ps1-linux

5. Set color change on command failure

If you have a broken command or the command ended with non zero output with some kind of bad nasty message and you want to make, that more appearing making it red heighlighted, here is how:

PROMPT_COMMAND='PS1="\[\033[0;33m\][\!]\`if [[ \$? = “0” ]]; then echo “\\[\\033[32m\\]”; else echo “\\[\\033[31m\\]”; fi\`[\u.\h: \`if [[ `pwd|wc -c|tr -d ” “` > 18 ]]; then echo “\\W”; else echo “\\w”; fi\`]\$\[\033[0m\] “; echo -ne “\033]0;`hostname -s`:`pwd`\007"'

6. Other beautiful PS1 Color Prompts with statistics

PS1="\n\[\e[32;1m\](\[\e[37;1m\]\u\[\e[32;1m\])-(\[\e[37;1m\]jobs:\j\[\e[32;1m\])-(\[\e[37;1m\]\w\[\e[32;1m\])\n(\[\[\e[37;1m\]! \!\[\e[32;1m\])-> \[\e[0m\]"

another-very-beuatiful-bash-colorful-prompt

7. Add Muliple Colors to Same Shell prompt

function prompt { local BLUE="\[\033[0;34m\]” local DARK_BLUE=”\[\033[1;34m\]” local RED=”\[\033[0;31m\]” local DARK_RED=”\[\033[1;31m\]” local NO_COLOR=”\[\033[0m\]” case $TERM in xterm*|rxvt*) TITLEBAR=’\[\033]0;\u@\h:\w\007\]’ ;; *) TITLEBAR=”” ;; esac PS1=”\u@\h [\t]> ” PS1=”${TITLEBAR}\ $BLUE\u@\h $RED[\t]>$NO_COLOR " PS2='continue-> ' PS4='$0.$LINENO+ ' }

colorful-prompt-blue-and-red-linux-console-PS1

8. Setting / Change Shell background Color

export PS1="\[$(tput bold)$(tput setb 4)$(tput setaf 7)\]\u@\h:\w $ \[$(tput sgr0)\]"

tput Color Capabilities:

tput setab [1-7] – Set a background color using ANSI escape
tput setb [1-7] – Set a background color
tput setaf [1-7] – Set a foreground color using ANSI escape
tput setf [1-7] – Set a foreground color

tput Text Mode Capabilities:

tput bold – Set bold mode
tput dim – turn on half-bright mode
tput smul – begin underline mode
tput rmul – exit underline mode
tput rev – Turn on reverse mode
tput smso – Enter standout mode (bold on rxvt)
tput rmso – Exit standout mode
tput sgr0 – Turn off all attributes

Color Code for tput:

0 – Black
1 – Red
2 – Green
3 – Yellow
4 – Blue
5 – Magenta
6 – Cyan
7 – White

9. Howto Use bash shell function inside PS1 variable

If you administrate Apache or other HTTPD servers or any other server whose processes are forked and do raise drastically at times to keep an eye while actively working on the server.

function httpdcount { ps aux | grep apache2 | grep -v grep | wc -l } export PS1="\u@\h [`httpdcount`]> "

10. PS2, PS3, PS4 little known variables

I'll not get much into detail to PS2, PS3, PS4 but will mention them as perhaps many people are not even aware they exist.
They're rarely used in the daily system administrator's work but useful for Shell scripting purposes of Dev Ops and Shell Scripting Guru Programmers.

PS2 – Continuation interactive prompt

A very long unix command can be broken down to multiple line by giving \ at the end of the line. The default interactive prompt for a multi-line command is “> “. Let us change this default behavior to display “continue->” by using PS2 environment variable as shown below.

hipo@db-host :~$ myisamchk –silent –force –fast –update-state \
> –key_buffer_size=512M –sort_buffer_size=512M \
> –read_buffer_size=4M –write_buffer_size=4M \
> /var/lib/mysql/bugs/*.MYI
[Note: This uses the default “>” for continuation prompt]

PS3 – Prompt used by “select” inside shell script (usefulif you write scripts with user prompts)
PS4 – Used by “set -x” to prefix tracing output
The PS4 shell variable defines the prompt that gets displayed.

You can find example with script demonstrating PS2, PS3, PS4 use via small shell scripts in thegeekstuff's article Take control of PS1, PS2, PS3, PS4 read it here

Summary

In this article, I've shortly reviewed on what is a TTY, how it evolved into Pseudo TTY and how it relates to current shells which are the interface communicating with the modern UNIX like Operating systems's userland and kernel.
Also it was reviewed shortly how the current definitions of shell variables could be viewed with declare cmd. Also I went through on how to display the PS1 variable and on how to modify PS1 and make the prompt different statistics and monitoring parameters straight into the command shell. I've shown some common PS1 strings that report on current date hour, minute, seconds, modify the coloring of the bash prompt shell, show processes count, and some PS1 examples were given that combines beuatiful shell coloring as well as how the Prompt background color can be changed.
Finally was shown how a combination of commands can be executed by exporting to PS1 to update process counf of Apache on every shell prompt iteration.
Other shell goodies are mostly welcome

Tags: access, adding, administrate, administrator, after, algorithm, ALL, amp, bash shell, Below, command, current time, default shell, echo, email, home, How to, root linux, Virtual Terminal Line, VTL
Posted in Linux, Linux and FreeBSD Desktop | No Comments »

Check weather forecast from console (terminal) on GNU / Linux and FreeBSD howto

Friday, August 23rd, 2019

how to get weather forecast prognosis from command line text terminal / console on Linux and FreeBSD

Doing everything in Linux console / terminal is a question perhaps every Linux / BSD hacker wants to do as Graphical user interface and using web search or using Graphical Environment plugins is an unneded complexity + googling or duckduckgoing for weather to check your next vacation destination city has been more and more of a terrible experience (for me) as I'm not a big fan of using the OS in a GUI.
In that manner of thoughts, as a Linux console geek and hard core ASCII art fan. I was recently happy to find that possible to check weather forecast in tty console or Linux terminal in a beautiful ascii art way easily through a Web wttr.in service – a web application weather forecast service that supports displaying the current and few days in future, weather forecast either in browser as a plain text or from the command line by simply accessing it with your favourite web access / transfer tool such as;
wget / curl or any of your favourite text browser elinks / lynx / w3m or if on *BSDs use fetch command.

Install Curl data transfer tool if it is not already

Wget is installed by default across most Linux distributions and fetch is present by default on BSDs, displaying it in text browser would perhaps be never used but if you decide to give it a try maybe try with elinks (to get colorful output), w3m and lynx will display a black and white results.

In case if you miss curl, install it:

On Debian distro

aptitude install -y curl

or Fedora

yum install -y curl
…

Of course to use wttr.in as it is Internet based Weather Forecast service the minimum you need to have is to have Internet connection to your Linux / BSD desktop computer.

Text based Weather Forecast Web App currently supports:

display the current weather as well as a 3-day weather forecast, split into morning, noon, evening and night

Temperature is displayed for morning, noon, evening and night (includes temperature range, wind speed and direction, viewing distance, precipitation amount and probability)
Provide results for Weather based on City / town / village location
Supports display of Moon Phases Forecast in calendar days
Supports multilingual names (Bulgarian Phonetic cyrillic / Russian and other exotic UTF-8 encodings such as Chineese and Japanese), 50+ languages are currently supported
Has ability for prognosis for hostname (domain) location based on an its IP GeoIP location on the Globe
Geographical locations / landmarks such as Lakes / Mountains etc. can be easily queried
Query results metrics could be configured, e.g. USCS units or EU and rest of world accepted ones (SI) metric
Displayed result could be either in ANSI (if from terminal / console / HTML if queried from browser or in PNG – if needed)

Where wttr.in could be useful ?

The best applications use, I can think of are for server (shell) / perl scripting automation purposes, it could be useful especially in TOO HOT, TOO, COLD, TOO WET location in Small and Middle sized Data Centers Green Energy (Sun Panel) Parks / Wind Energy situated Linux monitoring hosts to track possible problems of overheats or overcolding of servers due to abnormal excessive temperatures such as the ones we experienced this summer here All across in Europe or in too Cold DC locations such as heat locations Deserts in African Countries, Saudi Arabia or Chukotka or Siberia in Russia.
Other application is as a backup option to other normal Weather report services by PHP or Python scripts that fetch data, from multiple places.
Of course since this is a third party controlled service, the downtime is due to excessive connection requests, the service could get flooded and stopped working, but I guess for any Commercial use, wttr.in creator Igor Chubin would be happy to sell a specific crafted service for any end user candidates.

Here is few examples of the beautiful returned ASCII art formatted output of wttr.in.

1. Getting a three days Weather Forecast prognosis for city / town location

To get what is current weather in my current city of Living, Sofia Bulgaria just pass the city to the URL address

curl http://wttr.in/Sofia

text-console-wttr.in-Weather-forecast-Sofia-for-Linux

links http://wttr.in/Dobrich

curl-Linux-show--Dobrich-Weather-forecast-in-lynx-text-browser

Default links (Linux) www text browser produces ugly black and white

2. Displaying Weather forecast with wget

wget -O- -q http://wttr.in

getting-weather-forecast-on-linux-terminal-console-with-wget-command

If you're lazy you can even omit the http:// as wget will look for HyperText Transmission Protocol by itself

wget -O- -q wttr.in

3. Getting Forecast results for a Tourist Destination

Lets get the weather forecast for the popular tourist Bulgarian destination of the Seven Rila Lakes (near Rila Monastery), situated in the Rila Mountain BG.

curl http://wttr.in/Seven+Rila+Lakes

Console-terminal-Weather-forecast-Linux-Seven-Rila-Lakes

4. Display Forecast for a specific server IP

Displaying information on specific server IP address current situated in GeoIP database, of course could be not really true, as the IP could be just a Load Balancer a router that does NAT to some internal DMZ-ed location server, but anyways it is a cool feature.

Lets get information on what is the weather on Google Global's Public DNS server IP 8.8.8.8 so commonly used to guarantee a Windows and Linux Desktop client machines Internet connectivity.

curl wttr.in/@8.8.8.8

wttr.in-Linux-text--forecast-service-curl-screenshot Google Public DNS location weather forecast

5. Download PNG image picture from wttr.in service

Lets say you want to get a 3 days standard Weather forecast for the popular Black Sea Resort town in Bulgaria Pomorie (a beautiful sea city which has even a functioning 5 Monks Monastery Pomorie Monastery situated near sea coast)

curl http://wttr.in/Pomorie.png

–2019-08-22 20:15:51– http://wttr.in/Pomorie.png
Resolving wttr.in (wttr.in)… 5.9.243.187
Connecting to wttr.in (wttr.in)|5.9.243.187|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 42617 (42K) [image/png]
Saving to: ‘Pomorie.png’
Pomorie.png 100%[=======================================================================================================>] 41.62K –.-KB/s in 0.07s
2019-08-22 20:15:52 (586 KB/s) – ‘Pomorie.png’ saved [42617/42617]

Note: The generated .png is again the ASCII art produced by a direct text fetch bug in pic format

6. Displaying Current Moon Phase

If you want to enjoy a text based Moon phase picture through wttr.in 🙂

wget -O- -q wttr.in/Moon

You can also get a Moon Phase prognosis for a current future date or get a previous date phase

curl wttr.in/moon@2019-09-15

Full-Moon-Weather-forecast-text-console-reporting-via-wttr.in-on-Gnu_Linux

Full Moon Madness !! – Vampires are out beaware and Enjoy the ultra kewl ASCII Colorful Art 🙂

7. Getting help for wttr.in terminal Waether Forecast results

$ curl wttr.in/:help
Usage:

    $ curl wttr.in          # current location
    $ curl wttr.in/muc      # weather in the Munich airport
Supported location types:
    /paris                  # city name
    /~Eiffel+tower          # any location
    /Москва                 # Unicode name of any location in any language
    /muc                    # airport code (3 letters)
    /@stackoverflow.com     # domain name
    /94107                  # area codes
    /-78.46,106.79          # GPS coordinates
Special locations:
    /moon                   # Moon phase (add ,+US or ,+France for these cities)
    /moon@2016-10-25        # Moon phase for the date (@2016-10-25)
Units:
    m                       # metric (SI) (used by default everywhere except US)
    u                       # USCS (used by default in US)
    M                       # show wind speed in m/s
View options:
    0                       # only current weather
    1                       # current weather + 1 day
    2                       # current weather + 2 days
    A                       # ignore User-Agent and force ANSI output format (terminal)
    F                       # do not show the "Follow" line
    n                       # narrow version (only day and night)
    q                       # quiet version (no "Weather report" text)
    Q                       # superquiet version (no "Weather report", no city name)
    T                       # switch terminal sequences off (no colors)
PNG options:
    /paris.png              # generate a PNG file
    p                       # add frame around the output
    t                       # transparency 150
    transparency=…        # transparency from 0 to 255 (255 = not transparent)
Options can be combined:
    /Paris?0pq
    /Paris?0pq&lang=fr
    /Paris_0pq.png          # in PNG the file mode are specified after _
    /Rome_0pq_lang=it.png   # long options are separated with underscore
Localization:
    $ curl fr.wttr.in/Paris
    $ curl wttr.in/paris?lang=fr
    $ curl -H "Accept-Language: fr" wttr.in/paris
Supported languages:
    af da de el et fr fa hu id it nb nl pl pt-br ro ru tr uk vi (supported)
    az be bg bs ca cy cs eo es fi ga hi hr hy is ja jv ka kk ko ky lt lv mk ml nl fy nn pt pt-br sk sl sr sr-lat sv sw th te uz zh zu he (in progress)
Special URLs:
    /:help                  # show this page
    /:bash.function         # show recommended bash function wttr()
    /:translation           # show the information about the translators

8. Comparing two cities weather from command line

One useful use of wttr.in if you plan to travel from Location city A to Location city B is to compare the temperatures with a simple bash one liner script:

diff -Naur <(curl -s http://wttr.in/Sofia ) <(curl -s http://wttr.in/Beograd )

9. Using ansiweather command to get Weather Temperature / Wind / Humidity in one line beuatiful text

If you go and install answeather Linux package

apt-get install –yes ansiweather

You will get a shell script wrapper with ANSI colors and Unicode symbols support. Weather data comes from OpenWeatherMap, this is useful if wttr.in is not working due to some URL malfunction (due to service is DoS-ed) etc.

ansiweather -l Atina

ansiweather-Atina-weather-forecast-result-linux-text-console

Lets use ansiweather to print the weather prognosis for upcoming 5 days for near port of Burgas, BG

ansiweather -F -l Burgas

ansiweather-print-weather-forecast-prognosis-for-5-days-in-Linux-text-terminal

10. Get all Weather current forecast for each Capital in the world

You can download and use this simple plain text file list of All Country Capitals in the World (country-capitals-all-world.txt) with ansiweather and a bash loop to get displayed each and every current day Weather Forecast in the World, here is how:

while read line; do ansiweather -l $line; sleep 3; done < country-capitals-all-world.txt

ansiweather-all-countires-capitals-result

As you can see some of the very exotic third world capitals does not return data so 'ERROR: Cannot fetch weather data' is returned.

You can also substitute ansiweather with curl wttr.in/$line to do get the beautiful ASCII art 3 days weather forecast via wttr.in

while read line; do curl http://wttr.in/$line; sleep 3; done < country-capitals-all-world.txt

I'll be happy to know other nice ASCII Art supporting Web service to enjoy from text terminal on Linux (nomatter useful or) just funny joyful prank maniacal pranks such as Watching text ASCII version remake of Star Wars Classic Movie by simply telnetting to towel.blinkenlights.nl (if you haven't so just telnet and enjoy the streamed ASCIIs ! 🙂

telnet towel.blinkenlights.nl

Talking about fun and ASCII, its worthy to mention hollywood Linux package

hipo@jeremiah:~/Desktop$ apt-cache show hollywood|grep -i desc -A 3
Description-en: fill your console with Hollywood melodrama technobabble
This utility will split your console into a multiple panes of genuine
technobabble, perfectly suitable for any Hollywood geek melodrama.
It is particularly suitable on any number of computer consoles in the
—
Description-md5: 768f44c76220ea2b35f855ea34c8bc35
Homepage: http://launchpad.net/hollywood
Section: games
Priority: optional

Once installed on Debian with:

aptitude install -y hollywood
…

You can get in a rapid manner plenty of tmux (screen like – virtual console emulator) split screen statistics about your notebook / workstation / server CPU usage, mlocate.db status, info about plugged in machine voltage, Speedometer (statistics about Network bandwidth usage), System load avarage (CPU Count, Memory Utilization) and some other random info coming out of dmesg kernel log and more. The information displayed in splitted windows changes rapidly and (assuming you run it at home Desktop with a soundblaster) and not remotely, a james bond Agent 007 soundtrack is played on the back, that brings up one's adrenaline and makes it look even cooler.

hollywood-melodrama-technobubble-split-console-multiple-panes-for-genuine-technobubble

To give you an idea what to expect, here is shot of /usr/games/hollywood (the program start binary location) on Debian GNU / Linux running, Enjoy! 🙂

Tags: about, access, console, default, destination city, Displaying Current Moon Phase, domain location, ip, languages, linux?, look, png, Resolving, specific, terminal, tourist destination, Usage, web access, wget, Windows
Posted in Curious Facts, Educational, Linux, Linux and FreeBSD Desktop | No Comments »

What is inode and how to find out which directory is eating up all your filesystem inodes on Linux, Increase inode count on a ext3 ext4 and ufs filesystems

Tuesday, August 20th, 2019

what-is-inode-find-out-which-filesystem-or-directory-eating-up-all-your-system-inodes-linux_inode_diagram

If you're a system administrator of multiple Linux servers used for Web serving delivery / Mail server sysadmin, Database admin or any High amount of Drives Data Storage used for backup servers infra, Data Repository administrator such as Linux hosted Samba / CIFS shares, etc. or using some Linux Hosting Provider to host your website or any other UNIX like Infrastructure servers that demands a storage of high number of files under a Directory you might end up with the common filesystem inode depletion issues ( Maximum Inode number for a filesystem is predefined, limited and depending on the filesystem configured size).

In case a directory stored files end up exceding the amount of possible addressable inodes could prevent any data to be further assiged and stored on the Filesystem.

When a device runs out of inodes, new files cannot be created on the device, even though there may be plenty free space available and the first time it happened to me very long time ago I was completely puzzled how this is possible as I was not aware of Inodes existence …

Reaching maximum inodes number (e.g. inode depletion), often happens on Busy Mail servers (receivng tons of SPAM email messages) or Content Delivery Network (CDN – Website Image caching servers) which contain many small files on EXT3 or EXT4 Journalled filesystems. File systems (such as Btrfs, JFS or XFS) escape this limitation with extents or dynamic inode allocation, which can 'grow' the file system or increase the number of inodes.

Hence ending being out of inodes could cause various oddities on how stored data behaves or communicated to other connected microservices and could lead to random application disruptions and odd results costing you many hours of various debugging to find the root cause of inodes (index nodes) being out of order.

In below article, I will try to give an overall explanation on what is an I-Node on a filesystem, how inodes of FS unit could be seen, how to diagnose a possible inode poblem – e.g. see the maximum amount of inodes available per filesystem and how to prepare (format) a new filesystem with incrsed set of maximum inodes.

What are filesystem i-nodes?

This is a data structure in a Unix-style file system that describes a file-system object such as a file or a directory.
The data structure described in the inodes might vary slightly depending on the filesystem but usually on EXT3 / EXT4 Linux filesystems each inode stores the index to block that contains attributes and disk block location(s) of the object's data.
– Yes for those who are not aware on how a filesystem is structured on *nix it does allocate all stored data in logical separeted structures called data blocks. Each file stored on a local filesystem has a file descriptor, there are virtual unit structures file tables and each of the inodes that are a reference number has a own data structure (inode table).

Inodes / "Index" are slightly unusual on file system structure that stored the access information of files as a flat array on the disk, with all the hierarchical directory information living aside from this as explained by Unix creator and pioneer- Dennis Ritchie (passed away few years ago).

what-is-inode-very-simplified-explanation-diagram-data

Simplified explanation on file descriptors, file table and inode, table on a common Linux filesystem

Here is another description on what is I-node, given by Ken Thompson (another Unix pioneer and father of Unix) and Denis Ritchie, described in their paper published in 1978:

" As mentioned in Section 3.2 above, a directory entry contains only a name for the associated file and a pointer to the file itself. This pointer is an integer called the i-number (for index number) of the file. When the file is accessed, its i-number is used as an index into a system table (the i-list) stored in a known part of the device on which the directory resides. The entry found thereby (the file's i-node) contains the description of the file:…
— The UNIX Time-Sharing System, The Bell System Technical Journal, 1978 "

What is typical content of inode and how I-nodes play with rest of Filesystem units?

The inode is just a reference index to a data block (unit) that contains File-system object attributes. It may include metadata information such as (times of last change, access, modification), as well as owner and permission data.

On a Linux / Unix filesystem, directories are lists of names assigned to inodes. A directory contains an entry for itself, its parent, and each of its children.

Structure-of-inode-table-on-Linux-Filesystem-diagram

Structure of inode table-on Linux Filesystem diagram (picture source GeeksForGeeks.org)

Information about files(data) are sometimes called metadata. So you can even say it in another way, "An inode is metadata of the data."
Inode : Its a complex data-structure that contains all the necessary information to specify a file. It includes the memory layout of the file on disk, file permissions, access time, number of different links to the file etc.
Global File table : It contains information that is global to the kernel e.g. the byte offset in the file where the user's next read/write will start and the access rights allowed to the opening process.
Process file descriptor table : maintained by the kernel, that in turn indexes into a system-wide table of files opened by all processes, called the file table .

The inode number indexes a table of inodes in a known location on the device. From the inode number, the kernel's file system driver can access the inode contents, including the location of the file – thus allowing access to the file.

Inodes do not contain its hardlink names, only other file metadata.
Unix directories are lists of association structures, each of which contains one filename and one inode number.
The file system driver must search a directory looking for a particular filename and then convert the filename to the correct corresponding inode number.

The operating system kernel's in-memory representation of this data is called struct inode in Linux. Systems derived from BSD use the term vnode, with the v of vnode referring to the kernel's virtual file system layer.

But enough technical specifics, lets get into some practical experience on managing Filesystem inodes.

Listing inodes on a Fileystem

Lets say we wan to to list an inode number reference ID for the Linux kernel (files):

root@linux: # ls -i /boot/vmlinuz-*
3055760 /boot/vmlinuz-3.2.0-4-amd64 26091901 /boot/vmlinuz-4.9.0-7-amd64
3055719 /boot/vmlinuz-4.19.0-5-amd64 26095807 /boot/vmlinuz-4.9.0-8-amd64

To list an inode of all files in the kernel specific boot directory /boot:

root@linux: # ls -id /boot/
26091521 /boot/

Listing inodes for all files stored in a directory is also done by adding the -i ls command flag:

Note the the '-1' flag was added to to show files in 1 column without info for ownership permissions

root@linux:/# ls -1i /boot/
26091782 config-3.2.0-4-amd64
3055716 config-4.19.0-5-amd64
26091900 config-4.9.0-7-amd64
26095806 config-4.9.0-8-amd64
26091525 grub/
3055848 initrd.img-3.2.0-4-amd64
3055644 initrd.img-4.19.0-5-amd64
26091902 initrd.img-4.9.0-7-amd64
3055657 initrd.img-4.9.0-8-amd64
26091756 System.map-3.2.0-4-amd64
3055703 System.map-4.19.0-5-amd64
26091899 System.map-4.9.0-7-amd64
26095805 System.map-4.9.0-8-amd64
3055760 vmlinuz-3.2.0-4-amd64
3055719 vmlinuz-4.19.0-5-amd64
26091901 vmlinuz-4.9.0-7-amd64
26095807 vmlinuz-4.9.0-8-amd64

To get more information about Linux directory, file, such as blocks used by file-unit, Last Access, Modify and Change times, current External Symbolic or Static links for filesystem object:

root@linux:/ # stat /etc/
File: /etc/
Size: 16384        Blocks: 32         IO Block: 4096   catalog
Device: 801h/2049d   Inode: 6365185     Links: 231
Access: (0755/drwxr-xr-x) Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-08-20 06:29:39.946498435 +0300
Modify: 2019-08-14 13:53:51.382564330 +0300
Change: 2019-08-14 13:53:51.382564330 +0300
Birth: –

Within a POSIX system (Linux-es) and *BSD are more or less such, a file has the following attributes[9] which may be retrieved by the stat system call:

   – Device ID (this identifies the device containing the file; that is, the scope of uniqueness of the serial number).
    File serial numbers.
    – The file mode which determines the file type and how the file's owner, its group, and others can access the file.
    – A link count telling how many hard links point to the inode.
    – The User ID of the file's owner.
    – The Group ID of the file.
    – The device ID of the file if it is a device file.
    – The size of the file in bytes.
    – Timestamps telling when the inode itself was last modified (ctime, inode change time), the file content last modified (mtime, modification time), and last accessed (atime, access time).
    – The preferred I/O block size.
    – The number of blocks allocated to this file.

Getting more extensive information on a mounted filesystem

Most Linuxes have the tune2fs installed by default (in debian Linux this is through e2fsprogs) package, with it one can get a very good indepth information on a mounted filesystem, lets say about the ( / ) root FS.

root@linux:~# tune2fs -l /dev/sda1
tune2fs 1.44.5 (15-Dec-2018)
Filesystem volume name:   <none>
Last mounted on:          /
Filesystem UUID:          abe6f5b9-42cb-48b6-ae0a-5dda350bc322
Filesystem magic number: 0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Filesystem flags:         signed_directory_hash
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              30162944
Block count:              120648960
Reserved block count:     6032448
Free blocks:              13830683
Free inodes:              26575654
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      995
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Filesystem created:       Thu Sep 6 21:44:22 2012
Last mount time:          Sat Jul 20 11:33:38 2019
Last write time:          Sat Jul 20 11:33:28 2019
Mount count:              6
Maximum mount count:      22
Last checked:             Fri May 10 18:32:27 2019
Check interval:           15552000 (6 months)
Next check after:         Wed Nov 6 17:32:27 2019
Lifetime writes:          338 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:              256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
First orphan inode:       21554129
Default directory hash:   half_md4
Directory Hash Seed:      d54c5a90-bc2d-4e22-8889-568d3fd8d54f
Journal backup:           inode blocks

Important note to make here is file's inode number stays the same when it is moved to another directory on the same device, or when the disk is defragmented which may change its physical location. This also implies that completely conforming inode behavior is impossible to implement with many non-Unix file systems, such as FAT and its descendants, which don't have a way of storing this invariance when both a file's directory entry and its data are moved around. Also one inode could point to a file and a copy of the file or even a file and a symlink could point to the same inode, below is example:

$ ls -l -i /usr/bin/perl*
266327 -rwxr-xr-x 2 root root 10376 Mar 18 2013 /usr/bin/perl
266327 -rwxr-xr-x 2 root root 10376 Mar 18 2013 /usr/bin/perl5.14.2

A good to know is inodes are always unique values, so you can't have the same inode number duplicated. If a directory is damaged, only the names of the things are lost and the inodes become the so called “orphan”, e.g. inodes without names but luckily this is recoverable. As the theory behind inodes is quite complicated and is complicated to explain here, I warmly recommend you read Ian Dallen's Unix / Linux / Filesystems – directories inodes hardlinks tutorial – which is among the best academic Tutorials explaining various specifics about inodes online.

How to Get inodes per mounted filesystem

root@linux:/home/hipo# df -i
Filesystem       Inodes IUsed   IFree IUse% Mounted on

dev             2041439     481   2040958   1% /dev
tmpfs            2046359     976   2045383   1% /run
tmpfs            2046359       4   2046355   1% /dev/shm
tmpfs            2046359       6   2046353   1% /run/lock
tmpfs            2046359      17   2046342   1% /sys/fs/cgroup
/dev/sdb5        1221600    2562   1219038   1% /usr/var/lib/mysql
/dev/sdb6        6111232 747460   5363772 13% /var/www/htdocs
/dev/sdc1      122093568 3083005 119010563   3% /mnt/backups
tmpfs            2046359      13   2046346   1% /run/user/1000

As you see in above output Inodes reported for each of mounted filesystems has a specific number. In above output IFree on every mounted FS locally on Physical installed OS Linux is good.

Here is an example on how to recognize a depleted Inodes on a OpenXen Virtual Machine with attached Virtual Hard disks.

linux:~# df -i
Filesystem         Inodes     IUsed      IFree     IUse%   Mounted on
/dev/xvda         2080768    2080768     0      100%    /
tmpfs             92187      3          92184   1%     /lib/init/rw
varrun            92187      38          92149   1%    /var/run
varlock            92187      4          92183   1%    /var/lock
udev              92187     4404        87783   5%    /dev
tmpfs             92187       1         92186   1%    /dev/shm

Finding files with a certain inode

At some cases if you want to check all the copy files of a certain file that have the same i-node pointer it is useful to find them all by their shared inode this is possible with simple find (below example is for /usr/bin/perl binary sharing same inode as perl5.28.1:

ls -i /usr/bin/perl
23798851 /usr/bin/perl*

find /usr/bin -inum 435308 -print
/usr/bin/perl5.28.1
/usr/bin/perl

Find directory that has a large number of files in it?

To get an overall number of inodes allocated by a certain directory, lets say /usr /var

root@linux:/var# du -s –inodes /usr /var
566931 /usr
56020 /var/

To get a list of directories use by inode for a directory with its main contained sub-directories sorted from 1 till highest number use:

du -s –inodes * 2>/dev/null |sort -g

Usually running out of inodes means there is a directory / fs mounts that has too many (small files) that are depleting the max count of possible inodes.

The most simple way to list directories and number of files in them on the server root directory is with a small bash shell loop like so:

for i in /*; do echo $i; find $i |wc -l; done

Another way to identify the exact directory that is most likely the bottleneck for the inode depletion in a sorted by file count, human readable form:

find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n

This will dump a list of every directory on the root (/) filesystem prefixed with the number of files (and subdirectories) in that directory. Thus the directory with the largest number of files will be at the bottom.

The -xdev switch is used to instruct find to narrow it's search to only the device where you're initiating the search (any other sub-mounted NAS / NFS filesystems from a different device will be omited).

Print top 10 subdirectories with Highest Inode Usage

Once identifed the largest number of files directories that is perhaps the issue, to further get a list of Top subdirectories in it with highest amount of inodes used, use below cmd:

for i in `ls -1A`; do echo "`find $i | sort -u | wc -l` $i"; done | sort -rn | head -10

To list more than 10 of the top inodes used dirs change the head -10 to whatever num needed.

N.B. ! Be very cautious when running above 2 find commands on a very large filesystems as it will be I/O Excessive and in filesystems that has some failing blocks this could create further problems.

To omit putting a high I/O load on a production filesystem, it is possible to also use du + very complex regular expression:

cd /backup
du –inodes -S | sort -rh | sed -n '1,50{/^.\{71\}/s/^$.\{30\}$.*$.\{37\}$$/\1…\2/;p}'

Results returned are from top to bottom.

How to Increase the amount of Inodes count on a new created volume EXT4 filesystem

Some FS-es XFS, JFS do have an auto-increase inode feature in case if their is physical space, whether otheres such as reiserfs does not have inodes at all but still have a field reported when queried for errors. But the classical Linux ext3 / ext4 does not have a way to increase the inode number on a live filesystem. Instead the way to do it there is to prepare a brand new filesystem on a Disk / NAS / attached storage.

The number of inodes at format-time of the block storage can be as high as 4 billion inodes. Before you create the new FS, you have to partition the new the block storage as ext4 with lets say parted command (or nullify the content of an with dd to clean up any previous existing data on a volume if there was already existing data:

parted /dev/sda
…

dd if=/dev/zero of=/dev/path/to/volume

then format it with this additional parameter:

mkfs.ext4 -N 3000000000 /dev/path/to/volume

Here in above example the newly created filesystem of EXT4 type will be created with 3 Billion inodes !, for setting a higher number on older ext3 filesystem max inode count mkfs.ext3 could be used instead.

Bear in mind that 3 Billion number is a too high number and if you plan to have some large number of files / directories / links structures just raise it up to your pre-planning requirements for FS. In most cases it will be rarely anyone that want to have this number higher than 1 or 2 billion of inodes.

On FreeBSD / NetBSD / OpenBSD setting inode maximum number for a UFS / UFS2 (which is current default FreeBSD FS), this could be done via newfs filesystem creation command after the disk has been labeled with disklabel:

freebsd# newfs -i 1024 /dev/ada0s1d

Increase the Max Count of Inodes for a /tmp filesystem

Sometimes on some machines it is necessery to have ability to store very high number of small files (e.g. have a very large number of inodes) on a temporary filesystem kept in memory. For example some web applications served by Web Server Apache + PHP, Nginx + Perl-FastCGI are written in a bad manner so they kept tons of temporary files in /tmp, leading to issues with exceeded amount of inodes.
If that's the case to temporary work around you can increase the count of Inodes for /tmp to a very high number like 2 billions using:

mount -o remount,nr_inodes=<bignum> /tmp

To make the change permanent on next boot if needed don't forget to put the nr_inodes=whatever_bignum as a mount option for the temporary fs to /etc/fstab

Eventually, if you face this issues it is best to immediately track which application produced the mess and ask the developer to fix his messed up programs architecture.

Conclusion

It was explained on the very common issue of having maximum amount of inodes on a filesystem depleted and the unpleasent consequences of inability to create new files on living FS.
Then a general overview was given on what is inode on a Linux / Unix filesystem, what is typical content of inode, how inode addressing is handled on a FS. Further was explained how to get basic information about available inodes on a filesystem, how to get a filename/s based on inode number (with find), the well known way to determine inode number of a directory or file (with ls) and get more extensive information on a FS on inodes with tune2fs.
Also was explained how to identify directories containing multitudes of files in order to determine a sub-directories that is consuming most of the inodes on a filesystem. Finally it was explained very raughly how to prepare an ext4 filesystem from scratch with predefined number to inodes to much higher than the usual defaults by mkfs.ext3 / mkfs.ext4 and *bsds newfs as well as how to raise the number of inodes of /tmp tmpfs temporary RAM filesystem.

Tags: access, bottleneck, called, CDN, content delivery network, data structure, Database, directory, ext3, ext4, fil, filesystem, filesystems, finding, flags, get list of inodes on sub-directory, Highest Inode Usage, increase inode number linux, indexes, information, inodes, Inodes Index, issues, Journal, Linux Unix, linux?, Listing, number, parameter, Print, root linux, statistics inode on filesystem, subdirectories, tutorial, ufs, xfs
Posted in Linux, Storage, System Administration | No Comments »

Helpful Hints For Starting A Small WordPress Website or Ecomerce Business

Wednesday, August 14th, 2019

Wordpress is the web application collection of PHP program behind thirty four percent (43%) of the internet’s websites, and fifteen percent (50%) of the top one hundred websites in the world, so if you’re considering it for your website then you’re perhaps thinking in the right direction. Small start-up projects a community website or even a small personal owned blog or mid to even large business presentation site can benefit greatly from setting up their Web Platrform or Ecommerce shops on a WordPress website platform (that of itself depends just on a small number of technologies such as a Linux server with a Web Server installed on it to serve PHP as well as some kind of Linux host installed Database backend engine such as MYSQL / PostgreSQL etc. …

But if you really want to create a successful ecommerce website on WordPress, that can seem a little intimidating at first as the general complexity to start up with WordPress looks very scary in the beginning. However in this article I’ll point to fewhelpful hints should get you off on the right foot, and make your entry into the world of Wodpress / WP Ecommerce a little easier and less scary.

This article is to be less technical than expected and in that will contrast slightly with many of the articles on this blog, the target audience is more of Web Marketing Manager or a Start-up Search Engine Optimization person at a small personal project or employed in the big bad corporate world.This is no something new that is going to be outlined in this article but a general rules that are known for the professional SEO Gurus but is most likely to be helpful for the starting persons.

If you happen to be one of these you should know you have to follow a set of well known rules on the website structure text, descriptions, text, orientation, ordering of menus and data etc. in order to have the WordPress based website running at full speed attracting more visitors to your site.

Photos

Importance of Photos on a Webiste
Although the text for your website is very important – more on that later – when a user first opens up your website in their browser, their eyes are going to be caught by the images that you have laid out on your website. Not using images is a big mistake, since it bores users’ eyes and makes your website seem amateur and basic, but using low quality images or irrelevant images can also harm your chances of appearing authentic to a user (yes here on this blog there are some of this low quality pictures but this is due to fact this website is more of information blog and not ecommerce. Thus at best case always make sure that you find the best, high-quality images for your website – make sure that you have the correct rights to use the images as well (as copyright infrignmenets) could cause you even a law suits ending in hundred or thousand dollar fines or even if this doesn't happen any publicity of such would reduce your website indexing rating. The images placed should always be relevant to your website. If you find a breath-taking sunset or tech-gadget picture, that’s great, but maybe not for your healthy food ecommerce store, but for your personal ranting or describing a personal experience.

Product Photos

Assuming that sooner or later even if you have a community website you will want to monerize it to bring back to yourself in material form at least part of the many years effort to bring the site to the web rank gained.
Leading on from that point, you’re going to be selling or advertise items – that’s the whole point of ecommerce. But users often find ads / online shopping frustrating due to not being able to properly see and understand what they’re buying before they make their purchase. This can lead to ‘buyer’s remorse’, and, consequently, refunds galore, which is not what you want. Make sure that images of your products are always available and of a high quality – investing in a fairly high quality camera might be a good idea – and consider many pictures for different angles or even rotating images so that the user can decide for themself which angle they want to look at.

Engaging Descriptions

“I can guarantee that you can’t remember the last five product descriptions you read – not even word-for-word, but the general ideas and vocabulary used will have been tossed into your short-term memory and forgotten in an instant. This is where your website can shine, and become better than ninety percent of those lingering on the internet,” Matthew Kelly, a project manager at WriteMyX and NextCoursework, suggests, “since putting effort into writing your product descriptions and making them lively and engaging will make your website memorable, and your subscribers will turn helpfully soon loyal customers will be more likely to come back time and time again and become repeat business, as well as mention you to their friends (social mounth to mouth marketing) and that way working as free advertising for you and making your website incredibly effective.”

Mobile-Friendly

Which device is most used to check email Laptop / PC or Mobile statistics as of year 2019

These days with the bloom of Mobile Devices that are currently overrunning the user of normal Desktop PCs, Laptops and Tablets and this trend is likely to stay and even increase, “If your website isn’t mobile-friendly in this day and age, then you won’t get anywhere with it.” Anne Baker, a marketer at BritStudent and Australia2Write, states. “Most people use their phones when they access websites, especially when they go shopping on the internet.

Statistics on user stay (secs / mins) stay on a website from Desktop PC and Mobile devices

On WordPress, this means finding a more recent theme – an older theme, maybe four-five years old, will probably not support mobile, and you just can’t afford to lose out on the mobile market.” In short, find yourself a mobile-friendly theme or install the right WordPress Pluguin that will enable you to have a Mobile Friendly theme in case if blog is accessed from a Mobile Dev or many of your customers will become frustrated with the badly formatted ‘mobile’ version of your website that they end up using, which might be for instance meant for a much larger screen. It can also ruin the atmosphere (experience) created at the accessed user site and have negative impact on your audience opion of your site or business. This is even more the case if your website or webapp is targetting to be modern and keeping with the times – or especially if it deals with IT and electronics (where the competition is huge)!

Registration

Registration Ecommerce website

Registration form (Sign Up) on a website and the overall business cycle idea behind web product or business is of critical importance as this is the point that will guarantee intimidation with the customer, failing to have the person be engaged will quickly make your website rank lower and your producs less wanted. The general rule here is to make your registration be an easy (to orientate for the user) and be present on a very visible place on the site.

Registration steps should be as less as possible as this might piss off the user and repel him out of the site before the registration is completed. Showing oportunity to register with a Pop-Up window (while the user clicks on a place showing interest for the produce might be useful in some cases but generally might also push the user back so if you decide to implement it do it with a lot of care (beware of too much aggressive marketing on our site).

An example

The registration process should be as intimidating as possible to leave joy in the user that might later return and log in to your site or ecommerce platform, e.g. be interested to stay for a longer time. The marketing tactic aiming to make the user stay for a longer time on the website (dragging his attention / interest to stuff) is nothing new by the way as it is well known marketing rule integrated in every supermarket you buy groceries, where all is made to keep you in the shop for as longer as possible. Research has shown that spending longer time within the supermarket makes the user buy more.

Returning customers can be intimidated with membership or a free gift (be it even virtual picture gift – free email whatever) or information store place could be given or if products are sold, registration will be obligatory to make them use their payment method or delivery address on next login to easify the buy out process. But if registration is convoluted and forced (e.g. user is somehow forced to become meber) then many customers will turn away and find another website for their shopping needs. Using a method like Quora’s ‘login to see more’ in that case might be a good idea even though for me this is also a very irritating and irritating – this method however should never be used if you run a ecommerce selling platform, on ecommerce site gatekeeping will only frustrate customers. Login is good to be implmeneted as a popup option (and not taking too much of the screen). Sign up and Login should be simplistic and self-explanatory – always not required but optioned and user should get the understanding of the advantage to be a member of the website if possible before the sign up procedure. Then, customers are more likely to sign up and won’t feel like they’ve been pushed into the decision – or pushed away, as the case may be.

Katrina Hatchett works as a lifestyle blogger at both Academic Brits and Assignment Help, due to a love of literature and writing, which she has had since youth. Throughout her career, she has become involved with many projects, such as writing for the PhD Kingdom blog.

Tags: access, Advantage, again, age, ALL, and, another, ANY, application, are, article, Database, linux?, little, personal, right, small, target audience, websites, Wordpress
Posted in Educational, Everyday Life, Marketing, Wordpress | No Comments »

☩ Walking in Light with Christ – Faith, Computing, Diary

Posts Tagged ‘access’

Linux: Howto Disable logging for all VirtualHosts on Apache and NGINX Webservers one liner

Howto Pass SSH traffic through a Secured Corporate Proxy server with corkscrew, using sshd as a standalone proxy service with no proxy installed on remote Linux server or VPS

1. How to view your corporate used (default) proxy / Check Proxy.pac file definitions

2. Few of the reasons corporations proxy all their employee's work PC web traffic

3. Run SSH Socks proxy to remote SSHd server running on common SSL 443 port

4. How to access SSH through Proxy using jumphost SSH hop

5. How to get around paranoid corporate setup where only remote access to Corporate proxy on TCP Port 80 and TCP 443 is available in Browser only

6. How to Transfer Files / Data via SSH Protocol through Proxy with SCP and SFTP

Conclusion

Windows 7 how to configure Separator delimiter Divider between Pinned and Running Opened programs on Taskbar Classic theme pane

Helpful Hints For Starting A Small WordPress Website or Ecomerce Business

Photos

Product Photos

Engaging Descriptions

Mobile-Friendly

Registration

An example

Daily Bible quote

GET ARTICLE UPDATES

Useful blog? Help it:

Links to Other Places

Recent Posts

Ads

Categories

About Myself

Recent Comments

Top Post Views

blogtopsites

Posts Tagged ‘access’

1. Create the userpameter_active_node.conf Below script is 3 nodes Haproxy cluster

3. Configure in Zabbix for active.dc key Trigger and Item

1. How to view your corporate used (default) proxy / Check Proxy.pac file definitions

2. Few of the reasons corporations proxy all their employee's work PC web traffic

3. Run SSH Socks proxy to remote SSHd server running on common SSL 443 port

4. How to access SSH through Proxy using jumphost SSH hop

5. How to get around paranoid corporate setup where only remote access to Corporate proxy on TCP Port 80 and TCP 443 is available in Browser only

6. How to Transfer Files / Data via SSH Protocol through Proxy with SCP and SFTP

Conclusion

Why run an Own Home or Dedicated Own server nowadays is a must in the Digital Age ?

– The Downsides of using Public free hosted email address such as Google / Yahoo / Yandex / Mail.com etc.

– Why using Android Phone with Gmail is terrible for your privacy / myaccount.google.com and what Google knows about you

1. Install prerequired packages for Email setup and set hostname

2. Install Postfix Mail Server Debian package

3. Setup Postfix Mail server configuration main.cf

4. Testing email sent is working on Postfix Mail Server

5. Install and configure Dovecot IMAP daemon

6. Restart dovecot and test if configuration works as expected

7. Create new local email user and test mail delivery and IMAP dovecot connectivity

8. Install and Configure RainLoop (PHP) WebMail

1. Scanning for TCP opened / filtered ports remote Linux / Windows server

2. Performing a standard HTTP request with netcat

3. Scanning a range of opened / filtered UDP ports

4. Port Scanning TCP listening ports with Netcat

5. Port scanning range of IP addresses with netcat

6. Checking whether UDP port traffic is allowed to destination server

7. Checking whether TCP port traffic allowed to DST remote server

8. Proxying traffic with netcat

Using netcat as a bind shell (make any local program / process listen and deliver via nc)

9. Copying files over network with netcat

10. Copying piped archived directory files with netcat

11. Creating a one page webserver with netcat and ncat

12. Cloning Hard disk partitions with netcat

Setting the Bash Prompt in Terminal / Console on GNU / Linux

1. How to show current hour:minute:seconds / print full date in Prompt Shell (PS)

2. How to make the prompt of a System user appear Green

3. Print New line, username@hostname, base PTY, shell level, history (number), newline and full working directory $PWD

4. Showing the numbert of jobs the shell is currently managing.

5. Set color change on command failure

6. Other beautiful PS1 Color Prompts with statistics

7. Add Muliple Colors to Same Shell prompt

8. Setting / Change Shell background Color

9. Howto Use bash shell function inside PS1 variable

10. PS2, PS3, PS4 little known variables

Summary

Install Curl data transfer tool if it is not already

Where wttr.in could be useful ?

1. Getting a three days Weather Forecast prognosis for city / town location

3. Getting Forecast results for a Tourist Destination

4. Display Forecast for a specific server IP

5. Download PNG image picture from wttr.in service

6. Displaying Current Moon Phase

7. Getting help for wttr.in terminal Waether Forecast results

8. Comparing two cities weather from command line

9. Using ansiweather command to get Weather Temperature / Wind / Humidity in one line beuatiful text

10. Get all Weather current forecast for each Capital in the world

What are filesystem i-nodes?

What is typical content of inode and how I-nodes play with rest of Filesystem units?

Listing inodes on a Fileystem

Getting more extensive information on a mounted filesystem

How to Get inodes per mounted filesystem

Finding files with a certain inode

Find directory that has a large number of files in it?

Print top 10 subdirectories with Highest Inode Usage

How to Increase the amount of Inodes count on a new created volume EXT4 filesystem

Increase the Max Count of Inodes for a /tmp filesystem

Conclusion

Photos

Product Photos

Engaging Descriptions

Mobile-Friendly

Registration

An example

Daily Bible quote

GET ARTICLE UPDATES

Useful blog? Help it:

Links to Other Places

Recent Posts

Ads

1. Create the userpameter_active_node.conf
Below script is 3 nodes Haproxy cluster