Archive for the ‘Performance Tuning’ Category

Linux extending life time for a damaged hard drive server tricks on a live server. Force fcsk on next reboot.Read-only file system error solutions

Friday, February 17th, 2023

linux-extending-life-time-for-a-damaged-hard-drive-server-tricks-can-not-read-superblock-linux-force-fsck-on-next-reboot

In our daily work as system administrators we have some very old Legacy systems running Clustered High Availability proxies using CRM (Cluster Resource Manager) and some legacy systems still using Heartbeat to manage the cluster instead of the newer and modern Corosync variant.

The HA cluster is only 2 nodes Linux machine and running the obscure already long time unsupported version of Redhat 5.11 (Ootpa) who was officially became stable distant year 1998 (yeath the years were good) and whose EOL (End of Life) has been reached long time ago and the OS is no longer supported, however for about 14 years the machines has been running perfectly fine until one of the Cluster nodes managed by ocf::heartbeat:IPAddr2 , that is  /etc/ha.d/resource.d/IPAddr2 shell script. Yeah for the newbies Heartbeat Application Cluster in Linux does work like that it uses a number of extendable pair of shell scripts written for different kind of Network / Web / Mail / SQL or whatever services HA management.

The first node configured however, started failing due to some errors like:
 

EXT3-fs error (device dm-1): ext3_journal_start_sb: Detected aborted journal
sd 0:2:0:0: rejecting I/O to offline device
Aborting journal on device sda1.
sd 0:2:0:0: rejecting I/O to offline device
printk: 159 messages suppressed.
Buffer I/O error on device sda1, logical block 526
lost page write due to I/O error on sda1
sd 0:2:0:0: rejecting I/O to offline device
sd 0:2:0:0: rejecting I/O to offline device
ext3_abort called.
EXT3-fs error (device sda1): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
sd 0:2:0:0: rejecting I/O to offline device
sd 0:2:0:0: rejecting I/O to offline device
sd 0:2:0:0: rejecting I/O to offline device
sd 0:2:0:0: rejecting I/O to offline device
sd 0:2:0:0: rejecting I/O to offline device
megaraid_sas: FW was restarted successfully, initiating next stage…
megaraid_sas: HBA recovery state machine, state 2 starting…
megasas: Waiting for FW to come to ready state
megasas: FW in FAULT state!!
FW state [-268435456] hasn't changed in 180 secs
megaraid_sas: out: controller is not in ready state
megasas: waiting_for_outstanding: after issue OCR. 
megasas: waiting_for_outstanding: before issue OCR. FW state = f0000000
megaraid_sas: pending commands remain even after reset handling. megasas[0]: Dumping Frame Phys Address of all pending cmds in FW
megasas[0]: Total OS Pending cmds : 0 megasas[0]: 64 bit SGLs were sent to FW
megasas[0]: Pending OS cmds in FW :

The result out of that was a frequently the filesystem of the machine got re-mounted as Read Only and of course that is
quite bad if you have a running processess of haproxy that should be able to be living their and take up some Web traffic
for high availability and you run all the traffic only on the 2nd pair of machine.

This of course was a clear sign for a failing disks or some hit bad blocks regions or as the messages indicates, some
problem with system hardware or Raid SAS Array.

The physical raid on the system, just like rest of the hardware is very old stuff as well.

[root@haproxy_lb_node1 ~]# lspci |grep -i RAI
01:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 05)

The produced errors not only made the machine to auto-mount its root / filesystem in Read-Only mode but besides has most
likely made the machine to automatically reboot every few days or few times every day in a raw.

The second Load Balancer node2 did operated perfectly, and we thought that we might just keep the broken machine in that half running
and inconsistent state for few weeks until we have built the new machines with Pre-Installed new haproxy cluster with modern
RedHat Linux 8.6 distribution, but since we have to follow SLAs (Service Line Agreements) with Customers and the end services behind the
High Availability (HA) Haproxy cluster were at danger … 

We as sysadmins had the task to make our best to try to stabilize the unstable node with disk errors for the system to servive
and be able to normally serve traffic (if node2 that is in a separate Data center fails due to a hardware or electricity issues etc.)
.

Here is few steps we took, that has hopefully improved the situation.

1. Make backups of most important files of high importance

Always before doing anything with a broken system, prepare backup of the most important files, if that is a cluster that should be a backup of the cluster configurations (if you don't have already ones) backup of /etc/hosts / backup of any important services configs /etc/haproxy/haproxy.cfg /etc/postfix/postfix.cfg (like it was my case), preferrably backup of whole /etc/  any important files from /root/ or /home/users* directories backup of at leasts latest logs from /var/log etc.
 

2. Clear up all unnecessery services scripts from the server

Any additional Softwares / Services and integrity checking tools (daemons) / scripts and cron jobs, were immediately stopped and wheter unused removed.

E.g. we had moved through /etc/cron* to check what's there,

# ls -ld /etc/cron.*
drwx—— 2 root root 4096 Feb  7 18:13 /etc/cron.d
drwxr-xr-x 2 root root 4096 Feb  7 17:59 /etc/cron.daily
-rw-r–r– 1 root root    0 Jul 20  2010 /etc/cron.deny
drwxr-xr-x 2 root root 4096 Jan  9  2013 /etc/cron.hourly
drwxr-xr-x 2 root root 4096 Jan  9  2013 /etc/cron.monthly
drwxr-xr-x 2 root root 4096 Aug 26  2015 /etc/cron.weekly

 

And like well professional butchers removed everything unnecessery that could trigger any extra unnecessery disk read / writes to HDD.

E.g. just create

# mkdir -p /root/etc_old/{/etc/cron.d,\
/etc/cron.daily,/etc/cron.hourly,/etc/cron.monthly\
,/etc/cron.weekly}

 

And moved all unnecessery cron job scripts like:

1. nmon (old school network / memory / hard disk console tool for monitoring and tuning server parameters)
2. clamscan / freshclam crons
3. mlocate (the script that is taking care for periodic run of updatedb command to keep the locate command to easily search
for files inside the DB to put less read operations on disk in case if you need to find file (e.g. prevent yourself to everytime
run cmd like: find / . -iname '*whatever_you_look_for*'
4. cups cron jobs
5. logwatch cron
6. rkhunter stuff
7. logrotate (yes we stopped even logrotation trigger job as we found the server was crashing sometimes at the same time when
the lograte job to rotate logs inside /var/log/* was running perhaps leading to a hit of the I/O read error (bad blocks).


Also inspected the Administrator user root cron job for any unwated scripts and stopped two report bash scripts that were part of the PCI tightened Security procedures.
Therein found script responsible to periodically report the list of installed packages and if they have not changed, as well a script to periodically report via email the list of
/etc/{passwd,/etc/shadow} created users, used to historically keep an eye on the list of users and easily see if someone
has created new users on the machine. Those were enabled via /var/spool/cron/root cron jobs, in other cases, on other machines if it happens for you
it is a good idea to check out all the existing user cron jobs and stop anything that might be putting Read / Write extra heat pressure on machine attached the Hard drives.

# ls -al /var/spool/cron/
total 20
drwx——  2 root root 4096 Nov 13  2015 .
drwxr-xr-x 12 root root 4096 May 11  2011 ..
-rw——-  1 root root  133 Nov 13  2015 root


3. Clear up old log files and any files unnecessery

Under /var/log and /home /var/tmp /var/spool/tmp immediately try to clear up the old log files.
From my past experience this has many times made the FS file inodes that are storing on a unbroken part (good blocks) of the hard drive and
ready to be reused by newly written rsyslog / syslogd services spitted files.

!!! Note that during the removal of some files you might hit a files stored on a bad blocks that might lead to a unexpected system reboot.

But that's okay, don't worry most likely after a hard reset by a technician in the Datacenter the machine will boot again and you can enjoy
removing remaining still files to send them to the heaven for old files.

 

4. Trigger an automatic system file system check with fsck on next boot

The standard way to force a Linux to aumatically recheck its Root filesystem is to simply create the /forcefsck to root partition or any other secondary disk partition you would like to check.

# touch /forcefsck

# reboot


However at some occasions you might be unable to do it because, the / (root fs) has been remounted in ReadOnly mode, yackes …

Luckily old Linux distibutions like this RHEL 5.1, has a way to force a filesystem check after reboot fsck and identify any
unknown bad-blocks and hopefully succceed in isolating them, so you don't hit into the same auto-reboots if the hard drive or Software / Hardware RAID
is not in terrible state
, you can use an option built in in /sbin/shutdown command the '-F'

   -F     Force fsck on reboot.


Hence to make the machine reboot and trigger immediately fsck:

# shutdown -rF now


Just In case you wonder why to reboot before check the Filesystem. Well simply because you need to have them unmounted before you check.

In that specific case this produced so far a good result and the machine booted just fine and we crossed the fingers and prayed that the machine would work flawlessly in the coming few weeks, before we finalize the configuration of the substitute machines, where this old infrastructure will be migrated to a new built cluster with new Haproxy and Corosync / Pacemaker Cluster on a brand new RHEL.

NB! On newer machines this won't work however as shutdown command has been stripped off this option because no SystemV (SystemInit) or Upstart and not on SystemD newer services architecture.
 

5. Hints on checking the hard drives with fsck

If you happen to be able to have physical access to the remote Hardare machine via a TTY[1-9] Console, that's even better and is the standard way to do it but with this specific case we had no easy way to get access to the Physical server console.

It is even better to go there and via either via connected Monitor (Display) or KVM Switch (Those who hear KVM switch first time this is a great device in server rooms to connect multiple monitors to same Monitor Display), it is better to use a some of the multitude of options to choose from for USB Distro Linux recovery OS versions or a CDROM / DVD on older machines like this with the Redhat's recovery mode rolled on.
After mounting the partition simply check each of the disks
e.g. :

# fsck -y /dev/sdb
# fsck -y /dev/sdc

Or if you want to not waste time and look for each hard drive but directly check all the ones that are attached and known by Linux distro via /etc/fstab definition run:

# fsck -AR

If necessery and you have a mixture of filesystems for example EXT3 , EXT4 , REISERFS you can tell it to omit some filesystem, for example ext3, like that:

# fsck -AR -t noext3 -y


To skip fsck on mounted partitions with fsck:

# fsck -M /dev/sdb


One remark to make here on fsck is usually fsck to complete its job on various filesystem it uses other external component binaries usually stored in /sbin/fsck*

ls -al /sbin/fsck*
-rwxr-xr-x 1 root root  55576 20 яну 2022 /sbin/fsck*
-rwxr-xr-x 1 root root  43272 20 яну 2022 /sbin/fsck.cramfs*
lrwxrwxrwx 1 root root      9  4 юли 2020 /sbin/fsck.exfat -> exfatfsck*
lrwxrwxrwx 1 root root      6  7 юни 2021 /sbin/fsck.ext2 -> e2fsck*
lrwxrwxrwx 1 root root      6  7 юни 2021 /sbin/fsck.ext3 -> e2fsck*
lrwxrwxrwx 1 root root      6  7 юни 2021 /sbin/fsck.ext4 -> e2fsck*
-rwxr-xr-x 1 root root  84208  8 фев 2021 /sbin/fsck.fat*
-rwxr-xr-x 2 root root 393040 30 ное 2009 /sbin/fsck.jfs*
-rwxr-xr-x 1 root root 125184 20 яну 2022 /sbin/fsck.minix*
lrwxrwxrwx 1 root root      8  8 фев 2021 /sbin/fsck.msdos -> fsck.fat*
-rwxr-xr-x 1 root root    333 16 дек 2021 /sbin/fsck.nfs*
lrwxrwxrwx 1 root root      8  8 фев 2021 /sbin/fsck.vfat -> fsck.fat*


6. Using tune2fs to  adjust tunable filesystem parameters on ext2/ext3/ext4 filesystems (few examples)

a) To check whether really the filesystem was checked on boot time or check a random filesystem on the server for its last check up date with fsck:

#  tune2fs -l /dev/sda1 | grep checked
Last checked:             Wed Apr 17 11:04:44 2019

On some distributions like old Debian and Ubuntu, it is even possible to enable fsck to log its operations during check on reboot via changing the verbosity from NO to YES:

# sed -i "s/#VERBOSE=no/VERBOSE=yes/" /etc/default/rcS


If you're having the issues on old Debian Linuxes  and not on RHEL  it is possible to;

b) Enable all fsck repairs automatic on boot

by running via:
 

# sed -i "s/FSCKFIX=no/FSCKFIX=yes/" /etc/default/rcS


c) Forcing fcsk check on for server attached Hard Drive Partitions with tune2fs

# tune2fs -c 1 /dev/sdXY

Note that:
tune2fs can force a fsck on each reboot for EXT4, EXT3 and EXT2 filesystems only.

tune2fs can trigger a forced fsck on every reboot using the -c (max-mount-counts) option.
This option sets the number of mounts after which the filesystem will be checked, so setting it to 1 will run fsck each time the computer boots.
Setting it to -1 or 0 resets this (the number of times the filesystem is mounted will be disregarded by e2fsck and the kernel).


 For example you could:

d) Set fsck to run a filesystem check every 30 boots, by using -c 30 
 

# tune2fs -c 30 /dev/sdXY


e) Checking whether a Hard Drive has been really checked on the boot

 

#  tune2fs -l /dev/sda1 | grep checked
Last checked:             Wed Apr 17 11:04:44 2019


e) Check when was the last time the file system /dev/sdX was checked:
 

# tune2fs -l /dev/sdX | grep Last\ c
Last checked:             Thu Jan 12 20:28:34 2017


f) Check how many times our /dev/sdX filesystem was mounted

# tune2fs -l /dev/sdX | grep Mount
Mount count:              157

g) Check how many mounts are allowed to pass before filesystem check is forced
 

# tune2fs -l /dev/sdX | grep Max
Maximum mount count:      -1


7. Repairing disk / partitions via GRUB fsck.mode and fsck.repair kernel module options

It is also possible to force a fsck.repair on boot via GRUB, but that usually is not an option someone would like as the machine might fail too boot if it hards to repair hardly, however in difficult situations with failing disks temporary enabling it is good idea.

This can be done by including for grub initial config

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash fsck.mode=force fsck.repair=yes"

fsck.mode=force – will force a fsck each time a system boot and keeping that value enabled for a long time inside GRUB is stupid for servers as

sometimes booting could be severely prolonged because of the checks especially with servers with many or slow old hard drives.

fsck.repair=yes – will make the fsck try to repair if it finds bad blocks when checking (be absolutely sure you know, what you're doing if passing this options)

The options can be also set via editing the GRUB boot screen, if you have physical access to the server and don't want to reload the grub loader and possibly make the machine unbootable on next boot.
 

8. Few more details on how /etc/fstab disk fsck check parameters values for Systemd Linux machines works

The "proper" way on systemd (if we can talk about proper way on Linux) to runs fsck for each filesystem that has a fsck is to pass number greater than 0 set in
/etc/fstab (last column in /etc/fstab), so make sure you edit your /etc/fstab if that's not the case.

The root partition should be set to 1 (first to be checked), while other partitions you want to be checked should be set to 2.

Example /etc/fstab:
 

# /etc/fstab: static file system information.

/dev/sda1  /      ext4  errors=remount-ro  0  1
/dev/sda5  /home  ext4  defaults           0  2

The values you can put here as a second number meaning is as follows:
0 – disabled, that is do not check filesystem
1 – partition with this PASS value has a higher priority and is checked first. This value is usually set to the root / partition
2 – partitions with this PASS value will be checked last

a) Check the produced log out of fsck

Unfortunately on the older versions of Linux distros with SystemV fsck log output might be not generated except on the physical console so if you have a kind of duplicator device physical tty on the display port of the server, you might capture some bad block reports or fixed errors messages, but if you don't you might just cross the fingers and hope that anything found FS irregularities was recovered.

On systemd Linux machines the fsck log should be produced either in /run/initramfs/fsck.log or some other location depending on the Linux distro and you should be able to see something from fsck inside /var/log/* logs:

# grep -rli fsck /var/log/*


Close it up

Having a system with failing disk is a really one of the worst sysadmin nightmares to get. The good news is that most of the cases we're prepared with some working backup or some work around stuff like the few steps explained to mitigate the amount of Read / Writes to hard disks on the failing machine HDDs. If the failing disk is a primary Linux filesystem all becomes even worse as every next reboot, you have no guarantee, whether the kernel / initrd or some of the other system components required to run the Core Linux system won't break up the normal boot. Thus one side changes on the hard drives is a risky business on ther other side, if you're in a situation where you have a mirror system or the failing system is just a Linux server installed without a Cluster pair, then this is not a big deal as you can guarantee at least one of the nodes still up, unning and serving. Still doing too much of operations with HDD is always a danger so the steps described, though in most cases leading to improvement on how the system behaves, the system should be considered totally unreliable and closely monitored not only by some monitoring stuff like Zabbix / Prometheus whatever but regularly check the systems state via normal SSH logins. It is important if you have some important datas or logs on the system that are not synchronized to a system node to copy them before doing any of the described operations. After all minimal is backuped, proceed to clear up everything that might be cleared up and still the machine to continue providing most of its functionalities, trigger fsck automatic HDD check on next reboot, reboot, check what is going on and monitor the machine from there on.

Hopefully the few described steps, has helped some sysadmin. There is plenty of things which I've described that might go wrong, even following the described steps, might not help if the machines Storage Drives / SAS / SSD has too much of a damage. But as said in most cases following this few steps would improve the machine state.

Wish you the best of luck!

 

Remove pre-installed HP, Dell, Asus, Acer, Toshiba not needed default vendor software on a new bought PC notebook quickly with Decrap My Computer

Thursday, April 14th, 2016

remove-default-unneded-software-from-manufacturer-hp-asus-acer-dell-toshiba
While browsing today and looking for software to clean up all the spy software from my corporate HP laptop, I've come across an interesting tool called Decrap which aims at removing Bloatware from a Windows PC / notebook (mirrored here because original software site was down)

So what is Bloatware ?

The term Bloatware (also called jokingly crapware) is term is the one that was coined to describe, the default pre-installed software that comes to you together with Windows pre-installed OS by the hardware manufacturer.
I'm sure anyone who bought brand new branded PC or laptop over the last 10 years have already suffered the unwanted and unnecessery
bunch of software that comes pre-installed freeware programs aiming to help you in your daily work but in reality just slowing down your PC
and showing annoying popups or at best keeping useless in Windows system apptray.

Let me give you an example:

Cleanup (Remove) common ASUS, HP, Dell default  installed unneded (Bloatware) software

Many users may want to uninstall ASUS Crapware software such as ASUS Tutor, ASUS LifeFrame3, ASUS WebStorage and ASUSVibe.

Like Asus, HP computers often come preloaded with useless software from factory, example for this is HP Customer Service enhancements, HP Update, HP Total Care Setup and ProtectSmart.

If you buy Dell PC notebook soon you'll discover that there are several preloaded software (often unnecessery software) such as Dell Stage, Dell Digital Delivery and Dell DataSafe.

Toshiba computers and notebooks contain pre-loeaded "crapware" software from Toshiba.
Just to mention a few of those: Toshiba Disc Creator, Toshiba ReelTime, Service Station, Bulletin Board and Toshiba Assist.

Often there are fingerprint reader programs, Wi-Fi connection managers, Bluetooth managers, Audio Management sofware and other third party vendor software which tend to be not working as good as others softwares from third vendor, so Decrap is to help you to identify and remove these too in a easy GUI manner.

Non-experienced Computer users often leave the bloatware to hang around for even years and only if some relative that is an IT involved person / sysadmin / Even once you're aware that the Bloatware is on the system the ordinary user is hard to remove it as he is scared not to break the system.
Besides that fften this bloatware just soft comes so much integrated into Windows that removing it costs hours of tries and research online on
all the Bloatware components and even then could mislead you so you break the PC. programmer etc. comes home of such users finds out about the happily existing of the useless software on the notebook.


So here is Decrap My Computer coming at place aiming to help to remove the unnecessery Hardware vendor software in few easy (Click, Click, Click Next ..) steps.

decrap-my-computer-clean-up-bloatware-crapware-on-windows-laptop-main-gui

Then decrap does silently all the complex operations and suggestions to make the HP, Dell, Asus, Toshiba manufacturer prebundled software to be stopped and uninstalled.
Decrap My Computer is a freeware, lightweight and easy to use and lets you safely remove crapware and bloatware, or any software, from any Windows PC.

Even for Old computers, Decrap comes handy for the unexperienced avarage user who used his laptop with this useless  default vendors programs silently killing the performance respectively user experience for years.

Using Decrap is quite intuitive PC is scanned for Bloatware and then after a backup Windows Restore Point is offered you're offered to review and Uninstall the unwanted softwares. There is also an automatic mode but those one still could be a bit dangerous, so use the automatic mode only on multiple machines with the same model  / brand notebooks that comes prebundled with same sofware after testing and confirming the automatic mode on 1 initial machine will not break up some needed functionality.

decrap-your-pc-clean-up-windows-from-hp-dell-toshiba-asus-bloatware-unuseful-programs

Here is few screenshots of the tool in action:
decrap-choose-what-default-laptop-manufacturer-software-you-want-to-clean-from-new-bought-pc

decrap-choose-what-default-laptop-manufacturer-software-you-want-to-clean-from-new-bought-pc-1

decrap-choose-what-default-laptop-manufacturer-software-you-want-to-clean-from-new-bought

Another good alternative (since decrap seems to be not maintained anymore) as I just leardned from Natasha Myles (thanks for pointing me about the broken link to decrap website) is SpeedUpPC more on speeding up old PC or laptop is her article 

Graphic tool to get Hardware Information on Linux / How to view Hardware Information easy in Linux

Tuesday, October 3rd, 2017

 

 

howto-get-graphically-system-hardware-info-linux-gui-program-for-hardware-recognition-linu

 

IS THERE A GRAPHIC ( GUI ) TOOL TO VIEW HARDWARE INFORMATION ON LINUX?

 


If you are a console maniac like myself, perhaps you never think that you might need anything graphical besides to view hardware information on Linux, but as we're growing older sometimes it becomes much less easier to just use a graphical tool that can show us all the information we need regarding a Notebook / Desktop PC with Linux or even Server machine with enabled Graphical Environment with a brand new installed GNU / Linux whatever version (I hope you don't own server with running Xorg / Gnome / Mate / Xfce etc. as that's pretty much a waste of hardware resource and opens a dozen of other security risks for the server running services ).

 

 

 There are at least 2 ways to quickly check hardware on both PC WorkStation or Server, the easiest and quickest for PC / Notebook Linux users if you have installed GTK libraries or Gnome Desktop Environment is with;
 

LSHW-GTK


LSHW-GTK is simply a GTK frontend over the command line tool for hardware information gathering LSHW
 

HardiInfo

 

HardInfo – is a small application that displays information about your hardware and operating system. Currently it knows about PCI, ISA PnP, USB, IDE, SCSI, Serial and parallel port devices.


1. Howto Install LSHW-GTK / HardInfo on Debian / Ubuntu / Mint GNU / Linux to easy view hardware information


To install both of them on Debian / Ubuntu GNU / Linux, run:
 

apt-get install –yes lshw lshw-gtk hardinfo

 

2. Howto install LSHW-GTK on Fedora, CentOS and OpenSuSE Linux to view easy hardware information

On RedHat RPM based Linux distributions, the package to install is called lshw-gui

Install with yum RPM package manager:
 

yum install –yes lshw lshw-gui  hardinfo


3. Run lshw-gtk / hardinfo

Again, find them and run from GUI environment menus or run manually like in below example:

$ lshw-gtk


graphic-program-to-view-computer-hardware-on-linux-lshw-gtk-on-debian-linux-screenshot-view-hardware-easy-linux1

 

$ hardinfo


hardinfo-a-gui-program-to-view-computer-hardware-info-on-linux-and-freebsd

As you see hardinfo is really interactive and it gives you pretty much all the information, you might need, the only information that was missing at my case and I guess, that would happen to others is information about the SSD Hard Disk, which   180GB

HardInfo is really amazing program as it even includes various common Benchmark Tests and comparison with other Computers:

hardinfo-get-hardware-information-easily-on-linux-and-freebsd-benchmark-info-screenshot-debian-stretch

True that the tests, are pretty simple but still could be useful.

Now run it either from GNOME / Cinnamon (The default graphical environment of Debian Linux) or PLASMA (The new name for the second most popular Linux Graphical Environment – KDE desktop environment)

 

$ lshw


Here is few more screenshots from hardware info reported from my ThinkPad T410 Laptop Running Debian 9 Stretch at the moment.

 

MotherBoard -> BIOS Information

(thatnks God this old but gold Thinkpad T420 business notebook does not run UEFI substitute for BIOS 🙂

graphic-program-to-view-computer-hardware-on-linux-lshw-gtk-on-debian-linux-screenshot-view-hardware-easy-linux2

CPU Information (with all the supported CPU capabilities (extensions)

graphic-program-to-view-computer-hardware-on-linux-lshw-gtk-on-debian-linux-screenshot-view-hardware-easy-linux3

Host Bridge Info

graphic-program-to-view-computer-hardware-on-linux-lshw-gtk-on-debian-linux-screenshot-view-hardware-easy-linux4

Thinkpad BATTERY (45N1005) Info

graphic-program-to-view-computer-hardware-on-linux-lshw-gtk-on-debian-linux-screenshot-view-hardware-easy-linux5

By the way another Way to GUI View your Computer is to just generate HTML from lshw command line tool (as it supports export to HTML), here is how:

 

$ lshw -html > ~/hardware-specs.html


Then just open it with Browser, for example I like GNOME Epiphany browser, so I'll read HTML with it:

 

$ epiphany ~/hardware-specs.html


graphical-software-to-view-hardware-on-linux-lshw-command-to-generate-html-and-view-it-graphically-in-browser-on-home-pc-or-server


The great thing about generating HTML report for hardware is that on Staging / Production / Development servers which you inherited from some other administrator who for some reason (laziness 🙂 ) didn't left necessery documentation, you can easily map the machine hardware and even if it is a group of machines, you can automate report generation for all of them write a short script that parses the data on each of the HTML reports and finally creates a merged document with main important information about hardware of a cluster of computers etc.

If you still want to stick to console run the console version of lshw or use dmidecode or lshw:

 

$ lshw

hipo@jericho:~$ lshw
WARNING: you should run this program as super-user.
jericho                     
    description: Computer
    width: 64 bits
    capabilities: smp vsyscall32
  *-core
       description: Motherboard
       physical id: 0
     *-memory
          description: System memory
          physical id: 0
          size: 7870MiB
     *-cpu
          product: Intel(R) Core(TM) i7-2640M CPU @ 2.80GHz
          vendor: Intel Corp.
          physical id: 1
          bus info: cpu@0
          size: 891MHz
          capacity: 3500MHz
          width: 64 bits
          capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp x86-64 constant_tsc arch_perfmon pebs bts nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm epb tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts cpufreq
     *-pci
          description: Host bridge
          product: 2nd Generation Core Processor Family DRAM Controller
          vendor: Intel Corporation
          physical id: 100
          bus info: pci@0000:00:00.0
          version: 09
          width: 32 bits
          clock: 33MHz
        *-pci:0
             description: PCI bridge
             product: Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port
             vendor: Intel Corporation
             physical id: 1
             bus info: pci@0000:00:01.0
             version: 09
             width: 32 bits
             clock: 33MHz
             capabilities: pci normal_decode bus_master cap_list
             configuration: driver=pcieport
             resources: irq:24 ioport:5000(size=4096) memory:f0000000-f10fffff ioport:c0000000(size=301989888)
           *-generic UNCLAIMED
                description: Unassigned class
                product: Illegal Vendor ID
                vendor: Illegal Vendor ID
                physical id: 0
                bus info: pci@0000:01:00.0
                version: ff
                width: 32 bits
                clock: 66MHz
                capabilities: bus_master vga_palette cap_list
                configuration: latency=255 maxlatency=255 mingnt=255
                resources: memory:f0000000-f0ffffff memory:c0000000-cfffffff memory:d0000000-d1ffffff ioport:5000(size=128) memory:f1000000-f107ffff
        *-display
             description: VGA compatible controller
             product: 2nd Generation Core Processor Family Integrated Graphics Controller
             vendor: Intel Corporation
             physical id: 2
             bus info: pci@0000:00:02.0
             version: 09
             width: 64 bits
             clock: 33MHz
             capabilities: vga_controller bus_master cap_list rom
             configuration: driver=i915 latency=0
             resources: irq:30 memory:f1400000-f17fffff memory:e0000000-efffffff ioport:6000(size=64) memory:c0000-dffff
        *-communication:0
             description: Communication controller
             product: 6 Series/C200 Series Chipset Family MEI Controller #1
             vendor: Intel Corporation
             physical id: 16
             bus info: pci@0000:00:16.0
             version: 04
             width: 64 bits
             clock: 33MHz
             capabilities: bus_master cap_list
             configuration: driver=mei_me latency=0
             resources: irq:27 memory:f3925000-f392500f
        *-communication:1
             description: Serial controller
             product: 6 Series/C200 Series Chipset Family KT Controller
             vendor: Intel Corporation
             physical id: 16.3
             bus info: pci@0000:00:16.3
             version: 04
             width: 32 bits
             clock: 66MHz
             capabilities: 16550 bus_master cap_list
             configuration: driver=serial latency=0
             resources: irq:19 ioport:60b0(size=8) memory:f392c000-f392cfff
        *-network
             description: Ethernet interface
             product: 82579LM Gigabit Network Connection
             vendor: Intel Corporation
             physical id: 19
             bus info: pci@0000:00:19.0
             logical name: enp0s25
             version: 04
             serial: 00:21:cc:cc:b2:27
             capacity: 1Gbit/s
             width: 32 bits
             clock: 33MHz
             capabilities: bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
             configuration: autonegotiation=on broadcast=yes driver=e1000e driverversion=3.2.6-k firmware=0.13-3 latency=0 link=no multicast=yes port=twisted pair
             resources: irq:25 memory:f3900000-f391ffff memory:f392b000-f392bfff ioport:6080(size=32)
        *-usb:0
             description: USB controller
             product: 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2
             vendor: Intel Corporation
             physical id: 1a
             bus info: pci@0000:00:1a.0
             version: 04
             width: 32 bits
             clock: 33MHz
             capabilities: ehci bus_master cap_list
             configuration: driver=ehci-pci latency=0
             resources: irq:16 memory:f392a000-f392a3ff
        *-multimedia
             description: Audio device
             product: 6 Series/C200 Series Chipset Family High Definition Audio Controller
             vendor: Intel Corporation
             physical id: 1b
             bus info: pci@0000:00:1b.0
             version: 04
             width: 64 bits
             clock: 33MHz
             capabilities: bus_master cap_list
             configuration: driver=snd_hda_intel latency=0
             resources: irq:29 memory:f3920000-f3923fff
        *-pci:1
             description: PCI bridge
             product: 6 Series/C200 Series Chipset Family PCI Express Root Port 1
             vendor: Intel Corporation
             physical id: 1c
             bus info: pci@0000:00:1c.0
             version: b4
             width: 32 bits
             clock: 33MHz
             capabilities: pci normal_decode cap_list
             configuration: driver=pcieport
             resources: irq:16
        *-pci:2
             description: PCI bridge
             product: 6 Series/C200 Series Chipset Family PCI Express Root Port 2
             vendor: Intel Corporation
             physical id: 1c.1
             bus info: pci@0000:00:1c.1
             version: b4
             width: 32 bits
             clock: 33MHz
             capabilities: pci normal_decode bus_master cap_list
             configuration: driver=pcieport
             resources: irq:17 memory:f3800000-f38fffff
           *-network
                description: Wireless interface
                product: Centrino Advanced-N 6205 [Taylor Peak]
                vendor: Intel Corporation
                physical id: 0
                bus info: pci@0000:03:00.0
                logical name: wlp3s0
                version: 34
                serial: 26:ad:26:50:f1:db
                width: 64 bits
                clock: 33MHz
                capabilities: bus_master cap_list ethernet physical wireless
                configuration: broadcast=yes driver=iwlwifi driverversion=4.9.0-3-amd64 firmware=18.168.6.1 ip=192.168.0.102 latency=0 link=yes multicast=yes wireless=IEEE 802.11
                resources: irq:28 memory:f3800000-f3801fff
        *-pci:3
             description: PCI bridge
             product: 6 Series/C200 Series Chipset Family PCI Express Root Port 4
             vendor: Intel Corporation
             physical id: 1c.3
             bus info: pci@0000:00:1c.3
             version: b4
             width: 32 bits
             clock: 33MHz
             capabilities: pci normal_decode bus_master cap_list
             configuration: driver=pcieport
             resources: irq:19 ioport:4000(size=4096) memory:f3000000-f37fffff ioport:f1800000(size=8388608)
        *-pci:4
             description: PCI bridge
             product: 6 Series/C200 Series Chipset Family PCI Express Root Port 5
             vendor: Intel Corporation
             physical id: 1c.4
             bus info: pci@0000:00:1c.4
             version: b4
             width: 32 bits
             clock: 33MHz
             capabilities: pci normal_decode bus_master cap_list
             configuration: driver=pcieport
             resources: irq:16 ioport:3000(size=4096) memory:f2800000-f2ffffff ioport:f2000000(size=8388608)
           *-generic
                description: System peripheral
                product: MMC/SD Host Controller
                vendor: Ricoh Co Ltd
                physical id: 0
                bus info: pci@0000:0d:00.0
                version: 08
                width: 32 bits
                clock: 33MHz
                capabilities: bus_master cap_list
                configuration: driver=sdhci-pci latency=0
                resources: irq:16 memory:f2800000-f28000ff
        *-usb:1
             description: USB controller
             product: 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1
             vendor: Intel Corporation
             physical id: 1d
             bus info: pci@0000:00:1d.0
             version: 04
             width: 32 bits
             clock: 33MHz
             capabilities: ehci bus_master cap_list
             configuration: driver=ehci-pci latency=0
             resources: irq:23 memory:f3929000-f39293ff
        *-isa
             description: ISA bridge
             product: QM67 Express Chipset Family LPC Controller
             vendor: Intel Corporation
             physical id: 1f
             bus info: pci@0000:00:1f.0
             version: 04
             width: 32 bits
             clock: 33MHz
             capabilities: isa bus_master cap_list
             configuration: driver=lpc_ich latency=0
             resources: irq:0
        *-storage
             description: SATA controller
             product: 6 Series/C200 Series Chipset Family 6 port SATA AHCI Controller
             vendor: Intel Corporation
             physical id: 1f.2
             bus info: pci@0000:00:1f.2
             version: 04
             width: 32 bits
             clock: 66MHz
             capabilities: storage ahci_1.0 bus_master cap_list
             configuration: driver=ahci latency=0
             resources: irq:26 ioport:60a8(size=8) ioport:60bc(size=4) ioport:60a0(size=8) ioport:60b8(size=4) ioport:6060(size=32) memory:f3928000-f39287ff
        *-serial
             description: SMBus
             product: 6 Series/C200 Series Chipset Family SMBus Controller
             vendor: Intel Corporation
             physical id: 1f.3
             bus info: pci@0000:00:1f.3
             version: 04
             width: 64 bits
             clock: 33MHz
             configuration: driver=i801_smbus latency=0
             resources: irq:18 memory:f3924000-f39240ff ioport:efa0(size=32)
     *-scsi
          physical id: 2
          logical name: scsi1
          capabilities: emulated
        *-cdrom
             description: DVD-RAM writer
             product: DVDRAM GT50N
             vendor: HL-DT-ST
             physical id: 0.0.0
             bus info: scsi@1:0.0.0
             logical name: /dev/cdrom
             logical name: /dev/cdrw
             logical name: /dev/dvd
             logical name: /dev/dvdrw
             logical name: /dev/sr0
             version: LT20
             capabilities: removable audio cd-r cd-rw dvd dvd-r dvd-ram
             configuration: ansiversion=5 status=nodisc
WARNING: output may be incomplete or inaccurate, you should run this program as super-user.

 

Enjoy Life ! 🙂

Optimizing Linux TCP/IP Networking to increase Linux Servers Performance

Tuesday, April 8th, 2008

optimize-linux-servers-for-network-performance-to-increase-speed-and-decrease-hardware-costs-_tyan-exhibits-hpc-optimized-server-platforms-featuring-intel-xeon-processor-e7-4800-v3-e5-2600-supercomputing-15_full

Some time ago I thought of ways to optimize my Linux Servers network performance.

Even though there are plenty of nice articles on the topic on how to better optimize Linux server performance by tunning up the kernel sysctl (variables).

Many of the articles I found was not structed in enough understandable way so I decided togoogle around and  found few interesting websites which gives a good overview on how one can speed up a bit and decrease overall server loads by simply tuning few basic kernel sysctl variables.

Below article is a product of my research on the topic on how to increase my GNU / Linux servers performance which are mostly running LAMP (Linux / Apache / MySQL / PHP) together with Qmail mail servers.

The article is focusing on Networking as networking is usual bottleneck for performance.
Below are the variables I found useful for optimizing the Linux kernel Network stack.

Implementing the variables might reduce your server load or if not decrease server load times and CPU utilization, they would at lease increase thoroughput so more users will be able to access your servers with (hopefully) less interruptions.
That of course would save you some Hardware costs and raise up your Servers efficiency.

Here are the variables themselves and some good example:
 

# values.net.ipv4.ip_forward = 0 ( Turn off IP Forwarding )

net.ipv4.conf.default.rp_filter = 1

# ( Control Source route verification )
net.ipv4.conf.default.accept_redirects = 0

# ( Disable ICMP redirects )
net.ipv4.conf.all.accept_redirects = 0 ( same as above )
net.ipv4.conf.default.accept_source_route = 0

# ( Disable IP source routing )
net.ipv4.conf.all.accept_source_route = 0
( - || - )net.ipv4.tcp_fin_timeout = 40

# ( Decrease FIN timeout ) - Useful on busy/high load
serversnet.ipv4.tcp_keepalive_time = 4000 ( keepalive tcp timeout )
net.core.rmem_default = 786426 - Receive memory stack size ( a good idea to increase it if your server receives big files )
net.ipv4.tcp_rmem = "4096 87380 4194304"
net.core.wmem_default = 8388608 ( Reserved Memory per connection )
net.core.wmem_max = 8388608
net.core.optmem_max = 40960
( maximum amount of option memory buffers )

# like a homework investigate by yourself what the variables below stand for :)
net.ipv4.tcp_max_tw_buckets = 360000
net.ipv4.tcp_reordering = 5
net.core.hot_list_length = 256
net.core.netdev_max_backlog = 1024

 

# Below are newly added experimental
#net.core.rmem_max = 16777216
#net.core.wmem_max = 16777216
##kernel.msgmni = 1024
##kernel.sem = 250 256000 32 1024
##vm.swappiness=0
kernel.sched_migration_cost=5000000

 

Also a good sysctl.conf file which one might want to substitite or use as a skele for some productive server is ready for download here


Even if you can't reap out great CPU reduction benefits from integrating above values or similar ones, your overall LAMP performance to end customers should increase – at some occasions dramatically, at others little bit but still noticable.

If you're unsure on exact kernel variable values to use check yourself what should be the best values that fits you according to your server Hardware – usually this is done by experimenting and reading the kernel documentation as provided for each one of uplisted variables.

Above sysctl.conf is natively created to run on Debian and on other distributions like CentOS, Fedora Slackware some values might either require slight modifications.

Hope this helps and gives you some idea of how network optimization in Linux is usually done. Happy (hacking) tweakening !

Finding top access IPs in Webserver or how to delay connects from Bots (Web Spiders) to your site to prevent connect Denial of Service

Friday, September 15th, 2017

analyze-log-files-most-visited-ips-and-find-and-stopwebsite-hammering-bot-spiders-neo-tux

If you're a sysadmin who has to deal with cracker attemps for DoS (Denial of Service) on single or multiple servers (clustered CDN or standalone) Apache Webservers, nomatter whether working for some web hosting company or just running your private run home brew web server its very useful thing to inspect Web Server log file (in Apache HTTPD case that's access.log).

Sometimes Web Server overloads and the follow up Danial of Service (DoS) affect is not caused by evil crackers (mistkenly often called hackers but by some data indexing Crawler Search Engine bots who are badly configured to aggressively crawl websites and hence causing high webserver loads flooding your servers with bad 404 or 400, 500 or other requests, just to give you an example of such obstructive bots.

1. Dealing with bad Search Indexer Bots (Spiders) with robots.txt

Just as I mentioned hackers word above I feel obliged to expose the badful lies the press and media spreading for years misconcepting in people's mind the word cracker (computer intruder) with a hacker, if you're one of those who mistakenly call security intruders hackers I recommend you read Dr. Richard Stallman's article On Hacking to get the proper understanding that hacker is an cheerful attitude of mind and spirit and a hacker could be anyone who has this kind of curious and playful mind out there. Very often hackers are computer professional, though many times they're skillful programmers, a hacker is tending to do things in a very undstandard and weird ways to make fun out of life but definitelely follow the rule of do no harm to the neighbor.

Well after the short lirical distraction above, let me continue;

Here is a short list of Search Index Crawler bots with very aggressive behaviour towards websites:

 

# mass download bots / mirroring utilities
1. webzip
2. webmirror
3. webcopy
4. netants
5. getright
6. wget
7. webcapture
8. libwww-perl
9. megaindex.ru
10. megaindex.com
11. Teleport / TeleportPro
12. Zeus
….

Note that some of the listed crawler bots are actually a mirroring clients tools (wget) etc., they're also included in the list of server hammering bots because often  websites are attempted to be mirrored by people who want to mirror content for the sake of good but perhaps these days more often mirror (duplicate) your content for the sake of stealing, this is called in Web language Content Stealing in SEO language.


I've found a very comprehensive list of Bad Bots to block on Mike's tech blog his website provided example of bad robots.txt file is mirrored as plain text file here

Below is the list of Bad Crawler Spiders taken from his site:

 

# robots.txt to prohibit bad internet search engine spiders to crawl your website
# Begin block Bad-Robots from robots.txt
User-agent: asterias
Disallow:/
User-agent: BackDoorBot/1.0
Disallow:/
User-agent: Black Hole
Disallow:/
User-agent: BlowFish/1.0
Disallow:/
User-agent: BotALot
Disallow:/
User-agent: BuiltBotTough
Disallow:/
User-agent: Bullseye/1.0
Disallow:/
User-agent: BunnySlippers
Disallow:/
User-agent: Cegbfeieh
Disallow:/
User-agent: CheeseBot
Disallow:/
User-agent: CherryPicker
Disallow:/
User-agent: CherryPickerElite/1.0
Disallow:/
User-agent: CherryPickerSE/1.0
Disallow:/
User-agent: CopyRightCheck
Disallow:/
User-agent: cosmos
Disallow:/
User-agent: Crescent
Disallow:/
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow:/
User-agent: DittoSpyder
Disallow:/
User-agent: EmailCollector
Disallow:/
User-agent: EmailSiphon
Disallow:/
User-agent: EmailWolf
Disallow:/
User-agent: EroCrawler
Disallow:/
User-agent: ExtractorPro
Disallow:/
User-agent: Foobot
Disallow:/
User-agent: Harvest/1.5
Disallow:/
User-agent: hloader
Disallow:/
User-agent: httplib
Disallow:/
User-agent: humanlinks
Disallow:/
User-agent: InfoNaviRobot
Disallow:/
User-agent: JennyBot
Disallow:/
User-agent: Kenjin Spider
Disallow:/
User-agent: Keyword Density/0.9
Disallow:/
User-agent: LexiBot
Disallow:/
User-agent: libWeb/clsHTTP
Disallow:/
User-agent: LinkextractorPro
Disallow:/
User-agent: LinkScan/8.1a Unix
Disallow:/
User-agent: LinkWalker
Disallow:/
User-agent: LNSpiderguy
Disallow:/
User-agent: lwp-trivial
Disallow:/
User-agent: lwp-trivial/1.34
Disallow:/
User-agent: Mata Hari
Disallow:/
User-agent: Microsoft URL Control – 5.01.4511
Disallow:/
User-agent: Microsoft URL Control – 6.00.8169
Disallow:/
User-agent: MIIxpc
Disallow:/
User-agent: MIIxpc/4.2
Disallow:/
User-agent: Mister PiX
Disallow:/
User-agent: moget
Disallow:/
User-agent: moget/2.1
Disallow:/
User-agent: mozilla/4
Disallow:/
User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows ME)
Disallow:/
User-agent: mozilla/5
Disallow:/
User-agent: NetAnts
Disallow:/
User-agent: NICErsPRO
Disallow:/
User-agent: Offline Explorer
Disallow:/
User-agent: Openfind
Disallow:/
User-agent: Openfind data gathere
Disallow:/
User-agent: ProPowerBot/2.14
Disallow:/
User-agent: ProWebWalker
Disallow:/
User-agent: QueryN Metasearch
Disallow:/
User-agent: RepoMonkey
Disallow:/
User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow:/
User-agent: RMA
Disallow:/
User-agent: SiteSnagger
Disallow:/
User-agent: SpankBot
Disallow:/
User-agent: spanner
Disallow:/
User-agent: suzuran
Disallow:/
User-agent: Szukacz/1.4
Disallow:/
User-agent: Teleport
Disallow:/
User-agent: TeleportPro
Disallow:/
User-agent: Telesoft
Disallow:/
User-agent: The Intraformant
Disallow:/
User-agent: TheNomad
Disallow:/
User-agent: TightTwatBot
Disallow:/
User-agent: Titan
Disallow:/
User-agent: toCrawl/UrlDispatcher
Disallow:/
User-agent: True_Robot
Disallow:/
User-agent: True_Robot/1.0
Disallow:/
User-agent: turingos
Disallow:/
User-agent: URLy Warning
Disallow:/
User-agent: VCI
Disallow:/
User-agent: VCI WebViewer VCI WebViewer Win32
Disallow:/
User-agent: Web Image Collector
Disallow:/
User-agent: WebAuto
Disallow:/
User-agent: WebBandit
Disallow:/
User-agent: WebBandit/3.50
Disallow:/
User-agent: WebCopier
Disallow:/
User-agent: WebEnhancer
Disallow:/
User-agent: WebmasterWorldForumBot
Disallow:/
User-agent: WebSauger
Disallow:/
User-agent: Website Quester
Disallow:/
User-agent: Webster Pro
Disallow:/
User-agent: WebStripper
Disallow:/
User-agent: WebZip
Disallow:/
User-agent: WebZip/4.0
Disallow:/
User-agent: Wget
Disallow:/
User-agent: Wget/1.5.3
Disallow:/
User-agent: Wget/1.6
Disallow:/
User-agent: WWW-Collector-E
Disallow:/
User-agent: Xenu’s
Disallow:/
User-agent: Xenu’s Link Sleuth 1.1c
Disallow:/
User-agent: Zeus
Disallow:/
User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow:/
Crawl-delay: 20
# Begin Exclusion From Directories from robots.txt
Disallow: /cgi-bin/

Veryimportant variable among the ones passed by above robots.txt is
 

Crawl-Delay: 20

 


You might want to tune that variable a Crawl-Delay of 20 instructs all IP connects from any Web Spiders that are respecting robots.txt variables to delay crawling with 20 seconds between each and every connect client request, that is really useful for the Webserver as less connects means less CPU and Memory usage and less degraded performance put by aggressive bots crawling your site like crazy, requesting resources 10 times per second or so …

As you can conclude by the naming of some of the bots having them disabled would prevent your domain/s clients from Email harvesting Spiders and other not desired activities.


 

2. Listing IP addresses Hits / How many connects per IPs used to determine problematic server overloading a huge number of IPs connects

After saying few words about SE bots and I think it it is fair to also  mention here a number of commands, that helps the sysadmin to inspect Apache's access.log files.
Inspecting the log files regularly is really useful as the number of malicious Spider Bots and the Cracker users tends to be
raising with time, so having a good way to track the IPs that are stoning at your webserver and later prohibiting them softly to crawl either via robots.txt (not all of the Bots would respect that) or .htaccess file or as a last resort directly form firewall is really useful to know.
 

– Below command Generate a list of IPs showing how many times of the IPs connected the webserver (bear in mind that commands are designed log fields order as given by most GNU / Linux distribution + Apache default logging configuration;

 

webhosting-server:~# cd /var/log/apache2 webhosting-server:/var/log/apache2# cat access.log| awk '{print $1}' | sort | uniq -c |sort -n


Below command provides statistics info based on whole access.log file records, sometimes you will need to have analyzed just a chunk of the webserver log, lets say last 12000 IP connects, here is how:
 

webhosting-server:~# cd /var/log/apache2 webhosting-server:/var/log/apache2# tail -n 12000 access.log| awk '{print $1}' | sort | uniq -c |sort -n


You can combine above basic bash shell parser commands with the watch command to have a top like refresh statistics every few updated refreshing IP statistics of most active customers on your websites.

Here is an example:

 

webhosting-server:~# watch "cat access.log| awk '{print $1}' | sort | uniq -c |sort -n";

 


Once you have the top connect IPs if you have a some IP connecting with lets say 8000-10000 thousand times in a really short interval of time 20-30 minues or so. Hence it is a good idea to investigate further where is this IP originating from and if it is some malicious Denial of Service, filter it out either in Firewall (with iptables rules) or ask your ISP or webhosting to do you a favour and drop all the incoming traffic from that IP.

Here is how to investigate a bit more about a server stoner IP;
Lets assume that you found IP: 176.9.50.244 to be having too many connects to your webserver:
 

webhosting-server:~# grep -i 176.9.50.244 /var/log/apache2/access.log|tail -n 1
176.9.50.244 – – [12/Sep/2017:07:42:13 +0300] "GET / HTTP/1.1" 403 371 "-" "Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)"

 

webhosting-server:~# host 176.9.50.244
244.50.9.176.in-addr.arpa domain name pointer static.244.50.9.176.clients.your-server.de.

 

webhosting-server:~# whois 176.9.50.244|less

 

The outout you will get would be something like:

% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: this output has been filtered.
%       To receive output for a database update, use the "-B" flag.

% Information related to '176.9.50.224 – 176.9.50.255'

% Abuse contact for '176.9.50.224 – 176.9.50.255' is 'abuse@hetzner.de'

inetnum:        176.9.50.224 – 176.9.50.255
netname:        HETZNER-RZ15
descr:          Hetzner Online GmbH
descr:          Datacenter 15
country:        DE
admin-c:        HOAC1-RIPE
tech-c:         HOAC1-RIPE
status:         ASSIGNED PA
mnt-by:         HOS-GUN
mnt-lower:      HOS-GUN
mnt-routes:     HOS-GUN
created:        2012-03-12T09:45:54Z
last-modified:  2015-08-10T09:29:53Z
source:         RIPE

role:           Hetzner Online GmbH – Contact Role
address:        Hetzner Online GmbH
address:        Industriestrasse 25
address:        D-91710 Gunzenhausen
address:        Germany
phone:          +49 9831 505-0
fax-no:         +49 9831 505-3
abuse-mailbox:  abuse@hetzner.de
remarks:        *************************************************
remarks:        * For spam/abuse/security issues please contact *
remarks:        * abuse@hetzner.de, not this address. *
remarks:        * The contents of your abuse email will be *
remarks:        * forwarded directly on to our client for *
….


3. Generate list of directories and files that are most called by clients
 

webhosting-server:~# cd /var/log/apache2; webhosting-server:/var/log/apache2# awk '{print $7}' access.log|cut -d? -f1|sort|uniq -c|sort -nk1|tail -n10

( take in consideration that this info is provided only on current records from /var/log/apache2/ and is short term for long term statistics you have to merge all existing gzipped /var/log/apache2/access.log.*.gz )

To merge all the old gzipped files into one single file and later use above shown command to analyize run:

 

cd /var/log/apache2/
cp -rpf *access.log*.gz apache-gzipped/
cd apache-gzipped
for i in $(ls -1 *access*.log.*.gz); do gzip -d $i; done
rm -f *.log.gz;
for i in $(ls -1 *|grep -v access_log_complete); do cat $i >> access_log_complete; done


Though the accent of above article is Apache Webserver log analyzing, the given command examples can easily be recrafted to work properly on other Web Servers LigHTTPD, Nginx etc.

Above commands are about to put a higher load to your server during execution, so on busy servers it is a better idea, to first go and synchronize the access.log files to another less loaded servers in most small and midsized companies this is being done by a periodic synchronization of the logs to the log server used usually only to store log various files and later used to do various analysis our run analyse software such as Awstats, Webalizer, Piwik, Go Access etc.

Worthy to mention one great text console must have Apache tool that should be mentioned to analyze in real time for the lazy ones to type so much is Apache-top but those script will be not installed on most webhosting servers and VPS-es, so if you don't happen to own a self-hosted dedicated server / have webhosting company etc. – (have root admin access on server), but have an ordinary server account you can use above commands to get an overall picture of abusive webserver IPs.

logstalgia-visual-loganalyzer-in-reali-time-windows-linux-mac
 

If you have a Linux with a desktop GUI environment and have somehow mounted remotely the weblog server partition another really awesome way to visualize in real time the connect requests to  web server Apache / Nginx etc. is with Logstalgia

Well that's all folks, I hope that article learned you something new. Enjoy

Thanks for article neo-tux picture to segarkshtri.com.np)

Migrate Webserver and SQL data from old SATA Hard drive to SSD to boost websites performance / Installing new SSD KINGSTON 120GB hard disk on Linux

Monday, March 28th, 2016

ssd-linux-migrate-webserver-and-mysql-from-old-SATA-to-SSD-Kingston-Hard-drive-to-boost-performance-installing-new-SSD-on-Debian-linux
Blog and websites hosted on a server were giving bad performance lately and the old SATA Hard Disk on the Lenovo Edge server seemed to be overloaded from In/Out operations and thus slowing down the websites opeining time as well as SQL queries (especially the ones from Related Posts WordPress plugin was quite slow. Sometimes my blog site opening times were up to 8-10 seconds.

To deal with the issue I obviously needed a better speed of I/O of hard drive thus as I've never used SSD hard drives so far,  I decided to buy a new SSD (Solid State Drive) KINGSTON SV300S37A120G, 605ABBF2, max UDMA/133  hard disk.
Mounting the hard disk physically on the computer tower case wasn't a big deal as there are no rotating elements of the SSD it doesn't really matter how it is mounted main thing is that it is being hooked up somewhere to the case.

I was not sure whether the SSD HDD is supported by my Debian GNU / Linux so I had see whether Linux Operating System has properly detected your hard disk use dmesg

1. Check if SSD Hard drive is supported in Linux

 

linux:~# dmesg|grep -i kingston
[    1.182734] ata5.00: ATA-8: KINGSTON SV300S37A120G, 605ABBF2, max UDMA/133
[    1.203825] scsi 4:0:0:0: Direct-Access     ATA      KINGSTON SV300S3 605A PQ: 0 ANSI: 5

 

linux:~# dmesg|grep -i sdb
[    1.207819] sd 4:0:0:0: [sdb] 234441648 512-byte logical blocks: (120 GB/111 GiB)
[    1.207847] sd 4:0:0:0: [sdb] Write Protect is off
[    1.207848] sd 4:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[    1.207860] sd 4:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.207928]  sdb: unknown partition table
[    1.208319] sd 4:0:0:0: [sdb] Attached SCSI disk

 

Well great news as you see from above output obviously the Kingston SSD HDD was detected by the kernel.
I've also inspected whether the proper dimensions of hard drive (all 120 Gigabytes are being detected by the OS):

 

linux:~# fdisk -l /dev/sdb
Disk /dev/sdb: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders, total 234441648 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Even better as the proper HDD sizing was detected by Linux kernel.
Next thing to do was of course to create ext4 filesystem on the SSD HDD.
I wanted to give 2 separate partitions for my Webserver Websites DocumentRoot directories which all lay under the standard Apache location inside /var/www as well as MySQL data folder which is also under the standard Debian based Linuces – /var/lib/mysql as the SQL data directory was just 3.3 GB size, I've decided to reserve 20GB gigabytes for the MySQL and another 100 GB for my PHP / CSS / JS / HTML and other data files /var/www.
 

2. Create SSD partitions with cfdisk

Hence I needed to create:

1. SSD partition of 100GB
2. SSD partition of 20GB

I have cfdisk installed and I believe, the easiest way to create the partitions is using interactive partitioner as CFDISK instead of fdisk: so in order to make the proper partitions I've ran

 

linux:~# cfdisk /dev/sdb


I' will skip explainig details on how to use CFDISK as it is pretty standard – display or manipulate disk partition table tool.
Just press on NEW button (moving with arrow keys buttons) and choose the 2 partitions size 100000 and 20000 MB (one thing to note here is that you have to choose between Primary and  Logical creation of partitions, as my SSD is a secondary drive and I already have a ) and then press the
WRITE button to save all the partition changes.

!!! Be very careful here as you might break up your other disks data make sure you're really modifying the SSD Hard Drive and not your other /dev/sda or other attached external Hard drive or ATA / SATA disk.
Press the WRITE button only once you're absolutely sure, you do it at your own (always create backup of your other data and don't blame me if something goes wrong) …

Once created the two partitions will look like in the screenshot below:
creating-linux-partitions-with-cfdisk-linux-partitioning-tool.png

 


3. Create ext4 filesystem 100 and 20 GB partitions

Next thing to do before the two partitions are ready to mount under Webserver's files documentroot /var/www and /var/lib/mysql is to create ext4 filesystem, though some might prefer to stick to ext3 or reiserfs partition, I would recommend you use ext4 for the reason ext4 according to my quick research is said to perform much better with SSD Hard Drives.

The tool to create the ext4 filesystems is mkfs4.ext4 it is provided by debian package e2fsprogs I have it already installed on my server, if you don't have it just go on and install it with:
 

linux:~# apt-get install –yes e2fsprogs

 

To create the two ext4 partitions run:
 

linux:~# mkfs4.ext4 /dev/sdb5

 

linux:~# mfs4.ext4 /dev/sdb6


Here the EXT4 filesystem on partition that is supposed to be 100 Gigabytes will take 2, 3 minutes as the dimensions of partition are a bit bigger, so if you don't want to get boring go grab a coffee, once the partitions are ready you can evaluate whether everyhing is properly created with fdisk you should get output like the one below

 

linux:~# fdisk -l /dev/sdb
Disk /dev/sdb: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders, total 234441648 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1              63   234441647   117220792+   5  Extended
/dev/sdb5             126    39070079    19534977   83  Linux
/dev/sdb6        39070143   234441647    97685752+  83  Linux

 

4. Mount newly created SSD partitions under /var/www and /var/lib/mysql

Before I mounted /var/www and /var/lib/mysql in order to be able to mount under the already existing directories I had to:

1. Stop Apache and MySQL server
2. Move Mysql and Apache Documentroot and Data directories to -bak
3. Create new empty /var/www and /var/lib/mysql direcotries
4. Copy backpups ( /var/www-bak and /var/lib/mysql-bak ) to the newly mounted ext4 SSD partitions

To achieve that I had to issue following commands:
 

linux:~# /etc/init.d/apache2 stop
linux:~# /etc/init.d/mysql stop

linux:~# mv /var/www /var/www-bak
linux:~# mv /var/lib/mysql /var/lib/mysql-bak

linux:~# mkdir /var/www
linux:~# mkdir /var/lib/mysql
linux:~# chown -R mysql:mysql /var/lib/mysql


Then to manually mount the SSD partitions:
 

linux:~# mount  /dev/sdb5 /var/lib/mysql
linux:~# mount /dev/sdb6 /var/www


To check that the folders are mount into the SSD drive, ran mount cmd:

 

linux:~# mount
/dev/sda1 on / type ext3 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
/dev/sdc1 on /backups type ext4 (rw)
/dev/sdb5 on /var/lib/mysql type ext4 (rw,relatime,discard,data=ordered))
/dev/sdb6 on /var/www type ext4 (rw,relatime,discard,data=ordered))

 

That's great now the filesystem mounts fine, however as it an SSD drive and SSD drives are being famous for having a number of limited writes on disk before the drive lifetime is over it is a good idea to increase a bit the lifetime of the SSD by mounting the SSD partitions with noatime and errors=remount-ro (in order to not log file access times to filesystem table and to remount the FS read only in case of some physical errors of the drive).

5. Configure SSD partitions to boot every time Linux reboots

Now great, the filesystems gets mounted fine so next thing to do is to make it automatically mount every time the Linux OS boots up, this on GNU / Linux is done through /etc/fstab, for my 2 ext4 partitions this is the content to add at the end of /etc/fstab:

 

/dev/sdb5               /var/lib/mysql      ext4        noatime,errors=remount-ro       0       1
/dev/sdb6               /var/www        ext4    noatime,errors=remount-ro       0       1

 

quickest way to add it without a text editor is to echo to the end of file:
 

linux:~# cp -rpf /etc/fstab /etc/fstab.bak_25_03_2016
linux:~# echo ' /dev/sdb5               /var/lib/mysql      ext4        noatime,errors=remount-ro,discard       0       1' >> /etc/fstab
linux:~# echo ' /dev/sdb6               /var/www        ext4    noatime,errors=remount-ro,discard       0       1 ' >> /etc/fstab


Then mount again all the filesystems including the 2 new created SSD (100 and 20 GB) partitions:
 

linux:~# umount /var/www
linux:~# umount /var/lib/mysql
linux:~# mount -a


To assure properly mounted with noatime and remount-ro on errors options:


linux:~# mount | grep -i sdb
/dev/sdb5 on /var/lib/mysql type ext4 (rw,noatime,errors=remount-ro)
/dev/sdb6 on /var/www type ext4 (rw,noatime,errors=remount-ro)

 

It is also a good idea to check a statistics of disk free command:
 

linux:~# df -h|grep -i sdb
/dev/sdb5         19G  0G    19G  0% /var/lib/mysql
/dev/sdb6         92G   0G    92G  0% /var/www


6. Copy all Webserver and SQL data from backupped directories to new SSD mounted

Last but not least is to copy all original content files from /var/www-bak and /var/lib/mysql-bak to the new freshly  created SSD partitions, though copying the files can be made with normal linux copy command (cp),
I personally prefer rsync because rsync is much quicker and more efficient in copying large amount of files in my case this were 48 Gigabytes.

To copy files from with rsync:

 

linux:~# rsync -av –log-file /var/log/backup.log  /var/www-bak /var/www
linux:~# rsync -av –log-file /var/log/backup.log  /var/lib/mysql-bak /var/lib/mysql


Then ofcourse, finally to restore my websites normal operation I had to bring up the Apache Webservers and MySQL service

 

linux:~# /etc/init.d/apache2 start
linux:~# /etc/init.d/mysql start


7. Optimizing SSD performance with periodic trim (discard of unused blocks on a mounted filesystem)

As I digged deeper into how to even further optimize SSD drive performance I learned about the cleaning action TRIM of the partitions for a long term performance proper operation, to understand it better think about trimming like Windows degrament operatin.
 

NAME
fstrim – discard unused blocks on a mounted filesystem

SYNOPSIS
fstrim [-o offset] [-l length] [-m minimum-free-extent] [-v] mountpoint

DESCRIPTION
fstrim is used on a mounted filesystem to discard (or "trim") blocks which are not in use by the filesystem. This is useful for
solid-state drives (SSDs) and thinly-provisioned storage.

By default, fstrim will discard all unused blocks in the filesystem. Options may be used to modify this behavior based on range or
size, as explained below.


Trimming is really necessery, otherwise SSD become very slow after some time. All modern SSD's support TRIM, but older SSD's from before 2010 usually don't.
Thus for an older SSD you'll want to check this on the website of the manufacturer.

As I mentioned earlier TRIM is not supported by all SSD drives, to check whether TRIM is supported by SSD:

linux:~# hdparm -I /dev/sdb|grep -i -E 'trim|discard'
                  *          Data set Management TRIM supported (limit 1 block)

It's easiest to let the system perform an automatic TRIM. That can be done in several ways.

The quickest way for trimming is to place into /etc/rc.local trim  commands, in my case it was the following commands:

 

fstrim -v /var/lib/mysql
fstrim -v /var/www

To add it I've used my favourite vim text editor.
Adding commands to rc.local will make SSD trimming be executed at boot time so this will reduce a bit the downtime during the trim with some time so perhaps for those like me which are running a crually important websites a better

An alternative way is to schedule a daily cron job to do just place a new job in /etc/cron.daily/trim e.g.:
 

linux:~# vim /etc/cron.daily/trim

 

#!/bin/sh
fstrim -v /var/lib/mysql
fstrim -v /var/www

linux:~# chmod +x /etc/cron.daily/trim

However the best way to enable automatic trimming to SSD  is to just add the discard parameter to /etc/fstab I've already done that earlier in this article.

Not really surprising the increase of websites opening (page load times) were decreased dramatically web page loading waiting time fall down 2 to 2.5 times, so the moral of story for me is always when possible from now on to use SSD in order to have superb websites opening times.

To sum it up what was achieved with moving my data into SSD Drive, before moving websites and SQL data to SSD drive the websites were opening for 6 to 10 seconds now sites open in 2 to 4.5 seconds which is below 5 seconds (the normal waiting time for a user to see your website).
By the way it should be not a news forfor people that are into Search Engine Optimization but might be for some of unexperienced new Admins and Webmasters that, all that all page opening times that  exceeds 5 secs is considered to be a slow website (and therefore perhaps not worthy to read).
The high load page times >5 secs makes the website also less interesting not only for end users but also for search engines (Google / Yahoo / Bing / Baidoo etc.) will is said to crawl it less if website is slow.
Search Engines are said to Index much better and crawl more frequently into more responsive websites.
Hence implementing SSD to a server and decreasing the page load time should bring up my visitors stats a bit too.

Well that's all for today, hope you enjoyed 🙂

Howto Fix “sysstat Cannot open /var/log/sysstat/sa no such file or directory” on Debian / Ubuntu Linux

Monday, February 15th, 2016

sysstast-no-such-file-or-directory-fix-Debian-Ubuntu-Linux-howto
I really love sysstat and as a console maniac I tend to install it on every server however by default there is some <b>sysstat</b> tuning once installed to make it work, for those unfamiliar with <i>sysstat</i> I warmly recommend to check, it here is in short the package description:<br /><br />
 

server:~# apt-cache show sysstat|grep -i desc -A 15
Description: system performance tools for Linux
 The sysstat package contains the following system performance tools:
  – sar: collects and reports system activity information;
  – iostat: reports CPU utilization and disk I/O statistics;
  – mpstat: reports global and per-processor statistics;
  – pidstat: reports statistics for Linux tasks (processes);
  – sadf: displays data collected by sar in various formats;
  – nfsiostat: reports I/O statistics for network filesystems;
  – cifsiostat: reports I/O statistics for CIFS filesystems.
 .
 The statistics reported by sar deal with I/O transfer rates,
 paging activity, process-related activities, interrupts,
 network activity, memory and swap space utilization, CPU
 utilization, kernel activities and TTY statistics, among
 others. Both UP and SMP machines are fully supported.
Homepage: http://pagesperso-orange.fr/sebastien.godard/

 

If you happen to install sysstat on a Debian / Ubuntu server with:

server:~# apt-get install –yes sysstat


, and you try to get some statistics with sar command but you get some ugly error output from:

 

server:~# sar Cannot open /var/log/sysstat/sa20: No such file or directory


And you wonder how to resolve it and to be able to have the server log in text databases periodically the nice sar stats load avarages – %idle, %iowait, %system, %nice, %user, then to FIX that Cannot open /var/log/sysstat/sa20: No such file or directory

You need to:

server:~# vim /etc/default/sysstat


By Default value you will find out sysstat stats it is disabled, e.g.:

ENABLED="false"

Switch the value to "true"

ENABLED="true"


Then restart sysstat init script with:

server:~# /etc/init.d/sysstat restart

However for those who prefer to do things from menu Ncurses interfaces and are not familiar with Vi Improved, the easiest way is to run dpkg reconfigure of the sysstat:

server:~# dpkg –reconfigure


sysstat-reconfigure-on-gnu-linux

 

root@server:/# sar
Linux 2.6.32-5-amd64 (pcfreak) 15.02.2016 _x86_64_ (2 CPU)

0,00,01 CPU %user %nice %system %iowait %steal %idle
0,15,01 all 24,32 0,54 3,10 0,62 0,00 71,42
1,15,01 all 18,69 0,53 2,10 0,48 0,00 78,20
10,05,01 all 22,13 0,54 2,81 0,51 0,00 74,01
10,15,01 all 17,14 0,53 2,44 0,40 0,00 79,49
10,25,01 all 24,03 0,63 2,93 0,45 0,00 71,97
10,35,01 all 18,88 0,54 2,44 1,08 0,00 77,07
10,45,01 all 25,60 0,54 3,33 0,74 0,00 69,79
10,55,01 all 36,78 0,78 4,44 0,89 0,00 57,10
16,05,01 all 27,10 0,54 3,43 1,14 0,00 67,79


Well that's it now sysstat error resolved, text reporting stats data works again, Hooray! 🙂

Apache Webserver disable hostnamelookups “HostnameLookups off” for minor performance increase

Friday, February 12th, 2016

apache-disable-dns-lookups-for-speed-hostnamelookups-off-directive-building-scalable-php-applications

If you don't much care about logging in logs from which domain / hostnames requests to webserver originate and you want to boost up the Apache Webserver performance a bit especially on a heavy loaded Websites, where no need for stuff like Webalizer, Awstats etc. , e.g. you're using GoogleAnalytics to already track requests (beware as sometimes GoogleAnalytics could be missing requests to your webserver, so having some kind of LogAnalyzer software on server is always a plus). But anyways accepting that many of us already trust GoogleAnalytitcs.


Then a great tuning option to use in default domain configuration or in multiple VirtualHosts config is:

HostnameLookups off

If you want to make the HostnameLookups off as a default behaviour to all your virtualhosts on  Debian / Ubuntu / CentOS / SuSE / RHEL distro virtualhosts add either to default config /etc/apache2/sites-enabled/000-default (on Deb based Linuxes) or (on RPM based ones), add directive to /etc/httpd/httpd.conf

For self-hosted websites (if run your own small hosting) or for a home situated webservers with up to 20-50 websites it is also a useful optimization tip to include in /etc/hosts file all the IPs of sites with respective domain names following the normal syntax of /etc/hosts, e.g. in my own /etc/hosts, I have stuff like:
 

pcfreak:~$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain
127.0.1.1 pcfreak.www.pc-freak.net pcfreak mail.www.pc-freak.net
192.168.0.14 new-pcfreak
219.22.88.70 fw
212.36.0.70 ftp.bg.debian.org
212.211.132.32 security.debian.org
83.228.93.76 pcfreak.biz www.pc-freak.net www.pc-freak.net
# for wordpress plugins
216.58.209.3 gstatic.com
91.225.248.129 www.linkedin.com
74.50.119.198 www.blogtopsites.com
94.31.29.40 static.addtoany.com
216.58.209.202 fonts.googleapis.com
216.58.209.14 www.google-analytics.com
216.58.209.14 feeds.feedburner.com
93.184.220.241 wprp.zemanta.com
199.30.80.32 stumbleupon.com
156.154.168.17 stumbleupon.com
2.18.89.251 platform.linkedin.com
# The following lines are desirable for IPv6 capable hosts

# … etc. put IPs and hostnames following above syntax


As you see from above commented section for wordpress plugins, I've included some common websites used by WordPress enabled plugins to prevent my own hosting server to query DNS server every time. The normal way the Linux / Unix works is it first checks in /etc/hosts and only if the hostname is not defined there then it queries the DNS caching server in my case this is a local DJBDNS cache server, however defining the hosts in /etc/hosts saves a lot of milisecons on every request and often if multiple hosts are defined could save (decrease site opening for end users) with seconds.


Well now use some website speed testing plugin like Yslow, Firebug Fiddler or HTTPWatch

 

Windows “God Mode” one shortcut to see and configure all setttings in Microsoft Windows 7 / 8 / 10 – Windows Master Control Panel hidden feature

Monday, January 25th, 2016

GodMode-secret-master-control-panel-in-windows-operating-system
One very handy "secret" feature of Windows Operating System which is very useful to people who administrate a dozen of Windows servers daily is called "God Mode".
The idea behind "God Mode" is pretty simple it aims to give you maximum control and viability concentrated in one single Window interface.

God Mode was quite a lot ranted over the past years so it is likely that many of my blog readers are already aware of that Windows secret, but for those who didn't it will be
nice to check it out. To see the God Mode Windows functionality just create a New Folder in Windows Desktop or Anywhere on the Windows PC and Rename the New Folder to:
 

GodMode.{ED7BA470-8E54-465E-825C-99712043E01C}

Windows-7-God-mode-screenshot-or-how-to-easily-get-one-point-of-control-in-windows-OS

By creating folder witth his text string you will be able to do almost everything you ever tend to do on Windows from changing the outlook of theme and mouse cursor, changing,
Win explorer's folder's options, modify fonts, change cursor blink rate, get windows performance tools and information, add / remove programs, modify language, modify
firewall settings and in short do everything that is provided by Control Panel + some other goodies like Administrative Tools, Restore Options, Event logs etc. grouped in a fantastic readable manner.
GodMode naming says it all more or less it aims to give you "Godlike" accessibility to the Windows. Of course to be able to properly use the feature you will have to create
the Folder named GodMode.{ED7BA470-8E54-465E-825C-99712043E01C} with Administrator user.
The GodMode is available in Windows OSes since quite a long (2007) and is documented officially by Microsoft

Another alternative shortcut that gives the Godmode Master control panel is:
 

God.{ed7ba470-8e54-465e-825c-99712043e01c}

 

Enjoy 🙂

No space left on device with free disk space / Why no space left on device while there is plenty of disk space on drive – Running out of Inodes

Tuesday, November 17th, 2015

no_space_left-on-device-while-there-is-disk-space-running-out-of-file-inodes-unix_linux_file_system_diagram.gif

 

On one of the servers, I'm administrating the websites started showing some Mysql database table corrup errors like:
 

 

Table './database_name/site_news_list_com' is marked as crashed and last (automatic?) repair failed

The server is using Oracle MySQL server community stable edition on Debian GNU / Linux 6.0, so I first thought during work the server crashed either due to some bug issue in MySQL or it crashed due to some PHP cron job that did something messy. Thus to solve the crashed tables, tried using mysqlcheck tool which helped pretty fine, at many times whether there were database / table corruptions. I've run the following set of mysqlcheck commands with root (superuser) in a bash shell after logging in through SSH:

:

server:~# /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–check –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log
server:~# /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf –analyze –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log
server:~# /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–auto-repair –optimize –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log
server:~# /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–optimize –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log


In order for above commands to work, I've created the /root/.my.cnf containing my root (mysql CLI) mysql username and password, e.g. file has content like below:

 

[client]
user=root
password=MySecretPassword8821238

 

Btw a good note here is its generally a good idea (if you want to have consistent mysql databases) to automatically execute via a cron job 2 times a month, I've in root cronjob the following:

 

crontab -u root -l |grep -i mysqlcheck
04 06 5,10,15,20,25,1 * * /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–check –all-databases –silent -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log 07 06 5,10,15,20,25,1 * * /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf –analyze –all-databases –silent -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log 12 06 5,10,15,20,25,1 * * /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–auto-repair –optimize –all-databases –silent -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log 17 06 5,10,15,20,25,1 * * /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–optimize –all-databases –silent -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log


Strangely I got a lot of errors that some .MYI / .MYD .frm temp files, necessery for the mysql tables recovery can't be written inside /home/mysql/database_name

That was pretty weird and I thought there might be some issues with permissions, causing the inability to write, due to some bug or something so I went straight and checked /home/mysql/database_name permissions, e.g.::

 

server:/home/mysql/database_name# ls -ld soccerfame
drwx—— 2 mysql mysql 36864 Nov 17 12:00 soccerfame
server:/home/mysql/database_name# ls -al1|head -n 10
total 1979012
drwx—— 2 mysql mysql 36864 Nov 17 12:00 .
drwx—— 36 mysql mysql 4096 Nov 17 11:12 ..
-rw-rw—- 1 mysql mysql 8712 Nov 17 10:26 1_campaigns_diez.frm
-rw-rw—- 1 mysql mysql 14672 Jul 8 18:57 1_campaigns_diez.MYD
-rw-rw—- 1 mysql mysql 1024 Nov 17 11:38 1_campaigns_diez.MYI
-rw-rw—- 1 mysql mysql 8938 Nov 17 10:26 1_campaigns.frm
-rw-rw—- 1 mysql mysql 8738 Nov 17 10:26 1_campaigns_logs.frm
-rw-rw—- 1 mysql mysql 883404 Nov 16 22:01 1_campaigns_logs.MYD
-rw-rw—- 1 mysql mysql 330752 Nov 17 11:38 1_campaigns_logs.MYI


As seen from above output, all was perfect with permissions, so it should have been something else, so I decided to try to create a random file with touch command inside /home/mysql/database_name directory:

 

touch /home/mysql/database_name/somefile-to-test-writtability.txt touch: cannot touch ‘/scr1/data/somefile-to-test-writtability.txt‘: No space left on device


Then logically I thought the /home/mysql/ mounted ext4 partition got filled, because of crashed SQL database or a bug thus, checked with disk free command df whether there is enough space on server:

server:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md1 20G 7.6G 11G 42% /
udev 10M 0 10M 0% /dev
tmpfs 13G 1.3G 12G 10% /run
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/md2 256G 134G 110G 55% /home

Well that's weird? Obviously only 55% of available disk space is used and available 134G which was more than enough so I got totally puzzled why, files can't be written.

Then very logically, I thought it might be that /home directory has remounted as read only, because the SSD memory disk on server is failing and checked for errors in dmesg, i.e.:

 

server:~# dmesg|grep -i error


Also checked how exactly was partition mounted, to check whether it is (RO) read-only:

 

server:~# mount -l|grep -i /home
/dev/md2 on /home type ext4 (rw,relatime,discard,data=ordered)


Now everything become even more weirder, as obviously the disk continued to be claiming no space left on device, while in reality there was plenty of disk space.

Then after running a quick research on the internet for the no space left on device with free disk space, I've come across this great superuser.com thread which let me realize the partition run out of inodes and that's why no new file inodes could be assigned and therefore, the linux kernel is refusing to write the file on ext4 partition.

For those who haven't heard of Linux Partition Inodes here is link to Wikipedia and a quick quote:

 

In a Unix-style file system, the inode is a data structure used to represent a filesystem object, which can be one of various things including a file or a directory. Each inode stores the attributes and disk block location(s) of the filesystem object's data.[1] Filesystem object attributes may include manipulation metadata (e.g. change,[2] access, modify time), as well as owner and permission data (e.g. group-id, user-id, permissions).[3]
Directories are lists of names assigned to inodes. The directory contains an entry for itself, its parent, and each of its children.


Once I understood it is the inodes, I checked how many of them are occupied with cmd:

 

server:~# df -i /home
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md2 17006592 17006592 0 100% /home


You see, there were 0 (zero) free file inodes on server and that was the reason for no space left on device while there was actually free disk space

To clean up (free) some inodes on partition, first thing I did is to delete all old logs which were inside /home and files I positively know not to be necessery, then to find which directories allocating most innodes used:

 

server:~# find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n


If you're on a regular old fashined IDE Hard Drive and not SSD or you have too much files inside this command will take really long …:

Therefore a better solution might be to frist:

a) Try to find root folders with large inodes count:

for i in /home/*; do echo $i; find $i |wc -l; done
Try to find specific folders:


You should get output like:

 

/home/new_website
606692
/home/common
73
/home/pcfreak
5661
/home/hipo
33
/home/blog
13570
/home/log
123
/home/lost+found
1

b) Then once you know the directory allocating most inodes, run the command again to see the sub-directories with most files (eating) partition innodes:

 

for i in /home/webservice/*; do echo $i; find $i |wc -l; done

 

One usual large folder which could free you some nodes is the linux source headers, but in my case it was simply a lot of tiny old logs being logged on the system for few years in the past without cleaning:

After deleting the log dirs and cache folder in my case /home/new_website/{log,cache}:

server:~# rm -rf /home/new_website/log/*
server:~# rm -rf /home/new_website/cache/*

 

 

a) Then, stopping Apache webserver to check prevent Apache to use MySQl databases while running database repair and restaring MySQL:
 

server:~# /etc/init.d/apache2 stop Restarting MySQL server
..
server:~# /etc/init.d/mysql restart
..


b) And re-issuing MySQL Check / Repair / Optimize database commands:
 

 

mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–check –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log

mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf –analyze –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log

mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–auto-repair –optimize –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log

mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–optimize –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log

c) And finally starting the Apache Webserver again:
 

server:~# /etc/init.d/apache2 start


Some innodse got freed up:
 

server:~# df -i /home Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md2 17006592 16797196 209396 99% /home


And hooray by God's Grace and with help of prayers of The most Holy Theotokos (Virgin) Mary, websites started again !