Posts Tagged ‘home directory’

How to Increase virtualbox Linux install machine VM .VDI hard disk size to free space on root partition – Move /usr to a new partition

Tuesday, October 10th, 2017

extend-vdi-virtual-machine-partition-with-virtualbox-howto-virtualbox-logo
 

How to Increse Hard Disk size of VirtualBox Virtual Machine .VDI file to Free Space on root Partition or Howto move /usr large partition to separate new partition?


I just wondered how to increase hard disk size of Virtualbox Virtual Machine image .VDI, because for some stupid reason I've created my initial hard disk size for Linux partition to be the default 10 Gigabytes.

The problem is the packages I need to have installed on the Virtual Machine which will be a testbed for future tests of a production website applications are taking up too much space, once installed so I'm left with no space
in /var/lib/mysql for the database import. So what can I do in that case is to simply free up disk space or Merge ROOT partition with another partition.

Since merging the partition is not a trivial job and would require me to have installable CD with the Linux distro (in my case that's Debian Linux) or have a bootable USB flash drive, I preferred second approach to problem e.g. to free up disk space on ROOT partition by creating a second partition and move the /usr folder to reside there.

Before that it is of course necessery to  have extended the .VDI file using VirtualBox, so more space than the default 10GB preconfigured are available, this is easily done on Windows OS as, VBox is provided with GUI clickable option to do it, but for who knows what reason that is not the case with Linux, so Linux users only option to increase VDI file is to manually run command part of the virtualbox package, that is not a hard task really but it requires some typing and basic knowledge on how to run commands in terminal.


To .VDI resize (extend), we first go to default location where VirtualBox stores its image .VDI files (by default as of moment of writting this article – this is ~/"VrtualBox VMs"  (or home directory of logged in user dir VirtualBox VMs), the command to use is VBoxManage

 

root@jericho:/home/hipo# cd VirtualBox VMs/
root@jericho:/home/hipo/VirtualBox VMs# ls
Debian 6  Debian 9  Windows 10
root@jericho:/home/hipo/VirtualBox VMs# cd Debian 6/
r
oot@jericho:/home/hipo/VirtualBox VMs/Debian 6# ls
Debian 6.vbox  Debian 6.vbox-prev  Debian 6.vdi  Logs  NewVirtualDisk1.vdi  Snapshots

root@jericho:/home/hipo/VirtualBox VMs/Debian 6# VBoxManage modifyhd Debian 6.vdi –resize 20000
0%…10%…20%…30%…40%…50%…60%…70%…80%…90%…100%
root@jericho:/home/hipo/VirtualBox VMs/Debian 6#


 

Above command does resize the 10GB default created partition for Linux, where I have installed Linux which was 99% full of data, because of the many packages I installed to 20GB size, to make it bigger just use the respective size, be it 30000 (for 30GB) or 100000 (for 100GB) etc.

Even though in this example VBoxManage virtual partition resize command was done for GNU / Linux Operating System, it can be done for any other Operating as well to resize the size of the Virtual .VDI file (Virtual Machine) partition, be it Windows 7 / 8 / 10 or the rest of Free Operating systems FreeBSD / OpenBSD / BSD that are installed in a VM etc.

Next Launch the Virtual Machine with VBox Server client Program and install there Gparted (GNU Parted), as we'll need it to create a new Hard Disk Partition:

 

$ VirtualBox

 


Inside virtualmachine's in gnome-terminal / xterm etc. depending on the graphical environment used do install with apt-get:

 

debian:# apt-get install –yes gparted

 

debian~:# gparted


Notice that gparted has to be ran as a root superuser.

 

Run GParted and create new EXT3 filesystem that is 10GB (the size of the new created partition).

If you have installed Debian to place all partitions under / (root directory /dev/sda1) then the fresh new partition to create should be
/dev/sda3, anyways just look closely in EXT3 and in your case if the partiition is named differently create according to proper partition /dev/ naming.

I'll not run into details on how to create the partition with GParted as the program interface is very self-explanatory, the only thing is to apply the update to create partition and the ext3 filesystem, that's being done
with a green tick:

gparted-create-ext3-partition-howto-linux

Next step is to check with fdisk whether, we have ext3 properly created  filesystem as we've done already with GPARTED:

Once we have the partition created with EXT3 filesystem, we're ready to move /usr temporary to other folder, I use usually /root for the move but you can create anywhere a new folder for that and move to there.

To move to /root directory run again in terminal:

 

debian:~# mv /usr /root
debian:~# mkdir /usr

 

howto-extend-root-filesystem-disk-space-linux-move-usr-folder-to-root-temporary-debian-gnu-linux

 

Note that during the move operations, your Desktop icons will become without (with broken) pictures and the default Debian background picture is to disappear, that's because the GUI environment will soon realize /usr/ libraries that're periodically reloaded in memory are missing and will be unable to reload them as it does in a cycle.

That should take a few minutes, so grab a coffee or if you're a smoker (hope not as smoking kills 🙂 ), in 5 / 10 minutes time depending on your computer / server configuration, it will be over, so we're ready to create new /usr dir and mount the  new partition:

 

debian:~# mount /dev/sda3 /usr

 

howto-extend-root-filesystem-disk-space-linux-move-usr-folder-to-root-temporary-debian-gnu-linux1

 

Now we check with mount command whether mount is fine:

 

check-if-filesystem-is-properly-mounted-linux-debian-screenshot


Now  /dev/sda3 is mounted under /usr  and we have to move back /root/usr directory content back to the newly mounted /usr so we run command:
 

debian:~#  mv /root/usr/* /usr/*


Finally we need to create proper records for the new partition inside /etc/fstab (fstab –FileSystem Tab file – the file which describes instructs the Linux OS what partition to boot where, what)

HOW TO CHECK LINUX UUID FOR A PARTITION??

Before adding anything to /etc/fstab you need to check the UUID of /dev/sda3 (or whatever the partition is called), without proper UUID, the system might fail to boot.
So here is how to check the UUID we'll need for config:

 

hipo@debian:~$ /sbin/blkid /dev/sda3
/dev/sda3: UUID="2273db4b-3069-4f78-90fc-e7483c0305bd" SEC_TYPE="ext2" TYPE="ext3"

hipo@debian:~$ ls -al /dev/disk/by-uuid/
total 0
drwxr-xr-x 2 root root 100 Oct  6 05:16 .
drwxr-xr-x 6 root root 120 Oct  6 05:16 ..
lrwxrwxrwx 1 root root  10 Oct  6 05:16 2273db4b-3069-4f78-90fc-e7483c0305bd -> ../../sda3
lrwxrwxrwx 1 root root  10 Oct  6 05:16 b98d92cd-41aa-4e18-a474-9b8df445dbe5 -> ../../sda1
lrwxrwxrwx 1 root root  10 Oct  6 05:16 f27f7448-f200-4983-b54f-b9e5206f77ac -> ../../sda5

As you can see our /dev/sda3 UUID is 2273db4b-3069-4f78-90fc-e7483c0305bd

Further on lets view and edit /etc/fstab you can also download a copy of my Virtual Machine fstab here

 

debian:~# cat /etc/fstab

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
# / was on /dev/sda1 during installation
UUID=b98d92cd-41aa-4e18-a474-9b8df445dbe5 /               ext3    errors=remount-ro 0       1
# swap was on /dev/sda5 during installation
UUID=f27f7448-f200-4983-b54f-b9e5206f77ac none            swap    sw              0       0
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto     0       0
/dev/scd1       /media/cdrom1   udf,iso9660 user,noauto     0       0


We need to add following line to  /etc/fstab:
 

UUID=2273db4b-3069-4f78-90fc-e7483c0305bd    /usr        ext3 error=remount-ro    0    1

 


Open the file with your favourite text editor (gedit / nano / pico / vim / joe) etc.

debian:~# vim /etc/fstab

 

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
# / was on /dev/sda1 during installation
UUID=b98d92cd-41aa-4e18-a474-9b8df445dbe5 /               ext3    errors=remount-ro 0       1
# swap was on /dev/sda5 during installation
UUID=f27f7448-f200-4983-b54f-b9e5206f77ac none            swap    sw              0       0
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto     0       0
/dev/scd1       /media/cdrom1   udf,iso9660 user,noauto     0       0

UUID=2273db4b-3069-4f78-90fc-e7483c0305bd     /usr        ext3 error=remount-ro    0    1    

Basicly it should be possible to add (for historic reasons) also instead of UUID=2273db4b-3069-4f78-90fc-e7483c0305bd  /dev/sda3
So it looks like so but, the better practice is to use UUID line given.

Well that's all folks now /usr directory will contain all your heavy root partition (disk filling) libraries and files, so you can happily use /var/lib/mysql or /var/lib/postgresql /var/www to store your web application files and import your databases.

Big thanks to Ubuntu Forums article – How do I increase the hard disk size of the Virtual Machine article for inspiring this post.

Hope that helps anyone and thanks and other comments are always welcome ! 🙂

Share this on

No space left on device with free disk space / Why no space left on device while there is plenty of disk space on drive – Running out of Inodes

Tuesday, November 17th, 2015

no_space_left-on-device-while-there-is-disk-space-running-out-of-file-inodes-unix_linux_file_system_diagram.gif

 

On one of the servers, I'm administrating the websites started showing some Mysql database table corrup errors like:
 

 

Table './database_name/site_news_list_com' is marked as crashed and last (automatic?) repair failed

The server is using Oracle MySQL server community stable edition on Debian GNU / Linux 6.0, so I first thought during work the server crashed either due to some bug issue in MySQL or it crashed due to some PHP cron job that did something messy. Thus to solve the crashed tables, tried using mysqlcheck tool which helped pretty fine, at many times whether there were database / table corruptions. I've run the following set of mysqlcheck commands with root (superuser) in a bash shell after logging in through SSH:

:

server:~# /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–check –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log
server:~# /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf –analyze –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log
server:~# /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–auto-repair –optimize –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log
server:~# /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–optimize –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log


In order for above commands to work, I've created the /root/.my.cnf containing my root (mysql CLI) mysql username and password, e.g. file has content like below:

 

[client]
user=root
password=MySecretPassword8821238

 

Btw a good note here is its generally a good idea (if you want to have consistent mysql databases) to automatically execute via a cron job 2 times a month, I've in root cronjob the following:

 

crontab -u root -l |grep -i mysqlcheck
04 06 5,10,15,20,25,1 * * /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–check –all-databases –silent -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log 07 06 5,10,15,20,25,1 * * /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf –analyze –all-databases –silent -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log 12 06 5,10,15,20,25,1 * * /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–auto-repair –optimize –all-databases –silent -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log 17 06 5,10,15,20,25,1 * * /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–optimize –all-databases –silent -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log


Strangely I got a lot of errors that some .MYI / .MYD .frm temp files, necessery for the mysql tables recovery can't be written inside /home/mysql/database_name

That was pretty weird and I thought there might be some issues with permissions, causing the inability to write, due to some bug or something so I went straight and checked /home/mysql/database_name permissions, e.g.::

 

server:/home/mysql/database_name# ls -ld soccerfame
drwx—— 2 mysql mysql 36864 Nov 17 12:00 soccerfame
server:/home/mysql/database_name# ls -al1|head -n 10
total 1979012
drwx—— 2 mysql mysql 36864 Nov 17 12:00 .
drwx—— 36 mysql mysql 4096 Nov 17 11:12 ..
-rw-rw—- 1 mysql mysql 8712 Nov 17 10:26 1_campaigns_diez.frm
-rw-rw—- 1 mysql mysql 14672 Jul 8 18:57 1_campaigns_diez.MYD
-rw-rw—- 1 mysql mysql 1024 Nov 17 11:38 1_campaigns_diez.MYI
-rw-rw—- 1 mysql mysql 8938 Nov 17 10:26 1_campaigns.frm
-rw-rw—- 1 mysql mysql 8738 Nov 17 10:26 1_campaigns_logs.frm
-rw-rw—- 1 mysql mysql 883404 Nov 16 22:01 1_campaigns_logs.MYD
-rw-rw—- 1 mysql mysql 330752 Nov 17 11:38 1_campaigns_logs.MYI


As seen from above output, all was perfect with permissions, so it should have been something else, so I decided to try to create a random file with touch command inside /home/mysql/database_name directory:

 

touch /home/mysql/database_name/somefile-to-test-writtability.txt touch: cannot touch ‘/scr1/data/somefile-to-test-writtability.txt‘: No space left on device


Then logically I thought the /home/mysql/ mounted ext4 partition got filled, because of crashed SQL database or a bug thus, checked with disk free command df whether there is enough space on server:

server:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md1 20G 7.6G 11G 42% /
udev 10M 0 10M 0% /dev
tmpfs 13G 1.3G 12G 10% /run
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/md2 256G 134G 110G 55% /home

Well that's weird? Obviously only 55% of available disk space is used and available 134G which was more than enough so I got totally puzzled why, files can't be written.

Then very logically, I thought it might be that /home directory has remounted as read only, because the SSD memory disk on server is failing and checked for errors in dmesg, i.e.:

 

server:~# dmesg|grep -i error


Also checked how exactly was partition mounted, to check whether it is (RO) read-only:

 

server:~# mount -l|grep -i /home
/dev/md2 on /home type ext4 (rw,relatime,discard,data=ordered)


Now everything become even more weirder, as obviously the disk continued to be claiming no space left on device, while in reality there was plenty of disk space.

Then after running a quick research on the internet for the no space left on device with free disk space, I've come across this great superuser.com thread which let me realize the partition run out of inodes and that's why no new file inodes could be assigned and therefore, the linux kernel is refusing to write the file on ext4 partition.

For those who haven't heard of Linux Partition Inodes here is link to Wikipedia and a quick quote:

 

In a Unix-style file system, the inode is a data structure used to represent a filesystem object, which can be one of various things including a file or a directory. Each inode stores the attributes and disk block location(s) of the filesystem object's data.[1] Filesystem object attributes may include manipulation metadata (e.g. change,[2] access, modify time), as well as owner and permission data (e.g. group-id, user-id, permissions).[3]
Directories are lists of names assigned to inodes. The directory contains an entry for itself, its parent, and each of its children.


Once I understood it is the inodes, I checked how many of them are occupied with cmd:

 

server:~# df -i /home
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md2 17006592 17006592 0 100% /home


You see, there were 0 (zero) free file inodes on server and that was the reason for no space left on device while there was actually free disk space

To clean up (free) some inodes on partition, first thing I did is to delete all old logs which were inside /home and files I positively know not to be necessery, then to find which directories allocating most innodes used:

 

server:~# find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n


If you're on a regular old fashined IDE Hard Drive and not SSD or you have too much files inside this command will take really long …:

Therefore a better solution might be to frist:

a) Try to find root folders with large inodes count:

for i in /home/*; do echo $i; find $i |wc -l; done
Try to find specific folders:


You should get output like:

 

/home/new_website
606692
/home/common
73
/home/pcfreak
5661
/home/hipo
33
/home/blog
13570
/home/log
123
/home/lost+found
1

b) Then once you know the directory allocating most inodes, run the command again to see the sub-directories with most files (eating) partition innodes:

 

for i in /home/webservice/*; do echo $i; find $i |wc -l; done

 

One usual large folder which could free you some nodes is the linux source headers, but in my case it was simply a lot of tiny old logs being logged on the system for few years in the past without cleaning:

After deleting the log dirs and cache folder in my case /home/new_website/{log,cache}:

server:~# rm -rf /home/new_website/log/*
server:~# rm -rf /home/new_website/cache/*

 

 

a) Then, stopping Apache webserver to check prevent Apache to use MySQl databases while running database repair and restaring MySQL:
 

server:~# /etc/init.d/apache2 stop Restarting MySQL server
..
server:~# /etc/init.d/mysql restart
..


b) And re-issuing MySQL Check / Repair / Optimize database commands:
 

 

mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–check –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log

mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf –analyze –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log

mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–auto-repair –optimize –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log

mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–optimize –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log

c) And finally starting the Apache Webserver again:
 

server:~# /etc/init.d/apache2 start


Some innodse got freed up:
 

server:~# df -i /home Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md2 17006592 16797196 209396 99% /home


And hooray by God's Grace and with help of prayers of The most Holy Theotokos (Virgin) Mary, websites started again !

Share this on

Adding another level of security to your shared Debian Linux webhosting server with SuPHP

Tuesday, April 7th, 2015

suphp_improve-apache-security-protect-against-virus-internal-server-infections-suphp-webserver-logo

There are plenty of security schemes and strategies you can implement if you're a Shared Web Hosting company sysadmin however probably the most vital one is to install on Apache + PHP Webserver SuPHP module.

# apt-cache show suphp-common|grep -i descrip -A 4

Description: Common files for mod suphp Suphp consists of an Apache module (mod_suphp for either Apache 1.3.x or Apache 2.x) and a setuid root binary (suphp) that is called by the Apache module to change the uid of the process executing the PHP interpreter to the owner of the php script.

So what SuPHP actuall  does is to run separate CPanel / Kloxo etc. Users with separate username and groupid permissions coinciding with the user present in /etc/passwd , /etc/shadow files existing users, thus in case if someone hacks some of the many customer sites he would be able to only write files and directories under the user with which the security breach occured.

On servers where SuPHP is not installed, all  systemusers are using the same UserID / GuID to run PHP executable scripts under separate domains Virtualhost which are coinciding with Apache (on Debian / Ubuntu  uid, gid – www-data) or on (CentOS / RHEL / Fedora etc. – user apache) so once one site is defaced  exploited by a worm all or most server websites might end up infected with a Web Virus / Worm which will be trying to exploit even more sites of a type running silently in the background.  This is very common scenarios as currently there are donezs of PHP / CSS / Javasripts / XSS vulnerability exploited on VPS and Shared hosting servers due to failure of a customer to update his own CMS  scripts / Website  (Joomla, Wordpress, Drupal etc.) and the lack of resource to regularly monitor all customer activities / websites.

Therefore installing SuPHP Apache module is essential one to install on new serverslarge hosting providers as it saves the admin a lot of headache from spreading malware across all hosted servers sites ..
Some VPS admins that are security freaks tend to also install SuPHP module together with many chrooted Apache / LiteSpeed / Nginx webservers each of which running in a separate Jailed environment.

Of course using SuPHP besides giving a improved security layer to the webserver has its downsides such as increased load for the server and making Apache PHP scripts being interpretted a little bit slower than with plain Apache + PHP but performance difference while running a site on top of SuPHP is often not so drastic so you can live it up ..

Installing SuPHP on a Debian / Ubuntu servers is a piece of cake, just run the as root superuser, usual:
 

# apt-get install libapache2-mod-suphp


Once installed only thing to make is to turn off default installed Apache PHP module (without SuPHP compiled support and restart Apache webserver):
 

# a2dismod php5 …

# /etc/init.d/apache2 restart


To test the SuPHP is properly working on the Apache Webserver go into some of many hosted server websites DocumentRoot

And create new file called test_suphp.php with below content:

# vim test_suphp.php
<?php
system('id');
?>

Then open in browser http://whatever-website/test_suphp.php assuming that system(); function is not disabled for security reasons in php.ini you should get an User ID, GroupID bigger than reserved system IDs on GNU / Linux e.g. ID > UID / GID 99

Its also a good idea to take a look into SuPHP configuration file /etc/suphp/suphp.conf and tailor options according to your liking 

If different hosted client users home directories are into /home directory, set in suphp.conf

;Path all scripts have to be in

docroot=/home/


Also usually it is a good idea to set 

umask=0022 

Share this on

Windows Explorer (Open directory in command prompt preserving dir PATH) – Add Dos Prompt Here feature via tiny registry tweak

Friday, January 10th, 2014

Windows explorer dos prompt here open directory in windows command line

If you have to use Windows on system administration level, you had to use command prompt daily, thus its useful to be able to be able to open Command Line starting from desired directory with no need to copy directory Path by hand and CD to it manually.

By default Command Prompt, cmd.exe always opens itself setting a path to user home directory, reading what is defined by win system variable %USERPROFILE% or %HOMEPATH% – MS Windows equivalent of UNIX's $HOME shell variable.

To add open in  DOS Prompt Here Command Prompt option to Windows Explorer menus its necessary to apply few rules to Windows registry DB
Use above Download link and launch it and from there on clicking with right Mouse button to any directory will enable you to open directory in CMD.EXE.

Here is content of little registry tweak adding the new menu Dos Prompt Here button
Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Directory\shell\DosHere]
@="Dos &Prompt Here"

[HKEY_CLASSES_ROOT\Directory\shell\DosHere\Command]
@="cmd.exe /k cd %1"

[HKEY_CLASSES_ROOT\Drive\shell\DosHere]
@="Dos &Prompt Here"

[HKEY_CLASSES_ROOT\Drive\shell\DosHere\Command]
@="cmd.exe /k cd %1"

Windows explorer open program files or any specific directory in windows command line

windows opening directory in command line program files screenshot win 7

This little registry code is originally for Windows 2000, anyways it is compatible with all NT technology based Windowses, Add DOS Prompt Here tweak works fine on Windows XP, Windows 7 and Windows 8 (Home, Pro and Business editions).

By Mentioning $HOME its interesting to say Windows equivalent of Linux's as it might be useful to know:

linux:~# echo $HOME
/home/hipo

is

C:\\> echo %USERPROFILE%
C:\\Users\\georgi

To list all Windows Command Prompt environment variable equivalent to Linux's bash shell env / setenv is SET command), here is example output from my Winblows;

C:\Users\georgi> SET
ALLUSERSPROFILE=C:\\ProgramData
APPDATA=C:\\Users\georgi\AppData\Roaming
CLASSPATH=.;C:\\Program Files (x86)\\Java\\jre6\\lib\\ext\\QTJava.zip
CommonProgramFiles=C:\\Program Files\Common Files
CommonProgramFiles(x86)=C:\\Program Files (x86)\\Common Files
CommonProgramW6432=C:\\Program Files\\Common Files
COMPUTERNAME=GEORGI
ComSpec=C:\\Windows\\system32\\cmd.exe
DEFLOGDIR=C:\\ProgramData\\McAfee\\DesktopProtection
FP_NO_HOST_CHECK=NO
HOMEDRIVE=C:
HOMEPATH=\\Users\\georgi
LOCALAPPDATA=C:\\Users\\georgi\\AppData\\Local
NUMBER_OF_PROCESSORS=4
OS=Windows_NT
Path=C:\\Program Files\\RA2HP\\;C:\\Windows\\system32;C:\\Windows;C:\\Windows\\S
Wbem;C:\\Windows\\System32\\WindowsPowerShell\v1.0\;C:\\Program Files\\WIDCOM
oth Software\\;C:\\Program Files\\WIDCOMM\\Bluetooth Software\\syswow64;C:\\Pr
les (x86)\\Hewlett-Packard\\HP ProtectTools Security Manager\\Bin\\;C:\\Progr
\\ActivIdentity\\ActivClient\\;C:\\Program Files (x86)\\ActivIdentity\\ActivCl
\\Program Files (x86)\\QuickTime\\QTSystem\\
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_ARCHITECTURE=AMD64
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 58 Stepping 9, GenuineIntel
PROCESSOR_LEVEL=6
PROCESSOR_REVISION=3a09
ProgramData=C:\\ProgramData
ProgramFiles=C:\\Program Files
ProgramFiles(x86)=C:\\Program Files (x86)
ProgramW6432=C:\\Program Files
PROMPT=$P$G
PSModulePath=C:\\Windows\\system32\\WindowsPowerShell\\v1.0\Modules\\
PTSMInstallPath_X86=C:\\Program Files (x86)\\Hewlett-Packard\\HP ProtectToo
ity Manager\\
PUBLIC=C:\\Users\\Public
QTJAVA=C:\\Program Files (x86)\\Java\\jre6\\lib\\ext\\QTJava.zip
SESSIONNAME=Console
SystemDrive=C:
SystemRoot=C:\\Windows
TEMP=C:\\Users\\georgi\\AppData\\Local\\Temp
TMP=C:\\Users\\georgi\\AppData\\Local\\Temp
USERDNSDOMAIN=EMEA.CPQCORP.NET
USERDOMAIN=EMEA
USERNAME=georgi
USERPROFILE=C:\\Users\\georgi
VSEDEFLOGDIR=C:\\ProgramData\\McAfee\\DesktopProtection
windir=C:\\Windows
windows_tracing_flags=3
windows_tracing_logfile=C:\\BVTBin\\Tests\\installpackage\\csilogfile.log

Hope this little trick hopes someone out there.
I will be glad to hear of other cool useful windows registry tweaks?

 

Share this on

How to create user with only FTP access on Linux

Saturday, May 11th, 2013

Linux access only to ftp How to prohibit ssh access on GNU Linux

Creating user with access only through FTP is vital in daily routine system administration job. The reason why it is good to disable SSH access to users which don't need it is of course better security. Disabling access to ssh shell for users which don't need it prevents you for user to run malicious code usually exploits or some DDoS Fork bombs – like the infamous Linux shell Denial of Service string;

:(){ :|:&};:

Better not try above string on productive server 😉
So back to the topic here how to add Linux FTP only user;

1. Create a regular user with adduser or useradd (depending) on GNU / Linux distribution

adduser is available across most Linux distributions nowadays, however I remember in past there was some distros which had useradd instead. Anyways for most adduser should be ok. As of time of writting both 3 main stream Linux distributions Slackware, Debian and Fedora has adduser.

linux:~#  adduser new-user-for-ftp-only

Adding user `new-user-for-ftp-only' …
Adding new group `new-user-for-ftp-only' (1006) …
Adding new user `new-user-for-ftp-only' (1005) with group `new-user-for-ftp-only' …
Creating home directory `/home/new-user-for-ftp-only' …
Copying files from `/etc/skel' …
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Changing the user information for new-user-for-ftp-only
Enter the new value, or press ENTER for the default
    Full Name []: New Linux User Only for FTP access  
    Room Number []:
    Work Phone []:
    Home Phone []:
    Other []:
Is the information correct? [Y/n] Y

linux:~#

2. Change user shell /bin/bash to /bin/false

Again depending on Linux distribution by default /bin/bash /bin/sh or /bin/whatever shell will get added. To make just created user access to SSH disabled. Change shell to /bin/false – a tiny program which just returns a FALSE value and quits immediately.

There are two ways to do so;

a) Edit directly /etc/passwd with vim / joe

linux:~# vim /etc/passwd

Go to end of file and find the record for user, should be smth like:

 

new-user-for-ftp-only:x:1005:1006:New Linux User Only for FTP access,,,:/home/new-user-for-ftp-only:/bin/bash

Change to;

new-user-for-ftp-only:x:1005:1006:New Linux User Only for FTP access,,,:/home/new-user-for-ftp-only:/bin/false

b) Use chsh cmd

linux:~# chsh new-user-for-ftp-only

Changing the login shell for new-user-for-ftp-only
Enter the new value, or press ENTER for the default
    Login Shell [/bin/bash]: /bin/false

linux:~# grep -i new-user-for-ftp-only /etc/passwd

new-user-for-ftp-only:x:1005:1006:New Linux User Only for FTP access,,,:/home/new-user-for-ftp-only:/bin/false

3. Testing if ssh access to new user is disabled

linux:~# ssh new-user-for-ftp-only@localhost

new-user-for-ftp-only@localhost's password:
Linux noah 2.6.32-5-amd64 #1 SMP Mon Feb 25 00:26:11 UTC 2013 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Connection to localhost closed.

Share this on

How to search text strings only in hidden files dot (.) files within a directory on Linux and FreeBSD

Saturday, April 28th, 2012

how-to-search-hidden-files-linux-freebsd-logo_grep
If there is necessity to look for a string in all hidden files with all sub-level subdirectories (be aware this will be time consuming and CPU stressing) use:
 

hipo@noah:~$ grep -rli 'PATH' .*

./.gftp/gftprc
./.gftp/cache/cache.OOqZVP
….

Sometimes its necessery to only grep for variables within the first-level directories (lets say you would like to grep a 'PATH' variable set, string within the $HOME directory, the command is:

hipo@noah:~$ grep PATH .[!.]*

.profile:PATH=/bin:/usr/bin/:${PATH}
.profile:export PATH
.profile:# set PATH so it includes user's private bin if it exists
.profile: PATH="$HOME/bin:$PATH"
.profile.language-env-bak:# set PATH so it includes user's private bin if it exists
.profile.language-env-bak: PATH="$HOME/bin:$PATH"
.viminfo:?/PATH.xcyrillic: XNLSPATH=/usr/X11R6/lib/X11/nls
.xcyrillic: export XNLSPATH

The regular expression .[!.]*, means exclude any file or directory name starting with '..', e.g. match only .* files

Note that to use the grep PATH .[!.]* on FreeBSD you will have to use this regular expression in bash shell, the default BSD csh or tsch shells will not recognize the regular expression, e.g.:

grep PATH '.[!.]*'
grep: .[!.]*: No such file or directory

Hence on BSD, if you need to look up for a string within the home directory, hidden files: .profile .bashrc .bash_profile .cshrc run it under bash shell:

freebsd# /usr/local/bin/bash
[root@freebsd:/home/hipo]# grep PATH .[!.]*

.bash_profile:# set PATH so it includes user's private bin if it exists
.bash_profile:# PATH=~/bin:"${PATH}"
.bash_profile:# do the same with …

Another easier to remember, alternative grep cmd is:

hipo@noah:~$ grep PATH .*
.profile:PATH=/bin:/usr/bin/:${PATH}
.profile:export PATH
.profile:# set PATH so it includes user's private bin if it exists
.profile: PATH="$HOME/bin:$PATH"
….

Note that grep 'string' .* is a bit different in meaning, as it will not prevent grep to match filenames with names ..filename1, ..filename2 etc.
Though grep 'string' .* will work note that it will sometimes output some unwanted matches if filenames with double dot in the beginning of file name are there …
That's all folks 🙂

Share this on

How to make a mirror of website on GNU / Linux with wget / Few tips on wget site mirroring

Wednesday, February 22nd, 2012

how-to-make-mirror-of-website-on-linux-wget

Everyone who used Linux is probably familiar with wget or has used this handy download console tools at least thousand of times. Not so many Desktop GNU / Linux users like Ubuntu and Fedora Linux users had tried using wget to do something more than single files download.
Actually wget is not so popular as it used to be in earlier linux days. I've noticed the tendency for newer Linux users to prefer using curl (I don't know why).

With all said I'm sure there is plenty of Linux users curious on how a website mirror can be made through wget.
This article will briefly suggest few ways to do website mirroring on linux / bsd as wget is both available on those two free operating systems.

1. Most Simple exact mirror copy of website

The most basic use of wget's mirror capabilities is by using wget's -mirror argument:

# wget -m http://website-to-mirror.com/sub-directory/

Creating a mirror like this is not a very good practice, as the links of the mirrored pages will still link to external URLs. In other words link URL will not pointing to your local copy and therefore if you're not connected to the internet and try to browse random links of the webpage you will end up with many links which are not opening because you don't have internet connection.

2. Mirroring with rewritting links to point to localhost and in between download page delay

Making mirror with wget can put an heavy load on the remote server as it fetches the files as quick as the bandwidth allows it. On heavy servers rapid downloads with wget can significantly reduce the download server responce time. Even on a some high-loaded servers it can cause the server to hang completely.
Hence mirroring pages with wget without explicity setting delay in between each page download, could be considered by remote server as a kind of DoS – (denial of service) attack. Even some site administrators have already set firewall rules or web server modules configured like Apache mod_security which filter requests to IPs which are doing too frequent HTTP GET /POST requests to the web server.
To make wget delay with a 10 seconds download between mirrored pages use:

# wget -mk -w 10 -np --random-wait http://website-to-mirror.com/sub-directory/

The -mk stands for -m/-mirror and -k / shortcut argument for –convert-links (make links point locally), –random-wait tells wget to make random waits between o and 10 seconds between each page download request.

3. Mirror / retrieve website sub directory ignoring robots.txt "mirror restrictions"

Some websites has a robots.txt which restricts content download with clients like wget, curl or even prohibits, crawlers to download their website pages completely.

/robots.txt restrictions are not a problem as wget has an option to disable robots.txt checking when downloading.
Getting around the robots.txt restrictions with wget is possible through -e robots=off option.
For instance if you want to make a local mirror copy of the whole sub-directory with all links and do it with a delay of 10 seconds between each consequential page request without reading at all the robots.txt allow/forbid rules:

# wget -mk -w 10 -np -e robots=off --random-wait http://website-to-mirror.com/sub-directory/

4. Mirror website which is prohibiting Download managers like flashget, getright, go!zilla etc.

Sometimes when try to use wget to make a mirror copy of an entire site domain subdirectory or the root site domain, you get an error similar to:

Sorry, but the download manager you are using to view this site is not supported.
We do not support use of such download managers as flashget, go!zilla, or getright

This message is produced by the site dynamic generation language PHP / ASP / JSP etc. used, as the website code is written to check on the browser UserAgent sent.
wget's default sent UserAgent to the remote webserver is:
Wget/1.11.4

As this is not a common desktop browser useragent many webmasters configure their websites to only accept well known established desktop browser useragents sent by client browsers.
Here are few typical user agents which identify a desktop browser:
 

  • Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0
  • Mozilla/5.0 (X11; Linux i686; rv:6.0) Gecko/20100101 Firefox/6.0
  • Mozilla/6.0 (Macintosh; I; Intel Mac OS X 11_7_9; de-LI; rv:1.9b4) Gecko/2012010317 Firefox/10.0a4
  • Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.2a1pre) Gecko/20110324 Firefox/4.2a1pre

etc. etc.

If you're trying to mirror a website which has implied some kind of useragent restriction based on some "valid" useragent, wget has the -U option enabling you to fake the useragent.

If you get the Sorry but the download manager you are using to view this site is not supported , fake / change wget's UserAgent with cmd:

# wget -mk -w 10 -np -e robots=off \
--random-wait
--referer="http://www.google.com" \--user-agent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6" \--header="Accept:text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5" \--header="Accept-Language: en-us,en;q=0.5" \--header="Accept-Encoding: gzip,deflate" \--header="Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7" \--header="Keep-Alive: 300"

For the sake of some wget anonimity – to make wget permanently hide its user agent and pretend like a Mozilla Firefox running on MS Windows XP use .wgetrc like this in home directory.

5. Make a complete mirror of a website under a domain name

To retrieve complete working copy of a site with wget a good way is like so:

# wget -rkpNl5 -w 10 --random-wait www.website-to-mirror.com

Where the arguments meaning is:
-r – Retrieve recursively
-k – Convert the links in documents to make them suitable for local viewing
-p – Download everything (inline images, sounds and referenced stylesheets etc.)
-N – Turn on time-stamping
-l5 – Specify recursion maximum depth level of 5

6. Make a dynamic pages static site mirror, by converting CGI, ASP, PHP etc. to HTML for offline browsing

It is often websites pages are ending in a .php / .asp / .cgi … extensions. An example of what I mean is for instance the URL http://php.net/manual/en/tutorial.php. You see the url page is tutorial.php once mirrored with wget the local copy will also end up in .php and therefore will not be suitable for local browsing as .php extension is not understood how to interpret by the local browser.
Therefore to copy website with a non-html extension and make it offline browsable in HTML there is the –html-extension option e.g.:

# wget -mk -w 10 -np -e robots=off \
--random-wait \
--convert-links http://www.website-to-mirror.com

A good practice in mirror making is to set a download limit rate. Setting such rate is both good for UP and DOWN side (the local host where downloading and remote server). download-limit is also useful when mirroring websites consisting of many enormous files (documental movies, some music etc.).
To set a download limit to add –limit-rate= option. Passing by to wget –limit-rate=200K would limit download speed to 200KB.

Other useful thing to assure wget has made an accurate mirror is wget logging. To use it pass -o ./my_mirror.log to wget.
 

Share this on

How to configure pine (alpine) console client to work with vpopmail pop3 and imap protocol

Monday, June 13th, 2011

I needed to check my mail via ssh connection, as my installed squirrelmail is curently broken and I’m away from my own personal computer.

I did some online research on how this can be achieved and thanksfully I finallyfound a way to check my pop3 and imap mailbox with a console client called alpine , better known in unix community under the name pine .

I installed pine on my Debian with apt:


debian:~# apt-get install alpine

Here is my pine configuration file .pinerc used to fetch my mail with pine:

a .pinerc conf file to check my pop3 mail

To use that file I placed it in my home directory ~/ , e.g.:


debian:~# wget http://www.pc-freak.net/files/.pinerc
...

To attune the pop3 server configuration in the sample .pinerc above one needs to change the value of:


inbox-path=

For example to configure pine to fetch mail from the pop3 server mail.pc-freak.net and store it locally in my home directory within a file called INBOX
I have configured the inbox-path .pinerc variable to look like so:


inbox-path={mail.pc-freak.net/pop3/user=hipo@pc-freak.net}INBOX

In above configuration’s inbox-path variable configuration the /pop3/ specifies I want to fetch my mail via the pop3 protocol , if one wants to use imap this has to be substituted with /imap/

The value user=hipo@pc-freak.net specifies my vpopmail created user which in my case is obviously hipo@pc-freak.net

The other variables which are good to be changed in .pinerc config are:


personal-name=

This variable has to be set to the name of the Email Sender which will be set, if pine is used to send email.

I also changed the user-domain variable as it’s used to set the domain name from which the pine client will send the emails from:

As my domain is pc-freak.net I’ve set the domain name variable to be:


user-domain=pc-freak.net

Now after launching pine it prompted me for my email password, putting in the pass did fetch all my new unread mails via pop3 protocol.

The only annoying thing was that each time I quit pine and start it up again, I’m now asked to enter the email password.

This behaviour is really shitty, but thanksfully one can easily workaround that by letting pine be constantly running detached in gni screen session.

Share this on