Posts Tagged ‘file’
Thursday, June 12th, 2014 If you're a Linux geek and forced to work on Windows for some reason. You will probably want to add some Linux & Feel to Windows console. Yesterday in a discussion with a colleague who is considered to be our team's Windows Guru, I noticed his Windows 7 command line looks slightly different and asked him, "is it some cmd.exe customization?" and this is how I learned about existence of ConEmu and Clink.
ConEmu is advanced console window where you can run any shell of your choice.
ConEmu (Console Emulator) + Clink combination is very much like MobaXterm (tabbed SSH client and win cmd line) in many sense except it doesn't have an embbed ssh support client. By using Conemu + Clink (Powerful bash style line editing) wrappers to cmd.exe you get the invaluable Linux / UNIX tab completion, command line shortcuts like CTRL + R, CTRL + A, CTRL + P etc. into Windows OS. ConEmu is very useful in Windows environments when you have to heavily use Windows PowerShell
If you're get used to Linux's Midnight Commander (MC) and you're looking for MC for Windows you will certainly want to check out Far Manager (File and Archive Manager). Far Manager provides simple and intuitive ncurses (mc) like interface on Windows and supports view, edit, copy rename and a lot of other basic operations supported also by mc.
Clink combines the native Windows shell cmd.exe with the powerful command line editing features of the GNU Readline library, which provides rich completion, history, and line-editing capabilities. Readline is well-known and used in Unix shell Bash, the standard shell for Mac OS X and most of Linux distributions.
In my view coolest Clink command shortcut is reverse-search allowing you to auto-type by letters commands (from command line history), lets say you want to invoke dir – you press CTRL + r and type D and all occurances earlier typed starting with "d", pops up as possible choice.
Everyone who had to administer Windows servers know how annoying is that by default cmd.exe doesn't support window resize.
Using ConEmu solves that as it supports resize of command prompt Window (horizontal and vertical resize).
ConEmu has a status bar, support (configuration colors) and even transperancy but most importantly for old school dos geeks is that it supports integration with dosbox run old DOS applications (games) in Windows 7.
To use the tools use gui installers, here is latest mirror copy of Conemu (for download), current latest Far Manager release and latest present release of Clink (download link). Once installed it is a must to spend some time configure ConEmu to match your preferences. To configure it press with right mouse button on top of Cemu window and do the needful 🙂
Once everything is installed, you will have to get into the habit to launch ConEmu instead of cmd.exe (as ConEmu doesn't interfere with cmd.exe). To launch far manager from ConEmu command line simply type:
C:> far
To open new tabs in ConEmu the well known Firefox used CTRL + T will not work, but 'open new tab' key shortcut can be customized and also set to CTRL + T. Default key shortcut for opening new tab in ConEmu is 'Windows Button' + SHIFT + W. While Opening new consoles in ConEmu, there is also the useful option of Splitting console (top and right).
Here is also list with all Clink command shortcuts
C-@ : set-mark
C-a : beginning-of-line
C-b : backward-char
C-c : ctrl-c
C-d : delete-char
C-e : end-of-line
C-f : forward-char
C-g : abort
C-h : backward-delete-char
C-i : clink-completion-shim
C-j : accept-line
C-k : kill-line
C-l : clear-screen
C-m : accept-line
C-n : next-history
C-p : previous-history
C-q : reload-lua-state
C-r : reverse-search-history
C-s : forward-search-history
C-t : transpose-chars
C-u : unix-line-discard
C-v : paste-from-clipboard
C-w : unix-word-rubout
C-y : yank
C-z : undo
C-] : character-search
C-_ : undo
A-C-c : copy-line-to-clipboard
A-C-e : expand-env-vars
A-C-g : abort
A-C-h : backward-kill-word
A-C-i : tab-insert
A-C-j : vi-editing-mode
A-C-m : vi-editing-mode
A-C-r : revert-line
A-C-u : up-directory
A-C-y : yank-nth-arg
A-C-[ : complete
A-C-] : character-search-backward
A- : set-mark
A-# : insert-comment
A-& : tilde-expand
A-* : insert-completions
A– : digit-argument
A-. : yank-last-arg
A-0 : digit-argument
A-1 : digit-argument
A-2 : digit-argument
A-3 : digit-argument
A-4 : digit-argument
A-5 : digit-argument
A-6 : digit-argument
A-7 : digit-argument
A-8 : digit-argument
A-9 : digit-argument
A-< : beginning-of-history
A-= : possible-completions
A-> : end-of-history
A-? : possible-completions
A- : delete-horizontal-space
A-_ : yank-last-arg
A-b : backward-word
A-c : capitalize-word
A-d : kill-word
A-f : forward-word
A-h : show-rl-help
A-l : downcase-word
A-n : non-incremental-forward-search-history
A-p : non-incremental-reverse-search-history
A-r : revert-line
A-t : transpose-words
A-u : upcase-word
A-y : yank-pop
A-~ : tilde-expand
C-x,C-g : abort
C-x,C-r : re-read-init-file
C-x,C-u : undo
C-x,C-x : exchange-point-and-mark
C-x,( : start-kbd-macro
C-x,) : end-kbd-macro
C-x,e : call-last-kbd-macro
Tags: cmd line, command, exe, explorer, Far Manager, file, line, mirror copy, support, tab completion, tabs, Windows, Windows Guru
Posted in Everyday Life, System Administration, Various, Windows | 1 Comment »
Thursday, August 7th, 2014 
If you have a sensitive data like a scan copy of your ID card, Driving License, Birth Certificate, Marriage Certificate or some revolutionary business / idea or technology and you want to transfer that over some kind of network lets say Internet vie some public unencrypted e-mail service like (Gmail.com / Yahoo Mail / Mail.com / (Bulgarian Mail Abv.bg)) etc. you will certainly want to transfer the file in encrypted form to prevent, someone sniffing your Network or someone having administrative permissions to servers of free mail where your mail data is stored.
Transferring your files in encrypted form become very important these days especially after recent Edward Snowden disclosures about American Mass Surveilance program PRISM – for those who didn't yet hear of PRISM (this is a American of America's NSA – National Security Agency aiming to sniff and log everyone's information transferred in digital form via the Internet and even Mobile Phone conversations)…
First step to mitigate surveilance is to use fully free software (100% free software) OS distribution like Trisquel GNU / Linux.
Second is to encrypt to use encryption – the process of transforming information (referred to as plaintext) using an algorithm (called cipher) to make it unreadable to anyone except those possessing special knowledge, usually referred to as a key.
There are many ways to encrypt your data on Linux and to later decrpyt it, I've earlier blogged about encryping files with GPG and OpenSSL on Linux, however encryption with GPG and OpenSSL is newer as concept than the old-school way to encrypt files on UNIX with crypt command which in Linux is replaced by mcrypt command.
mcrypt is provided by mcrypt package by default on most if not all Linux distributions, however mcrypt is not installed by default so to start using it you have to install it first.
1. Install mcrypt on Debian / Ubuntu / Mint (deb based) Linux
apt-get install –yes mcrypt
2. Install mcrypt on Fedora / CentOS rest of RPM bases Linux
yum -y install libmcrypt
3. Encrypting file with mcrypt
To get a list with all supported algorithms by mcrypt:
mcrypt –list
cast-128 (16): cbc cfb ctr ecb ncfb nofb ofb
gost (32): cbc cfb ctr ecb ncfb nofb ofb
rijndael-128 (32): cbc cfb ctr ecb ncfb nofb ofb
twofish (32): cbc cfb ctr ecb ncfb nofb ofb
arcfour (256): stream
cast-256 (32): cbc cfb ctr ecb ncfb nofb ofb
loki97 (32): cbc cfb ctr ecb ncfb nofb ofb
rijndael-192 (32): cbc cfb ctr ecb ncfb nofb ofb
saferplus (32): cbc cfb ctr ecb ncfb nofb ofb
wake (32): stream
blowfish-compat (56): cbc cfb ctr ecb ncfb nofb ofb
des (8): cbc cfb ctr ecb ncfb nofb ofb
rijndael-256 (32): cbc cfb ctr ecb ncfb nofb ofb
serpent (32): cbc cfb ctr ecb ncfb nofb ofb
xtea (16): cbc cfb ctr ecb ncfb nofb ofb
blowfish (56): cbc cfb ctr ecb ncfb nofb ofb
enigma (13): stream
rc2 (128): cbc cfb ctr ecb ncfb nofb ofb
tripledes (24): cbc cfb ctr ecb ncfb nofb ofb
mcrypt < File-To-Crypt.PDF > File-To-Crypt.PDF.cpy
Enter the passphrase (maximum of 512 characters)
Please use a combination of upper and lower case letters and numbers.
Enter passphrase:
Enter passphrase:
If crypt is invoked to create the encrypted file without OS redirects (< >), i.e.:
mcrypt -a blowfish File-To-Crypt.PDF
Please use a combination of upper and lower case letters and numbers.
Enter passphrase:
Enter passphrase:
File File-To-Crypt was encrypted.
mcrypt outputs encrypted file in .nc extension and the new file and file default mode of 0600 (read write only for root user) are set, while new file keeps the modification date of the original.
4. Decrypting file with mcrypt
Decryption of files is done mdecrypt
mdecrypt File-To-Crypt.PDF.cpy
Enter passphrase:
File File-To-Crypt.PDF.cpy was decrypted.
To make mcrypt behave in a certain way when invoked modify ~/.mcryptrd
mcrypt is also available as a module for php5 (php5-mcrypt).
Tags: American Mass Surveilance, Debian Ubuntu Linux, encryption, file, free software, GPG, How to, information, Install, Internet, Linux, password, PDF, root user, sensitive data
Posted in Everyday Life, Linux, Linux and FreeBSD Desktop, System Administration, Various | 1 Comment »
Wednesday, August 13th, 2014 
I had to fix recently another Windows XP infected with plenty of malicious malware. I've used tips from http://malwaretips.com/blogs/ and launch all suggested tools to cleanup the PC. Some few thousands of infections were cleaned and the PC started working much better than earlier, however still there was one weird issue on the Windows XP desktop there was an .exe file hanging with 0 bytes size and whenever I tried to delete the file either from GUI or command line with (del command) it refused to delete with error:
The process cannot access the file because it is being used by another process.
Well the message clearly shows, the file can't be deleted because some program was using it however I couldn't see any program that has locked the .exe file. I've checked that in (Windows Task Manager) run from
Start -> Run:
taskmgr
I've done a quick search online to see, whether someone has fixed the issues, and saw many threads talking about the issue suggesting that the issue got solved Windows XP OS re-install, as this was not an option for me because the PC's reinstall would be at least a 2 day time work, plus it contains some programs which will probably be hard to freshly install, because they're no longed supported.
Finally I've seen in this thread suggestion to try to kill Windows Explorer after you have opened windows with windows cmd.exe because this will prevent the .exe file being locked by Windows Explorer and will allow it to be deleted with del command. This actually worked. I've launched
Start -> Run
taskmgr
and
cmd.exe
Found explorer.exe process and killed it, this makes the whole desktop (icons, start menu etc. disappear) – as Windows Explorer is main managing Windows app. Then using Alt+Tab to switch to Command Prompt window deleted it with:
del Process_that_cannot_be_killed.exe
And finally succesfully deleted it 🙂
Tags: Cannot Delete, Desktop, desktop icons, exe, file, issue, online, Pc, process, threads, windows explorer, Windows Task Manager
Posted in Everyday Life, System Administration, Various, Windows | No Comments »
Monday, August 18th, 2014 
tar: Error exit delayed from previous errors
error is a very common error encountered when creating archives (or backing up server configurations / websites / sql binary data). The error is quite unexplanatory and whenever creating files verbose in order to see the files added to archve in "real time" with lets say:
tar -czvf /tmp/filename_backup_date-of-backup.tar.gz /home/websites /home/sql
its pretty hard to track on exactly which file is the backup producing the Error exit delayed from previous errors, this is especially the case whenever adding to archive directories containing millions of tiny few kilobyte sized files. Many novice on uncautious Linux admins , might simply ignore the warning if they're in a hurry / are having excessive work to be done as there will be .tar.gz backup produced and whenever uncompressed most of the files are there and the backup error would seem not of a big issue.
However as backuping files is vital stuff, especially when moving the files from a server to be decomissioned you have to be extra careful and make the backup properly, e.g. figure out the cause of the error, to do so log the full output of tar operations with tee command, like so:
tar -czvf /tmp/filename_backup_date-of-backup.tar.gz /home/websites/ /home/sql | tee /tmp/backup_tar_full_output.log
Then you will have to review the file and lookup for errors with less search string – / (slash) – look for "error" and "permission den" keywords and this should point you to what is causing the error. In cases when millions of files are to be archived, the log might grow really big and hard to process, therefore a much quicker way to understand what's happening is to only log and show in shell standard output last file error with > (shell redirect):
tar -czvf /tmp/filename_backup_date-of-backup.tar.gz /home/websites /home/sql > /tmp/backup_failure-cause.log
tar: www.ur-website.com-http/2.0.63/conf/tnsnames.ora.20080918: Cannot open: Permission denied
tar: Removing leading `/' from member names
The error indicates clearly the cause of error is lack of Permissions to read the file tnsnames.ora.20080918 so solution is to either grant permissions to non-root user with (chmod / chown) cmds, in my case grant perms to user hipo with which tar is ran, or run again the website backup with superuser, I usually just run with root user to prevent tampering with original permissions, e.g. to solve the error, either:
$ su root
# tar -czvf /tmp/filename_backup_date-of-backup.tar.gz /home/websites /home/sql
Or even better if sudo is installed and user is added to /etc/sudoers file
$ sudo tar -czvf /tmp/filename_backup_date-of-backup.tar.gz /home/websites /home/sql
Though permission errors is the most often reason for:
tar: Error exit delayed from previous errors, you should keep in mind that in some cases the error might be caused due to failing RAID membered disk drive or single hdd failure on systems that are not in some RAID array
Tags: Cannot, cause and solution, file, gz, home, log, root user, server, sudo, tar
Posted in Linux, Linux Backup tools, Remote System Administration, System Administration | No Comments »
Friday, November 28th, 2014 
I was working on a Application Migration Project whose goal was to Install a business application called Asset Guardian and then move current company Data from the old server to the new AppServer.
For that purpose the company vendor Asset Guardian shipped to a Public access FTP, a huge (12GB) ZIP archive file which had to be transferred into a well secured DMZ-ed corporation network with various implemented Traffic Shaping Network policies, a resctrictive firewall allowing access to Internal Network only and to Few (Restrictive configured) Proxy Server IPs on port 80 and 8080.
One of the proxy servers allowed access to the Internet and I set this one and tried downloading the Huge Archive file with the Windows 2012 server default browser Internet Explorer 10, though the download started it kept slow between ~ 300 – 500KB sec and when reached 3.4GB download failed. I tried resuming the download but as the remote Public FTP server where files resides doesn't support FTP RESUME function.
I thought it might be that Internet Explorer is badly managing the download so, I go forward and installed Portable Firefox (mirrored version 33.1.1 is here). Re-running download with firefox also failed, so the next logical step was for me to try downloading with Windows version of Wget (Wget) and with Portable Free Download Manager 3.9.14.1481 (mirrored here) using both of them was unable to complete download (probably due to firewall or Proxy screwing the proxy inspected traffic) thus I had to look for another way to copy the enormous archive into the company network.
To get around the issue I tried to download the file from FTP to another Server running Apache and tried re-downloading the big file archive (Asset-Guardian-data.zip) from Apache Webserver via HTTP protocol, this download method didn't work neither using plain HTTP protocol for download when downloaded file reached (3.4GB), thus I realized this is due to restrictive Proxy servers (dropping file downloads) bigger than 3.4GBs).
Then to be able to transfer the huge 12GB file, it seems the only left option was to to chop the big file on smaller file chunks and transfer them one by one.
In my case I had the Asset-Guardian-Files.zip transferred already to the Apache (Webserver) host which is running Linux so basicly the task was to Transfer Big archive file between the SuSE Linux Enterprise Server (SLES) 11 and Windows 2012 Server.
Quickesy way to do that is by using UNIX split command, i.e.:
split -b 1024m Asset-Guardian-Files.zip
The outputted files each 1GB are with naming (xaa, xab, xac, xad, xae, xaf, gaf etc.) in same folder where split command is run:
To later merge the files on the Windows 2012 server (copy) command is used:
copy /b file1 + file2 + file3 + file4 filetogether
In my case the command to issue on Win 2012 server was:
copy /b xaa + xab + xac + xae + xae + xaf + xaf + xag xah xai xaj xak Asset-Guardian-files.zip
This method to chop and transfer the file is most simple one and it doesn't require the two servers to have WinRAR or Console RAR / unrar installed.
If instead of Copy Huge File from Linux -> Windows host you need to copy too big file (lets say 100GB) between 2 Windows servers (Windows server host A and Windows server Host B – both situated in different firewall corporate networks) you will need to download to Win Host A and use Windows UNIX split equivalent tool called sfk (The Swiss File Knife) , sfk has port also for Mac OS so in case of need for need for migrating huge archive file from Mac OS X host it will serve as Linux's split – I've made SFK (current version) mirror here.
Another way to cut the 12GB file in parts and transfer to destination host via HTTP was to use rar (on the Linux host), then download the file on Win 2012 server and use Winrar Portable Free to extract the multiple files:
To make archive separate in parts set out to certain size out of a huge file with rar on Linux use:
cd /var/www
rar -a -v1000000k Asset_Guardian_Files.splitted.rar /var/www/Asset_Guardian_Files.zip
10000000Kbs = 10000000/1024 = 976MBs, hence rar produced parts will be sized to 976MB rar parts.
To find out archives check for *splitted*.rar in your /var/www
ls -al /var/www/*splitted*.rar
-rw-r–r– 1 root root 1048576 ное 28 18:34 Asset-Guardian-Files.splitted.part1.rar
-rw-r–r– 1 root root 1048576 ное 28 18:34 Asset-Guardian-Files.splitted.part2.rar
-rw-r–r– 1 root root 1048576 ное 28 18:34 Asset-Guardian-Files.splitted.part3.rar
-rw-r–r– 1 root root 1048576 ное 28 18:34 Asset-Guaridna-Filse.splitted.part4.rar
…
Then to download the files M$ Win 2012 server IE (http://my-linux-host.com/Asset-Guardian-Files.splitted.part1.rar, http://my-linux-host.com/Asset-Guardian-Files.splitted.part2.rar. etc.)
Thanks God, Problem Solved 🙂
Tags: application, archive files, chop, command, copy, data, DMZ, downloads, file, How to, Linux Create, Migration Project, rar, root root, transfer, Windows, Winrar Portable Free, working, zip
Posted in Business Management, Everyday Life, System Administration, Various, Web and CMS, Windows | No Comments »
Monday, December 1st, 2014 
Recently, I've blogged on how to move large files from source to destination server in parts on a slow / restricted networks or whenever the media is limtied in size. This is not a common scenario but it happens so if you're admin sooner or later you will need that. I give example with UNIX's split and unrar. However strip's file naming can get you insane (in case if you don't want to use cstrip command – split a file into sections determined by context lines instead) plus normal split Linux / *nix command doesn't support compression and encryption. On the other side on many Company internal Networks with Windows server hosts running – Winblows (2003, 2008, 1012) for security purposes it might be that WinRar is not installed, thus you might need to transfer the file parted between the GNU / Linux server and Windows server in standard OS supported by Windows ZIP format. Assuming that you have root (admin) access to the Linux host you can then archive your file in parts using ZIP encryption algorithm with 7zip.
1. Installing 7zip on CentOS / Fedora / RHEL and other Redhat based Linuces
If the Linux server is running:
Fedora / CentOS / RHEL and you don't have 7zip installed yet install it with:
yum -y install p7zip
According to distros version it might be the name could be a bit different if p7zip is different to find the one you need search with:
yum search p7zip
and install whatever you need
2. Installing 7zip on Debian / Ubuntu and other Debian based servers
apt-get install –yes p7zip-full
Depending on Deb based distro just like with fedora if p7zip-full pack is not installable, check 7zip's package distro version:
apt-cache search p7zip
3. Archiving ZIP file in multiple (sized) parts on GNU / Linux
7z a -v512m Large-file-separated-in-multi-parts.zip Large-Many-Gigabytes-File.SQL
This would output multiple files:
Large-file-separated-in-multi-parts.zip.001, Large-file-separated-in-multi-parts.zip.002, Large-file-separated-in-multi-parts.zip.003, Large-file-separated-in-multi-parts.004 etc.
If you want to add security to the transferred file to protect newly created ZIP archive with password use following command:
7z a -v512m Large-file-separated-in-multi-parts.zip Large-Many-Gigabytes-File.SQL
7-Zip [64] 9.20 Copyright (c) 1999-2010 Igor Pavlov 2010-11-18
p7zip Version 9.20 (locale=bg_BG.UTF-8,Utf16=on,HugeFiles=on,2 CPUs)
Scanning
Creating archive Large-file-separated-in-multi-parts.zip
Enter password (will not be echoed) :
…
Once you have transferred all the many parts via (SSH/ FTPS or not preferrably HTTP / HTTPS / FTP) place them in the same folder and use Windows standard ZIP to unarchive.
If the archived 7zip files are to be unarchived on another Linux host (in case if multi part zip transfer is between Linux -> Linux hosts) to unarchive, parted files:
7z x Large-file-separated-in-multi-parts.zip.* …
Tags: case, Creating, data, file, Linux, need, password, root admin, security, SQL, use, Windows, zip
Posted in Everyday Life, Linux, Linux and FreeBSD Desktop, System Administration, Various, Windows | 1 Comment »
Tuesday, December 2nd, 2014 
If you're migrating a website from Apache Webserver to Nginx to boost performance and better Utilize your servers hardware and the websites (Virtualhosts) has sections with implemented Apache .htaccess / .htaccess password authentication, you will have to migrate also Apache directory password protection to Nginx.
This is not a hard task as NginX's password protection uses same password format as Apache and Nginx password protection files are generated with standard htpasswd part of apache2-utils package (on Debian / Ubuntu servers) and httpd-tools on CentOS / Fedora / RHEL. If you're migrating the Apache websites to Nginx on a fresh new installed server and website developers are missing htpasswd tool to install it depending on Linux distro:
On Debian / Ubuntu deb based servers, install htpasswd with:
apt-get install –yes apache2-utils
…
On CentOS / Fedora … other RPM based servers:
yum -y install httpd-tools
…
Once installed if you need to protect new section site still being in development with password with Nginx, do it as usual with htpasswd
htpasswd -c /home/site/nginx-websitecom/.htpaswd admin
Note that if .htpasswd file has already exist and has other user records, to not overwritted multiple users / passes and let all users in file login to Nginx HTTP auth with separate passwords, do:
htpasswd /var/www/nginx-websietcom/.htpasswd elijah
Now open config file of Nginx Vhost and modify it to include configuration like this:
server {
listen 80;
server_name www.nginx-website.com nginx-website.com;
root /var/www/www.nginx-website.com/www;
[…]
location /test {
auth_basic "Restricted";
auth_basic_user_file /var/www/www.example.com/.htpasswd;
}
[…]
}
Do it for as many Vhosts as you have and to make the new settings take affect restart Nginx:
/etc/init.d/nginx restart
Enjoy 🙂
Tags: com, file, htpasswd, HTTP, passwords, protection, servers, var, website, www
Posted in Linux, Nginx, Various, Web and CMS | 3 Comments »
Tuesday, February 10th, 2015 
I've been recently writting this Apache webserver / Tomcat / JBoss / Java decomissioning bash script. Part of the script includes extraction from httpd.conf of DocumentRoot variable configured for Apache host.
I was using following one liner to grep and store DocumentRoot set directory into new variable:
documentroot=$(grep -i documentroot /usr/local/apache/conf/httpd.conf | awk '{ print $2 }' |sed -e 's#"##g');
Above line greps for documentroot prints 2nd column of the matchi (which is the Apache server set docroot and then removes any " chars).
However I faced the issue that parsed string contained in $documentroot variable there was mysteriously containing r – return carriage – this is usually Carriage Return (CR) sent by Mac OS and Apple computers. For those who don't know the End of Line of files in UNIX / Linux OS-es is LF – often abreviated as n – often translated as return new line), while Windows PCs use for EOF CR + LF – known as the infamous rn. I was running the script from the server which is running SuSE SLES 11 Linux, meaning the CR + LF end of file is standardly used, however it seem someone has editted the httpd.conf earlier with a text editor from Mac OS X (Terminal). Thus I needed a way to remove the r from CR character out of the variable, because otherwise I couldn't use it to properly exec tar to archive the documentroot set directory, cause the documentroot directory was showing unexistent.
Opening the httpd.conf in standard editor didn't show the r at the end of
"directory", e.g. I could see in the file when opened with vim
DocumentRoot "/usr/local/apache/htdocs/site/www"
However obviously the r character was there to visualize it I had to use cat command -v option (–show-nonprinting):
cat -v /usr/local/apache/conf/httpd.conf
…
DocumentRoot "/usr/local/apache/htdocs/site/wwwr"
…
1. Remove the r CR with bash
To solve that with bash, I had to use another quick bash parsing that scans through $directory and removes r, here is how:
documentroot=${documentroot%$'r'}
It is also possible to use same example to remove "broken" Windows rn Carriage Returns after file is migrated from Windows to Liunx / FreeBSD host:
documentroot=${documentroot%$'rn'}
2. Remove r Carriage Return character with sed
Other way to do remove (del) Windows / Mac OS Carriage Returns in case if Migrating to UNIX is with sed (stream editor).
sed -i s/r// filename >> filename_out.txt
3. Remove r CR character with tr
There is a third way also to do it with (tr) – translate or delete characters old shool *nix command:
tr -d 'r' < file_with_carriagereturns > file_without_carriage_returns
4. Remove r CRs with awk (pattern scanning and processing language)
awk 'sub("$", "r")' inputf_with_crs.txt > outputf_without_crs.txt
5. Delete r CR with VIM editor
:%s/r//g
6. Converting file DOS / UNIX OSes with dos2unix and unix2dos command line tools
For sysadmins who don't want to bother with writting code to convert CR when moving files between Windows and UNIX hosts there are dos2unix and unix2dos installable commands.
All done Cheers ! 🙂
Tags: apache, bash script, bash shell, conf, configured, Delete, directory, documentroot, file, hidden, httpd, vim, Windows
Posted in Curious Facts, Everyday Life, Programming, System Administration, Various | No Comments »
Sunday, October 30th, 2011 
Have you ever been in need to execute some commands scheduled via a crontab, every let’s say 5 seconds?, naturally this is not possible with crontab, however adding a small shell script to loop and execute a command or commands every 5 seconds and setting it up to execute once in a minute through crontab makes this possible.
Here is an example shell script that does execute commands every 5 seconds:
#!/bin/bash
command1_to_exec='/bin/ls';
command2_to_exec='/bin/pwd';
for i in $(echo 1 2 3 4 5 6 7 8 9 10 11); do
sleep 5;
$command1_to_exec; $command2_to_exec;
done
This script will issue a sleep every 5 seconds and execute the two commands defined as $command1_to_exec and $command2_to_exec
Copy paste the script to a file or fetch exec_every_5_secs_cmds.sh from here
The script can easily be modified to execute on any seconds interval delay, the record to put on cron to use with this script should look something like:
# echo '* * * * * /path/to/exec_every_5_secs_cmds.sh' | crontab -
Where of course /path/to/exec_every_5_secs_cmds.sh needs to be modified to a proper script name and path location.
Another way to do the on a number of seconds program / command schedule without using cron at all is setting up an endless loop to run/refresh via /etc/inittab with a number of predefined commands inside. An example endless loop script to run via inittab would look something like:
while [ 1 ]; do
/bin/ls
sleep 5;
done
To run the above sample never ending script using inittab, one needs to add to the end of inittab, some line like:
mine:234:respawn:/path/to/script_name.sh
A quick way to add the line from consone would be with echo:
echo 'mine:234:respawn:/path/to/script' >> /etc/inittab
Of course the proper paths, should be put in:
Then to load up the newly added inittab line, inittab needs to be reloaded with cmd:
# init q
I've also red, some other methods suggested to run programs on a periodic seconds basis using just cron, what I found in stackoverflow.com's as a thread proposed as a solution is:
* * * * * /foo/bar/your_script
* * * * * sleep 15; /foo/bar/your_script
* * * * * sleep 30; /foo/bar/your_script
* * * * * sleep 45; /foo/bar/your_script
One guy, even suggested a shorted way with cron:
0/15 * * * * * /path/to/my/script
Tags: bashcommand, basis, com, command, command2, copy, copy paste, course, cron, crontab, echo 1, echo echo, endless loop, exec, file, foo, foo bar, freebsd, gnu linux, How to, init, inittab, interval, line, Linux, location, loop, mine, minute, name, nbsp, number, Path, predefined commands, pwd, record, refresh, respawn, run, script, script name, secs, SHA, Shell, shell script, sleep, something, stackoverflow, thread, time, time interval, way
Posted in Linux, Linux and FreeBSD Desktop, System Administration, Various | 1 Comment »
Friday, July 8th, 2011 
Some long time ago I’ve written an article Optimizing Linux tcp/ip networking
In the article I’ve examined a number of Linux kernel sysctl variables, which significantly improve the way TCP/IP networking is handled by a non router Linux based servers.
As the time progresses I’ve been continuing to read materials on blogs and internet sites on various tips and anti Denial of Service rules which one could apply on newly installed hosting (Apache/MySql/Qmail/Proxy) server to improve webserver responce times and tighten the overall security level.
In my quest for sysctl 😉 I found a few more handy sysctl variables apart from the old ones I incorporate on every Linux server I adminstrate.
The sysctl variables improves the overall network handling efficiency and protects about common SYN/ACK Denial of service attacks.
Here are the extra sysctl variables I started incorporating just recently:
############ IPv4 Sysctl Settings ################
#Enable ExecShield protection (randomize virtual assigned space to protect against many exploits)
kernel.randomize_va_space = 1
#Increase the number of PIDs processes could assign this is very needed especially on more powerful servers
kernel.pid_max = 65536
# Prevent against the common 'syn flood attack'
net.ipv4.tcp_syncookies = 1
# Controls the use of TCP syncookies two is generally a better idea, though you might experiment
#net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2
##################################################
#
############## IPv6 Sysctl Settings ################
# Number of Router Solicitations to send until assuming no routers are present.
net.ipv6.conf.default.router_solicitations = 0
# Accept Router Preference in RA? Again not necessery if the server is not a router
net.ipv6.conf.default.accept_ra_rtr_pref = 0
# Learn Prefix Information in Router Advertisement (Unnecessery) for non-routers
net.ipv6.conf.default.accept_ra_pinfo = 0
# disable accept of hop limit settings from other routers (could be used for DoS)
net.ipv6.conf.default.accept_ra_defrtr = 0
# disable ipv6 global unicasts server assignments
net.ipv6.conf.default.autoconf = 0
# neighbor solicitations to send out per address (better if disabled)
net.ipv6.conf.default.dad_transmits = 0
# disable assigning more than 1 address per network interface
net.ipv6.conf.default.max_addresses = 1
#####################################################
To use this settings paste the above sysctl variables in /etc/sysctl.conf and ask sysctl command to read and apply the newly added conf settings:
server:~# sysctl -p
...
Hopefully you should not get errors while applying the sysctl settings, if you get some errors, it’s possible some of the variable is differently named (depending on the Linux kernel version) or the Linux distribution on which sysctl’s are implemented.
For some convenience I’ve created unified sysctl variables /etc/sysct.conf containing the newly variables I started implementing to servers with the ones I already exlpained in my previous post Optimizing Linux TCP/IP Networking
Here is the optimized / hardened sysctl.conf file for download
I use this exact sysctl.conf these days on both Linux hosting / VPS / Mail servers etc. as well as on my personal notebook 😉
Here is also the the complete content of above’s sysctl.conf file, just in case if somebody wants to directly copy/paste it in his /etc/sysctl.conf
# Sysctl kernel variables to improve network performance and protect against common Denial of Service attacks
# It's possible that not all of the variables are working on all Linux distributions, test to make sure
# Some of the variables might need a slight modification to match server hardware, however in most cases it should be fine
# variables list compiled by hip0
### https://www.pc-freak.net
#### date 08.07.2011
############ IPv4 Sysctl Kernel Settings ################
net.ipv4.ip_forward = 0
# ( Turn off IP Forwarding )
net.ipv4.conf.default.rp_filter = 1
# ( Control Source route verification )
net.ipv4.conf.default.accept_redirects = 0
# ( Disable ICMP redirects )
net.ipv4.conf.all.accept_redirects = 0
# ( same as above )
net.ipv4.conf.default.accept_source_route = 0
# ( Disable IP source routing )
net.ipv4.conf.all.accept_source_route = 0
# ( - || - )net.ipv4.tcp_fin_timeout = 40
# ( Decrease FIN timeout ) - Useful on busy/high load server
net.ipv4.tcp_keepalive_time = 4000
# ( keepalive tcp timeout )
net.core.rmem_default = 786426
# Receive memory stack size ( a good idea to increase it if your server receives big files )
##net.ipv4.tcp_rmem = "4096 87380 4194304"
net.core.wmem_default = 8388608
#( Reserved Memory per connection )
net.core.wmem_max = 8388608
net.core.optmem_max = 40960
# ( maximum amount of option memory buffers )
# tcp reordering, increase max buckets, increase the amount of backlost
net.ipv4.tcp_max_tw_buckets = 360000
net.ipv4.tcp_reordering = 5
##net.core.hot_list_length = 256
net.core.netdev_max_backlog = 1024
#Enable ExecShield protection (randomize virtual assigned space to protect against many exploits)
kernel.randomize_va_space = 1
#Increase the number of PIDs processes could assign this is very needed especially on more powerful servers
kernel.pid_max = 65536
# Prevent against the common 'syn flood attack'net.ipv4.tcp_syncookies = 1
# Controls the use of TCP syncookies two is generally a better idea, though you might experiment
#net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2
###################################################
############## IPv6 Sysctl Settings ################
# Number of Router Solicitations to send until assuming no routers are present.
net.ipv6.conf.default.router_solicitations = 0
# Accept Router Preference in RA? Again not necessery if the server is not a router
net.ipv6.conf.default.accept_ra_rtr_pref = 0
# Learn Prefix Information in Router Advertisement (Unnecessery) for non-routersnet.
ipv6.conf.default.accept_ra_pinfo = 0
# disable accept of hop limit settings from other routers (could be used for DoS)
net.ipv6.conf.default.accept_ra_defrtr = 0
# disable ipv6 global unicasts server assignmentsnet.
ipv6.conf.default.autoconf = 0
# neighbor solicitations to send out per address (better if disabled)
net.ipv6.conf.default.dad_transmits = 0
# disable assigning more than 1 address per network interfacenet.
ipv6.conf.default.max_addresses = 1
#####################################################
# Reboot if kernel panic
kernel.panic = 20
These sysctl settings will tweaken the Linux kernel default network settings performance and you will notice the improvements in website responsiveness immediately in some cases implementing this kernel level goodies will make the server perform better and the system load might decrease even 😉
This optimizations on a kernel level are not only handy for servers, their implementation on Linux Desktop should also have a positive influence on the way the network behaves and could improve significantly the responce times of opening pages in Firefox/Opera/Epiphany Torrent downloads etc.
Hope this kernel tweakenings are helpful to someone.
Cheers 😉
Tags: adminstrate, amount, anti, apache mysql, autoconf, default, default network, default router, Denial, denial of service, denial of service attacks, exploits, file, harden, How to, imprpove, internet sites, ip networking, ipv, ipv4, ipv6, kernel, kernel level, level, Linux, Linux Security, linux server, memory, network efficiency, non, number, protection, proxy server, Qmail, quot, ra, randomize, reordering, responce, responce times, router linux, routers, rtr, security, security level, serverskernel, Service, solicitations, Source, SYN, syn flood attack, sysctl, time, variables, wmem
Posted in Computer Security, Linux, Linux and FreeBSD Desktop, System Administration | 4 Comments »
