Posts Tagged ‘multiple’

Create and Configure SSL bundle file for GoGetSSL issued certificate in Apache Webserver on Linux

Saturday, November 3rd, 2018

gogetssl-install-certificate-on-linux-howto-sslcertificatechainfile-obsolete

I had a small task to configure a new WildCard SSL for domains on a Debian GNU / Linux Jessie running Apache 2.4.25.

The official documentation on how to install the SSL certificate on Linux given by GoGetSSL (which is by COMODO was obsolete as of time of writting this article and suggested as install instructions:
 

SSLEngine on
SSLCertificateKeyFile /etc/ssl/ssl.key/server.key
SSLCertificateFile /etc/ssl/ssl.crt/yourDomainName.crt
SSLCertificateChainFile /etc/ssl/ssl.crt/yourDomainName.ca-bundle


Adding such configuration to domain Vhost and testing with apache2ctl spits an error like:

 

root@webserver:~# apache2ctl configtest
AH02559: The SSLCertificateChainFile directive (/etc/apache2/sites-enabled/the-domain-name-ssl.conf:17) is deprecated, SSLCertificateFile should be used instead
Syntax OK

 


To make issued GoGetSSL work with Debian Linux, hence, here is the few things done:

The files issued by Gogetssl.COM were the following:

 

AddTrust_External_CA_Root.crt
COMODO_RSA_Certification_Authority.crt
the-domain-name.crt


The webserver had already SSL support via mod_ssl Apache module, e.g.:

 

root@webserver:~# ls -al /etc/apache2/mods-available/*ssl*
-rw-r–r– 1 root root 3112 окт 21  2017 /etc/apache2/mods-available/ssl.conf
-rw-r–r– 1 root root   97 сеп 19  2017 /etc/apache2/mods-available/ssl.load
root@webserver:~# ls -al /etc/apache2/mods-enabled/*ssl*
lrwxrwxrwx 1 root root 26 окт 19  2017 /etc/apache2/mods-enabled/ssl.conf -> ../mods-available/ssl.conf
lrwxrwxrwx 1 root root 26 окт 19  2017 /etc/apache2/mods-enabled/ssl.load -> ../mods-available/ssl.load


For those who doesn't have mod_ssl enabled, to enable it quickly run:

 

# a2enmod ssl


The VirtualHost used for the domains had Apache config as below:

 

 

 

NameVirtualHost *:443

<VirtualHost *:443>
    ServerAdmin support@the-domain-name.com
    ServerName the-domain-name.com
    ServerAlias *.the-domain-name.com the-domain-name.com

    DocumentRoot /home/the-domain-namecom/www
    SSLEngine On
#    <Directory />
#        Options FollowSymLinks
#        AllowOverride None
#    </Directory>
    <Directory /home/the-domain-namecom/www>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride None
        Include /home/the-domain-namecom/www/htaccess_new.txt
        Order allow,deny
        allow from all
    </Directory>

    ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
    <Directory "/usr/lib/cgi-bin">
        AllowOverride None
        Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
        Order allow,deny
        Allow from all
    </Directory>

    ErrorLog ${APACHE_LOG_DIR}/error.log

    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    LogLevel warn

    CustomLog ${APACHE_LOG_DIR}/access.log combined

#    Alias /doc/ "/usr/share/doc/"
#   <Directory "/usr/share/doc/">
#       Options Indexes MultiViews FollowSymLinks
#       AllowOverride None
#       Order deny,allow
#       Deny from all
#       Allow from 127.0.0.0/255.0.0.0 ::1/128
#   </Directory>
SSLCertificateKeyFile /etc/apache2/ssl/the-domain-name.com.key
SSLCertificateFile /etc/apache2/ssl/chain.crt

 

</VirtualHost>

The config directives enabling and making the SSL actually work are:
 

SSLEngine On
SSLCertificateKeyFile /etc/apache2/ssl/the-domain-name.com.key
SSLCertificateFile /etc/apache2/ssl/chain.crt

 

The chain.crt file is actually a bundle file containing a bundle of the gogetssl CA_ROOT and RSA_Certification_Authority 3 files, to prepare that file, I've used bundle.sh small script found on serverfault.com here I've made a mirror of bundle.sh on pc-freak.net here   the script content is as follows:

To prepare the chain.crt  bundle, I ran:

 

sh create-ssl-bundle.sh _iq-test_cc.crt chain.crt
sh create-ssl-bundle.sh _iq-test_cc.crt >chain.crt
sh create-ssl-bundle.sh COMODO_RSA_Certification_Authority.crt >> chain.crt
sh create-ssl-bundle.sh bundle.sh AddTrust_External_CA_Root.crt >> chain.crt


Then I copied the file to /etc/apache2/ssl together with the-domain-name.com.key file earlier generated using openssl command earlier explained in my article how to install RapidSSL certificate on Linux

/etc/apache2/ssl was not previously existing (on Debian Linux), so to create it:

 

root@webserver:~# mkdir /etc/apache2/ssl
root@webserver:~# ls -al /etc/apache2/ssl/chain.crt
-rw-r–r– 1 root root 20641 Nov  2 12:27 /etc/apache2/ssl/chain.crt
root@webserver:~# ls -al /etc/apache2/ssl/the-domain-name.com.key
-rw-r–r– 1 root root 6352 Nov  2 20:35 /etc/apache2/ssl/the-domain-name.com.key

 

As I needed to add the SSL HTTPS configuration for multiple domains, further on I've wrote and used a tiny shell script add_new_vhost.sh which accepts as argument the domain name I want to add. The script works with a sample Skele (Template) file, which is included in the script itself and can be easily modified for the desired vhost config.
To add my multiple domains, I've used the script as follows:
 

sh add_new_vhost.sh add-new-site-domain.com
sh add_new_vhost.sh add-new-site-domain1.com


etc.

Here is the complete script as well:

 

#!/bin/sh
# Shell script to add easily new domains for virtual hosting on Debian machines
# arg1 should be a domain name
# This script takes the domain name which you type as arg1 uses it and creates
# Docroot / cgi-bin directory for the domain, create seperate site's apache log directory
# then takes a skele.com file and substitutes a skele.com with your domain name and directories
# This script's aim is to easily enable sysadmin to add new domains in Debian
sites_base_dir=/var/www/jail/home/www-data/sites/;
# the directory where the skele.com file is
skele_dir=/etc/apache2/sites-available;
# base directory where site log dir to be created
cr_sep_log_file_d=/var/log/apache2/sites;
# owner of the directories
username='www-data';
# read arg0 and arg1
arg0=$0;
arg1=$1;
if [[ -z $arg1 ]]; then
echo "Missing domain name";
exit 1;
fi

 

# skele template
echo "#
#  Example.com (/etc/apache2/sites-available/www.skele.com)
#
<VirtualHost *>
        ServerAdmin admin@design.bg
        ServerName  skele.com
        ServerAlias www.skele.com


        # Indexes + Directory Root.
        DirectoryIndex index.php index.htm index.html index.pl index.cgi index.phtml index.jsp index.py index.asp

        DocumentRoot /var/www/jail/home/www-data/sites/skelecom/www/docs
        ScriptAlias /cgi-bin "/var/www/jail/home/www-data/sites/skelecom/cgi-bin"
        
        # Logfiles
        ErrorLog  /var/log/apache2/sites/skelecom/error.log
        CustomLog /var/log/apache2/sites/skelecom/access.log combined
#       CustomLog /dev/null combined
      <Directory /var/www/jail/home/www-data/sites/skelecom/www/docs/>
                Options FollowSymLinks MultiViews -Includes
                AllowOverride None
                Order allow,deny
                allow from all
                # This directive allows us to have apache2's default start page
                # in /apache2-default/, but still have / go to the right place
#               RedirectMatch ^/$ /apache2-default/
        </Directory>

        <Directory /var/www/jail/home/www-data/sites/skelecom/www/docs/>
                Options FollowSymLinks ExecCGI -Includes
                AllowOverride None
                Order allow,deny
                allow from all
        </Directory>

</VirtualHost>
" > $skele_dir/skele.com;

domain_dir=$(echo $arg1 | sed -e 's/\.//g');
new_site_dir=$sites_base_dir/$domain_dir/www/docs;
echo "Creating $new_site_dir";
mkdir -p $new_site_dir;
mkdir -p $sites_base_dir/cgi-bin;
echo "Creating sites's Docroot and CGI directory";
chown -R $username:$username $new_site_dir;
chown -R $username:$username $sites_base_dir/cgi-bin;
echo "Creating site's Log files Directory";
mkdir -p $cr_sep_log_file_d/$domain_dir;
echo "Creating sites's VirtualHost file and adding it for startup";
sed -e "s#skele.com#$arg1#g" -e "s#skelecom#$domain_dir#g" $skele_dir/skele.com >> $skele_dir/$arg1;
ln -sf $skele_dir/$arg1 /etc/apache2/sites-enabled/;
echo "All Completed please restart apache /etc/init.d/apache restart to Load the new virtual domain";

# Date Fri Jan 11 16:27:38 EET 2008


Using the script saves a lot of time to manually, copy vhost file and then edit it to change ServerName directive, for vhosts whose configuration is identical and only the ServerName listener has to change, it is perfect to create all necessery domains, I've created a simple text file with each of the domains and run it in a loop:
 

while :; do sh add_new_vhost.sh $i; done < domain_list.txt
 

 

Ansible Quick Start Cheatsheet for Linux admins and DevOps engineers

Wednesday, October 24th, 2018

ansible-quick-start-cheetsheet-ansible-logo

Ansible is widely used (Configuration management, deployment, and task execution system) nowadays for mass service depoyments on multiple servers and Clustered environments like, Kubernetes clusters (with multiple pods replicas) virtual swarms running XEN / IPKVM virtualization hosting multiple nodes etc. .

Ansible can be used to configure or deploy GNU / Linux tools and services such as Apache / Squid / Nginx / MySQL / PostgreSQL. etc. It is pretty much like Puppet (server / services lifecycle management) tool , except its less-complecated to start with makes it often a choose as a tool for mass deployment (devops) automation.

Ansible is used for multi-node deployments and remote-task execution on group of servers, the big pro of it it does all its stuff over simple SSH on the remote nodes (servers) and does not require extra services or listening daemons like with Puppet. It combined with Docker containerization is used very much for later deploying later on inside Cloud environments such as Amazon AWS / Google Cloud Platform / SAP HANA / OpenStack etc.

Ansible-Architechture-What-Is-Ansible-Edureka

0. Instaling ansible on Debian / Ubuntu Linux


Ansible is a python script and because of that depends heavily on python so to make it running, you will need to have a working python installed on local and remote servers.

Ansible is as easy to install as running the apt cmd:

 

# apt-get install –yes ansible
 

The following additional packages will be installed:
  ieee-data python-jinja2 python-kerberos python-markupsafe python-netaddr python-paramiko python-selinux python-xmltodict python-yaml
Suggested packages:
  sshpass python-jinja2-doc ipython python-netaddr-docs python-gssapi
Recommended packages:
  python-winrm
The following NEW packages will be installed:
  ansible ieee-data python-jinja2 python-kerberos python-markupsafe python-netaddr python-paramiko python-selinux python-xmltodict python-yaml
0 upgraded, 10 newly installed, 0 to remove and 1 not upgraded.
Need to get 3,413 kB of archives.
After this operation, 22.8 MB of additional disk space will be used.

apt-get install –yes sshpass

 

Installing Ansible on Fedora Linux is done with:

 

# dnf install ansible –yes sshpass

 

On CentOS to install:
 

# yum install ansible –yes sshpass

sshpass needs to be installed only if you plan to use ssh password prompt authentication with ansible.

Ansible is also installable via python-pip tool, if you need to install a specific version of ansible you have to use it instead, the package is available as an installable package on most linux distros.

Ansible has a lot of pros and cons and there are multiple articles already written on people for and against it in favour of Chef or Puppet As I recently started learning Ansible. The most important thing to know about Ansible is though many of the things can be done directly using a simple command line, the tool is planned for remote installing of server services using a specially prepared .yaml format configuration files. The power of Ansible comes of the use of Ansible Playbooks which are yaml scripts that tells ansible how to do its activities step by step on remote server. In this article, I'm giving a quick cheat sheet to start quickly with it.
 

1. Remote commands execution with Ansible
 

First thing to do to start with it is to add the desired hostnames ansible will operate with it can be done either globally (if you have a number of remote nodes) to deploy stuff periodically by using /etc/ansible/hosts or use a custom host script for each and every ansible custom scripts developed.

a. Ansible main config files

A common ansible /etc/ansible/hosts definition looks something like that:

 

# cat /etc/ansible/hosts
[mysqldb]
10.69.2.185
10.69.2.186
[master]
10.69.2.181
[slave]
10.69.2.187
[db-servers]
10.69.2.181
10.69.2.187
[squid]
10.69.2.184

Host to execute on can be also provided via a shell variable $ANSIBLE_HOSTS
b) is remote hosts reachable / execute commands on all remote host

To test whether hour hosts are properly configure from /etc/ansible/hosts you can ping all defined hosts with:

 

ansible all -m ping


ansible-check-hosts-ping-command-screenshot

This makes ansible try to remote to remote hosts (if you have properly configured SSH public key authorization) the command should return success statuses on every host.

 

ansible all -a "ifconfig -a"


If you don't have SSH keys configured you can also authenticate with an argument (assuming) all hosts are configured with same password with:

 

ansible all –ask-pass -a "ip all show" -u hipo –ask-pass


ansible-show-ips-ip-a-command-screenshot-linux

If you have configured group of hosts via hosts file you can also run certain commands on just a certain host group, like so:

 

ansible <host-group> -a <command>

It is a good idea to always check /etc/ansible/ansible.cfg which is the system global (main red ansible config file).

c) List defined host groups
 

ansible localhost -m debug -a 'var=groups.keys()'
ansible localhost -m debug -a 'var=groups'

d) Searching remote server variables

 

# Search remote server variables
ansible localhost -m setup -a 'filter=*ipv4*'

 

 

ansible localhost -m setup -a 'filter=ansible_domain'

 

 

ansible all -m setup -a 'filter=ansible_domain'

 

 

# uninstall package on RPM based distros
ansible centos -s -m yum -a "name=telnet state=absent"
# uninstall package on APT distro
ansible localhost -s -m apt -a "name=telnet state=absent"

 

 

2. Debugging – Listing information about remote hosts (facts) and state of a host

 

# All facts for one host
ansible -m setup
  # Only ansible fact for one host
ansible
-m setup -a 'filter=ansible_eth*'
# Only facter facts but for all hosts
ansible all -m setup -a 'filter=facter_*'


To Save outputted information per-host in separate files in lets say ~/ansible/host_facts

 

ansible all -m setup –tree ~/ansible/host_facts

 

3. Playing with Playbooks deployment scripts

 

a) Syntax Check of a playbook yaml

 

ansible-playbook –syntax-check


b) Run General Infos about a playbook such as get what a playbook would do on remote hosts (tasks to run) and list-hosts defined for a playbook (like above pinging).

 

ansible-playbook –list-hosts
ansible-playbook
–list-tasks


To get the idea about what an yaml playbook looks like, here is example from official ansible docs, that deploys on remote defined hosts a simple Apache webserver.
 


– hosts: webservers
  vars:
    http_port: 80
    max_clients: 200
  remote_user: root
  tasks:
  – name: ensure apache is at the latest version
    yum:
      name: httpd
      state: latest
  – name: write the apache config file
    template:
      src: /srv/httpd.j2
      dest: /etc/httpd.conf
    notify:
    – restart apache
  – name: ensure apache is running
    service:
      name: httpd
      state: started
  handlers:
    – name: restart apache
      service:
        name: httpd
        state: restarted

To give it a quick try save the file as webserver.yml and give it a run via ansible-playbook command
 

ansible-playbook -s playbooks/webserver.yml

 

The -s option instructs ansible to run play on remote server with super user (root) privileges.

The power of ansible is its modules, which are constantly growing over time a complete set of Ansible supported modules is in its official documenation.

Ansible-running-playbook-Commands-Task-script-Successful-output-1024x536

There is a lot of things to say about playbooks, just to give the brief they have there own language like a  templates, tasks, handlers, a playbook could have one or multiple plays inside (for instance instructions for deployment of one or more services).

The downsides of playbooks are they're so hard to write from scratch and edit, because yaml syntaxing is much more stricter than a normal oldschool sysadmin configuration file.
I've stucked with problems with modifying and writting .yaml files and I should say the community in #ansible in irc.freenode.net was very helpful to help me debug the obscure errors.

yamllint (The YAML Linter tool) comes handy at times, when facing yaml syntax errors, to use it install via apt:
 

# apt-get install –yes yamllint


a) Running ansible in "dry mode" just show what ansible might do but not change anything
 

ansible-playbook playbooks/PLAYBOOK_NAME.yml –check


b) Running playbook with different users and separate SSH keys

 

ansible-playbook playbooks/your_playbook.yml –user ansible-user
 
ansible -m ping hosts –private-key=~/.ssh/keys/custom_id_rsa -u centos

 

c) Running ansible playbook only for certain hostnames part of a bigger host group

 

ansible-playbook playbooks/PLAYBOOK_NAME.yml –limit "host1,host2,host3"


d) Run Ansible on remote hosts in parallel

To run in raw of 10 hosts in parallel
 

# Run 10 hosts parallel
ansible-playbook <File.yaml> -f 10            


e) Passing variables to .yaml scripts using commandline

Ansible has ability to pre-define variables from .yml playbooks. This variables later can be passed from shell cli, here is an example:

# Example of variable substitution pass from command line the var in varsubsts.yaml if present is defined / replaced ansible-playbook playbooks/varsubst.yaml –extra-vars "myhosts=localhost gather=yes pkg=telnet"

 

4. Ansible Galaxy (A Docker Hub) like large repository with playbook (script) files

 

Ansible Galaxy has about 10000 active users which are contributing ansible automation playbooks in fields such as Development / Networking / Cloud / Monitoring / Database / Web / Security etc.

To install from ansible galaxy use ansible-galaxy

# install from galaxy the geerlingguy mysql playbook
ansible-galaxy install geerlingguy.mysql


The available packages you can use as a template for your purpose are not so much as with Puppet as Ansible is younger and not corporate supported like Puppet, anyhow they are a lot and does cover most basic sysadmin needs for mass deployments, besides there are plenty of other unofficial yaml ansible scripts in various github repos.

How to merge two or more wordpress sites from separete domains into one without impacting SEO

Monday, October 23rd, 2017

merge-multiple-wordpres-websites-howto-merge-two-or-more-websites-together-without-loosing-seo-credability

If you run multiple WordPress websites or blogs, but one of the websites is much more developed at a certain point you might decide that the best of the two worlds would be two merge two WordPress blogs into one.
 

So how can we do merge your multiple wordpress websites into single ?


1. Export content from each of the old websites you will be closing soon

First thing to do is to immediately create SQL data and WordPress sites backup, so if something goes wrong you can easily restore.

 


Next From wordpress admin area wp-admin

export-content-wordpress-website-screenshot-howto

Visit

Tools -> Export

Make sure, the All Content option is selected like shown in above screenshot and click on

Download Export File button

The downloaded files will have a copy of all yours

– Posts and pages
– Custom Post Types
– All included categories and tags

Once we have transferred the file to the Website with which we're going to merge old WP content, we need to install the Import plugin

install-wordpress-importer-to-merge-multiple-wordpress-sites

To do so navigate to:

Tools -> Import -> (Intall now)

wordpress-merge-two-or-more-websites-run-importer-screenshot.

Tools -> Run Importer

You will be asked for the file you just dumped into the second wordpress install which you'll be merging

export-and-import-wp-file-screenshot

Wordpress will start importing, your Export file and analyze it, then you will be redirected to

Import Settings Page
 

wordpress-import-settings-page-screenshot-howto-merge-sites

Assign Authors (that should be your account or the account with which the primary blog has most posts and with which you'll be merging).

Hopefully all will be fine with import and you will get the success page:

wordpress-success-page-importing-two-websites

One last step is to set properly redirects so that the transition merge from the Old Website to the New one does not impact your website Search Engine Optimization Ratings.

The redirects are made via .htaccess file it will be located in the Document Root (Or main directory) of your wordpress installed websites for example if your website is located under /var/www/wordpress-site your .htaccess file will be located under /var/www/wordpress-site/.htaccess

Open the file (SITUATED ON THE OLD WORDPRESS WEBSITE THAT YOU'LL BE MERGING) with editor directly on the server or if it is a shared hosting, download it and edit it at your side.

To the beginning of it add the following Apache ModRewrite code:
 

#Options +FollowSymLinks
RewriteEngine on
RewriteRule ^(.*)$ http://www.newsite.com/$1 [R=301,L]


Replace inside above example http://www.newsite.com with the name of the primary domain name to which we're merging (e.g. the merging domain and not the one we merge).

If you experience some problems with pictures or you like to Merge Bulk Edit categories  and tags  read import external images to wordpress as well as how to merge bulk edit categories in tags in WordPress

 

Must have software on freshly installed windows – Essential Software after fresh Windows install

Friday, March 18th, 2016

Install-update-multiple-programs-applications-at-once-using-ninite

If you're into IT industry even if you don't like installing frequently Windows or you're completely Linux / BSD user, you will certainly have a lot of friends which will want help from you to re-install or fix their Windows 7 / 8 / 10 OS. At least this is the case with me every year, I'm kinda of obliged to install fresh windowses on new bought friends or relatives notebooks / desktop PCs.

Of course according to for whom the new Windows OS installed the preferrences of necessery software varies, however more or less there is sort of standard list of Windows Software which is used daily by most of Avarage Computer user, such as:
 

I tend to install on New Windows installs and thus I have more or less systematized the process.

I try to usually stick to free software where possible for each of the above categories as a Free Software enthusiast and luckily nowadays there is a lot of non-priprietary or at least free as in beer software available out there.

For Windows sysadmins or College and other public institutions networks including multiple of Windows Computers which are not inside a domain and also for people in computer repair shops where daily dozens of windows pre-installs or a set of software Automatic updates are  necessery make sure to take a look at Ninite

ninite-automate-windows-program-deploy-and-update-on-new-windows-os-openoffice-screenshot

As official website introduces Ninite:

Ninite – Install and Update All Your Programs at Once

Of course as Ninite is used by organizations as NASA, Harvard Medical School etc. it is likely the tool might reports your installed list of Windows software and various other Win PC statistical data to Ninite developers and most likely NSA, but this probably doesn't much matter as this is probably by the moment you choose to have installed a Windows OS on your PC.

ninite-choises-to-build-an-install-package-with-useful-essential-windows-software-screenshot
 

For Windows System Administrators managing small and middle sized network PCs that are not inside a Domain Controller, Ninite could definitely save hours and at cases even days of boring install and maintainance work. HP Enterprise or HP Inc. Employees or ex-employees would definitely love Ninite, because what Ninite does is pretty much like the well known HP Internal Tool PC COE.

Ninite could also prepare an installer containing multiple applications based on the choice on Ninite's website, so that's also a great thing especially if you need to deploy a different type of Users PCs (Scientific / Gamers / Working etc.)

Perhaps there are also other useful things to install on a new fresh Windows installations, if you're using something I'm missing let me know in comments.

chmod all directories permissions only and omit files (recursively) on Linux howto

Friday, March 11th, 2016

execute-write-read-of-user-group-and-others-on-linux-unix-bsd-explanationary-picture

If you mistakenly chmod-ed all files within directory full of multiple other subdirectories and files and you want to revert back and set a certain file permissions (read, wite execute) privileges only to all directories:
 

find /path/to/base/dir -type d -exec chmod 755 {} +


If there are too many files or directories you need to change mod use
 

chmod 755 $(find /path/to/base/dir -type d) chmod 644 $(find /path/to/base/dir -type f)

Above willl run evaluate $() all files searched and print them and pass them to chmod so if you have too many files / directories to change it will drastically reduce execution time.

An alternative and perhaps a better way to do it for those who don't remember by heart the chmod permission (numbers), use something like:
 

chmod -R u+rwX,go+rX,go-w /path

Below is arguments meaning:

    -R = recursively;
    u+rwX = Users can read, write and execute;
    go+rX = group and others can read and execute;
    go-w = group and others can't write

If like piping, a less efficient but still working way to change all directory permissions only is with:
 

find /path/to/base/dir -type d -print0 | xargs -0 chmod 755
find /path/to/base/dir -type f -print0 | xargs -0 chmod 644


For those who wish to automate and often do change permissions of only files or only directories it might be also nice to look at (chmod_dir_files-recursive.sh) shell script

Tadadam 🙂

 

How to find and Delete Duplicate files in directory on Linux server with find and fdupes command

Monday, March 16th, 2015

search-duplcate-files-linux-command-and-graphical-tools-how-to-find-duplicate-files-on-linux-mac-and-windows-os

Linux / UNIX find command is very helpful to do a lot of tasks to us admins such as Deleting empty directories to free up occupied inodes or finding and printing only empty files within a root file system within all sub-directories
There is too much of uses of find, however one that is probably rarely used known by sysadmins find command use is how to search for duplicate files on a Linux server:
 

find -not -empty -type f -printf “%s\n” | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 –all-repeated=separate

If you're curious how does duplicate files finding works, they are found by comparing file sizes and MD5 signatures, followed by a byte-by-byte comparison.

Most common application of below command is when you want to search and get rid of some old obsolete files which you forgot to delete such as old /etc/ configurations, old SQL backups and PHP / Java / Python programming code files etc.

If you have to do a regular duplicate file find on multiple servers Linux servers perhaps you should install and use  fdupes command.
On Debian Linux to install it:

root@pcfreak:/# apt-cache show fdupes|grep -i descr -A 4
Description: identifies duplicate files within given directories
 FDupes uses md5sums and then a byte by byte comparison to find
 duplicate files within a set of directories. It has several useful
 options including recursion.
Homepage: http://code.google.com/p/fdupes/

 

root@pc-freak.net:/# apt-get install –yes fdupes

To search for duplicate files with fdupes in lets /etc/ just run fdupes without arguments:

 

root@pcfreak:/# fdupes /etc/
/etc/magic
/etc/magic.mime

/etc/odbc.ini
/etc/.pwd.lock
/etc/environment
/etc/odbcinst.ini

/etc/shadow-
/etc/shadow


If you want to look up for all duplicate files within root directory:
 

root@pcfreak:/# fdupes -r /etc/
Building file list /

 

You can also find duplicate files for multiple directories by just passing all directories as arguments to fdupes

 

root@pcfreak:/# fdupes -r /etc/ /usr/ /root /disk /nfs_mount /nas


The -r argument (makes a recursive subdirectory search for duplicates), if you want to also see what is the size of duplicate files found add -S option

 

fdupes -r -S /etc/ /usr/ /root /disk /nfs_mount /nas

 


If you want to delete all duplicate files within lets say /etc/

 

root@pcfreak:/# fdupes -d /etc/

fdupes is also available and installable also on RPM based Linux distros Fedora / RHEL / CentOS etc., install on CentOS with:
 

[root@centos~ ]# yum -y install fdupes


There is also a port available for those who want to run it on FreeBSD on BSD install it from ports:

 

freebsd# cd /usr/ports/sysutils/fdupes
freebsd# make install clean


If you have a GUI environment installed on the server and you don't want to bother with command line to search for all duplicate files under main filesystem and other lint (junk) files take a look at FSlint

FSlint-2.02-search-for-duplicate-and-lint-files-linux-gui-tool

If you're looking for a GUI cross platform duplicate file finder tool that runs on all major used Operating Systems Mac OS X / Windows / Linux take a look at dupeGuru

 

How to stop / start services in boot time and install / remove / update SuSE SLES (Suse Enterprise Linux Server)?

Friday, February 6th, 2015

Suse-Logo-stop-start-services-on-linux-boot-time-howto-chkconfig
If you're long time Linux sysadmin but you haven't need to adminster SuSE Linux still and your company buys other business / company which already owns some SuSE servers and you need to deal with them, even though you're just starting up with SuSE Linux but you had already plenty of experience with other Linux distributions Fedora / RHEL / CentOS, don't worry set up / stop / start a service (daemon) to boot on Linux boot time is just the same as any other Redhat (RPM) Linux based distributions. it is done by multiple shell scripts located in /etc/init.d directory which can be manually stopped start by issuing the script with an argument e.g
 

suse:/etc/init.d# cd /etc/init.d/
suse:/etc/init.d# ./snmpd 
Usage: ./snmpd {start|stop|try-restart|restart|force-reload|reload|status}


To configure how each of the /etc/init.d/ existent service boots you can the use good old /sbin/chkconfig (a script written in perl) – which you already know from Fedora / CentOS and other RPM distros.

1. Get a list of all enabled on boot SuSE Linux services

To get a list of all set up to run on boot SuSE server services with chkconfig:

 

suse:/etc/init.d# /sbin/chkfong –list 

Makefile                  0:off  1:off  2:off  3:off  4:off  5:off  6:off
OVCtrl                    0:off  1:off  2:off  3:on   4:on   5:on   6:off
SuSEfirewall2_init        0:off  1:off  2:off  3:off  4:off  5:off  6:off
SuSEfirewall2_setup       0:off  1:off  2:off  3:off  4:off  5:off  6:off
Tivoli_lcfd1              0:off  1:off  2:on   3:on   4:off  5:on   6:off
Tivoli_lcfd1.bkp          0:off  1:off  2:off  3:off  4:off  5:off  6:off
aaeventd                  0:off  1:off  2:off  3:off  4:off  5:off  6:off
acpid                     0:off  1:off  2:on   3:on   4:off  5:on   6:off
alsasound                 0:off  1:off  2:on   3:on   4:off  5:on   6:off
apache2-eis               0:off  1:off  2:off  3:off  4:off  5:off  6:off
atd                       0:off  1:off  2:off  3:off  4:off  5:off  6:off
auditd                    0:off  1:off  2:off  3:on   4:off  5:on   6:off
autofs                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
autoyast                  0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.apparmor             0:off  1:off  2:on   3:on   4:off  5:on   6:off  B:on
cron                      0:off  1:off  2:on   3:on   4:off  5:on   6:off
dbus                      0:off  1:off  2:off  3:on   4:off  5:on   6:off
earlykbd                  0:off  1:off  2:off  3:off  4:off  5:on   6:off
earlysyslog               0:off  1:off  2:off  3:off  4:off  5:on   6:off
esound                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
evms                      0:off  1:off  2:off  3:off  4:off  5:off  6:off
fbset                     0:off  1:on   2:on   3:on   4:off  5:on   6:off
firstboot                 0:off  1:off  2:off  3:off  4:off  5:off  6:off
fixperms                  0:off  1:off  2:off  3:off  4:off  5:off  6:off
gpm                       0:off  1:off  2:off  3:off  4:off  5:off  6:off
gssd                      0:off  1:off  2:off  3:off  4:off  5:off  6:off
gwproxy                   0:off  1:off  2:on   3:on   4:off  5:on   6:off
haldaemon                 0:off  1:off  2:off  3:on   4:off  5:on   6:off
hp-health                 0:off  1:off  2:on   3:on   4:on   5:on   6:off
hp-ilo                    0:off  1:off  2:off  3:on   4:off  5:on   6:off
hp-snmp-agents            0:off  1:off  2:on   3:on   4:on   5:on   6:off
hpsmhd                    0:off  1:off  2:off  3:on   4:on   5:on   6:off
idmapd                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
ipmi                      0:off  1:off  2:off  3:off  4:off  5:off  6:off
ipmi.hp                   0:off  1:off  2:off  3:off  4:off  5:off  6:off
irq_balancer              0:off  1:on   2:on   3:on   4:off  5:on   6:off
itcaIBMTivoliCommonAgent0  0:off  1:off  2:on   3:on   4:off  5:on   6:off
jboss                     0:off  1:off  2:off  3:off  4:off  5:off  6:off
joystick                  0:off  1:off  2:off  3:off  4:off  5:off  6:off
kadmind                   0:off  1:off  2:off  3:off  4:off  5:off  6:off
kbd                       0:off  1:on   2:on   3:on   4:off  5:on   6:off  S:on
kdump                     0:off  1:off  2:off  3:off  4:off  5:off  6:off
kpropd                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
krb524d                   0:off  1:off  2:off  3:off  4:off  5:off  6:off
krb5kdc                   0:off  1:off  2:off  3:off  4:off  5:off  6:off
ldap                      0:off  1:off  2:off  3:on   4:off  5:on   6:off
lm_sensors                0:off  1:off  2:off  3:off  4:off  5:off  6:off
lw_agt                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
mdadmd                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
microcode                 0:off  1:on   2:on   3:on   4:off  5:on   6:off  S:on
multipathd                0:off  1:off  2:off  3:off  4:off  5:off  6:off
mysql                     0:off  1:off  2:off  3:off  4:off  5:off  6:off
network                   0:off  1:off  2:on   3:on   4:off  5:on   6:off
nfs                       0:off  1:off  2:off  3:on   4:off  5:on   6:off
nfsboot                   0:off  1:off  2:off  3:on   4:off  5:on   6:off
nfsserver                 0:off  1:off  2:off  3:off  4:off  5:off  6:off
nohup.out                 0:off  1:off  2:off  3:off  4:off  5:off  6:off
novell-zmd                0:off  1:off  2:off  3:off  4:off  5:off  6:off
nscd                      0:off  1:off  2:off  3:on   4:off  5:on   6:off
ntp                       0:off  1:off  2:on   3:on   4:off  5:on   6:off
openct                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
opsware-agent             0:off  1:off  2:off  3:on   4:on   5:on   6:off
osddownt                  0:off  1:off  2:off  3:on   4:on   5:on   6:off
ovpa                      0:on   1:off  2:on   3:on   4:off  5:on   6:off
pcscd                     0:off  1:off  2:off  3:off  4:off  5:off  6:off
pctl                      0:off  1:off  2:on   3:on   4:off  5:on   6:off
portmap                   0:off  1:off  2:off  3:on   4:off  5:on   6:off
postfix                   0:off  1:off  2:off  3:on   4:off  5:on   6:off
powerd                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
powersaved                0:off  1:off  2:off  3:off  4:off  5:off  6:off
random                    0:off  1:off  2:on   3:on   4:off  5:on   6:off
raw                       0:off  1:off  2:off  3:off  4:off  5:off  6:off
resmgr                    0:off  1:off  2:on   3:on   4:off  5:on   6:off
rpasswdd                  0:off  1:off  2:off  3:off  4:off  5:off  6:off
rpmconfigcheck            0:off  1:off  2:off  3:off  4:off  5:off  6:off
rrdtools                  0:off  1:off  2:off  3:on   4:off  5:on   6:off
rsyncd                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
saslauthd                 0:off  1:off  2:off  3:off  4:off  5:off  6:off
skeleton.compat           0:off  1:off  2:off  3:off  4:off  5:off  6:off
slurpd                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
smartd                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
smpppd                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
snmpd                     0:off  1:off  2:on   3:on   4:off  5:on   6:off
splash                    0:off  1:on   2:on   3:on   4:off  5:on   6:off  S:on
splash_early              0:off  1:off  2:on   3:on   4:off  5:on   6:off
sshd                      0:off  1:off  2:off  3:on   4:off  5:on   6:off
suseRegister              0:off  1:off  2:off  3:off  4:off  5:off  6:off
svcgssd                   0:off  1:off  2:off  3:off  4:off  5:off  6:off
syslog                    0:off  1:off  2:on   3:on   4:off  5:on   6:off
sysstat                   0:off  1:off  2:off  3:off  4:off  5:off  6:off
tecad_logfile             0:off  1:off  2:off  3:on   4:off  5:on   6:off
tomcat55                  0:off  1:off  2:off  3:off  4:off  5:off  6:off
tomcat_eis                0:off  1:off  2:off  3:off  4:off  5:off  6:off
tpmgwproxy.sh             0:off  1:off  2:on   3:on   4:off  5:on   6:off
uc4_smgrp                 0:off  1:off  2:off  3:on   4:off  5:on   6:off
uc4_smgrq1                0:off  1:off  2:off  3:on   4:off  5:on   6:off
xbis-ldap-tool            0:off  1:off  2:off  3:off  4:off  5:off  6:off
xdm                       0:off  1:off  2:off  3:off  4:off  5:on   6:off
xfs                       0:off  1:off  2:off  3:off  4:off  5:off  6:off
xinetd                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
ypbind                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
xinetd based services:
        chargen:            off
        chargen-udp:        off
        daytime:            off
        daytime-udp:        off
        echo:               off
        echo-udp:           off
        netstat:            off
        rsync:              off
        servers:            off
        services:           off
        systat:             off
        time:               off
        time-udp:           off

 

2. Stop / Disable a service in all Linux boot runlevels or in a concrete one

As you should know already in Linux there are multiple runlevels in which server can boot, under normal circumstances SuSE servers (as of time of writting) this article boots into runlevel 3, if you'r'e  unsure about the runlevel you can check it with runlevel command:
 

 suse:/etc/init.d# /sbin/runlevel
N 3

To stop a service on all possible boot runlevels – 1,2,3,4,5
 

suse:/etc/init.d# /sbin/chkconfig xinetd off


If you want to stop xinetd or any other service just for certain runlevels (lets say run-level 3,4,5):
 

suse:/etc/init.d# chkconfig –level 345 xinetd off

 

3. Start / Enable a service for a runlevel or all boot levels 1,2,3,4,5

To disable boot.apparmor on all boot runlevels –  kernel enhancement that enabled to set a limited set of resources for services (good for tightened security, but often creating issues with some external server configured services).
 

suse:/etc/init.d# chkconfig boot.apparmor off


Or for single boot modes again with –level option:
 

suse:/etc/init.d# chkconfig –level 345 boot.apparmor off

 

suse:/etc/init.d# chkconfig xfs off 


4. SuSE Linux Package management zypper console tool

If you need / wonder how to install /remove / update a service on a SuSE Linux server, take a look at zypper tool.
zypper is  a  command-line  interface to ZYPP system management library.

To install a package / service with zypper the syntax is very much like yum, for example:
 

suse: ~# zypper install vim -emacs

 


will remove emacs editor and install Vi Improved

The equivalent of yum -y  Fedora command in SuSE Enterprise Linux is –non-interactive option

 

suse:~# zypper –non-interactive install

 

In SuSE it is pretty annoying when you're asked for accepting licensing on some proprietary (external vendor) non-free software packages to get around this:
 

suse:~# zypper patch –auto-agree-with-licenses


To keep the SuSE server up2date – i.e. SLES equivalent of CentOS's yum update && yum upgrade

 

suse:~# zypper list-patches
Loading repository data…
Reading installed packages…
 
Repository                          | Name      | Version | Category | Status
————————————+———–+———+———-+——-
Updates for openSUSE 11.3 11.3-1.82 | lxsession | 2776    | security | needed

 

 

 

suse:~# zypper patch-check
Loading repository data…
Reading installed packages…
5 patches needed (1 security patch)


To look for a certain package with Zypper (equivalent of yum search packagename)

suse: ~# zypper search apache


To verify whether an RPM installed package dependecies are OK:
 

suse:~# zypper verify


The equivalent of Fedora yum update command in SuSE (SLES) are:

suse:~# zypper refresh


To force a complete refresh and rebuild of the database, including a forced download of raw metadata.
 

suse:~# zypper refresh -fdb

 


For people that are used to ncurses (midnight commander) like text interface you can also use yoast2 (text GUI) package manager:
 

suse:~# yoast2


update-linux-suse-server-with-yoast2-ncurses-package-text-gui-management-tool

If a package is messed you can always go back and use good old RPM (Redhat Package Manager) to solve it.

 

Turn your Windows PC / notebook to Wireless Router with My Wifi Router 3.0 and TP-Link TL-WN722N

Wednesday, January 7th, 2015

my-wifi-router-3.0-turn-regular-windows-notebook-to-wifi-router

I've been to home of my wife's parents and for this Christmas, they have received second hand Acer Aspire notebook as a Christmas gift. So far they were using internet using their Windows XP PC which is getting internet here in Belarus via UTP network cable using ByFly ZTE ADSL router modem. As ADSL modem lacked wifh WI-FI Antenna (support) and there was already the Acer notebook which had to access the internet preferrably via Wireless connection, the option was to get a WI-FI router and connect it to the ADSL modem but as this would cost (20 EUR at minimum) and there was alreay Wireless (Receiver) adapter TP-LINK TL-WN722N unused, I decided to try make the TP-LINK Receiver and Windows XP PC to act as a small Home Made Wireless  (software) router.

Until I succeeded I've tried multiple softwares which all failed to turn the Windows PC to Wi-Fi Hotspot.
Here is list of few of the softwares I tried that didn't worked for some reason:

1. Wifi HotSpot Creator

WiFi-HotSpot-Creator-screenshot

Turn your PC into a Wi-Fi Router for Free!

Instantly share your internet connection with your friends and peers over Wi-Fi. Turns your PC into a Wi-Fi Router! And its Free! Wifi HotSpot Creator is said to be able to convert any Mac OS X and Windows PC to hotspot here is Wifi HotSpot Creator download website
Wifi HotSpot Creator is said to work with Windows Vista / XP / 7 / 8, however as I said it doesn't work for me on Windows XP.
 

2. Virtual Wifi router

Here is

With Virtual WiFi Router you can create a WiFi hotspot for WiFi Reverse Tethering on Windows 7 and for wifi supported mobiles and other wifi enabled computer to create a network and to share internet. Virtual Wifi Router in a minute converts your PC into a WiFi hot spot for free.

To make the program working it depends on .NET 4.5. Though the program looked like a superb it unfortunately was crashing on Windows XP. Below are few screenshots from program working on Windows 7.

virtual-wifi-router-screenshot

virtual-wifi-router-connected-device-screenshot-windows
3. My Wifi Router 3.0

Finally I've come across My Wifi Router 3.0 which is just another program that makes necessery Windows configuration to TP-Link TL-WN722N Wireless receiver Adapter to turn it into a homemade Wi-Fi router.
my-wifi-router-on-windows-xp-desktop-pc-noteboko-creenshot

By default Amiss_papp Wifi is configured, this can be changed, however in my case when I tried changing it there were some issues, so finally I've had to re-install My Wifi Router to make it working again.
Once configured My Wifi Router there is the green button (Activate / Deactivate Free Wifi) as seen from screenshot.

As you can see My Wifi Router also allows to Share Videos over WiFi. Once I've tested the program and confirmed it as working, I had to configure it to automatically start on Windows PC boot.

This is done from Settings (located on backward triangle button, next to minimize function).
I had to set check in to  Auto Start and Software Conflict Detection.

make-windows-pc-with-wireless-wifi-router-my-wifi-router-settings-screenshot

Once connected to the TP-Link TL-WN722N (USB) Wi-Fi (High Gain) Receiver adapter in Windows Tray a new indicator will popup that a device has been connected. I've tested My Wifi Router and it seem to be working fine with 3 remote connected Wi-Fi devices (1 Notebook and 2 Nokia Lumia mobile phones). The speed of internet was fast and if I didn't know the connection is done in a software way via such an improvised Windows XP Wi-Fi network router I would think it is just a regular Wi-Fi network router.

One more thing I had to do to make the internet working I had to share the the LAN Network (ethernet card) Interface's internet from
 

Control Panel -> Networking -> Local Area Connection (Properties) -> Advanced

http://pc-freak.net/images/windows-xp-internet-connection-sharing-how-to-networking-screenshot-make-network-address-translation-windows-xp.jpg

A downside of My Wifi Router is I couldn't find a way to save password while connection to the newly created WiFi router with it, so each time I had to login I had to manually type in the password (default my wifi router password is 123567890). Re-typing password on each login is annoying but if you have to do it once per day in the morning when you turn on your notebook it is not such a big deal.

Once connected to My Wifi Router in Connection Management in Friends (tab) you will see a list with connected devices.

my-wifi-router-on-windows-xp-with-htc-and-ipad-iphone-connected
As visible from above screenshot default IPs which will be assigned to new connected clients to My Wifi Router will be in local network IP range 192.168.23.2 – 192.168.23.254.

Now all left is to Enjoy  your new Software Wi-Fi router 🙂