Create user and password on Linux non interactive and add it to sudo a tiny Dev Ops script


September 20th, 2018

Bash-Final-the-Bourne-again-shell-logo
A common task for SysAdmins who managed a multitude of servers remotely via Secure Shell was to add a user and assign password by using a script, this was sometimes necessery to set-up some system users and create access for university users on 10 / 20 testing Linux servers.

Nowadays this task of adding user to a list of remote servers and granting the new user superuser permissions through /etc/sudoers is practiced heavily by the so called Dev Ops (Just another Buziness Word for Senior System Admiistrators with good scripting skills and a little bit of development experience – same game different name.

The Dev Ops System Integration Engineers use this useful add non-interactive user via SSH in Cloud environments in order to prepare superuser (root permissioned through /etc/sudoers) user, that is later be used for lets say deployment on a few hundred of servers of lets say LAMP (Linux + Apache + MySQL + PHP) or LEMP (Linux NGINX MySQL PHP) or Software Load Balancer HAProxy  balacing for MySQL clusters / Nginx Application servers / JIRAs etc, through a Playbook script with some deployment automation tool such as Ansible.

Well enough talk here is the few lines of code which does create a user locally:
 

linux:~# apt-get install –yes sudo
linux:~# useradd devops –home /home/devops -s /bin/bash
linux:~# mkdir /home/devops
linux:~# chown -R devops:devops /home/devops
linux:~# echo 'username:testpass' | chpasswd


Though this lines could be invoked easily by passing it as arguments via ssh it is often unhandy to run them on remote host, because some of the remote hosts against executed, might have already the user existent with granted permissions for sudo

Thus a much better way to do things is use below script and first upload it to remote servers by running the scp command in a loop:

while read line; do
scp  root@$i:/root/
ssh "
create_user_noninteractive_and_add_to_sudoers.sh"
done < servers_list.txt


Where servers_list.txt contains a list of remote IPs:

#!/bin/bash
# Create new user/group and add nopasswd login to sudoers
# Author: Georgi Georgiev
# has to be run sa root – sudo devops
# hipo@pc-freak.net

 

u_id='devops';
g_id='devops';
pass='testpass';
sudoers_f='/etc/sudoers';

check_install_sudo ()  {
if [ $(dpkg –get-selections | cut -f1|grep -E ‘^sudo’) ]; then
apt-get install –yes sudo
else
        printf "Nothing to do sudo installed";
fi
}

check_install_user () {

if [ “$(sed -n “/$u_id/p” /etc/passwd|wc -l)” -eq 0 ]; then
apt-get install –yes sudo
apt-get install –yes sudo
useradd $u_id –home /home/$u_id
mkdir /home/$u_id
chown -R $u_id:$g_id /home/$u_id
echo "$u_id:$pass" | chpasswd
cp -rpf /etc/bash.bashrc /home/$u_id
if [ “$(sed -n “/$u_id/p” $sudoers_f|wc -l)” -eq “0” ]; then
echo "$u_id ALL=(ALL) NOPASSWD: ALL" >> $sudoers_f
else
        echo "$u_id existing. Exiting ..";
        exit 1;
fi

else
        echo "Will do nothing because $u_id exists";
fi

}

check_install_sudo;
check_install_user;


By the way this task was the simplest task given by a Company where I applied for a Dev Ops System Engineer, so I hope this will help someone else too.

P.S. If you prefer Shell scripts (even though much harder, time consuming etc.) as a mean of automation as an alternative to Ansible / Chef I suggest you check out and perhaps try to do the task with http://fuckingshellscripts.org 🙂

Install Slack and Mattermost clients for Start Up Business communication on Linux


September 19th, 2018

install-slack-and-mattermost-clients-for-start-up-business-communication-on-Linux
Many businesses nowadays are lookig for alternatives to the Microsoft dominated market of communication – Skype / Skype for Business Chat Audio and Video desktop client.
The two are the defacto standard for most of Corporate Businesses and is heavily used across most largest Corporations (companies) such as IBM / Xeror / DXC / CSC / Oracle / SAP / Microsoft / Amazon / Adobe … the list goes on and on.

However even though Skype is so easy to use across Microsoft Domain connected Computers the many start-up companies of today often try to avoid its use. The reason, well Skype is totally Proprietary non-transparent and by using it you probably get spied by Microsoft the CIA and God knows how many other Country Agencies. Besides that Skype has a bad history often had problems with Audio (Linux microphone and Video settings) in Free Software (Linux, FreeBSD etc.) realm and even though nowdays situation is improving and Skype Video / Audio runs fine on GNU / Linuxes its Skype for Business has no working release by Microsoft and has left-up Free Software users and Staring Business companies platform of the size of 20 to 1000 people  that choose Linux as a main Desktop / Work had to look for other ways to communicate internally within company and with clients.

Jabber XMPP communication protocol has been one alternative for a long time and historically many compainies that were running out of Skype use for their work often were using Small internally company hosted jabber servers, however as Jabber's  communication clients such as Gajim development is lagging behind seriously over the last 7 years and it prooved so buggy many businesses were looking for ways to avoid it.

Slack_Technologies_-corporate-communication-alternative-to-skype-Logo.png


Slack is multi platform just like Skype and has versions for Linux / Windows for macOS but its power comes mostly because most of its users use it via Skype Web Client (while Skype is a Desktop app and heavily used in Web Browser.

slack-web-communication-client-in-webbrowser-screenshot

Slack reminds in a what of things to the good old IRC chats and has channels in a similar fashion, it support Audio conversations but unfortunately at the moment didn't support Video.

The emergency of New Age of Computing and the quick adoption of Clouds as an aim to cut business costs put Jabber totally out of the game and in the niche and in August 2013 on the scene raised Slack (Team Messaging) which is an acronymi of (Searchable Log of All Conversations and Knowledge) – a cloud-based set of proprietary team collaboration tools and services, founded by Stewart Butterfield for the purpose of online Game (now defunct) called Glitch.

asana-for-slack-integration-2018-2-linux
The problem with Slack is that it is a freemium product, whose main paid features are the ability to search more than 10,000 archived messages (the ordinary free version allows the user to make up to 10000 searches in chat history), the paid Slack versions adds also unlimited apps and integrations and theoretically unlimited number of users (though this is seriously doubtful).

slack-10000-messages-per-user-limitation-shot

One very handy feature of Slack is its integration with "The World's Leading Software Development Platform" – GitHub .

To solve the problem with the little amount of Slack Chat history in Users conversations many Start-Up Business Companies do use Slack as a communication media with Clients and does often use as a communication Media another very popular Cloud messaging Open Source Software called Mattermost deploys to Cloud infrastruture but is at IT control of your company or your hired support and not third party vendor supports, making it a great communication tool for small and mid-sized companies who want to save money of purchasing a special server and hiring an admin or paying for one to support it all the time but instead directly use their Cloud account and deploy it there.

Mattermost is capable to Reach anyone, anywhere on any device. From the airport to data center, safely connect teams with EMM apps, hybrid cloud deployment and enterprise-grade flexibility to meet the unique needs of enterprise.

It is capable to Integrate with existing applications and build new workflows and empower your teams especially operations and DevOps – to perform faster and effectively. Mattermost


To install Slack on Linux:

Go and Download slack from Slack Linux download (the DEB / RPM 64 bit package)

As of time of writting this article latest Slack Desktop packages are: slack-desktop-3.3.1-amd64.deb and slack-3.3.1-0.1.fc21.x86_64.rpm

Depending on the type of Linux distribution install it with dpkg or rpm

1. Installing Slack Desktop client on Debian / Ubuntu Linux

On Debian / Ubuntu / Mint install Slack with:

 

root@ubuntu:~# dpkg -i slack-desktop-*.deb


For Ubuntu users there is also unofficial third-party Slack app ScoudCloud

it integrates well with Ubuntu Unity desktop (which I personally dislike 🙂 ) and gives you some extra goodies such as showing in an Unity manner unread message count, notification, bubbles, unity quicklists for fast-switching between Slack channels etc.

2. Installing Slack Desktop client on Redhat / Fedora / CentOS Linux

On Redhat / Fedora / CentOS install it with:

 

[root@fedora ~]:# rpm -ivh slack-*.rpm

 


3. Installing Mattermost Desktop client on Linux

mattermost-open-source-communication-in-the-cloud

Download Mattermost Linux package from download URL here

As of time of writting the DEB versions are mattermost-desktop-4.1.2-linux-amd64.deb mattermost-desktop-4.1.2-linux-i386.deb and there is no official RPM package for Fedora / CentOS users however I guess the .deb package can easily be converted to .rpm with alien tool.

To install Mattermost on Debian (in moment of writting September 2018):

 

root@debian:~# wget https://releases.mattermost.com/desktop/4.1.2/mattermost-desktop-4.1.2-linux-amd64.deb

 

root@debian:~# dpkg -i mattermost-desktop-4.1.2-linux-amd64.deb


mattermost-linux-client-2-screenshot

Mattermost supports file attachment (send) / Video previews (you can play sent Web videos directly within the Mattermost client) and on experimental level even supports Video and Audio Calls.

mattermost-linux-client-screenshot-1

mattermost-markdown-help-linux-screenshot

One nice feature of Mattermost for those who love coding is using tags to format messages

There is plenty of features of Mattermost among the best ones are integrations Private Cloud open source integrations (Jira, Jenkins, Bots, clients), supports Webhooks, Restful APIS, CLI and Public Cloud connections via Zapier – a connect and uatomate workflows e.g. gives you ability to move info between web apps automatically.

Install Jenkins software development automation server on GNU / Linux


September 18th, 2018

jenkins-automate-installation-on-logo-title

As I have gone through a hiring procedures for Dev Ops system administration (Senior System Engineer) positions in the largest IT company in Belarus EPAM (A Global provider for software engineering and IT consulting.one of the Dev Ops test tasks to do was to automate installation of Jenkins software development automation server on a Virtual Machine running Linust Guest of choice (VMWare / VirtualBox) with Ansible / Docker or Pure Shell Script as I love simplicity I choose to do it via Bash Shell script.

As Dev Ops position is more and more transforming into a programmer job into the New Age of non-sense Cloud Computing (I hate Clouds guys – I share Richard Stallman opinion that "Clouds are clouding your minds" !!! ) … the need for environments such as Jenkins allowing multiple pseudo "sys admins" (mostly copy / paste new age coders) to write and build there programs in Go Language / Python / Perl / Bash with a single environment for Continuous Integration (CI) that could deploy and keep software versionings in GitHub / Mercurial / SVN is exponentionally raising.

Old School Computer Geeks would definitely be amazed (reasonably) on why on earth would someone need a Web Based Java Crazy environment that Jenkins is to build a multiple language code and submit it to a source repository system
But as the people like to hype and make easy things harder adding more and more layers of complexity, the product is the new buziness hype terms Continuous Integration / Continuous Integration raise exponentially together with the softwares to do stuff.
As IT people are becoming more and more lazy and illitetelarete things like Jenkins is the next web development CI environment that is about to die in the coming 5 to 10 years.

Jenkins doesn't really cut the need for writting scrpits to make your application (Makefiles), for the Business Corporate world it is heavily used nowadays, because it is used to building projects using Web UI, running tests, doing static code analysis, and deploying.

What makes Jenkins in terms of IT architecture design solution is that every project that it builds is build via its Java Virtuam Machine Processor backend (the server itself runs in the background of the OS runs it through Java VM as a WAR file (Web Application Resource) with the help of Few Ruby and other scripted files.
Besides that JNS has in the moment of writting more than 1400 years spanning across platforms, UI, administration, source code management, and, most frequently, build management.

Jenkins is either used as a standalone server or as a servlet in Java app servers such as Tomcat.

This is the short script I came up with in bash that when executed installs Jenkins on the remote VM host server that ran Debian 9 Linux, the install_jenkins.sh script is downloadable here.

#!/bin/bash
# Install jenkins and test whether it runs prints password on prompt or send via email
# if email variable is set Jenkins password will be set to your email of choice using mail command
# NOTE: bsd-mailx package should be installed in order for email sent to work and local machine should be running a properly configured
# relay SMTP
# Author: Georgi Georgiev 
# hipo@pc-freak.net
email='hipo@mail.com';

add_repos_install_jenkins () {
apt-get install –yes -qq apt-transport-https git curl

wget -q -O – https://pkg.jenkins.io/debian/jenkins.io.key | sudo apt-key add –

if [ “$(sed -n ‘/jenkins/p’ /etc/apt/sources.list|wc -l)” -eq 0 ]; then
echo 'deb https://pkg.jenkins.io/debian binary/' >> /etc/apt/sources.list
fi

apt-get update -qq && apt-get install –yes -qq jenkins
}

check_j_install () {
if [[ “$(dpkg –get-selections | cut -f1|grep -i jenkins)” ]]; then echo 'succesfully installed'; 

else printf 'Problem in installing please check'; 
exit 1; 

fi

}

check_j_running_s_pass () {
if [[ $(ps -e -o command|grep -i jenkins) ]]; then 
echo 'Jenkins process working.'; 
echo ‘… do more here if necessery with some more commands’; 
else 
echo 'not working log to file' >> jenkins.log 
exit 1; 
fi

JENKINS_PASSWORD=`cat cat /var/lib/jenkins/secrets/initialAdminPassword`;
echo "Jenkins Admin password is $JENKINS_PASSWORD" | tee -a "jenkins_credentials.log";
if [ ! -z $email ]; then
echo $JENKINS_PASSWORD | mail -s "NEW Jenkins password" $email


fi

}

main () {
        add_repos_install_jenkins;
        check_j_install;
        check_j_running_s_pass;

}

main;

To run the script on the remote VM server started for the purpose I created a passwordless ssh key authentication with
 

ssh-keygen -t rsa


and 

 

ssh-copy-id -i ~/.ssh/id_dsa.pub root@remote-vm-host.com

 


command … for more check out my previous article "How to execute command to Multiple Servers / Establishing passwordless SSH key authentication on 50+ servers"

Once the passwordless authentication was established to remote Private Virtual Server I've used scp command to upload my install_jenkins.sh script with:

 

# scp -v install_jenkins.sh root@remote-vm-host.com:/root/install_jenkins.sh

Sending file modes: C0644 726 install_jenkins.sh
Sink: C0644 726 install_jenkins.sh
install_jenkins.sh              


Next to run the install_jenkins.sh on remote host I used remote SSH run command capability, the syntax goes like this:

 

 

 

ssh [USER-NAME]@[REMOTE-HOST] [command or script]


In that case the command I used was:

 

ssh root@remote-vm-host.com "chmod +x; /root/jenkins.sh"

 


Next I launched Firefox browser and accessed http://localhost:8080 on the VM host and used the long password generated from the script by command:

 

 

 

JENKINS_PASSWORD=`cat cat /var/lib/jenkins/secrets/initialAdminPassword`;


echo $JENKINS_PASSWORD

In the process of Initial Jenkins setup I selected the GitHub plugins necessery for me to connect Jenkins with GitHub WebHooks (for that perhaps I will write another article when I have time).

jenkins-plugin-installer-screenshot-linux-large

Jenkins Getting Started Initial Screen

jenkins-selecting-plugins-to-use-getting-started

Jenkins Selecting Plugins Screen

Once successfully set-up Jenkins Initial Project creation / Configuration ( Control Panel ) screen looks like so

jenkins-main-screen-successfully-deployed-on-gnu-linux

There was also a task to create a simple print Jenkins and Shell environment variables with a sample Jenkins Free Style Project.
Following 
the New Item menu and setting it to Execute Shell commands as a Build Parameters, the parameters set for the Jenkins Free Style Project to Print the Environment Varibles were like so:

 

 

 

 

 

 

#!/bin/sh

echo “Jenskins Environment Variables”

echo "BUILD_NUMBER" :: $BUILD_NUMBER

echo "BUILD_ID" :: $BUILD_ID

echo "BUILD_DISPLAY_NAME" :: $BUILD_DISPLAY_NAME

echo "JOB_NAME" :: $JOB_NAME

echo "JOB_BASE_NAME" :: $JOB_BASE_NAME

echo "BUILD_TAG" :: $BUILD_TAG

echo "EXECUTOR_NUMBER" :: $EXECUTOR_NUMBER

echo "NODE_NAME" :: $NODE_NAME

echo "NODE_LABELS" :: $NODE_LABELS

echo "WORKSPACE" :: $WORKSPACE

echo "JENKINS_HOME" :: $JENKINS_HOME

echo "JENKINS_URL" :: $JENKINS_URL

echo "BUILD_URL" ::$BUILD_URL

echo "JOB_URL" :: $JOB_URL

echo “===Linux Shell Variables ===”
env


Well, it wasn't really a rocket science was it?! 🙂

That's all folks, see you soon.

Fix Mac OS X camera problems – Tell which application is using Mac OS X builtin Camera


September 16th, 2018

http://pc-freak.net/images/macosx-check-what-process-is-using-camera-screenshot
It is a common problem on Mac OS X notebooks (MacBook Air , MacBook Proc)  with builtin Video Camera to have issues with Camera in Facetime, Skype and other applications which use it.

Considering that the Camera is physically working on the Mac (it did not burn etc.) and it stooped working suddenly (is not detected by Mac OS applications which support it), the most common cause for that is the fact that another application running on the system is using it.
With the spread of spyware and malware that can easily hit your computer by exploiting Javascript bugs in browser intepreter (Firefox, Chrome, Chrome) it is not impossible that your Mac PC got infected with a kind of WebCam spy software that keeps your Video Camera active all time.

Webcam spying is a real issue of today so to secure yourself partially you can place Oversight App to get notifications when an application starts using Mac's Webcam or audio.

Open Finder and run Terminal to check whether the Web Camera is used by some of the Mac running processes.

 

Applications -> Utilities -> Terminal

 


macosx-utilities-terminal-osx-screenshot

 

 

 

MacBook-Air:Volumes root#  lsof | grep "AppleCamera"

 

You should see one or more results. If you don’t see any results, try running the following commands as well.
One of the below commands may be necessary if you’re using an older version of macOS.

 

MacBook-Air:Volumes root#  lsof | grep "iSight"

 

MacBook-Air:Volumes root#  lsof | grep "VDC"

 

If VDCAssistant process shows running kill it.
 

MacBook-Air:Volumes root#  killall -9 VDCAssistant

 

 

 

http://pc-freak.net/images/macosx-check-what-process-is-using-camera-screenshot

You can also check whether the Mac Camera is being detected by Mac OS with system_profiler command (this is Mac's equivalent of Linux's lspci / lsusb / lshw / dmidecode for more on the topic you can check my previous article Get hardware system info on Linux etc.)
 

/usr/sbin/system_profiler

 

 

   Type8Camera::12.781 system_profiler[1075:84585] Exception NSInvalidArgumentE
      Version: 10,1
      Obtained from: Apple
      Last Modified: 13.12.2017, 9:34
      Kind: Intel
      64-Bit (Intel): Yes
      Signed by: Software Signing, Apple Code Signing Certification Authority, Apple Root CA
      Location: /System/Library/Image Capture/Devices/Type8Camera.app
      Get Info String: 10.1, © Copyright 2002-2014 Apple Inc. All rights reserved.

    Type5Camera:

      Version: 10,1
      Obtained from: Apple
      Last Modified: 13.12.2017, 9:34
      Kind: Intel
      64-Bit (Intel): Yes
      Signed by: Software Signing, Apple Code Signing Certification Authority, Apple Root CA
      Location: /System/Library/Image Capture/Devices/Type5Camera.app
      Get Info String: 10.1, © Copyright 2001-2014 Apple Inc. All rights reserve

.

    Type4Camera:

      Version: 10,1
      Obtained from: Apple
      Last Modified: 13.12.2017, 9:34
      Kind: Intel
      64-Bit (Intel): Yes
      Signed by: Software Signing, Apple Code Signing Certification Authority, Apple Root CA
      Location: /System/Library/Image Capture/Devices/Type4Camera.app
      Get Info String: 10.1, © Copyright 2001-2014 Apple Inc. All rights reserved.

    PTPCamera:

      Version: 10,1
      Obtained from: Apple
      Last Modified: 13.12.2017, 9:34
      Kind: Intel
      64-Bit (Intel): Yes
      Signed by: Software Signing, Apple Code Signing Certification Authority, Apple Root CA

How to list and uninstall installed Mac OS X packages command


September 15th, 2018

how-to-list-and-remove-packages-on-Mac-OS-X-MacBook-Air-Pro-notebooks-with-commands-Terminal
1. Listing All installed packages (.pkg) files on Mac OS X

If you are used to Linux package management (with dpkg .deb package tool or Redhat's .rpm package manager rpm command) and out of a suddeny you have to use for some time or permanently choose to migrate to Mac OS X and joined the Jailed closed Computer model of Mr. Steve Jobs.

As a Linux command addict,yYou might be wondering on how to list all installed packages on Mac OS with a command just like you use Debian / Ubuntus terminal to list installed packages and their overall state for example with dpkg with:

dpkg -l

or Fedora / CentOS to list all installed packages you're pretty much accustomed to:
 

rpm -qa

The same is possible in Mac OS X by opening Terminal and running cmd:

 

MacBook-Air:Volumes root# pkgutil –pkgs

om.apple.pkg.OSX_10_13_IncompatibleAppList.16U1254
com.apple.pkg.MRTConfigData.16U4001
com.apple.pkg.ChineseWordlistUpdate.14U1355
com.apple.pkg.GatekeeperConfigData.16U1300
com.apple.pkg.MRTConfigData.16U4003
com.apple.pkg.ChineseWordlistUpdate.14U1356
com.apple.pkg.ChineseWordlistUpdate.14U1346
com.apple.pkg.GatekeeperConfigData.16U1259
com.apple.pkg.GatekeeperConfigData.16U1265
com.apple.pkg.XProtectPlistConfigData.16U4009
com.apple.pkg.ChineseWordlistUpdate.14U1353
com.apple.pkg.ChineseWordlistUpdate.14U1347
com.apple.pkg.ChineseWordlistUpdate.14U1351
com.apple.pkg.MRTConfigData.16U4038
com.apple.pkg.iTunesX.12.7.3.delta
com.apple.pkg.XProtectPlistConfigData.14U4058
com.apple.pkg.GatekeeperConfigData.16U1138
com.apple.pkg.MLVUpdate_en_GB_daniel.16U1127
com.apple.pkg.EmbeddedOSFirmware
com.apple.pkg.MRTConfigData.16U4005
com.apple.pkg.ChineseWordlistUpdate.14U1323
com.apple.pkg.GatekeeperConfigData.16U1572
com.apple.pkg.iTunesX.12.7.delta
com.apple.pkg.MRT.14U2321

For the sake of convenience run it with a pipe to less cmd:

MacBook-Air:Volumes root# pkgutil –pkgs | less

To find whether a package with a certain name is installed or not

mac-os-x-listing-installed-operating-system-packages-with-pkgutil-command

MacBook-Air:Volumes root# pkgutil –pkgs|grep -i oracle
com.oracle.jre|


To further list the content of the package (the package files contained and directory structure etc.)

 

MacBook-Air:Volumes root# pkgutil –files the-package-name.pkg

 

this is like the Linux equivalent of
dpkg -L packagename and rpm -qa rpm-packagename:

Checking the package directory location on Mac OS is done with:

 

MacBook-Air:Volumes root#  pkgutil –pkg-info com.oracle.jre
package-id: com.oracle.jre
version: 1.0
volume: /
location: Library/Internet Plug-Ins/JavaAppletPlugin.plugin
install-time: 1523533193

 


2. Deleting a Mac OS X package .pkg 
 

After reviewing the .pkg file content for smaller sized packages the best way to remove it is to manually remove all files belonging to the package (but before make sure you are not deleting an important packageand have visually closely inspected them otherwise you might break badly your Mac OS X  …) you can run below 2 commands to delete package files and directory:

 

# pkgutil –only-files –files the-package-name.pkg | tr '\n' '\0' | xargs -n 1 -0 sudo rm -f
# pkgutil –only-dirs –files the-package-name.pkg | tail -r | tr '\n' '\0' | xargs -n 1 -0 sudo rmdir

 

Once you have removed the files you can remove the receipt (e.g. remove it from Mac package database), with:
 

# pkgutil –forget the-package-name.pkg

 

pkgutil –forget is useful if you get errors during removal attempt with pkgutil –only-files … , pkgutil –only-dirs …. cmds.
 

Note that to remove a package you have to be root  or run the commands via sudo with admin privileges if you have it installed and using it.

When removing the files be extremely careful as some of the packages you might try to remove might be updating important Mac OS X system components or used as a backbone for the overall Mac OS GUI / background operations thus removing a system related package might leave your MacBook unbootable because of the removal of the necessery component …

If you get some errors while issuing some of the 2 above commands like:

"Operation not permitted when trying to uninstall" this is because  the command or directory you're trying to remove is in current use by a running process on the Mac OS or because the file or directory has a special set permissions to prevent them from easy removal (as a precaution OS safety measures).

You might wonder what might be the purpose of a manual command removal of Mac package but, If you have to regularly delete Mac OS X packages by scripting the removal of unnecessery packages or Mac OS X updates on a many numbers of computers to automate removal jobs.
I've found also a github repository bash shell script that is automating the process of removal and could even save further time if in need to script a package information about Macs and be able to easily manage packages of mac Desktops via (lets say SSH), check out pkg-remove.sh script here.

Mass substitute WordPress site Old domain URL to new URL in MySQL (MariaDB) database after website migration


September 13th, 2018

mass-substitute-old-urls-to-new-urls-when-moving-wordpress-website-migrate-wordpress

Mass substitute WordPress site Old domain URL to new URL in MySQL (MariaDB) database after website migration

If you have just migrated a wordpress blog or site to a new server (domain URL) and you have many articles pointing to the old URL. Out of sudden the new domain will end up with many broken links and that would have a severe negative SEO effect on your website leading to a certain downfall of your number of daily unique visits.

Of course manually changing the URL links is achievable by going through all Published Posts when migrating small websites with 10-20 pages,  however it is an impossible tedious task you would definitely want to avoid when you're migrating large WP based websites with few hundred or thousands of posts / pages,
bacause this would be a few weeks of mindless repeatable job to go through each and every post and substite the broken URLs.

Fortunately with a little bit of SQL magic either through MySQL CLI or PHPMyAdmin (if the website is moved to a shared hosting where you have disabled access to MySQL (MariaDB) default connect tcp port 3306.

Depending on the type of WordPress or Website the old broken URLs might be located in various Database tables.
 

– So when Mass URL substitution is might be required ?


1. You migrate a Website http://what-ever-website.com with (PHP / CSS / HTML / Templates) etc. from Hosting Provider Hostgator.com to UK2.com (because the website target client changed lately to United Kingdom customers) to http://what-ever-website.co.uk and the site is moved to a new domain beacause of Business rebranding
 

2. Other reason for changing internal URLs from one URL to another might be if you're migrating your website from HTTP to HTTPS for security.

3. You are restructuring file storage / image directories on the server or due to migration of files to external CDN (Content Delivery Network).
For example (http://your-server.com/images/ , http://your-server.com/files )  URLs pointing to old website location subdirectory has to be changed to the new one (http://your-server.com/img/ , http://your-server.com/data)

 

– So what is the automated approach to solve the task of Mass URL substitution across WP site ?

 

  •  Create full backup of all your website database and double-check the backup (try to restore on a test (home) server or other hosting account to make sure the backup is consistent and restore would work normally if necessery
     
  • You can Create Backup either with mysqldump command tool manually … with the right command arguments or use some kind of script such as My tiny mysqlbackupper.sh shell script which I shared under my previous article Make daily MySQL on Linux backup with Shell Script  via PhpMyAdmin.


2. Change old website URL to new one directly from Database using MySQL text client
 

To change incorrect URL with the new correct one the general query to run is:

 

update TABLE_NAME set FIELD_NAME = replace(FIELD_NAME, ‘find string’, ‘replace string’);

 

To change old website URL to the new website URL across every table within the wordpress database use below queries:
 

hipo@linux:~$ mysql -u root -p
Enter password:

 

USE blog;

 

UPDATE wp_options SET option_value = replace(option_value, 'Existing (old) URL', 'New URL') WHERE option_name = 'home' OR option_name = 'siteurl';

UPDATE wp_posts SET post_content = replace(post_content, 'Existing (old)URL', 'New URL');

UPDATE wp_postmeta SET meta_value = replace(meta_value,'Existing (old) URL','New URL');

UPDATE wp_usermeta SET meta_value = replace(meta_value, 'Existing (old) URL','New URL');

UPDATE wp_links SET link_url = replace(link_url, 'Existing (old) URL','New URL');

UPDATE wp_comments SET comment_content = replace(comment_content , 'Existing (old) URL','New URL');


3. Replace Old website URL to New one after migration using PHPMyAdmin web interface
 

If you don't have access  ssh shell, you can also run the queries via PhpMyAdmin to do so:

1. Open PHPMyadmin URL Panel in browser and login with your user / pass

2. Choose the wordpress database of the wordpress site / blog

3. Select SQL tab and in the panel type on above given SQL queries
 

web-hosting-phpmyadmin-sql-query-tab-screenshot-how-to-run-sql-queries-via-phpmyadmin

If you're lazy to type there is also a web based SQL queries generator tool for moving websites to a new domain


4. Using Search and Replace WordPress plugin to do the old URL to new URL (strings) transition
 

If you have never used SQL queries and you're totally new to it and don't want to risk breaking up something there is also a bunch of wordpress plugins available that do the URL string substitution throughout each wordpress table in a WP database one such WordPress plugin is Search and Replace I have written earlier an article Change string in all WordPress Posts with Search and Replace plugin.


5. Problems with data-serialization
 

If you do a simple search and replace of Old domain urls to New ones, using above given commands and you still end up with some broken links on WP Pages that might be due to data-serialization issues (for the cause of issues check out what is data serialization).
Data serialization in wordpress terms is an array of PHP encrypted data that contains the actual URL, thus a simple search and replace as explained above if URLs use data-serialization would not work. There are available tools online that does URL search and replace operation through  "serialize-data sensitively" if you stuck with data-serializatoin caused issues.

Besides that for there are written scripts that does URL substitution to a WordPress or Joomla websites so an alternative to above WP plugin to replace the URL after migration is to use one of the scripts available a very famous one that will do pain-free all URL / string substitutions inside your WP, Drupal,  Joomla databses is interconnect/it.
 

Few closure words
 

As a system administrator and webmaster I have migrated wordpress installations many times with the need to change the old URLs to a new ones for both customer websites and my own wordpress based. On many ocassions because of lack of attention and hurry, I've messed up things.
The moral I got out of this is when you're doing a WordPress migration just like everything you have to be very attentive and do everything step by step slowly and have a good idea on what you're doing in advance …

Even as a person who had overall idea on how MySQL Server works and have experience in writting SQL queries, I have to confess I've  made mistakes during URL substitution operations when doing it via the MySQL CLI every now and then.

Thus I would recommend you better use some of the many plugins for wordpress and script tools (few of which I mentioned above), especially if you're not having at least few years with some kind of UNIX variation / Linux / MySQL.

Virtualbox Shared folder set up on Linux between Host and Guest OS – Set up Virtualbox shared folder to Copy files from PC Host to Guest


September 12th, 2018

mount-shares-between-host-OS-and-guest-virtual-machine-howto-virtualbox-vbox-logo

How to set-up Virtualbox shared folder to Copy files from PC Host  and Guest Virtualized OS?

Running VirtualBox Host is an easy thing to set-up across all Operating Systems.  Once you have it sooner or later you will need to copy files from the VM Host OS (that in my case is GNU / Linux) to the virtualized Guest operating system (again in my case that's again another Linux ISO running indide the Virtual Machine).

Below are steps to follow To use Virtualbox Shared Folder functionality to copy files between VBox and your Desktop / server Linux install.

1. Install Virtualbox Guest Additions CD Image ISO

I've explained how to add the Guest Additions CD image thoroughfully in my previous article Howto enable Copy / Paste Virtualbox betwen Linux guest and Host OS
Anyways I'll repeat myself below for sake of clarity:

To do so use Oracle VBox menus (on the booted virtualized OS VBox window):

 

Devices -> Insert Guest additions CD Image

 

Mount the ISO inside the Linux Virtual Machine:

root@debian:~# mount /media/cdrom1/
 

If the mount fails and there are no files inside the mount point it might be because the virtualbox-dkms and virtualbox-guest-dkms packages might be missing on the Host OS.

To install them (on Debian GNU / Linux) assuming that you're using virtualbox default distro packages /etc/apt/sources.list :
 

apt-get install –yes -qq virtualbox-dkms virtualbox-guest-dkms


and run:

 

root@debian:/media/cdrom1# cd /media/cdrom1; sh VBoxLinuxAdditions.run


2. Create directory for Shared Folder that will be used to access Host / OS files from the Guest Virtualized OS
 

root@debian:~# mkdir /mnt/shared_folder

 

3. Map from VBox program interface Shared folder settings and Mount /mnt/shared_folder location

virtualbox-virtual-machine-devices-shared-folders-shared-folder-settings-linux-screenshot

 

Devices -> Shared Folder -> Shared Folder settings -> Transient Folders (click blue folder add small button right)

 

From Transient Folders add whatever directory you want to be shared from your local notebook / PC to the VM.

virtualbox-devices-Shared-Folder-Add-Shared-Folder-add-share-linux-screenshotDepending on whether you would like to mount the shared folder only for reading files (choose Read Only) to make it a permanent shared folder (and not just for the one session of current running Virtual Machine until its killed use Make Permanent) or check Auto-Mount tick if you want the shared_folder mapping to be mounted on every VM boot.

Once the shared_folder directory location is set-up from GUI menu click OK and in order for the settings to take effect, you'll need to restart the VM Guest with Linux (use halt command from terminal) or Power Off the Machine via the VBox menus.

To mount use command like:

mount -t vboxsf name_of_folder_linked_from_vbox  /mnt/name_folder_guest_os/


mount-vboxsf-shared-folder-mnt-shared-linux-guest-screenshot

In my case I wanted to share home folder /home so the command I used is:

root@debian:~# mount -t vboxsf  shared_folder /mnt/shared_folder


If everything is fine your Host OS file content from /home will be visible (for read and write if you Mapped it so) 
under /mnt/shared_folder …

And as Turtles Ninja used to heavily say Cowabunga !!! 🙂
You have it mounted and ready for file share between Desktop -> Virtualized OS.

 

Bear in mind that above mount command has to run as root (superuser) to succeed.

You now could copy files from your Host OS (running the Virtual Machine) and the Guest OS (Virtualized OS) using /mnt/shared_folder mount point without problems.

The example is if you want to share files between VirtualBox installed Linux and the Guest (Desktop / server) OS, however at many cases mounting your Host OS directory for root users might be not very practical but, instead you might prefer to do the mount for specific non admin user, for example I prefer to do the shared folder mount with my pointed non-root username hipo.

Here is how to do above VM shared_folder mount for non-root user:

First you need to know the exact UID / GID (User ID / Group ID) of user, you can get that with id command:

 

hipo@linux:~$  id
uid=1000(hipo) gid=1000(hipo) groups=1000(hipo),24(cdrom),25(floppy),29(audio),30(dip),44(video),46(plugdev),108(netdev),114(bluetooth),115(lpadmin),119(scanner)

 

As you see UID / GID in my case are 1000 / 1000

hipo@linux:~$ sudo mount -t vboxsf -o rw,uid=1000,gid=1000 shared_folder /mnt/shared_folder

 

mount-virtual-box-shared_folder-with-non-administration-permissions-non-root-permissions-id-and-mount-command-screenshot-linux


4. Mounting configured shared_folder to automatically mount into the Guest OS Linux on every boot

a) Configuring shared_folder auto-mount using /etc/rc.local

If you need the shared_folder to automatically mount next-time you boot the virtual machine quickest way is to add the mount command to /etc/rc.local (on Debian 8 and Debian 9 and newer Ubuntu Linuxes rc.local is missing by default to enable it to work like it worked before read follow my previous article ).

b) Configuring auto-mount for shared_folder through /etc/fstab

The more professional way to auto-mount on emulated OS VM boot time,  you could add the vboxsf mount definitions to /etc/fstab with your favourite text-editor mcedit, nano, joe etc. … (for me that's vim).

Syntax of /etc/fstab is as follows:
 

<Device> <Mount Point> <Type> <Options> <Dump> <Pass>

root@linux:~# vim /etc/fstab

 

shared_folder /mnt/shared_folder                                vboxsf rw,uid=1000,gid=1000 0 0

Note that you will want to change 1000 / 1000, id / gid with the ones of the non-admin user you would like to add to mount it for.

A quick way to add it to /etc/fstab with a shell one-liner is with command
 

root@linux:~# echo 'shared_folder /mnt/shared_folder                                vboxsf rw,uid=1000,gid=1000 0  0' >> /etc/fstab

An alternative way to add a user to have permissions for vboxsf file system (without specifying the long -o uid=1000,gid=1000 options is to simply add the username in question to group vboxsf like so:

c) Adding non super user username to vboxsf group

root@linux:~# usermod -G vboxsf hipo
root@linux:~# grep -i vboxsf /etc/group
vboxsf:x:999:hipo

 

hipo@linux:~$ sudo mount -t vboxsf  shared_folder /mnt/shared_folder

 

without the extra arguments and the options to pass to /etc/fstab (for eventual requirement to auto mount the shared_folder) would be more simple e.g.:

 

echo 'shared_folder /mnt/shared_folder                                vboxsf ' >> /etc/fstab

 

One note to make here is if the uesr is added to vboxsf the line for /etc/fstab to auto mount to mount for root user and non-root will be identical.

Then you can get the /etc/fstab auto-mount configured tested by running:

c) Checking auto-mount is working

hipo@linux:~# mount -a
hipo@linux:~# mount |grep -i vboxsf
shared_folder on /mnt/shared_folder type vboxsf (rw,nodev,relatime)


5. What if you end up with mounting failed errors ? – What might be causing the mounting failed Protocol error (a few things to check to solve)


In case of troubles with the mount you might get an error like:

hipo@linux:~# mount -t vboxsf  share_folder /mnt/shared_folder

/sbin/mount.vboxsf: mounting failed with the error: Protocol error


This error might be caused because of Insert Guest Additions CD Image might be not properly enabled and installed using the ISO provided VBoxLinuxAdditions.sh shell script.
Other common reason you might get this error if you have mistyped the Folder name: given in Shared Folders -> Folder Path -> Add Share for example I have given shared_folder as a Map name but as you can see in above mount -t vboxsf, I've mistyped share_folder instead of the correct one shared_folder inserted.
In some VBox releases this error was caused by bugs in the Virtual Machine.
 

virtualbox-virtual-machine-shared-folder-transient-folder-add-folder-linux-VM-guest-linux

One useful tip is to be able to check whether a Virtualbox Virtual Machine has a configured shared_folder (if you're logging to manage the machine on remote server – nomatter whether you have logged in with VNC / Teamviewer / Citrix etc. or via SSH session.

To do so use VBoxControl as of time of writting usually located on most distributions under (/usr/bin/VBoxControl)
 

 

hipo@linux:~# VBoxControl sharedfolder list -automount
Oracle VM VirtualBox Guest Additions Command Line Management Interface Version 5.2.18
(C) 2008-2018 Oracle Corporation
All rights reserved.

 

Auto-mounted Shared Folder mappings (0):

No Shared Folders available.

You can use VBoxControl command to get set and list a number of settings on the VBox VM, here is an useful example with it where you get information about numerous VBox info values:

 

root@linux:~# VBoxControl guestproperty enumerate
Oracle VM VirtualBox Guest Additions Command Line Management Interface Version 5.2.18
(C) 2008-2018 Oracle Corporation
All rights reserved.

 

Name: /VirtualBox/GuestInfo/OS/Product, value: Linux, timestamp: 1536681633430852000, flags: <NULL>
Name: /VirtualBox/GuestInfo/Net/0/V4/IP, value: 10.0.2.15, timestamp: 1536681633438717000, flags: <NULL>
Name: /VirtualBox/HostInfo/GUI/LanguageID, value: en_US, timestamp: 1536697521395621000, flags: RDONLYGUEST
Name: /VirtualBox/GuestInfo/Net/0/MAC, value: 08002762FA1C, timestamp: 1536681633442120000, flags: <NULL>
Name: /VirtualBox/GuestInfo/OS/ServicePack, value: <NULL>, timestamp: 1536681633431259000, flags: <NULL>
Name: /VirtualBox/HostInfo/VBoxVerExt, value: 5.2.18, timestamp: 1536681619002646000, flags: TRANSIENT, RDONLYGUEST
Name: /VirtualBox/GuestInfo/Net/0/V4/Netmask, value: 255.255.255.0, timestamp: 1536681633440157000, flags: <NULL>
Name: /VirtualBox/GuestInfo/OS/Version, value: #1 SMP Debian 4.9.110-3+deb9u2 (2018-08-13), timestamp: 1536681633431125000, flags: <NULL>
Name: /VirtualBox/GuestAdd/VersionExt, value: 5.2.18, timestamp: 1536681633431582000, flags: <NULL>
Name: /VirtualBox/GuestAdd/Revision, value: 124319, timestamp: 1536681633432515000, flags: <NULL>
Name: /VirtualBox/HostGuest/SysprepExec, value: <NULL>, timestamp: 1536681619002355000, flags: TRANSIENT, RDONLYGUEST
Name: /VirtualBox/GuestInfo/OS/LoggedInUsers, value: 1, timestamp: 1536681673447293000, flags: TRANSIENT, TRANSRESET
Name: /VirtualBox/GuestInfo/Net/0/Status, value: Up, timestamp: 1536681633443911000, flags: <NULL>
Name: /VirtualBox/GuestInfo/Net/0/Name, value: enp0s3, timestamp: 1536681633445302000, flags: <NULL>
Name: /VirtualBox/HostGuest/SysprepArgs, value: <NULL>, timestamp: 1536681619002387000, flags: TRANSIENT, RDONLYGUEST
Name: /VirtualBox/GuestAdd/Version, value: 5.2.18, timestamp: 1536681633431419000, flags: <NULL>
Name: /VirtualBox/HostInfo/VBoxRev, value: 124319, timestamp: 1536681619002668000, flags: TRANSIENT, RDONLYGUEST
Name: /VirtualBox/GuestInfo/Net/0/V4/Broadcast, value: 10.0.2.255, timestamp: 1536681633439531000, flags: <NULL>
Name: /VirtualBox/HostInfo/VBoxVer, value: 5.2.18, timestamp: 1536681619002613000, flags: TRANSIENT, RDONLYGUEST
Name: /VirtualBox/GuestInfo/OS/LoggedInUsersList, value: hipo, timestamp: 1536681673446498000, flags: TRANSIENT, TRANSRESET
Name: /VirtualBox/GuestInfo/Net/Count, value: 1, timestamp: 1536698949773993000, flags: <NULL>
Name: /VirtualBox/GuestInfo/OS/Release, value: 4.9.0-7-amd64, timestamp: 1536681633431001000, flags: <NULL>
Name: /VirtualBox/GuestInfo/OS/NoLoggedInUsers, value: false, timestamp: 1536681673447965000, flags: TRANSIENT, TRANSRESET
Name: /VirtualBox/GuestAdd/HostVerLastChecked, value: 5.2.18, timestamp: 1536681702832389000, flags: <NULL>

Hope you enjoyed ! Have phun! 🙂

Mail send from command line on Linux and *BSD servers – useful for scripting


September 10th, 2018

mail-send-email-from-command-line-on-linux-and-freebsd-operating-systems-logo

Historically Email sending has been very different from what most people use it in the Office, there was no heavy Email clients such as Outlook Express no MX Exchange, no e-mail client capabilities for Calendar and Meetings schedule as it is in most of the modern corporate offices that depend on products such as Office 365 (I would call it a connectedHell 365 days a year !).

There was no free webmail and pop3 / imap providers such as Mail.Yahoo.com, Gmail.com, Hotmail.com, Yandex.com, RediffMail, Mail.com the innumerous lists goes and on.
Nope back in the day emails were doing what they were originally supposed to like the post services in real life simply send and receive messages.

For those who remember that charming times, people used to be using BBS-es (which were basicly a shared set-up home system as a server) or some of the few University Internal Email student accounts or by crazy sysadmins who received their notification and warnings logs about daemon (services) messages via local DMZ-ed network email servers and it was common to read the email directly with mail (mailx) text command or custom written scripts … It was not uncommon also that mailx was used heavily to send notification messages on triggered events from logs. Oh life was simple and clear back then, and even though today the email could be used in a similar fashion by hard-core old school sysadmins and Dev Ops / simple shell scriptings tasks or report cron jobs such usage is already in the deep history.

The number of ways one could send email in text format directly from the GNU / Linux / *BSD server to another remote mail MTA node (assuming it had properly configured Relay server be it Exim or Postifix) were plenty.

In this article I will try to rewind back some of the UNIX history by pinpointing a few of the most common ways, one used to send quick emails directly from a remote server connection terminal or lets say a cheap VPS few cents server, through something like (SSH or Telnet) etc.
 

1. Using the mail command client (part of bsd-mailx on Debian).
 

In my previous article Linux: "bash mail command not found" error fix
I ended the article with a short explanation on how this is done but I will repeat myself one more time here for the sake of clearness of this article.

root@linux:~# echo "Your Sample Message Body" | mail -s "Whatever … Message Subject" remote_receiver@remote-server-email-address.com


The mail command will connect to local server TCP PORT 25 on local configured MTA and send via it. If the local MTA is misconfigured or it doesn't have a proper MX / PTR DNS records etc. or not configure as a relay SMTP remote mail will not get delivered. Sent Email should be properly delivered at remote recipient address.

How to send HTML formatted emails using mailx command on Linux console / terminal shell using remote server through SSH ?

Connect to remote SSH server (VPS), dedicated server, home Linux router etc. and run:

 

root@linux:~# mailx -a 'Content-Type: text/html'
      -s "This is advanced mailx indeed!" < email_content.html
      "first_email_to_send_to@gmail.com, mail_recipient_2@yahoo.com"

 


email_content.html should be properly formatted (at best w3c standard compliant) HTML.

Here is an example email_content.html (skeleton file)

 

    To: your_customer@gmail.com
    Subject: This is an HTML message
    From: marketing@your_company.com
    Content-Type: text/html; charset="utf8"

    <html>
    <body>
    <div style="
        background-color:
        #abcdef; width: 300px;
        height: 300px;
        ">
    </div>
Whatever text mixed with valid email HTML tags here.
    </body>
    </html>


Above command sends to two email addresses however if you have a text formatted list of recipients you can easily use that file with a bash shell script for loop and send to multiple addresses red from lets say email_addresses_list.txt .

To further advance the one liner you can also want to provide an email attachment, lets say the file email_archive.rar by using the -A email_archive.rar argument.

 

root@linux:~# mailx -a 'Content-Type: text/html'
      -s "This is advanced mailx indeed!" -A ~/email_archive.rar < email_content.html
      "first_email_to_send_to@gmail.com, mail_recipient_2@yahoo.com"

 

For those familiar with Dan Bernstein's Qmail MTA (which even though a bit obsolete is still a Security and Stability Beast across email servers) – mailx command had to be substituted with a custom qmail one in order to be capable to send via qmail MTA daemon.
 

2. Using sendmail command to send email
 

Do you remember that heavy hard to configure MTA monster sendmail ? It was and until this very day is the default Mail Transport Agent for Slackware Linux.

Here is how we were supposed to send mail with it:

 

[root@sendmail-host ~]# vim email_content_to_be_delivered.txt

 

Content of file should be something like:

Subject: This Email is sent from UNIX Terminal Email

Hi this Email was typed in a file and send via sendmail console email client
(part of the sendmail mail server)

It is really fun to go back in the pre-history of Mail Content creation 🙂

 

[root@sendmail-host ~]# sendmail -v user_name@remote-mail-domain.com  < /tmp/email_content_to_be_delivered.txt

 

-v argument provided, will make the communication between the mail server and your mail transfer agent visible.
 

3. Using ssmtp command to send mail
 

ssmtp MTA and its included shell command was used historically as it was pretty straight forward you just launch it on the command line type on one line all your email and subject and ship it (by pressing the CTRL + D key combination).

To give it a try you can do:

 

root@linux:~# apt-get install ssmtp
Reading package lists… Done
Building dependency tree       
Reading state information… Done
The following additional packages will be installed:
  libgnutls-openssl27
The following packages will be REMOVED:
  exim4-base exim4-config exim4-daemon-heavy
The following NEW packages will be installed:
  libgnutls-openssl27 ssmtp
0 upgraded, 2 newly installed, 3 to remove and 1 not upgraded.
Need to get 239 kB of archives.
After this operation, 3,697 kB disk space will be freed.
Do you want to continue? [Y/n] Y
Get:1 http://ftp.us.debian.org/debian stretch/main amd64 ssmtp amd64 2.64-8+b2 [54.2 kB]
Get:2 http://ftp.us.debian.org/debian stretch/main amd64 libgnutls-openssl27 amd64 3.5.8-5+deb9u3 [184 kB]
Fetched 239 kB in 2s (88.5 kB/s)         
Preconfiguring packages …
dpkg: exim4-daemon-heavy: dependency problems, but removing anyway as you requested:
 mailutils depends on default-mta | mail-transport-agent; however:
  Package default-mta is not installed.
  Package mail-transport-agent is not installed.
  Package exim4-daemon-heavy which provides mail-transport-agent is to be removed.

 

(Reading database … 169307 files and directories currently installed.)
Removing exim4-daemon-heavy (4.89-2+deb9u3) …
dpkg: exim4-config: dependency problems, but removing anyway as you requested:
 exim4-base depends on exim4-config (>= 4.82) | exim4-config-2; however:
  Package exim4-config is to be removed.
  Package exim4-config-2 is not installed.
  Package exim4-config which provides exim4-config-2 is to be removed.
 exim4-base depends on exim4-config (>= 4.82) | exim4-config-2; however:
  Package exim4-config is to be removed.
  Package exim4-config-2 is not installed.
  Package exim4-config which provides exim4-config-2 is to be removed.

Removing exim4-config (4.89-2+deb9u3) …
Selecting previously unselected package ssmtp.
(Reading database … 169247 files and directories currently installed.)
Preparing to unpack …/ssmtp_2.64-8+b2_amd64.deb …
Unpacking ssmtp (2.64-8+b2) …
(Reading database … 169268 files and directories currently installed.)
Removing exim4-base (4.89-2+deb9u3) …
Selecting previously unselected package libgnutls-openssl27:amd64.
(Reading database … 169195 files and directories currently installed.)
Preparing to unpack …/libgnutls-openssl27_3.5.8-5+deb9u3_amd64.deb …
Unpacking libgnutls-openssl27:amd64 (3.5.8-5+deb9u3) …
Processing triggers for libc-bin (2.24-11+deb9u3) …
Setting up libgnutls-openssl27:amd64 (3.5.8-5+deb9u3) …
Setting up ssmtp (2.64-8+b2) …
Processing triggers for man-db (2.7.6.1-2) …
Processing triggers for libc-bin (2.24-11+deb9u3) …

 

As you see from above output local default Debian Linux Exim is removed …

Lets send a simple test email …

 

hipo@linux:~# ssmtp user@remote-mail-server.com
Subject: Simply Test SSMTP Email
This Email was send just as a test using SSMTP obscure client
via SMTP server.
^d

 

What is notable about ssmtp is that even though so obsolete today it supports of STARTTLS (email communication encryption) that is done via its config file

 

/etc/ssmtp/ssmtp.conf

 

4. Send Email from terminal using Mutt client
 

Mutt was and still is one of the swiff army of most used console text email clients along with Alpine and Fetchmail to know more about it read here

Mutt supports reading / sending mail from multiple mailboxes and capable of reading IMAP and POP3 mail fetch protocols and was a serious step forward over mailx. Its syntax pretty much resembles mailx cmds.

 

root@linux:~# mutt -s "Test Email" user@example.com < /dev/null

 

Send email including attachment a 15 megabytes MySQL backup of Squirrel Webmail

 

root@linux:~# mutt  -s "This is last backup small sized database" -a /home/backups/backup_db.sql user@remote-mail-server.com < /dev/null

 


5. Using simple telnet to test and send email (verify existence of email on remote SMTP)
 

As a Mail Server SysAdmin this is one of my best ways to test whether I had a server properly configured and even sometimes for the sake of fun I used it as a hack to send my mail 🙂
telnet is and will always be a great tool for doing SMTP issues troubleshooting.
 

It is very useful to test whether a remote SMTP TCP port 25 is opened or a local / remote server firewall prevents connections to MTA.

Below is an example connect and send example using telnet to my local SMTP on pc-freak.net (QMail powered (R) 🙂 )

sending-email-using-telnet-command-howto-screenshot

 

root@pcfreak:~# telnet localhost 25
Trying 127.0.0.1…
Connected to localhost.
Escape character is '^]'.
220 This is Mail Pc-Freak.NET ESMTP
HELO mail.pc-freak.net
250 This is Mail Pc-Freak.NET
MAIL FROM:<hipo@pc-freak.net>
250 ok
RCPT TO:<roots_bg@yahoo.com>
250 ok
DATA
354 go ahead
Subject: This is a test subject

 

This is just a test mail send through telnet
.
250 ok 1536440787 qp 28058
^]
telnet>

 

Note that the returned messages are native to qmail, a postfix would return a slightly different content, here is another test example to remote SMTP running sendmail or postfix.

 

root@pcfreak:~# telnet mail.servername.com 25
Trying 127.0.0.1…
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
220 mail.servername.com ESMTP Sendmail 8.13.8/8.13.8; Tue, 22 Oct 2013 05:05:59 -0400
HELO yahoo.com
250 mail.servername.com Hello mail.servername.com [127.0.0.1], pleased to meet you
mail from: systemexec@gmail.com
250 2.1.0 hipo@pc-freak.net… Sender ok
rcpt to: hip0d@yandex.ru
250 2.1.5 hip0d@yandex.ru… Recipient ok
data
354 Enter mail, end with "." on a line by itself
Hey
This is test email only

 

Thanks
.
250 2.0.0 r9M95xgc014513 Message accepted for delivery
quit
221 2.0.0 mail.servername.com closing connection
Connection closed by foreign host.


It is handy if you want to know whether remote MTA server has a certain Emailbox existing or not with telnet by simply trying to send to a certian email and checking the Email server returned output (note that the message returned depends on the remote MTA version and many qmails are configured to not give information on the initial SMTP handshake but returns instead a MAILER DAEMON failure error sent back to your sender address. Some MX servrers are still vulnerable to this attack yet, historically dreamhost.com. Below attack screenshot is made at the times before dreamhost.com fixed the brute force email issue.

Terminal-Verify-existing-Email-with-telnet

6. Using simple netcat TCP/IP Swiss Army Knife to test and send email in console

netcat-logo-a-swiff-army-knife-of-the-hacker-and-security-expert-logo
Other tool besides telnet of testing remote / local SMTP is netcat tool (for reading and writting data across TCP and UDP connections).

The way to do it is analogous but since netcat is not present on most Linux OSes by default you need to install it through the package manager first be it apt or yum etc.

# apt-get –yes install netcat


 

First lets create a new file test_email_content.txt using bash's echo cmd.
 

 

# echo 'EHLO hostname
MAIL FROM: hip0d@yandex.ru
RCPT TO:   solutions@pc-freak.net
DATA
From: A tester <hip0d@yandex.ru>
To:   <solutions@pc-freak.net>
Date: date
Subject: A test message from test hostname

 

Delete me, please
.
QUIT
' >>test_email_content.txt

 

# netcat -C localhost 25 < test_email_content.txt

 

220 This is Mail Pc-Freak.NET ESMTP
250-This is Mail Pc-Freak.NET
250-STARTTLS
250-SIZE 80000000
250-PIPELINING
250 8BITMIME
250 ok
250 ok
354 go ahead
451 See http://pobox.com/~djb/docs/smtplf.html.

Because of its simplicity and the fact it has a bit more capabilities in reading / writing data over network it was no surprise it was among the favorite tools not only of crackers and penetration testers but also a precious debug tool for the avarage sysadmin. netcat's advantage over telnet is you can push-pull over the remote SMTP port (25) a non-interactive input.


7. Using openssl to connect and send email via encrypted channel

 

root@linux:~# openssl s_client -connect smtp.gmail.com:465 -crlf -ign_eof

    ===
               Certificate negotiation output from openssl command goes here
        ===

        220 smtp.gmail.com ESMTP j92sm925556edd.81 – gsmtp
            EHLO localhost
        250-smtp.gmail.com at your service, [78.139.22.28]
        250-SIZE 35882577
        250-8BITMIME
        250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH
        250-ENHANCEDSTATUSCODES
        250-PIPELINING
        250-CHUNKING
        250 SMTPUTF8
            AUTH PLAIN *passwordhash*
        235 2.7.0 Accepted
            MAIL FROM: <hipo@pcfreak.org>
        250 2.1.0 OK j92sm925556edd.81 – gsmtp
            rcpt to: <systemexec@gmail.com>
        250 2.1.5 OK j92sm925556edd.81 – gsmtp
            DATA
        354  Go ahead j92sm925556edd.81 – gsmtp
            Subject: This is openssl mailing

            Hello nice user
            .
        250 2.0.0 OK 1339757532 m46sm11546481eeh.9
            quit
        221 2.0.0 closing connection m46sm11546481eeh.9
        read:errno=0


8. Using CURL (URL transfer) tool to send SSL / TLS secured crypted channel emails via Gmail / Yahoo servers and MailGun Mail send API service


Using curl webpage downloading advanced tool for managing email send might be  a shocking news to many as it is idea is to just transfer data from a server.
curl is mostly used in conjunction with PHP website scripts for the reason it has a Native PHP implementation and many PHP based websites widely use it for download / upload of user data.
Interestingly besides support for HTTP and FTP it has support for POP3 and SMTP email protocols as well
If you don't have it installed on your server and you want to give it a try, install it first with apt:
 

root@linux:~# apt-get install curl

 


To learn more about curl capabilities make sure you check cURL –manual arg.
 

root@linux:~# curl –manual

 

a) Sending Emails via Gmail and other Mail Public services

Curl is capable to send emails from terminal using Gmail and Yahoo Mail services, if you want to give that a try.

gmail-settings-google-allow-less-secure-apps-sign-in-to-google-screenshot

Go to myaccount.google.com URL and login from the web interface choose Sign in And Security choose Allow less Secure Apps to be -> ON and turn on access for less secure apps in Gmail. Though I have not tested it myself so far with Yahoo! Mail, I suppose it should have a similar security settings somewhere.

Here is how to use curl to send email via Gmail.

Gmail-password-Allow-less-secure-apps-ON-screenshot-howto-to-be-able-to-send-email-with-text-commands-with-encryption-and-outlook

 

 

root@linux:~# curl –url 'smtps://smtp.gmail.com:465' –ssl-reqd \
  –mail-from 'your_email@gmail.com' –mail-rcpt 'remote_recipient@mail.com' \
  –upload-file mail.txt –user 'your_email@gmail.com:your_accout_password'


b) Sending Emails using Mailgun.com (Transactional Email Service API for developers)

To use Mailgun to script sending automated emails go to Mailgun.com and create account and generate new API key.

Then use curl in a similar way like below example:

 

curl -sv –user 'api:key-7e55d003b…f79accd31a' \
    https://api.mailgun.net/v3/sandbox21a78f824…3eb160ebc79.mailgun.org/messages \
    -F from='Excited User <developer@yourcompany.com>' \
    -F to=sandbox21a78f824…3eb160ebc79.mailgun.org \
    -F to=user_acc@gmail.com \
    -F subject='Hello' \
    -F text='Testing Mailgun service!' \
   –form-string html='<h1>EDMdesigner Blog</h1><br /><cite>This tutorial helps me understand email sending from Linux console</cite>' \
    -F attachment=@logo_picture.jpg

 

The -F option that is heavy present in above command lets curl (Emulate a form filled in button in which user has pressed the submit button).
For more info of the options check out man curl.
 

 

9. Using swaks command to send emails from

 

root@linux:~# apt-cache show swaks|grep "Description" -B 10
Package: swaks
Version: 20170101.0-1
Installed-Size: 221
Maintainer: Andreas Metzler <ametzler@debian.org>
Architecture: all
Depends: perl
Recommends: libnet-dns-perl, libnet-ssleay-perl
Suggests: perl-doc, libauthen-sasl-perl, libauthen-ntlm-perl
Description-en: SMTP command-line test tool
 swaks (Swiss Army Knife SMTP) is a command-line tool written in Perl
 for testing SMTP setups; it supports STARTTLS and SMTP AUTH (PLAIN,
 LOGIN, CRAM-MD5, SPA, and DIGEST-MD5). swaks allows one to stop the
 SMTP dialog at any stage, e.g to check RCPT TO: without actually
 sending a mail.
 .
 If you are spending too much time iterating "telnet foo.example 25"
 swaks is for you.
Description-md5: f44c6c864f0f0cb3896aa932ce2bdaa8

 

 

 

root@linux:~# apt-get instal –yes swaks

root@linux:~# swaks –to mailbox@example.com -s smtp.gmail.com:587
      -tls -au <user-account> -ap <account-password>

 


The -tls argument (in order to use gmail encrypted TLS channel on port 587)

If you want to hide the password not to provide the password from command line so (in order not to log it to user history) add the -a options.

10. Using qmail-inject on Qmail mail servers to send simple emails

Create new file with content like:
 

root@qmail:~# vim email_file_content.text
To: user@mail-example.com
Subject: Test


This is a test message.
 

root@qmail:~# cat email_file_content.text | /var/qmail/bin/qmail-inject


qmail-inject is part of ordinary qmail installation so it is very simple it even doesn't return error codes it just ships what ever given as content to remote MTA.
If the linux host where you invoke it has a properly configured qmail installation the email will get immediately delivered. The advantage of qmail-inject over the other ones is it is really lightweight and will deliver the simple message more quickly than the the prior heavy tools but again it is more a Mail Delivery Agent (MDA) for quick debugging, if MTA is not working, than for daily email writting.

It is very useful to simply test whether email send works properly without sending any email content by (I used qmail-inject to test local email delivery works like so).
 

root@linux:~# echo 'To: mailbox_acc@mail-server.com' | /var/qmail/bin/qmail-inject

 

11. Debugging why Email send with text tool is not being send properly to remote recipient

If you use some of the above described methods and email is not delivered to remote recipient email addresses check /var/log/mail.log (for a general email log and postfix MTAs – the log is present on many of the Linux distributions) and /var/log/messages or /var/log/qmal (on Qmail installations) /var/log/exim4 (on servers running Exim as MTA).

http://pc-freak.net/images/linux-email-log-debug-var-log-mail-output

 Closure

The ways to send email via Linux terminal are properly innumerous as there are plenty of scripted tools in various programming languages, I am sure in this article,  also missing a lot of pre-bundled installable distro packages. If you know other interesting ways / tools to send via terminal I would like to hear it.

Hope you enjoyed, happy mailing !

Linux “bash: mail: command not found” error fix – Installing mail cli on Fedora, Debian, CentOS, Ubuntu


September 7th, 2018

bash-mail-command-not-found-error-fix-linux-installing-bsd-mailx-linux-fedora-debian-centos

If you just hired a cheap VPS with some kind of Linux OS installed or just installed a home Linux test system environment inside a Virtual Machine (such as VirtualBox) on your notebook and you want to read system email reports delivered by services (e.g. track possible errors with the OS) or simply test whether Email relaying is configured properly

and you end up with shell error:

bash: mail: command not found

because mail command is missing and you wonder what Linux package you have to install in order to have the lovely mail / mailx command back on the OS … this article will help you how you can easily solve it by installing the mailx binary file (e.g. install the package providing it depending on the Linux Distribution Operating System you face the problem on.

1.  Install mailx command on Debian / Ubuntu / Mint and other .deb based Linux
 

root@linux:~# apt-get install –yes bsd-mailx

 


An alternative way to install the mailx system binary is to install  mailutils package

which will set up the system with essential mail related programs and set up a light Exim MTA and common server Email surrounding  so you can easily configure default installed Exim Mail Server to serve as an SMTP relay (through dpkg-reconfigure exim4-config) command

 

root@linux~:# apt-get install mailutils
Reading package lists… Done
Building dependency tree       
Reading state information… Done
The following additional packages will be installed:
  exim4-base exim4-config exim4-daemon-light guile-2.0-libs libgsasl7
  libkyotocabinet16v5 libmailutils5 libmariadbclient18 libntlm0
  mailutils-common mysql-common
Suggested packages:
  eximon4 exim4-doc-html | exim4-doc-info spf-tools-perl swaks mailutils-mh
  mailutils-doc
The following NEW packages will be installed:
  exim4-base exim4-config exim4-daemon-light guile-2.0-libs libgsasl7
  libkyotocabinet16v5 libmailutils5 libmariadbclient18 libntlm0 mailutils

 

root@linux:~# dpkg-reconfigure exim4-config

 

dpkg-reconfigure-exim4-config-debian-linux-exim4-email-server-screenshot

2. Install mailx Linux command on Fedora / CentOS / RHEL e.g. Redhat based distros

 

On RPM based distrubutions the package name is different from debians to install it there you have to fetch and install current distro mailx .rpm

To install mail command on older CentOS / Redhat 7/6/5 distributions

 

[root@centos:~]# yum install -y mailx

 

On Fedora 22+ version yum auto-dependency tool was substituted with dnf:

 

[root@fedora:~]# dnf install -y mailx

 


3. Send Test Email with mail command

 

root@linux:~# echo "Sample Message Body" | mail -s "Whatever … Message Subject" remote_receiver@remote-server-email-address.com

 


Check the mailbox, you just sent the sample email, hopefully if MTA is relaying correctly and the SMTP relay is properly delivering the email should be at the recipient address.

Copy Paste Virtualbox enable between Linux Host and Guest Virtual Machine


September 6th, 2018

enable-copy-paste-between-linux-host-and-guest-OS-virtual-machine-virtualbox-vbox-logo

Copy / Pasting has become a vital part of our digital lifes, not having it in our days is insane, most of our System Administration jobs / Programmer or Office (Secretary) stuff daily without Copy / Paste is a unthinkable.

Lately I have been playing with Virtual Machines as I need to test some Dev Ops related stuff I install Jenkins write and test scripts and tools inside a VM, for that to happen I needed a working VM and because I am a Free Software supporter my VM of choice is usually Virtualbox. Even though Virtualbox is not the best and most performance efficient Virtual Machine out there for a small tasks and home usage dev / testing it does a great job, I have faced a problem with Copy / Pasting not working in VBox between the VM -> Guest OS, that's why I decided to write this step by step guide to help people who face the same issue.

For people who are starting with Oracle VirtualBox and you just have Created a New Virtual Machine and installed GNU / Linux or FreeBSD one common set back problem you will experience is the the Copy / Paste functionality between the VM Host Machine and the Guest won't be working out of the box. That's pretty much annoying thing and since usually one installs different Virtual Machines with different Operating Systems as a test bed for Software installs on top of various versions of OSes the lack of Copy, Paste functionality which is not there by default makes copy / pasting programming code / commands etc. between your local Desktop notebook or remote  server Machine and the VBox Virtualized Guest mostly unusuable …

Besides that integration (sharing resources / improved VM performance) between the Host (which is for example your Desktop notebook or PC) and the Virtualized Operating system be it some kind of UNIX (FreeBSD) etc. is not optimal until you enable an extra integration between VirtualBox and your (Host OS – that in my case is Debian 9 GNU / Linux but it can be Microsoft Windows XX as well or any other *nix like OS).In order for this Integration between your Host OS and Guest OS and Copy / Paste features between the two to work you need to Enable Guest Additions CD image (Software) that is part of VirtualBox installation on the Host OS and once enabled to mount (Attach the Vbox provided Virtual CD) from within the Guest OS (that in my case is another Debian 9 Stretch Linux that is installed inside the VM) and run a script that will install the necessery VirtualBox software inside the Guest OS Linux.
 

This is done in 3 easy steps:

STEP 1: Enable Vbox (Guest Additions CD Image)


virtualbox-VM-enable-insert-guest-additions-CD-image-menu-screenshot-debian-linux

 

Devices -> Insert Guest Additions CD image

 


2. Enable Shared Clipboard and Drag and Drop from the Virtual Machine

virtualbox-VM-enable-devices-shared-slipboard-bidirectional-menu-screenshot-debian-linux
 

Devices -> Shared Clipboard -> Bidirectional

 

 

virtualbox-VM-enable-devices-drag-and-drop-bidirectional-menu-screenshot-debian-linux
 

Devices -> Drag and Drop -> Bidirectional


3. Install development packages that are required by VBoxLinuxAdditions.run script

a) First thing is to use Virtual Box program interface menu and enable Insert Guest Additions CD Image option (assuming that you have the Linux Guest OS running).

virtualbox-VM-enable-insert-guest-additions-CD-image-menu-screenshot-debian-linux
 

Devices -> Insert Guest Additions CD image


http://pc-freak.net/images/virtualbox-virtual-machine-guest-os-settings

You can do it also from main VirtualBox interface (without having a Virtual machine running) by clicking over the desired Guest OS or pressting CTRL + S key combination

b) Next step is to mount the Virtual CD Guest Additions inside the Guest OS VM

To mount the Virtual content provided by the just ticked VBOX "Insert Guest Additions CD" menu, inside the Guest OS Linux, you have to run simple mount command:

 

 

root@debian:~# mount /media/cdrom1
mount: /dev/sr1 is write-protected, mounting read-only

 

c) Install necessery packages that VBoxLinuxAdditions.run will use

Further on VBoxLinuxAdditions.run (additions script) that adds the integration between the Host operating system and the Guest OS depends on some essential tools necessery for building (compiling) / developing software on Linux, this are:

1.) build-essential – a meta package that will install the following set of packages:

 

dpkg-dev fakeroot g++ g++-6 libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libdpkg-perl libfakeroot libfile-fcntllock-perl libstdc++-6-dev


which are necessery for creation .deb packages, compiling code and other various activitions. a) build-essential is a very important Debian based distro package and if you happen to be new to Linux or just starting learning how to develop programs on Linux, you will soon realize you will need it installed across almost all Linux OS hosts you maintain or develop on …

VirtualBox-VBoxLinuxAdditions-run-script-missing-headers-gcc-perl-how-to-fix

 

root@debian:~# apt-get install –yes build-essential linux-headers-$(uname -r) gcc perl make

 


2.) The linux-headers package in my case that it is is linux-headers-4.9.0-7-amd64 is important as it provides kernel header files that are necessery when compiling and integrating kernel modules VBoxLinuxAdditions.run script depends on it as it builds a kernel module that adds the integration (that enables Copy / Paste and performance inprovements) between the Host VM machine and Guest OS

3.) gcc perl packages this two provides the C compiler and perl scripting programming language interpreter which are also used by VBoxLinuxAdditions.run script.


Now run VBoxLinuxAdditions.run:

As you see in above screenshot the

root@debian:~#
root@debian:/media/cdrom1# sh VBoxLinuxAdditions.run
Verifying archive integrity… All good.
Uncompressing VirtualBox 5.2.18 Guest Additions for Linux……..
VirtualBox Guest Additions installer
Removing installed version 5.2.18 of VirtualBox Guest Additions…
Copying additional installer modules …
Installing additional modules …
VirtualBox Guest Additions: Building the VirtualBox Guest Additions kernel modules.  This may take a while.
VirtualBox Guest Additions: Running kernel modules will not be replaced until the system is restarted
VirtualBox Guest Additions: Starting.

 


If everything is fine you should get command output like above.

Above are the steps given to follow on current latest Debian Linux 9.5 (Stretch) as a Guest OS (this tutorial should be working also on Ubuntu / Mint and other deb package based distributions for RPM package based Linuxes Fedora / CentOS check out my previous toturial how to enable Linux Guest Virtualbox Addition on Redhat based distos.)

P. S.

What about Mac OS X since it is based on FreeBSD and it includes a lot of GNU licensing programs?

If you wonder whether on Mac OS X you don't need to run guest additions script such as (VBoxLinuxAdditions.run) on Mac Host -> Guest OS integration is done only by enabling the feature from VBox GUI interface via:
 

Devices -> Insert Guest Additions CD Image

What about enabling Copy / Paste on Linux Host VirtualBox hosting a Windows Guest OS?

 

 

Enabling Copy / Paste on a Windows Virtual Machine (I wrote about it earlier here), follows the same steps as in this tutorial.
The only difference is instead of running the VBoxLinuxAdditions.sh you have to navigate to the CD from Windows Explorer and run the appropriate .exe file depending on your architecture (X86 or 64 bit), the files to run on the virtual CDROM are one of the 3:

1. VBoxWindowsAdditions-amd64.exe
2. VBoxWindowsAdditions.exe
3. VBoxWindowsAdditions-x86.exe

virtualbox-vm-guest-additions-autoplay-web-on-windows-os

Follow the click through interface and you will have the Copy / Paste between your Linux -> Windows OS working.

Happy Copy / Pasting ! 🙂