Posts Tagged ‘sysadmins’

Generate and Add UUID for every existing Redhat / CentOS / RHEL network interface to configuration if missing howto

Saturday, August 5th, 2023

linux-fix-missing-uid-on-redhat-centos-fedora-networking-logo

If you manage old Linux machines it might be after the update either due to update mess or because of old system administrators which manually included the UUID to the config forgot to include it in the present network configuration in /etc/sysconfig/networking-scripts/ifcfg-* Universally Unique IDentifier (UUID)128-bit label I used a small one liner after listing all the existing configured LAN interfaces reported from iproute2 network stack with ip command. As this might be useful to someone out there here is the simple command that returns a number of commands to later just copy paste to console once verified there are no duplicates of the UUID already in the present server configuration with grep.

In overall to correct the configs and reload the network with the proper UUIDs here is what I had to do:


# grep -rli UUID /etc/sysconfig/network-scripts/ifcfg-*

No output from the recursive grep means UUIDs are not present on any existing interface, so we can step further check all the existing machines network ifaces and generate the missing UUIDs with uuidgen command

# ip a s |grep -Ei ': <'|sed -e 's#:##g' |grep -v '\.' |awk '{ print $2 }'
ifcfg-venet0
ifcfg-eth0
ifcfg-eth1

ifcfg-eth2
ifcfg-eth3

I've stumbled on that case on some legacy Linux inherited from other people sysadmins and in order to place the correct 

# for i in $(ip a s |grep -Ei ': <'|sed -e 's#:##g' |grep -v '\.' |awk '{ print $2 }'); do echo "echo UUID=$(uuidgen $i)"" >> ifcfg-$i"; done|grep -v '\-lo' 
echo UUID=26819d24-9452-4431-a9ca-176d87492b75 >> ifcfg-venet0
echo UUID=3c7e8848-0232-436f-a52a-46db9a03eb33 >> ifcfg-eth0
echo UUID=1fc0454d-bf23-417d-b960-571fc04754d2 >> ifcfg-eth1
echo UUID=5793c1e5-4481-4f09-967e-2cceda85c35f >> ifcfg-eth2
echo UUID=65fdcaf6-d271-4845-a8f1-0ec478c375d1 >> ifcfg-eth3


As you can see I exclude the loopback interface -lo from the ouput as it is not necessery to have UUID for it.
That's all folks problem solved. Enjoy

How to add local user to admin access via /etc/sudoers with sudo su – root / Create a sudo admin group to enable users belonging to group become superuser

Friday, January 15th, 2021

sudo_logo-how-to-add-user-to-sysadmin-group

Did you had to have a local users on a server and you needed to be able to add Admins group for all system administrators, so any local user on the system that belongs to the group to be able to become root with command lets say sudo su – root / su -l root / su – root?
If so below is an example /etc/sudoers file that will allow your users belonging to a group local group sysadmins with some assigned group number

Here is how to create the sysadmins group as a starter

linux:~# groupadd -g 800 sysadmins

Lets create a new local user georgi and append the user to be a member of sysadmins group which will be our local system Administrator (superuser) access user group.

To create a user with a specific desired userid lets check in /etc/passwd and create it:

linux:~# grep :811: /etc/passwd || useradd -u 811 -g 800 -c 'Georgi hip0' -d /home/georgi -m georgi

Next lets create /etc/sudoers (if you need to copy paste content of file check here)and paste below configuration:

linux:~# mcedit /etc/sudoers

## Updating the locate database
# Cmnd_Alias LOCATE = /usr/bin/updatedb

 

## Storage
# Cmnd_Alias STORAGE = /sbin/fdisk, /sbin/sfdisk, /sbin/parted, /sbin/partprobe, /bin/mount, /bin/umount

## Delegating permissions
# Cmnd_Alias DELEGATING = /usr/sbin/visudo, /bin/chown, /bin/chmod, /bin/chgrp

## Processes
# Cmnd_Alias PROCESSES = /bin/nice, /bin/kill, /usr/bin/kill, /usr/bin/killall

## Drivers
# Cmnd_Alias DRIVERS = /sbin/modprobe

Cmnd_Alias PASSWD = /usr/bin/passwd [a-zA-Z][a-zA-Z0-9_-]*, \\
!/usr/bin/passwd root

Cmnd_Alias SU_ROOT = /bin/su root, \\
                     /bin/su – root, \\
                     /bin/su -l root, \\
                     /bin/su -p root


# Defaults specification

#
# Refuse to run if unable to disable echo on the tty.
#
Defaults   !visiblepw

#
# Preserving HOME has security implications since many programs
# use it when searching for configuration files. Note that HOME
# is already set when the the env_reset option is enabled, so
# this option is only effective for configurations where either
# env_reset is disabled or HOME is present in the env_keep list.
#
Defaults    always_set_home
Defaults    match_group_by_gid

Defaults    env_reset
Defaults    env_keep =  "COLORS DISPLAY HOSTNAME HISTSIZE KDEDIR LS_COLORS"
Defaults    env_keep += "MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE"
Defaults    env_keep += "LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES"
Defaults    env_keep += "LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE"
Defaults    env_keep += "LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY"

#
# Adding HOME to env_keep may enable a user to run unrestricted
# commands via sudo.
#
# Defaults   env_keep += "HOME"
Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin

## Next comes the main part: which users can run what software on
## which machines (the sudoers file can be shared between multiple
## systems).
## Syntax:
##
##      user    MACHINE=COMMANDS
##
## The COMMANDS section may have other options added to it.
##
## Allow root to run any commands anywhere
root    ALL=(ALL)       ALL

## Allows members of the 'sys' group to run networking, software,
## service management apps and more.
# %sys ALL = NETWORKING, SOFTWARE, SERVICES, STORAGE, DELEGATING, PROCESSES, LOCATE, DRIVERS

## Allows people in group wheel to run all commands
%wheel  ALL=(ALL)       ALL

## Same thing without a password
# %wheel        ALL=(ALL)       NOPASSWD: ALL

## Allows members of the users group to mount and unmount the
## cdrom as root
# %users  ALL=/sbin/mount /mnt/cdrom, /sbin/umount /mnt/cdrom
## Allows members of the users group to shutdown this system
# %users  localhost=/sbin/shutdown -h now

%sysadmins            ALL            = SU_ROOT, \\
                                   NOPASSWD: PASSWD

## Read drop-in files from /etc/sudoers.d (the # here does not mean a comment)
#includedir /etc/sudoers.d

zabbix  ALL=(ALL) NOPASSWD:/usr/bin/grep


Save the config and give it a try now to become root with sudo su – root command

linux:~$ id
uid=811(georgi) gid=800(sysadmins) groups=800(sysadmins)
linux:~$ sudo su – root
linux~#

w00t Voila your user is with super rights ! Enjoy 🙂

 

How to add colorful random ASCII art picture and a bible verse on each SSH server login, joyout sysadmins life with cowsay, fortune, caca-utils and others

Tuesday, November 24th, 2020

Jesus-Christ-loves-the-world-ascii-art

There are pleny of console ASCII stuff out there that can make your console sysadmin boring life a little bit more funny and cherish some memories from the old times of 8 bit computers :).

One of this as I blogged earlier is cowsay and cowthink to generate a ascii picture with a cow with your custom message.
I've earlier blogged about that in my previous article Create ASCII Art Text bannners in Linux console / terminal with figlet and toilet

One of this cool things I'm using daily on my servers  is a cowsay console goodie together with a bash shell script that does visualize a random ASCII picture from a preset of pictures on each and every ssh login to my server.
The script I use is cowrand below is code:

#!/bin/bash
# cowsay pix randomizer by hip0
# it shows random ascii from the cowsay prog during logging. :]
a=0
b=1
cowrand='/etc/cowrand';
dir='/usr/share/cowsay/cows';
var=`ls -1 $dir | wc -l | awk '{ print $1}'`
#RANGE=$var
number=$RANDOM
let "number %= $var"
var1=`ls -1 $dir | head -n $number | tail -n 1 | head -n 1`
if [ -z “$var1” ]; then
$cowrand;
else
/usr/bin/cowsay -f $var1 Welc0m3 t0 pC-fREAK … Enj0y.
fi

 

The script is set as executable under /etc/cowrand

hipo@pcfreak:~$ ls -al /etc/cowrand
-rwxr-xr-x 1 hipo hipo 432 Nov 24 19:21 /etc/cowrand*

I've set this script to my /etc/profile to auto start on every login on my Debian Linux systems right after the comments like so:

hipo@pcfreak:~$ grep -i cowrand -A 2 -B 3 /etc/profile
# /etc/profile: system-wide .profile file for the Bourne shell (sh(1))
# and Bourne compatible shells (bash(1), ksh(1), ash(1), …).
echo '';
/etc/cowrand | lolcat
echo '';
#/usr/bin/verse

As you can see to make my life even more funnier, I've installed another fun command lolcat

lolcat-screenshot

hipo@pcfreak:~$ apt-cache show lolcat |grep -i desc -A 3
Description-en: colorful `cat`
 lolcat concatenates files like the UNIX `cat` program, but colors it for the
 lulz in a rainbow animation. Terminals with 256 colors and animations are
 supported.

Description-md5: 86f992d66ac74197cda39e0bbfcb549d
Homepage: https://github.com/busyloop/lolcat
Ruby-Versions: all
Section: games


You can think of lolcat as a standard cat command that has been made to print in colors, this gives a funny results.

cowrand-script-lolcat-os-release-how-to-make-your-linux-login-prompt-funnier

To add some spice to everything nice as a recipee for thethe creation of powerpuff girls, I've come up with a way to use fortune
console tool that uses to print quotes out of a database to use as a source a big database containing the Holy Bible books of Old and New Testament Books. The fortune prints me out a quote extract from the bible on each and every remote SSH login to my machine. The content of this bible database for fortune bible_quotes_fortune.tar.gz can be downloaded and used from here.

The command used to print out a verse from the holy bible is:
 

 

hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
For if thou refuse to let them go, and wilt hold them still,
        — Exodus 9:2
hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
And when the queen of Sheba heard of the fame of Solomon concerning
the name of the LORD, she came to prove him with hard questions.
        — 1 Kings 10:1
hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
And Shelemiah, and Nathan, and Adaiah,
        — Ezra 10:39
hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
For by thee I have run through a troop: by my God have I leaped
over a wall.
        — 2 Samuel 22:30
hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
Unto the place of the altar, which he had make there at the first:
and there Abram called on the name of the LORD.
        — Genesis 13:4
hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
And there shall dwell in Judah itself, and in all the cities thereof
together, husbandmen, and they that go forth with flocks.
        — Jeremiah 31:24
hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
And he hath put a new song in my mouth, even praise unto our God:
many shall see it, and fear, and shall trust in the LORD.
        — Psalms 40:3
hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
And Jehoshaphat made peace with the king of Israel.
        — 1 Kings 22:44
 

 

The fortune is really awesome as it reminds me often of a verses from Holy Bible I often forget, the database is using the all famous King James Bible famous as (KJB) / (KJV) from 1611 this bible version that is like a protestant standard nowadays takes its name after James VI and I (James Charles Stuart; 19 June 1566 – 27 March 1625 – King of Scotland and Ireland) who was the sponsor of KJV collection and print.

Finally after adding the /usr/games/fortune -s /usr/local/fortune/ to the beginning of /etc/profile together with cowsay and cowrand I got this beautiful and educational result that combines fun with wisdom, below is example of what you will get after you  do a remote ssh login;

 

ssh your-machine.com

cowrand-script-lolcat-os-release-how-to-make-your-linux-login-prompt-funnier_1

cowrand-script-lolcat-os-release-how-to-make-your-linux-login-prompt-funnier_2

cowrand-script-lolcat-os-release-how-to-make-your-linux-login-prompt-funnier_3

Those who have a Linux Graphical Environment desktop might also enjoy xcowsay

Another must I recommend to the text geeks is the caca-utils package which contains cool things such as aafire (cacafire)

cacaview-fire-screenshot-ascii-art

Or (Image to text converter) img2txt / cacaview (a text console picture viewer) that could give you a raw idea on how a png / jpg picture looks like (or at least the picture shapes) without a need for a GUI picture viewer such as Eye of the Gnome.

bear-for-you-picture-rose

Here is a original bear

cacaview-a-bear-for-you-picture-in-plain-text-ascii

And here is the one you'll see in cacaview 🙂
To read more about cacaview I have and its uses, check my previous article Viewing JPEG,GIF and PNG in ASCII with cacaview in Linux.
If you want to show off even more as a '1337 h4x0r' you might also show your sysadm 1337 5K!11Z to colleagues by showng them how you check weather via console (i've a separate article for how to ASCII art check colorful weather forecast via console / terminal ).

If you're too bored in your daily sys admin job, you might make some fun and take some useless effort to install ASCII Art Aquarium ASCIIQUARIUM

asciiquarium1

asciiquarium2

asciiquarium3

If you're crazy enough and want to torture your other sysadmin colleagues and a get a nice prank, you might install and set asciiquarium to auto run for their specific account on each and every login to some server until they control C or if you're a bit evil you can even set a small auto load on account login via ~/.bashrc shell script to 'Disable CTRL + C' combination 🙂
 

Of course there is plenty of other cool ASCII games and stuff. I've collected some of them by launching the Play Cool Ascii games service on my machine for ASCII art geeks to test out some ASCII games here.

 

VIM and VI UNIX text editor syntax highlighting and howto add remove code auto indent

Tuesday, February 4th, 2014

vim-vi-linux-text-editor-logo-vim-highlighting how to turn vim syntax highlighting on linux

For my daily system administration job I have to login to many SuSE Linux servers and do various configugration edits.
The systems are configured in different ways and the only text editors available across all servers I can use are VI and VIM (VI Improved).

As I usually had to edit configuration files and scripts and I'm on SSH color terminal its rather annoying that on some of the servers opening a file with VIM is not displayed with SYNTAX HIGHLIGHTING. Not having syntax highlighting is ugly and makes editting ugly and unreadable.
Thus it is useful to enable VI syntax highlighting straight into the file being editted. I suspect many novice sysadmins might not know how to turn syntax highlighting in vi so here is how.
 

Turn Syntax Highlighting in VIM

 

1. Open file with vim lets say Apache configuration

# vim /etc/apache2/apache2.conf

2. Press (Esc) Escape and ":" from kbd and then type in syntax on

:syntax on

vim-syntax-highlighting-howto-syntax-on-picture-screenshot-apache-config

To Turn On / Off VI Syntax Highlighting permanent add ":syntax on"
into ~/.vimrc

~/.vimrc file is red automatically on VIM start, so right after :syntax on is appended in it on relaunch vim will start showing colorfully.

Enjoy ! 🙂

 

How to keep track of All User accounts executed commands, highest CPU consumers and user times on Linux

Tuesday, February 5th, 2013

Linux accounting keeping an eye on all user run commands time accounting find cpu eaters

For people interested into statistics of how Linux existing users are spending, there log in times and what kind of commands each of users is executing, take a look at acct
acct is existing on all mainstream Linux distributions is a great sysadmin tool. acct is a great tool whether you have a system where a multitude of users you don't trust has to be monitored. It is an absolutely must have for anyone willing to run, lets say  experimental honeypot or  free shell host. acct is useful for paranoid sysadmins who like to always knows what there users are running as well as in situation where some of users is suspected to be a potential cracker trying to root the host.

Below is description of acct package on Debian:

# apt-cache show acct| grep -i description -A 8
Description: The GNU Accounting utilities for process and login accounting
 GNU Accounting Utilities is a set of utilities which reports and summarizes
 data about user connect times and process execution statistics.
 .
 "Login accounting" provides summaries of system resource usage based on connect
 time, and "process accounting" provides summaries based on the commands
 executed on the system.
 .
 The 'last' command is provided by the sysvinit package and not included here.

To start using acct, just install it with usual:

# apt-get install --yes acct

(Whether on Debian / Ubuntu Linux);

On Fedora, CentOS and RHEL and other RPM based Linuxes issue;

yum --y install psacct

On deb based Linux distributions, whether acct collects statistics is controlled via:

/etc/default/acct

# cat /etc/default/acct
# Defaults for acct

# If you want to keep acct installed, but not started automatically, set this
# variable to 0. Because /etc/cron.daily/acct calls the initscript daily, it is
# not sufficient to stop acct once after booting if your machine remains up.
ACCT_ENABLE="1"

# Amount of days that the logs are kept.
ACCT_LOGGING="30"

After installed to start collecting user "process accounting" data run acct via init script;

# /etc/init.d/acct start
Turning on process accounting, file set to '/var/log/account/pacct'.
Done..

The file gathering info on system usage, CPU load, user ran commands /var/log/account/psacct is a binary and unreadable tailing it with tail -f .

On CentOS / Fedora Linux to Enable acct account statistics gathering in future boot and from present moment on do;

# chkconfig psacct on
# /etc/init.d/psacct start

1. Find out all commands executed by Linux user account (lastcomm)

Once user accounting is running to get information of every command ever executed on user shell use lastcomm cmd. For example:

# lastcomm hipo

bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.03 secs Tue Feb  5 00:20
sed                    hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
uname                  hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
dircolors              hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
uname                  hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.03 secs Tue Feb  5 00:20
sed                    hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
uname                  hipo     pts/1      0.00 secs Tue Feb  5 00:20
bash              F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
id                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
mesg                   hipo     pts/1      0.00 secs Tue Feb  5 00:20
verse                  hipo     pts/1      0.00 secs Tue Feb  5 00:20
cowrand                hipo     pts/1      0.00 secs Tue Feb  5 00:20
cowsay                 hipo     pts/1      0.03 secs Tue Feb  5 00:20
cowrand           F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
head                   hipo     pts/1      0.00 secs Tue Feb  5 00:20
tail                   hipo     pts/1      0.00 secs Tue Feb  5 00:20
head                   hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
cowrand           F    hipo     pts/1      0.00 secs Tue Feb  5 00:20
awk                    hipo     pts/1      0.00 secs Tue Feb  5 00:20
wc                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20

A lot of the initial commands shown to run on pts/1 is not actual commands, by the user but are just stuff run on user login time via /etc/bash.bashrc, /etc/profile, ~/.bashrc. ~/.bash_profile.

lastcomm displayed output from 2nd column is a special flag giving more information on how and for what purpose command was executed. In above output
F
– indicates the command run after a fork.
X – is returned if a command exit with SIGTERM (kill signal)
D – in case of generated command core dump (D is good one to look for whether checking a suspicious user profile, as it is so common exploits use core dumping to get root superuser access)
S – means the command is run with superuser privileges (this one you will see usually whether inspecting user profile of a cracker who run exploit using core dump – a lot of Ds followed by some shell code to run as superuser)

2. Get statistics on CPU use time of services (daemons) and user accounts

psacct is very handy, whether you have CPU server overloads and you have difficulty finding out what are the "CPU hungry processes". To get those use summarized accounting information tool;

# sa -m
                                     2619      31.06re       0.54cp         0avio      2907k
root                                 2448      30.19re       0.52cp         0avio      2817k
www-data                               33       0.06re       0.02cp         0avio      3687k
hipo                                   72       0.15re       0.01cp         0avio      6217k
qscand                                 11       0.36re       0.00cp         0avio      5326k
vpopmail                               48       0.25re       0.00cp         0avio      1486k
qmails                                  6       0.00re       0.00cp         0avio       968k
sshd                                    1       0.04re       0.00cp         0avio     12632k

-m (prints user summary).

3. Find all system users running certain commands

Another good use of lastcomm command is to grep over all users executed command for precise commands of interest. One very good use case is if you catch a system abuser running certain exploit or DoS tool on the host and you want to make sure no-one else on the system doesn't try running it.

# lastcomm ls
ls                     www-data __         0.00 secs Tue Feb  5 00:40
ls                     www-data __         0.00 secs Tue Feb  5 00:30
ls                     hipo     pts/7      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     hipo     pts/1      0.00 secs Tue Feb  5 00:20
ls                     www-data __         0.00 secs Tue Feb  5 00:20
ls                     root     pts/0      0.00 secs Tue Feb  5 00:10
ls                     root     pts/0      0.00 secs Tue Feb  5 00:10
ls                     www-data __         0.00 secs Tue Feb  5 00:10
 

4. Get statistics of most active system users in hours

There is one tool called ac, which is similar in what it does to last command, just like last it uses /var/log/wtmp binary log file to get its user login times stats . The difference is ac provides more and better structured user login time length info.

Its very useful if you want to have idea, which user spends most time connected to host.

$ ac -p
    sic                                  4.86
    hipo                                 4.80
    root                                25.80
    play                                 0.02

To get general info on how much overall hours all existing users spend doing stuff on node;

$ ac total 35.61

To know which days from the month users were most active:

$ ac -d
Feb 1 total 14.54
Feb 2 total 0.97
Feb 3 total 12.47
Feb 4 total 5.96
Today total 1.73

Using perl and sed to substitute strings in multiple files on Linux and BSD

Friday, August 26th, 2011

Using perl and sed to replace strings in files on Linux, FreeBSD, OpenBSD, NetBSD and other UnixOn many occasions when had to administer on Linux, BSD, SunOS or any other *nix, there is a need to substitute strings inside files or group of files containing a certain string with another one.

The task is not too complex and many of the senior sysadmins out there would certainly already has faced this requirement and probably had a good idea on files substitution with perl and sed, however I’m quite sure there are dozen of system administrators out there who did not know, how and still haven’t faced a situation where there i a requirement to substitute from a command shell or via a scripting language.

This article tagets exactly these system administrators who are not 100% sys op Gurus 😉

1. Substitute text strings inside files on Linux and BSD with perl

Perl programming language has originally been created to do a lot of text manipulation as well as most of the Linux / Unix based hosts today have installed working copy of perl , therefore using perl as a mean to substitute one string in a file to another one is maybe the best way to completet the task.
Another good thing about perl is that text processing with it is said to be in most cases a bit faster than sed .
However it is still dependent on the string to be substituted I haven’t done benchmark tests to positively say 100% that always perl is quicker, however my common sense suggests perl will be quicker.

Now enough talk here is a very simple way to substitute a reoccuring, text string inside a file with another chosen one is like so:

debian:~# perl -pi -e 's/foo/bar/g' file1 file2

This will substitute the string foo with bar everywhere it’s matched in file1 and file2

However the above code is a bit “dangerous” as it does not preserve a backup copy of the original files, where string is substituted is not made.
Therefore using the above command should only be used where one is 100% sure about the string changes to be made.

Hence a better idea whether conducting the text substitution is to keep also the original file backup under a let’s say .bak extension. To achieve that I use perl as follows:

freebsd# perl -i.bak -p -e 's/syzdarma/magdanoz/g;' file1 file2

This command creates copies of the original files file1 and file2 under the names file1.bak and file2.bak , the files file1 and file2 text occurance of strings syzdarma will get substituted with magdanoz using the option /g which means – (substitute globally).

2. Substitute string in all files inside directory using perl on Linux and BSD

Every now and then the there is a need to do manipulations with large amounts of files, I can’t right now remember a good scenario where I had to change all occuring matching strings to anther one to all files located inside a directory, anyhow I’ve done this on a number of occasions.

A good way to do a mass file string substitution on Linux and BSD hosts equipped with a bash shell is via the commands:

debian:/root/textfiles:# for i in $(echo *.txt); do perl -i.bak -p -e 's/old_string/new_string/g;' $i; done

Where the text files had the default txt file extension .txt

Above bash loop prints each of the files located in /root/textfiles and substitutes everywhere (globally) the old_string with new_string .

Another alternative to the above example to replace multiple occuring text string in all files in multiple directories is possible using a combination of shell commands grep, perl, sort, uniq and xargs .
Let’s say that one wants to match everywhere inside the root directory and all the descendant directories for files with a custom string and substitute it to another one, this can be done with the cmd:

debian:~# grep -R -files-with-matches 'old_string' / | sort | uniq | xargs perl -pi~ -e 's/old_string/new_string/g'

This command will lookup for string old_string in all files in the / – root directory and in case of occurance will substitute with new_string (This command’s idea was borrowed as an idea from http://linuxadmin.org so thx.).

Using the combination of 5 commands, however is not very wise in terms of efficiency.

Therefore to save some system resources, its better in terms of efficiency to take advantage of the find command in combination with xargs , here is how:

debian:~# find / | xargs grep 'old_string' -sl |uniq | xargs perl -pi~ -e 's/old_string/new_string/g'

Once again the find command example will do exactly the same as the substitute method with grep -R …

As enough is said about the way to substitute text strings inside files using perl, I will further explain how text strings can be substituted using sed

The main reason why using sed could be a better choice in some cases is that Unices are not equipped by default with perl interpreter. In general the amount of servers who contains installed sed compared to the ones with perl language interpreter is surely higher.

3. Substitute text strings inside files on Linux and BSD with sed stream editor

In many occasions, wether a website is hosted, one needs to quickly conduct a change in string inside all files located in a directory, to resolve issues with static urls directly encoded in html.
To achieve this task here is a code using two little bash script loops in conjunctions with sed, echo and mv commands:

debian:/var/www/website# for i in $(ls -1); do cat $i |sed -e "s#index.htm#http://www.webdomain.com/#g">$i.new; done
debian:/var/www/website# for i in $(ls *.new); do mv $i $(echo $i |sed -e "s#.new##g"); done

The above command sed -e “s#index.htm#http://www.webdomain.com/#g”, instructs sed to substitute all appearance of the text string index.htm to the new text string http://www.webdomain.com

First for bash loop, creates all the files with substituted string to file1.new, file2.new, file3.new etc.
The second for loop uses mv to overwrite the original input files file1, file2, file3, etc. with the newly created ones file1.new, file2.new, file3.new

There is a a way shorter way to conclude the same text substitutions task using a simpler one liner with only using sed and bash’s eval capabilities, here is how:

debian:/var/www/website# sed -i 's/old_string/new_string/g' *

Above command will change old_string to new_string inside all files in directory /var/www/website

Whether a change has to be made with less than 1024 files using this method might be more efficient, however whether a text substitute has to be done to let’s say 5000+ the above simplistic version will not work. An error of Argument list too long will prevent the sed -i ‘s/old_string/new_string/g’ to complete its task.

The above for loop 2 liner should be also working without problems with FreeBSD and the rest of BSD derivatives, though I have not tested it yet, hence any feedback from FreeBSD guys is mostly welcome.

Consider that in order to have the for loops commands work on FreeBSD or NetBSD, they have to be run under a bash shell.
That’s all folks thanks the Lord for letting me write this nice article, I hope it gives some insights on how multiple files text replace on Unix works .
Cheers 😉