Posts Tagged ‘gurus’

Linux Bash Logging log everything. Prevent user from delete his history and keep record of every command User ever Run

Tuesday, March 17th, 2026

make_bash_history_permanent-how-to-keep-every-user-command-forever-prevent-users-from-deleting-their-bash-history-on-linux

Whether you're managing servers, writing scripts, or troubleshooting complex systems, one of the most valuable tools at your disposal is your command history. But the default Bash history has serious limitations: it’s easy to lose, doesn't timestamp by default, and doesn't log everything in real time.

What if you could keep a permanent, timestamped, real-time log of every command you ever run in Bash?

Good news: you can.

In this guide, we’ll walk through how to set up robust, automatic Bash logging to track every command you type—across sessions, with full timestamps, and even with user and host information. Ideal for system administrators, developers, auditors, or anyone who wants to maintain a clear, searchable audit trail.

Why Bash Logging Persistence So Important ?

Before we dive into the how, let's understand the why:

  • Accountability – Know exactly what commands were run, by whom, and when.
  • Auditability – Great for security reviews or compliance requirements.
  • Troubleshooting – Trace back actions that caused issues.
  • Documentation – Reuse commands or share with teammates.
  • Forensics – Investigate suspicious activity.

How Bash History Behaves  ( By Default )

Without any config everyone knows , Bash uses a file called ~/.bash_history in $HOME to save command history.

What is tricky here:

  • .bash_history not written to immediately – only when the session exits.
  • It can be overwritten by other sessions.
  • It lacks timestamps unless explicitly configured.
  • It doesn’t log failed attempts or commands from other users.

In this short article I'll show you one of the ways on how to make .bash_history keeps the record for you even though some user tries to hide tihngs by running the commands and exiting the shell abnormally by killing it with the well known command by hackers and sysadmin gurus:


$ kill -9 $$

The command forces the user you have logged into to kill the process of the bash (-bash). 

Here is how.

Enable Advanced Bash Logging

1. Enable Timestamps in History

Add this line to your ~/.bashrc or ~/.bash_profile:

export HISTTIMEFORMAT="%F %T "

This formats the date/time as YYYY-MM-DD HH:MM:SS.

After modifying the file, run:

source ~/.bashrc

Now, run:

history

And you’ll see timestamps next to your commands.

2. Increase History Size

The default history size is often too small. Let’s increase it:

export HISTSIZE=100000

export HISTFILESIZE=200000


Add these to ~/.bashrc as well.

3. Log Commands Immediately (Across Sessions)

By default, Bash only writes history when the shell exits. To log commands in real time, add the following to ~/.bashrc:

# Append to the history file, don't overwrite it

shopt -s histappend

# Immediately append command to history file after execution

PROMPT_COMMAND='history -a; history -n'

Explanation:

  • history -a: Append current session's command to ~/.bash_history
  • history -n: Read any new lines from the file (from other sessions)

4. Log All Commands to a Separate File (for each User)

To keep a separate, detailed log, you can use the trap command in combination with logger, or write to a custom file.

Add this to your ~/.bashrc:

LOG_FILE="$HOME/.bash_command_log"

trap 'echo "$(date "+%F %T") | $(whoami)@$(hostname) | $(pwd) | $BASH_COMMAND" >> "$LOG_FILE"' DEBUG

This logs every command as for example:

2025-10-10 14:25:02 | master_app@server01 | /var/www | systemctl restart nginx

This file can grow large over time – consider rotating it regularly with logrotate or similar tools.
To prevent the file 100% from being modified by the user itself you can make the log file  immutable with command

# chattr +i $HOME/.bash_command_log


5. Guarantee log security, Make copy of Logs to prevent hackers to modify them

If logging for audit/security purposes:

  • Store logs in append-only files (chattr +a logfile on ext4 FS)
  • store files with rsyslog service (see below)
  • Use remote logging (e.g., send via logger to syslog  / rsyslog or any other centralized logging service) / logcollector etc.
  • Monitor for tampering or suspicious gaps

6. Store file with rsyslog service

Create the file and set it proper permissions

# touch /var/log/bash_audit.log
# chmod 600 /var/log/bash_audit.log
# chown root:root /var/log/bash_audit.log

# vim /etc/rsyslog.d/bash_audit.conf

Add:

if $programname == 'bash_audit' then /var/log/bash_audit.log
& stop

# systemctl restart rsyslog


To later verify it works fine

# tail -f /var/log/bash_audit.log
# journalctl -t bash_audit

 

6. Add Global Bash Logging for All Users

Assuming that the bash_audit set program / name tag is already done as in step 5.
To apply logging system-wide, Edit /etc/profile /etc/bash_profile or /etc/bash.bashrc and include the same trap cmd and logging is ready. Ensure:

  • The log file is writable by users (or add users to a group that can append to file) or modify the command to use sudo logger for centralized syslog.
  • You test it carefully before deploying to all users.

     

     

     

    An improved wide user version of trap command would be something like this

# Bash command logging (readable layer)

trap 'CMD=$(history 1 | sed "s/^[ ]*[0-9]\+[ ]*//");
MSG="$(date "+%F %T") | $(whoami)@$(hostname) | $(pwd) | $CMD";

/usr/bin/logger -t bash_audit "$MSG"
' DEBUG

Make these two env variables read only for additional hardening 

readonly PROMPT_COMMAND
readonly HISTFILE

Note that you will need to edit passwordless login for sudo to logger

  • Setup auditd to make file read only

# apt install auditd audispd-plugins –yes

  • Test it with auditctl

# auditctl -a always,exit -F arch=b64 -S execve -F auid>=1000 -F auid!=4294967295 -k cmdlog
# auditctl -a always,exit -F arch=b32 -S execve -F auid>=1000 -F auid!=4294967295 -k cmdlog

  • Make rules permanent via cmdlog.rules

# vim /etc/audit/rules.d/cmdlog.rules

-a always,exit -F arch=b64 -S execve -F auid>=1000 -F auid!=4294967295 -k cmdlog
-a always,exit -F arch=b32 -S execve -F auid>=1000 -F auid!=4294967295 -k cmdlog

  • Load and lock audit rules

# augenrules –load
# auditctl -e 2

  • Check audit logs

# ausearch -k cmdlog -i
exe="/usr/bin/ls" argc=1 a0="ls"

7. Rotate Log Files Automatically with logrotate

Create a logrotate config like /etc/logrotate.d/bash_command_log:

/home/*/.bash_command_log {
daily
rotate 7
compress
missingok
notifempty
}

/var/log/bash_audit.log {
daily
rotate 7
compress
missingok
notifempty
}


This keeps logs for 7 days and compresses old ones.

8. Test Every command Logging is permanenty stored

After setting bash logging up up:

  1. Open a new terminal client with SSH session
  2. Run a few commands
  3. Check ~/.bash_command_log (or your alternative configured log location)

You should see a real-time record of every command executed.

Use tools like grep, awk, or fzf Command fuzzy finder to search through your command log efficiently. Example:

grep apt ~/.bash_command_log

You can further automate it and deploy it to multiple servers with Ansible or some shell scripting.
If you need it Ask me how to automate it?
Ask me how to automate it with Ansible or a shell script.

Wrapping it Up

With just a few lines in Bash config, basic history feature becomes a persistent, and timestamped static record  that’s invaluable for system admins, developers, and security teams.

Summary Checklist

  • Enable HISTTIMEFORMAT
  • Increase history size
  • Append history in real time
  • Log every command with trap DEBUG
  • Optionally send to rsyslog / syslogd / systemd-journald or other central log server (Fluentd / ELK Stack / Graylog)
  • Rotate logs with logrotate

Tags: , , , , , , , , , , , , , , , , , , , ,
Posted in Curious Facts, Linux, Security, System Administration | No Comments »

Use rsync to copy from files from destination host to source host (rsync reverse copy) / few words on rsync

Monday, January 9th, 2012

I've recently had to set up a backup system to synchronize backup archive files between two remote servers and as I do usually with this situation I just set up a crontab job to periodically execute rsync to copy data from source server to the destination server . Copying SRC to DEST is the default behaviour rsync uses, however in this case I had to copy from the destination server to the source server host (in other words sync files the reversely.

The usual way to copy with rsync via SSH (from SRC to DEST) is using a cmd line like:

debian:~$ /usr/bin/rsync -avz -e ssh backup-user@xxx.xxx.xxx.xxx:/home/backup-user/my-directory .

Where the xxx.xxx.xxx.xxx is my remote server IP with which files are synched.
According to rsync manual, the proposed docs SYNOPSIS is in the format;
Local: rsync [OPTION…] SRC… [DEST

Obviusly the default way to use rsync is to copy source to destination which I used until now, but in this case I had to the other way around and copy files from a destination host to the source server. It was logical that swapping the SRC and DEST would complete my required task. Anyways I consulted with some rsync gurus in irc.freenode.net , just to make sure it is proper to just swap the SRC, DEST arguments.
I was told this is possible, so I swapped args;

debian:~$ /usr/bin/rsync -avz -e ssh . backup-user@xxx.xxx.xxx.xxx:/home/backup-user/my-directory
...

Surprisingly this worked 😉 Anyways I was adviced by by a good guy nick named scheel , that putting -e ssh to command line is generally unnecessery except if there is no some uncommon used SSH port over which the data is transferred. An example case in which -e 'ssh is necessery would be if transferring via lets say SSH port 1234;

rsync -avz -e 'ssh -p1234' /source user@host:/dest

In all other cases omitting '-e ssh' is better as '-e ssh' is rsync default. Therefore my final swapped line I put in cron to copy from a destinatio to source host with rsync looked like so:

05 03 2 * * /usr/bin/ionice -c 3 /usr/bin/rsync -avz my-directory backup-user@xxx.xxx.xxx.xxx:/home/backup-user/ >/dev/null 2>&1
 

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in FreeBSD, Linux, System Administration | No Comments »

Using perl and sed to substitute strings in multiple files on Linux and BSD

Friday, August 26th, 2011

Using perl and sed to replace strings in files on Linux, FreeBSD, OpenBSD, NetBSD and other UnixOn many occasions when had to administer on Linux, BSD, SunOS or any other *nix, there is a need to substitute strings inside files or group of files containing a certain string with another one.

The task is not too complex and many of the senior sysadmins out there would certainly already has faced this requirement and probably had a good idea on files substitution with perl and sed, however I’m quite sure there are dozen of system administrators out there who did not know, how and still haven’t faced a situation where there i a requirement to substitute from a command shell or via a scripting language.

This article tagets exactly these system administrators who are not 100% sys op Gurus 😉

1. Substitute text strings inside files on Linux and BSD with perl

Perl programming language has originally been created to do a lot of text manipulation as well as most of the Linux / Unix based hosts today have installed working copy of perl , therefore using perl as a mean to substitute one string in a file to another one is maybe the best way to completet the task.
Another good thing about perl is that text processing with it is said to be in most cases a bit faster than sed .
However it is still dependent on the string to be substituted I haven’t done benchmark tests to positively say 100% that always perl is quicker, however my common sense suggests perl will be quicker.

Now enough talk here is a very simple way to substitute a reoccuring, text string inside a file with another chosen one is like so:

debian:~# perl -pi -e 's/foo/bar/g' file1 file2

This will substitute the string foo with bar everywhere it’s matched in file1 and file2

However the above code is a bit “dangerous” as it does not preserve a backup copy of the original files, where string is substituted is not made.
Therefore using the above command should only be used where one is 100% sure about the string changes to be made.

Hence a better idea whether conducting the text substitution is to keep also the original file backup under a let’s say .bak extension. To achieve that I use perl as follows:

freebsd# perl -i.bak -p -e 's/syzdarma/magdanoz/g;' file1 file2

This command creates copies of the original files file1 and file2 under the names file1.bak and file2.bak , the files file1 and file2 text occurance of strings syzdarma will get substituted with magdanoz using the option /g which means – (substitute globally).

2. Substitute string in all files inside directory using perl on Linux and BSD

Every now and then the there is a need to do manipulations with large amounts of files, I can’t right now remember a good scenario where I had to change all occuring matching strings to anther one to all files located inside a directory, anyhow I’ve done this on a number of occasions.

A good way to do a mass file string substitution on Linux and BSD hosts equipped with a bash shell is via the commands:

debian:/root/textfiles:# for i in $(echo *.txt); do perl -i.bak -p -e 's/old_string/new_string/g;' $i; done

Where the text files had the default txt file extension .txt

Above bash loop prints each of the files located in /root/textfiles and substitutes everywhere (globally) the old_string with new_string .

Another alternative to the above example to replace multiple occuring text string in all files in multiple directories is possible using a combination of shell commands grep, perl, sort, uniq and xargs .
Let’s say that one wants to match everywhere inside the root directory and all the descendant directories for files with a custom string and substitute it to another one, this can be done with the cmd:

debian:~# grep -R -files-with-matches 'old_string' / | sort | uniq | xargs perl -pi~ -e 's/old_string/new_string/g'

This command will lookup for string old_string in all files in the / – root directory and in case of occurance will substitute with new_string (This command’s idea was borrowed as an idea from http://linuxadmin.org so thx.).

Using the combination of 5 commands, however is not very wise in terms of efficiency.

Therefore to save some system resources, its better in terms of efficiency to take advantage of the find command in combination with xargs , here is how:

debian:~# find / | xargs grep 'old_string' -sl |uniq | xargs perl -pi~ -e 's/old_string/new_string/g'

Once again the find command example will do exactly the same as the substitute method with grep -R …

As enough is said about the way to substitute text strings inside files using perl, I will further explain how text strings can be substituted using sed

The main reason why using sed could be a better choice in some cases is that Unices are not equipped by default with perl interpreter. In general the amount of servers who contains installed sed compared to the ones with perl language interpreter is surely higher.

3. Substitute text strings inside files on Linux and BSD with sed stream editor

In many occasions, wether a website is hosted, one needs to quickly conduct a change in string inside all files located in a directory, to resolve issues with static urls directly encoded in html.
To achieve this task here is a code using two little bash script loops in conjunctions with sed, echo and mv commands:

debian:/var/www/website# for i in $(ls -1); do cat $i |sed -e "s#index.htm#http://www.webdomain.com/#g">$i.new; done
debian:/var/www/website# for i in $(ls *.new); do mv $i $(echo $i |sed -e "s#.new##g"); done

The above command sed -e “s#index.htm#http://www.webdomain.com/#g”, instructs sed to substitute all appearance of the text string index.htm to the new text string http://www.webdomain.com

First for bash loop, creates all the files with substituted string to file1.new, file2.new, file3.new etc.
The second for loop uses mv to overwrite the original input files file1, file2, file3, etc. with the newly created ones file1.new, file2.new, file3.new

There is a a way shorter way to conclude the same text substitutions task using a simpler one liner with only using sed and bash’s eval capabilities, here is how:

debian:/var/www/website# sed -i 's/old_string/new_string/g' *

Above command will change old_string to new_string inside all files in directory /var/www/website

Whether a change has to be made with less than 1024 files using this method might be more efficient, however whether a text substitute has to be done to let’s say 5000+ the above simplistic version will not work. An error of Argument list too long will prevent the sed -i ‘s/old_string/new_string/g’ to complete its task.

The above for loop 2 liner should be also working without problems with FreeBSD and the rest of BSD derivatives, though I have not tested it yet, hence any feedback from FreeBSD guys is mostly welcome.

Consider that in order to have the for loops commands work on FreeBSD or NetBSD, they have to be run under a bash shell.
That’s all folks thanks the Lord for letting me write this nice article, I hope it gives some insights on how multiple files text replace on Unix works .
Cheers 😉

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in FreeBSD, Linux, System Administration, Web and CMS | 1 Comment »

How to add cron jobs from command line or bash scripts / Add crontab jobs in a script

Saturday, July 9th, 2011

I’m currently writting a script which is supposed to be adding new crontab jobs and do a bunch of other mambo jambo.

By so far I’ve been aware of only one way to add a cronjob non-interactively like so:

                  linux:~# echo '*/5 * * * * /root/myscript.sh' | crontab -

Though using the | crontab – would work it has one major pitfall, I did completely forgot | crontab – OVERWRITES CURRENT CRONTAB! with the crontab passed by with the echo command.
One must be extremely careful if he decides to use the above example as you might loose your crontab definitions permanently!

Thanksfully it seems there is another way to add crontabs non interactively via a script, as I couldn’t find any good blog which explained something different from the classical example with pipe to crontab –, I dropped by in the good old irc.freenode.net to consult the bash gurus there 😉

So I entered irc and asked the question how can I add a crontab via bash shell script without overwritting my old existing crontab definitions less than a minute later one guy with a nickname geirha was kind enough to explain me how to get around the annoying overwridding.

The solution to the ovewrite was expected, first you use crontab to dump current crontab lines to a file and then you append the new cron job as a new record in the file and finally you ask the crontab program to read and insert the crontab definitions from the newly created files.
So here is the exact code one could run inside a script to include new crontab jobs, next to the already present ones:

linux:~# crontab -l > file; echo '*/5 * * * * /root/myscript.sh >/dev/null 2>&1' >> file; crontab file

The above definition as you could read would make the new record of */5 * * * * /root/myscript.sh >/dev/null be added next to the existing crontab scheduled jobs.

Now I’ll continue with my scripting, in the mean time I hope this will be of use to someone out there 😉

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Linux, System Administration | 9 Comments »