If you're using some external bash script to log messages via rsyslogd to some of the multiple rsyslog understood data tubes (called in rsyslog language facility levels) and you want Rsyslog to move message string to external log file, then you had the same task as me few days ago.
For example you have a bash shell script that is writting a message to rsyslog daemon to some of the predefined facility levels be it:
kern,user,cron, auth etc. or some local
and your logged script data ends under the wrong file location /var/log/messages , /var/log/secure , var/log/cron etc. However you need to log everything coming from that service to a separate file based on the localX (fac. level) the usual way to do it is via some config like, as you would usually do it with rsyslog variables as:
local1.info /var/log/custom-log.log
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none;local0.none;local1.none /var/log/messages
Note the local1.none is instructing the rsyslog not to log anything from local1 facility towards /var/log/message.
But what if this due to some weirdness in configuration of rsyslog on the server or even due to some weird misconfiguration in
/etc/systemd/journald.conf such as:
[Journal]
Storage=persistent
RateLimitInterval=0s
RateLimitBurst=0
SystemMaxUse=128M
SystemMaxFileSize=32M
MaxRetentionSec=1month
MaxFileSec=1week
ForwardToSyslog=yes
SplitFiles=none
Due to that config and especially the FowardToSyslog=yes, the messages sent via the logger tool to local1 still end up inside /var/log/messages, not nice huh ..
The result out of that is anything being sent with a predefined TAGGED string via the whatever.sh script which uses the logger command (if you never use it check man logger) to enter message into rsyslog with cmd like:
# logger -p local1.info -t TAG_STRING
# logger -p local2.warn test
# tail -2 /var/log/messages
Dec 22 18:58:23 pcfreak rsyslogd: — MARK —
Dec 22 19:07:12 pcfreak hipo: test
was nevertheless logged to /var/log/message.
Of course /var/log/message becomes so overfilled with "junk" shell script data not related to real basic Operating system adminsitration, so this prevented any critical or important messages that usually should come under /var/log/message / /var/log/syslog to be lost among the big quantities of other tagged tata reaching the log.
After many attempts to resolve the issue by modifying /etc/rsyslog.conf as well as the messed /etc/systemd/journald.conf (which by the way was generated with this strange values with an OS install time automation ansible stuff). It took me a while until I found the solution on how to tell rsyslog to log the tagged message strings into an external separate file. From my 20 minutes of research online I have seen multitudes of people in different Linux OS versions to experience the same or similar issues due to whatever, thus this triggered me to write this small article on the solution to rsyslog.
The solution turned to be pretty easy but requires some further digging into rsyslog, Redhat's basic configuration on rsyslog documentation is a very nice reading for starters, in my case I've used one of the Propery-based compare-operations variable contains used to select my tagged message string.
1. Add msg contains compare-operations to output log file and discard the messages
[root@centos bin]# vi /etc/rsyslog.conf
# config to log everything logged to rsyslog to a separate file
:msg, contains, "tag_string:/" /var/log/custom-script-log.log
:msg, contains, "tag_string:/" ~
Substitute quoted tag_string:/ to whatever your tag is and mind that it is better this config is better to be placed somewhere near the beginning of /etc/rsyslog.conf and touch the file /var/log/custom-script-log.log and give it some decent permissions such as 755, i.e.
1.1 Discarding a message
The tilda sign – ~
as placed to the end of the msg, contains is the actual one to tell the string to be discarded so it did not end in /var/log/messages.
Alternative rsyslog config to do discard the unwanted message once you have it logged is with the
rawmsg variable, like so:
# config to log everything logged to rsyslog to a separate file
:msg, contains, "tag_string:/" /var/log/custom-script-log.log
:rawmsg, isequal, "tag_string:/" stop
Other way to stop logging immediately after log is written to custom file across some older versions of rsyslog is via the &stop
:msg, contains, "tag_string:/" /var/log/custom-script-log.log
& stop
I don't know about other versions but Unfortunately the &stop does not work on RHEL 7.9 with installed rpm package rsyslog-8.24.0-57.el7_9.1.x86_64.
1.2 More with property based filters basic exclusion of string
Property based filters can do much more, you can for example, do regular expression based matches of strings coming to rsyslog and forward to somewhere.
To select syslog messages which do not contain any mention of the words fatal and error with any or no text between them (for example, fatal lib error), type:
:msg, !regex, "fatal .* error"
2. Create file where tagged data should be logged and set proper permissions
[root@centos bin]# touch /var/log/custom-script-log.log
[root@centos bin]# chmod 755 /var/log/custom-script-log.log
3. Test rsyslogd configuration for errors and reload rsyslog
[root@centos ]# rsyslogd -N1
rsyslogd: version 8.24.0-57.el7_9.1, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.
[root@centos ]# systemctl restart rsyslog
[root@centos ]# systemctl status rsyslog
● rsyslog.service – System Logging Service
Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2021-12-22 13:40:11 CET; 3h 5min ago
Docs: man:rsyslogd(8)
http://www.rsyslog.com/doc/
Main PID: 108600 (rsyslogd)
CGroup: /system.slice/rsyslog.service
└─108600 /usr/sbin/rsyslogd -n
4. Property-based compare-operations supported by rsyslog table
| Compare-operation | Description |
|---|---|
contains
| Checks whether the provided string matches any part of the text provided by the property. To perform case-insensitive comparisons, use
contains_i
. |
isequal
| Compares the provided string against all of the text provided by the property. These two values must be exactly equal to match. |
startswith
| Checks whether the provided string is found exactly at the beginning of the text provided by the property. To perform case-insensitive comparisons, use
startswith_i
. |
regex
| Compares the provided POSIX BRE (Basic Regular Expression) against the text provided by the property. |
ereregex
| Compares the provided POSIX ERE (Extended Regular Expression) regular expression against the text provided by the property. |
isempty
| Checks if the property is empty. The value is discarded. This is especially useful when working with normalized data, where some fields may be populated based on normalization result. |
5. Rsyslog understanding Facility levels
Here is a list of facility levels that can be used.
Note: The mapping between Facility Number and Keyword is not uniform over different operating systems and different syslog implementations, so among separate Linuxes there might be diference in the naming and numbering.
| Facility Number | Keyword | Facility Description |
|---|---|---|
| 0 | kern | kernel messages |
| 1 | user | user-level messages |
| 2 | mail system | |
| 3 | daemon | system daemons |
| 4 | auth | security/authorization messages |
| 5 | syslog | messages generated internally by syslogd |
| 6 | lpr | line printer subsystem |
| 7 | news | network news subsystem |
| 8 | uucp | UUCP subsystem |
| 9 | clock daemon | |
| 10 | authpriv | security/authorization messages |
| 11 | ftp | FTP daemon |
| 12 | – | NTP subsystem |
| 13 | – | log audit |
| 14 | – | log alert |
| 15 | cron | clock daemon |
| 16 | local0 | local use 0 (local0) |
| 17 | local1 | local use 1 (local1) |
| 18 | local2 | local use 2 (local2) |
| 19 | local3 | local use 3 (local3) |
| 20 | local4 | local use 4 (local4) |
| 21 | local5 | local use 5 (local5) |
| 22 | local6 | local use 6 (local6) |
| 23 | local7 | local use 7 (local7) |
6. rsyslog Severity levels (sublevels) accepted by facility level
As defined in RFC 5424, there are eight severity levels as of year 2021:
| Code | Severity | Keyword | Description | General Description |
|---|---|---|---|---|
| 0 | Emergency | emerg (panic) | System is unusable. | A "panic" condition usually affecting multiple apps/servers/sites. At this level it would usually notify all tech staff on call. |
| 1 | Alert | alert | Action must be taken immediately. | Should be corrected immediately, therefore notify staff who can fix the problem. An example would be the loss of a primary ISP connection. |
| 2 | Critical | crit | Critical conditions. | Should be corrected immediately, but indicates failure in a primary system, an example is a loss of a backup ISP connection. |
| 3 | Error | err (error) | Error conditions. | Non-urgent failures, these should be relayed to developers or admins; each item must be resolved within a given time. |
| 4 | Warning | warning (warn) | Warning conditions. | Warning messages, not an error, but indication that an error will occur if action is not taken, e.g. file system 85% full – each item must be resolved within a given time. |
| 5 | Notice | notice | Normal but significant condition. | Events that are unusual but not error conditions – might be summarized in an email to developers or admins to spot potential problems – no immediate action required. |
| 6 | Informational | info | Informational messages. | Normal operational messages – may be harvested for reporting, measuring throughput, etc. – no action required. |
| 7 | Debug | debug | Debug-level messages. | Info useful to developers for debugging the application, not useful during operations. |
7. Sample well tuned configuration using severity and facility levels and immark, imuxsock, impstats
Below is sample config using severity and facility levels
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none;local0.none;local1.none /var/log/messages
Note the local0.none; local1.none tells rsyslog to not log from that facility level to /var/log/messages.
If you need a complete set of rsyslog configuration fine tuned to have a proper logging with increased queues and included configuration for loggint to remote log aggegator service as well as other measures to prevent the system disk from being filled in case if something goes wild with a logging service leading to a repeatedly messages you might always contact me and I can help 🙂
Other from that sysadmins might benefit from a sample set of configuration prepared with the Automated rsyslog config builder or use some fine tuned config for rsyslog-8.24.0-57.el7_9.1.x86_64 on Redhat 7.9 (Maipo) rsyslog_config_redhat-2021.tar.gz.
To sum it up rsyslog though looks simple and not an important thing to pre
