Building A Central Loghost On CentOS And RHEL 5 With rsyslog

Gathering log messages is important. In a lot of situations you'll want to store all entries of logfiles on another server. If a server crashes or gets hacked you want to be able to browse through logfiles from this machine and you want to be sure these log files are not altered in any way. This can be accomplished using a central logserver that receives messages from all other hosts. A syslog facility can receive messages from UNIX and Linux hosts but also network devices and certainly Windows hosts. Such a syslog host should make these logfiles available to auditors and sysops using a read-only interface or they should not be available to anyone until an incident occurs.

Technically the difference is in how you store the messages:
- in plain text on a filesystem
- in an sql database with a web-interface

This howto describes rsyslog putting log messages in one file per day per remote host. Rsyslog is the current standard in RHEL6 and available as a package in the current package streams in RHEL 5.5 (and CentOS 5.5). Setting up rsyslog is pretty simple. It all comes down to a single config file but (there is always a but) every setting needs some planning.

Ask yourself:

Who should access the data and how many times per day (week, month, year)?
How many messages should the host handle per minute/hour?
How long do I want to store the data?

This all has to do with storage specs....

Our host:

Any machine that can decently run RHEL 5.5
A separate filesystem, in this case: /var/opt/syslog_data

Overview

stop sysklogd (standard syslog on RHEL 5.5)
install rsyslog
configure rsyslog to run at boot (and remove sysklogd)
configure rsyslog to save plain text files and add a cronjob to compress the files
describe gotcha's (there is always a but)

Getting our hands dirty:

Step 1

Stop sysklogd:

# service syslog stop

Step 2

Install rsyslog:

# yum install rsyslog

Step 3

Configure rsyslog to run at boot (and remove sysklogd):

# chkconfig syslog off
# chkconfig rsyslog on
# yum erase sysklogd

Step 4

Configure rsyslog to save plain text files and add a cronjob to compress the files.

First edit the startup options of rsyslog to run in a certain compatibility mode, this is done in the file /etc/sysconfig/rsyslog. If you don't do this, you get messages about compatibility mode. This file should look like this:

# Options to syslogd
# -m 0 disables 'MARK' messages.
# -rPortNumber Enables logging from remote machines. The listener will listen to the specified port.
# -x disables DNS lookups on messages recieved with -r
# See syslogd(8) for more details
SYSLOGD_OPTIONS="-c3"
# Options to klogd
# -2 prints all kernel oops messages twice; once for klogd to decode, and
#    once for processing with 'ksymoops'
# -x disables all klogd processing of oops messages entirely
# See klogd(8) for more details
KLOGD_OPTIONS="-x"

Then we remove the rsyslog.conf file from the /etc directory (or save it with a different name). Don't worry this howto delivers a complete file.

# rm /etc/syslog.conf

Now we create a new /etc/rsyslog.conf file with all that is needed to write local files for our own messages. Rsyslog works with modules, this means that a plain install can only do so much. We can add functionality by loading modules most of them are installed in the original RPM.

(You can get a proper explanation in the man file, use:

man rsyslog.conf

)

We add some lines like this:

#load the kernel logger module
$ModLoad imklog
#load the UNIX sockets module to receive local messages from processen
$ModLoad imuxsock
#load UDP powers, to receive messages via the UDP protocol
$ModLoad imudp
#make rsyslog listen on all ip addresses, you could specify an address
$UDPServerAddress 0.0.0.0
#make rsyslog listen on UDP port 514
$UDPServerRun 514
#repeated lines will be reduced
$RepeatedMsgReduction on ()

We could add some lines like this:

*.info;mail.none;authpriv.none;cron.none        /var/log/messages

The result will be: All processed messages with the info severity and all mail, authpriv, cron facility will be logged in the file /var/log/messages. All messages, including the messages from other hosts. This is not what we want, luckily rsyslog let us make evaluations (if then...):

Please make sure you replace hostname with the hostname of your loghost:

if \
        $source == 'hostname' \
        and \
               $syslogseverity <= '6' \
        and ( \
                        $syslogfacility-text != 'mail' \
                and \
                        $syslogfacility-text != 'authpriv' \
                and \
                        $syslogfacility-text != 'cron' \
        ) \
then   /var/log/messages;TraditionalFormat

The lines above will filter the messages from the host itself. Now for the received messages from other hosts. This is really basic, but you can make this very difficult real fast. You could save all messages in separate files per host, using the origional files and than get logrotate to rotate them based on time or filesize. You would also need to send a -HUP signal to the rsyslog proces once logrotate is done. Theoretically the rsyslog daemon should use the use the new filename based on hostname and start a new file after the kill -HUP.

Instead we could add the date to the filename, making rsyslog use a new file each day. We could then add a cronjob with a find query based on time, sending the results to bzip2. Beautiful in it's simplicity:

$template DailyPerHostLogs,"/var/opt/syslog/%HOSTNAME%/%HOSTNAME%.%$YEAR%-%$MONTH%-%$DAY%.log"
*.* -?DailyPerHostLogs;TraditionalFormat

In this syntax we create a template specifying where want to store the files, using variables. Then we specify what action should be taken for what messages; all messages should be stored using the template, including the ones from our localhost.

This should do the same. As a proper masterchef i've prepared this for you, the following can be pasted in your /etc/rsyslog.conf. The result is exactly the same as with the origional sysklogd setup.

Make sure you replace hostname with your hostname!

$ModLoad imuxsock.so
$ModLoad imklog.so
$ModLoad imudp.so
#load the network stuff
$UDPServerAddress 0.0.0.0
$UDPServerRun 514
#reduce any duplicates
$RepeatedMsgReduction on
# The template that wil format the message as it is writen to the file
# you can edit this line if you want to customize te message format
$template TraditionalFormat,"%timegenerated% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n"
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
#*.info;mail.none;authpriv.none;cron.none                /var/log/messages
if \
        $source == 'hostname' \
        and \
               $syslogseverity <= '6' \
        and ( \
                        $syslogfacility-text != 'mail' \
                and \
                        $syslogfacility-text != 'authpriv' \
                and \
                        $syslogfacility-text != 'cron' \
        ) \
then   /var/log/messages;TraditionalFormat
#authpriv.* /var/log/secure;TraditionalFormat
# The authpriv file has restricted access.
#authpriv.*                                              /var/log/secure
if \
        $source == 'hostname' \
                and \
        $syslogfacility-text == 'authpriv' \
then    /var/log/secure;TraditionalFormat
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
# mail.* /var/log/maillog;TraditionalFormat
if \
        $source == 'hostname' \
                and \
        $syslogfacility-text == 'mail' \
then    /var/log/maillog;TraditionalFormat
 
# Log cron stuff
#cron.* /var/log/cron;TraditionalFormat
if \
        $source == 'hostname' \
                and \
        $syslogfacility-text == 'cron' \
then    /var/log/cron;TraditionalFormat
# Everybody gets emergency messages
#*.emerg *
if \
        $source == 'hostname' \
                and \
        $syslogseverity-text == 'emerg' \
then    *
# this line creates a template that will store the messages for each host in a seperate file.
# a new file will ben created daily because of the date in the filename.
$template DailyPerHostLogs,"/var/opt/syslog/%HOSTNAME%/%HOSTNAME%.%$YEAR%-%$MONTH%-%$DAY%.log"
*.* -?DailyPerHostLogs;TraditionalFormat

You can test this easily using logger:

# logger “this is a test”
# tail /var/log/messages

You should see your message.

And the cronjob:

Place this content in the new file /etc/cron.hourly/compress-syslog.cron:

# Compressing remote syslog messagefiles, older then one day using bzip2
find /var/opt/syslog -type f -mtime 1  -name "*.log" -exec bzip2 '{}' \;

That's it you're done. You can make the remote hosts log to this server by adding this line to their /etc/syslog.conf:

# logger "this is a test"
# tail /var/log/messages

You can repeat this line for every sysloghost you have (replacing ipaddress or dnsname with the actual data).

*.* @ipaddress or dnsname

Step 5. Gotchas:

- Storage tips: Normal ext3 filesystems reserve a amount of space for emergencies (only root can use this reserved space), this is nice for your root filesystem but it's nasty big filesystems that store data like logfiles. No special space should be reserved, with normal settings a percentage is. You can make this space available by this line:

tune2fs -m 0 blockdevice

- Performance tips: If your system is overrun by messages you can use the buffer system that rsyslog has in place for every action it performs. You can enable the mainqueue with the following lines in rsyslog.conf

$WorkDirectory /tmp          # default location for work (spool) files
$MainMsgQueueFileName mainq  # set file name, also enables disk mode

This howto was created using the following:
http://wiki.rsyslog.com/index.php/DailyLogRotation (for insight on the logrotation and comressing)
http://www.rsyslog.com/ (wealth of information on rsyslog)
http://wiki.rsyslog.com/index.php/Sysklogd_drop-in_with_remote_logs_separated_by_dynamic_directory (separating the local messages from the remote messages)

Log in or Sign up

On this page