How to setup rsyslog for Centralized Log Management

This tutorial will explain how to setup rsyslog as a centralized log management server. Centralized log management means to collect all sorts of logs from several physical or virtualized servers on one log server to monitor the health and security of the server services. We use rsyslog in this tutorial because it offers high-performance, great security and a modular design. It is also capable of storing logs in numerous database solutions like MySQL, Oracle, Hadoop and others for better consolidation.

1. Preliminary Note

For this tutorial, I am using Oracle Linux 6.4 in the 32bit version. Please note that eventhough the configuration are made under Oracle Linux, the same steps will work on CentOS and Red Hat OS Linux. In this tutorial, we will use 2 servers. The first one will act as the rsyslog server and the other will act as workstation / client server for the rsyslog client. By the end of this tutorial, we will see that once a user logs into the client server, the rsyslog server will log the activity made by him automatically.

2. Rsyslog Installation

For the installation phase, we only require to have the rsyslog packages and its dependencies installed. First, let's confirm our version of Operating System.

[root@RSYS01 ~]# cat /etc/issue
Oracle Linux Server release 6.4
Kernel \r on an \m

[root@RSYS01 ~]# arch
i686

[root@RSYS01 ~]# uname -a
Linux RSYS01 2.6.32-358.el6.i686 #1 SMP Fri Feb 22 13:37:29 PST 2013 i686 i686 i386 GNU/Linux

Next, I will configure a new repository to install the rsyslog packages via yum utility.

[root@RSYS01 ~]# cd /etc/yum.repos.d/
[root@RSYS01 yum.repos.d]# vi rsyslog.repo
[rsyslog-v7-devel]
name=Adiscon Rsyslog v7-devel for CentOS-$releasever-$basearch
baseurl=http://rpms.adiscon.com/v7-devel/epel-$releasever/$basearch
enabled=0
gpgcheck=0
protect=1

[rsyslog-v7-stable]
name=Adiscon Rsyslog v7-stable for CentOS-$releasever-$basearch
baseurl=http://rpms.adiscon.com/v7-stable/epel-$releasever/$basearch
enabled=1
gpgcheck=0
protect=1

[root@RSYS01 yum.repos.d]# yum list rsyslog
Loaded plugins: refresh-packagekit, security
rsyslog-v7-stable | 2.5 kB 00:00
rsyslog-v7-stable/primary_db | 188 kB 00:01
Available Packages
rsyslog.i686 7.6.7-1.el6 rsyslog-v7-stable

Done, now let's start to install the new version of rsyslog. The steps are shown below:

[root@RSYS01 yum.repos.d]# yum install rsyslog -y
Loaded plugins: refresh-packagekit, security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package rsyslog.i686 0:7.6.7-1.el6 will be installed
--> Processing Dependency: liblogging-stdlog.so.0 for package: rsyslog-7.6.7-1.el6.i686
--> Processing Dependency: libjson-c.so.2 for package: rsyslog-7.6.7-1.el6.i686
--> Processing Dependency: libgthttp.so.0 for package: rsyslog-7.6.7-1.el6.i686
--> Processing Dependency: libgtbase.so.0 for package: rsyslog-7.6.7-1.el6.i686
--> Processing Dependency: libgt for package: rsyslog-7.6.7-1.el6.i686
--> Processing Dependency: libestr.so.0 for package: rsyslog-7.6.7-1.el6.i686
--> Running transaction check
---> Package json-c.i686 0:0.11-3.el6 will be installed
---> Package libestr.i686 0:0.1.9-1.el6 will be installed
---> Package libgt.i686 0:0.3.11-1.el6 will be installed
---> Package liblogging.i686 0:1.0.4-1.el6 will be installed
--> Finished Dependency Resolution 
Dependencies Resolved 
===============================================================================================================
Package Arch Version Repository Size
===============================================================================================================
Installing:
rsyslog i686 7.6.7-1.el6 rsyslog-v7-stable 920 k
Installing for dependencies:
json-c i686 0.11-3.el6 rsyslog-v7-stable 46 k
libestr i686 0.1.9-1.el6 rsyslog-v7-stable 9.0 k
libgt i686 0.3.11-1.el6 rsyslog-v7-stable 55 k
liblogging i686 1.0.4-1.el6 rsyslog-v7-stable 23 k 
Transaction Summary
===============================================================================================================
Install 5 Package(s) 
Total download size: 1.0 M
Installed size: 3.2 M
Downloading Packages:
(1/5): json-c-0.11-3.el6.i686.rpm | 46 kB 00:00
(2/5): libestr-0.1.9-1.el6.i686.rpm | 9.0 kB 00:00
(3/5): libgt-0.3.11-1.el6.i686.rpm | 55 kB 00:00
(4/5): liblogging-1.0.4-1.el6.i686.rpm | 23 kB 00:00
(5/5): rsyslog-7.6.7-1.el6.i686.rpm | 920 kB 00:03
---------------------------------------------------------------------------------------------------------------
Total 114 kB/s | 1.0 MB 00:09
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : libgt-0.3.11-1.el6.i686 1/5
Installing : liblogging-1.0.4-1.el6.i686 2/5
Installing : libestr-0.1.9-1.el6.i686 3/5
Installing : json-c-0.11-3.el6.i686 4/5
Installing : rsyslog-7.6.7-1.el6.i686 5/5
Verifying : json-c-0.11-3.el6.i686 1/5
Verifying : libestr-0.1.9-1.el6.i686 2/5
Verifying : liblogging-1.0.4-1.el6.i686 3/5
Verifying : libgt-0.3.11-1.el6.i686 4/5
Verifying : rsyslog-7.6.7-1.el6.i686 5/5 
 
Installed:
rsyslog.i686 0:7.6.7-1.el6 
 
Dependency Installed:
json-c.i686 0:0.11-3.el6 libestr.i686 0:0.1.9-1.el6 libgt.i686 0:0.3.11-1.el6 liblogging.i686 0:1.0.4-1.el6 
 
Complete!

[root@RSYS01 yum.repos.d]# rsyslogd -v
rsyslogd 7.6.7, compiled with:
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
Runtime Instrumentation (slow code): No
uuid support: Yes
Number of Bits in RainerScript integers: 64 
 
See http://www.rsyslog.com for more information.

[root@RSYS01 ~]# rpm -qa|grep rsyslog
rsyslog-7.6.7-1.el6.i686

Next, let's proceed with the installation phase. For this tutorial, we will skip on the security layer to make things simple. We will disable SELINUX to ensure that there's no issue related security raise during the way. Be aware that you should not disable SELinux and the Firewall on a live setup. Below are the steps:

First, check what is the current status for our SELINUX policy.

[root@RSYS01 ~]# getenforce
Enforcing

To disable it permenantly, follow the step below:

[root@RSYS01 ~]# cd /etc/sysconfig/
[root@RSYS01 ~]# vi selinux 
 
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
# targeted - Targeted processes are protected,
# mls - Multi Level Security protection.
SELINUXTYPE=targeted

After that let's ensure we have disabled the firewall to avoid any blocking between server and client connection.

[root@RSYS01 ~]# iptables -nL
Chain INPUT (policy ACCEPT)
target prot opt source destination 
 
Chain FORWARD (policy ACCEPT)
target prot opt source destination 
 
Chain OUTPUT (policy ACCEPT)
target prot opt source destination

[root@RSYS01 ~]# /etc/init.d/iptables stop
iptables: Flushing firewall rules: [ OK ]
iptables: Setting chains to policy ACCEPT: filter [ OK ]
iptables: Unloading modules: [ OK ]

[root@RSYS01 ~]# /etc/init.d/iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num target prot opt source destination 
 
Chain FORWARD (policy ACCEPT)
num target prot opt source destination 
 
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination

Done, now as the installation phase has succeed. Let's move on to configuration phase.

3. Rsyslog Configuration

All package dependencies has been installed already, let's focus on the configuration of rsyslog. Let's go inside the configuration file and make the changes like below:

[root@RSYS01 ~]# vi /etc/rsyslog.conf 
 
module(load="imudp") # needs to be done just once
input(type="imudp" port="514") 
 
$template Auditlog, "/var/log/rsyslog_client/%HOSTNAME%/%PROGRAMNAME%.log"
$template TmplMsg, "/var/log/rsyslog_client/%HOSTNAME%/%PROGRAMNAME%.log"
authpriv.* ?Auditlog

Below is the explaination on the configuration changes we made:

• module(load="imudp") # needs to be done just once ==> We will provide ability to receive rsyslog message between server and client via UDP protocol
• input(type="imudp" port="514") ==> We will use port 514 for rsyslog services
• $template Auditlog, "/var/log/rsyslog_client/%HOSTNAME%/%PROGRAMNAME%.log" ==> For every client connected to rsyslog server, system will automatically create the client hostname folder and related services file name
• $template TmplMsg, "/var/log/rsyslog_client/%HOSTNAME%/%PROGRAMNAME%.log" ==> For every client connected to rsyslog server, system will automatically create the client hostname folder and related services file name

Once the configuration is done, let's start the rsyslog service.

[root@RSYS01 yum.repos.d]# /etc/init.d/rsyslog restart
Shutting down system logger: [FAILED]
Starting system logger: [ OK ]

Use netstat o ensure whether the rsyslog services are up and running:

[root@RSYS01 yum.repos.d]# netstat -uanp|grep rsyslog
udp 0 0 0.0.0.0:514 0.0.0.0:* 2430/rsyslogd
udp 0 0 :::514 :::* 2430/rsyslogd

Above you can see that our rsyslog service is running with the port assignements we have made. By default, rsyslog also audit its own server access as it treats the rsyslog server as a client. To ensure whether it's working, we can take a look into the /var/log folder. Let's check whether a folder with the name RSYS01 (that's the server hostname) exist or not.

[root@RSYS01 log]# cd /var/log/
[root@RSYS01 log]# ls -l|grep rsyslog
drwx------ 3 root root 4096 Oct 24 18:21 rsyslog_client

[root@RSYS01 log]# cd rsyslog_client
[root@RSYS01 rsyslog_client]# ls
RSYS01
[root@RSYS01 rsyslog_client]# cd RSYS01/
[root@RSYS01 RSYS01]# ls
rsyslogd.log

Nice, seems everything works like a charm! Now let's proceed with the testing phase to conclude all configurations are made as expected.

4. Testing Phase

As the server that runs the rsyslog service also works as a client, we can see on the server itself if monitoring of local logins is monitored. To ensure whether it is true, let's log into the rsyslog server with another session by using the SSH service. For this step, we will assume that the server itself already configured passwordless configuration. Below are the steps:

[root@RSYS01 RSYS01]# ssh root@RSYS01
Last login: Sat Oct 22 15:45:48 2016 from 172.20.181.70

[root@RSYS01 ~]# who
root pts/0 2016-10-22 00:21 (172.20.181.11)
root pts/1 2016-10-24 18:22 (127.0.0.1)

[root@RSYS01 ~]# exit
logout
Connection to RSYS01 closed.

Done, as simple as that. We just log into the rsyslog server itself then once the new session is created, we simply log again just to ensure the rsyslog service did audit the session. Now, let's check whether the session has been audited or not. Below are the steps:

[root@RSYS01 ~]# cd var/log/rsyslog_client/RSYS01
[root@RSYS01 RSYS01]# ls
rsyslogd.log sshd.log
[root@RSYS01 RSYS01]# tail -f sshd.log
Oct 24 18:22:46 RSYS01 sshd[2536]: Accepted password for root from 192.168.43.101 port 52862 ssh2
Oct 24 18:22:46 RSYS01 sshd[2536]: pam_unix(sshd:session): session opened for user root by (uid=0)
Oct 24 18:22:50 RSYS01 sshd[2536]: Received disconnect from 192.168.43.101: 11: disconnected by user
Oct 24 18:22:50 RSYS01 sshd[2536]: pam_unix(sshd:session): session closed for user root
^C

Excellent, the rsyslog service automatically created a sshd.log file when the session started on the server. Inside the log file, we can see that there's list of detail stated the time, port and user created under the session.

Now as everything works as expected. Let's setup a workstation for the rsyslog client to be audited by our rsyslog server. For the rsyslog client, you just need to install the rsyslog packages and make a simple change in the configuration file to link to the rsyslog server. Below are the steps:

[root@CLIENT01 ~]# cd /etc/yum.repos.d/
[root@CLIENT01 yum.repos.d]# yum list rsyslog
Loaded plugins: refresh-packagekit, security
rsyslog-v7-stable | 2.5 kB 00:00
rsyslog-v7-stable/primary_db | 188 kB 00:01
Available Packages
rsyslog.i686 7.6.7-1.el6 rsyslog-v7-stable

[root@RSYS01 yum.repos.d]# yum install rsyslog -y
Loaded plugins: refresh-packagekit, security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package rsyslog.i686 0:7.6.7-1.el6 will be installed
--> Processing Dependency: liblogging-stdlog.so.0 for package: rsyslog-7.6.7-1.el6.i686
--> Processing Dependency: libjson-c.so.2 for package: rsyslog-7.6.7-1.el6.i686
--> Processing Dependency: libgthttp.so.0 for package: rsyslog-7.6.7-1.el6.i686
--> Processing Dependency: libgtbase.so.0 for package: rsyslog-7.6.7-1.el6.i686
--> Processing Dependency: libgt for package: rsyslog-7.6.7-1.el6.i686
--> Processing Dependency: libestr.so.0 for package: rsyslog-7.6.7-1.el6.i686
--> Running transaction check
---> Package json-c.i686 0:0.11-3.el6 will be installed
---> Package libestr.i686 0:0.1.9-1.el6 will be installed
---> Package libgt.i686 0:0.3.11-1.el6 will be installed
---> Package liblogging.i686 0:1.0.4-1.el6 will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
 
===============================================================================================================
Package Arch Version Repository Size
===============================================================================================================
Installing:
rsyslog i686 7.6.7-1.el6 rsyslog-v7-stable 920 k
Installing for dependencies:
json-c i686 0.11-3.el6 rsyslog-v7-stable 46 k
libestr i686 0.1.9-1.el6 rsyslog-v7-stable 9.0 k
libgt i686 0.3.11-1.el6 rsyslog-v7-stable 55 k
liblogging i686 1.0.4-1.el6 rsyslog-v7-stable 23 k
 
Transaction Summary
===============================================================================================================
Install 5 Package(s)
 
Total download size: 1.0 M
Installed size: 3.2 M
Downloading Packages:
(1/5): json-c-0.11-3.el6.i686.rpm | 46 kB 00:00
(2/5): libestr-0.1.9-1.el6.i686.rpm | 9.0 kB 00:00
(3/5): libgt-0.3.11-1.el6.i686.rpm | 55 kB 00:00
(4/5): liblogging-1.0.4-1.el6.i686.rpm | 23 kB 00:00
(5/5): rsyslog-7.6.7-1.el6.i686.rpm | 920 kB 00:03
---------------------------------------------------------------------------------------------------------------
Total 114 kB/s | 1.0 MB 00:09
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : libgt-0.3.11-1.el6.i686 1/5
Installing : liblogging-1.0.4-1.el6.i686 2/5
Installing : libestr-0.1.9-1.el6.i686 3/5
Installing : json-c-0.11-3.el6.i686 4/5
Installing : rsyslog-7.6.7-1.el6.i686 5/5
Verifying : json-c-0.11-3.el6.i686 1/5
Verifying : libestr-0.1.9-1.el6.i686 2/5
Verifying : liblogging-1.0.4-1.el6.i686 3/5
Verifying : libgt-0.3.11-1.el6.i686 4/5
Verifying : rsyslog-7.6.7-1.el6.i686 5/5
 
Installed:
rsyslog.i686 0:7.6.7-1.el6
 
Dependency Installed:
json-c.i686 0:0.11-3.el6 libestr.i686 0:0.1.9-1.el6 libgt.i686 0:0.3.11-1.el6 liblogging.i686 0:1.0.4-1.el6
 
Complete!

Done, now we've installed the rsyslog package on our client workstation. Now let's make an amendment inside the rsyslog configuration file. For a client configuration, you just need to amend the configuration like below:

[root@CLIENT01 ~]# vi /etc/rsyslog.conf
 
*.* @192.168.43.101:514

That's all, noted that we have included IP 192.168.43.101 with port 514 in the configuration file. That IP is the IP for rsyslog server. Now as everything is done, let's restart the rsyslog service on the client workstation to load the changes. Below are the steps:

[root@CLIENT01 ~]# /etc/init.d/rsyslog restart
Shutting down system logger: [FAILED]
Starting system logger: [ OK ]

Now let's go back to our rsyslog server and see if a folder for the rsyslog client hostname has been created in the rsyslog log directory. Below are the steps:

[root@RSYS01 ~]# cd var/log/rsyslog_client/
[root@RSYS01 rsyslog_client]# ls
RSYS01 CLIENT01

Excellent, note that a folder with rsyslog client hostname has been created automatically. This confirms that our configuration is correct and the rsyslog client is able to make a UDP connection to the rsyslog server.

For the next test procedure, let's log into rsyslog client as another user and see if the rsyslog server manages to capture the activity or not. Below are the steps:

::CLIENT01::
login as: shahril
[email protected]'s password:
Last login: Sun Oct 23 00:21:40 2016 from 172.20.181.11

[shahril@CLIENT01 ~]$ who
shahril pts/0 2016-10-24 17:01 (192.168.43.80)

[shahril@CLIENT01 ~]$ exit

Now, let's check the log directory inside the rsyslog server to see if we managed to log the activity created from rsyslog client or not.

[root@RSYS01 ~]# cd var/log/rsyslog_client/
[root@RSYS01 rsyslog_client]# cd CLIENT01/
[root@RSYS01 CLIENT01]# ls
rsyslogd.log sshd.log
[root@RSYS01 CLIENT01]# tail -10 sshd.log
Oct 24 17:01:47 CLIENT01 sshd[2102]: Accepted password for shahril from 192.168.43.80 port 17002 ssh2
Oct 24 17:01:47 CLIENT01 sshd[2102]: pam_unix(sshd:session): session opened for user shahril by (uid=0)

Great, result shows the process works as expected. Now for the final testing, let's log back into the rsyslog client and install a package to test if the rsyslog service manages to log other activity than session creation. Below are the steps:

::CLIENT01::
login as: root
[email protected]'s password:
Last login: Sat Oct 22 10:21:40 2016 from 172.20.181.11

[root@CLIENT01 ~]# yum install firefox -y
Loaded plugins: refresh-packagekit, security
Repository 'OEL64' is missing name in configuration, using id
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package firefox.i686 0:10.0.12-1.0.1.el6_3 will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
 
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
firefox i686 10.0.12-1.0.1.el6_3 OEL64 20 M
 
Transaction Summary
================================================================================
Install 1 Package(s)
 
Total download size: 20 M
Installed size: 23 M
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : firefox-10.0.12-1.0.1.el6_3.i686 1/1
Verifying : firefox-10.0.12-1.0.1.el6_3.i686 1/1
 
Installed:
firefox.i686 0:10.0.12-1.0.1.el6_3
 
Complete!

The above shows that we have successfully installed a firefox browser installer inside our rsyslog client workstation. Now let's go back to our rsyslog server and check if rsyslog was able to log the process of installing a third party package into the workstation. Below are the steps:

[root@RSYS01 ~]# cd var/log/rsyslog_client/
[root@RSYS01 CLIENT01]# ls
rsyslogd.log sshd.log yum.log
[root@RSYS01 CLIENT01]# tail -20 yum.log
Oct 25 17:13:17 CLIENT01 yum[2319]: Installed: firefox-10.0.12-1.0.1.el6_3.i686

Excellent, rsyslog service managed to audit the information of the installation activity in the client workstation.