Enhanced Logging With rsyslog On Debian Etch And phpLogcon For Viewing
Well everybody knows the issues of reading and searching in log files. If you have more than one machine it even gets worse. This tutorial describes how to install and configure rsyslog on Debian Etch, but it can be adapted to other distributions.
Exerpt from rsyslog site:
"Rsyslog, the enhanced syslogd for Linux and Unix.
Rsyslog is an enhanced multi-threaded syslogd supporting, among others, MySQL, syslog/tcp, RFC 3195, permitted sender lists, filtering on any message part, and fine grain output format control. It is quite compatible to stock sysklogd and can be used as a drop-in replacement. Its advanced features make it suitable for enterprise-class, encryption protected syslog relay chains while at the same time being very easy to setup for the novice user. An optional web interface - phpLogCon - can be used to visualize all data online."
In this tutorial we will be building rsyslog from source and write the necessary config files.
Step 1: See That The Necessary Tools Are Installed
apt-get install binutils cpp fetchmail flex gcc libarchive-zip-perl libc6-dev libcompress-zlib-perl libdb4.3-dev libpcre3 libpopt-dev linux-kernel-headers lynx m4 make ncftp nmap openssl perl perl-modules unzip zip zlib1g-dev autoconf automake1.9 libtool bison autotools-dev g++ mysql-server mysql-client libmysqlclient15-dev
Next set a password for your mysql root user:
mysqladmin -u root password your_mysqlroot_password
Now we can create the rsyslog db:
mysqladmin -u root -p create rsyslog
Next we start the mysql command shell and create the rsyslog user:
mysql -u root -p
GRANT SELECT, INSERT, UPDATE, DELETE ON rsyslog.* TO 'rsyslog_user'@'localhost' IDENTIFIED BY 'rsyslog_user_password';
GRANT SELECT, INSERT, UPDATE, DELETE ON mail.* TO 'rsyslog_user'@'localhost.localdomain' IDENTIFIED BY 'rsyslog_user_password';
FLUSH PRIVILEGES;
quit
As you might have noticed we do not import any tables into the db. phpLogCon will do that for us.
Step 2: Get The rsyslog Source And Build rsyslog
Get the source:
cd /tmp
wget http://www.rsyslog.com/Downloads-req-getit-lid-58.phtml
Now let's build and install rsyslog:
tar xvzf rsyslog-1.9.6.tar.gz
cd rsyslog-1.9.6
./configure
make
make install
Since rsyslog has no configuration examples (and files) I'll provide the examples here. All of these are just default configurations, use the documentation provided by rsyslog to customize the configuration to your specific needs.
First the rsyslog configuration file /etc/rsyslog.conf.
vi /etc/rsyslog.conf
Just copy and paste the contents below:
Do not forget to change the line:
*.* >127.0.0.1,rsyslog,rsyslog_user,rsyslog_user_password
with the username and password you specified above.
# /etc/rsyslog.conf Configuration file for rsyslogd.
#
# For more information see
# /usr/share/doc/rsyslog/html/rsyslog_conf.html
#
# First some standard logfiles. Log by facility.
#
$ModLoad MySQL
*.* >127.0.0.1,rsyslog,rsyslog_user,rsyslog_user_password
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log
daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
user.* -/var/log/user.log
#
# Logging for the mail system. Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info -/var/log/mail.info
mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err
#
# Logging for INN news system
#
news.crit /var/log/news/news.crit
news.err /var/log/news/news.err
news.notice -/var/log/news/news.notice
#
# Some `catch-all' logfiles.
#
*.=debug;\
auth,authpriv.none;\
news.none;mail.none -/var/log/debug
*.=info;*.=notice;*.=warn;\
auth,authpriv.none;\
cron,daemon.none;\
mail,news.none -/var/log/messages
#
# Emergencies are sent to everybody logged in.
#
*.emerg *
#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
# news.=crit;news.=err;news.=notice;\
# *.=debug;*.=info;\
# *.=notice;*.=warn /dev/tty8
# The named pipe /dev/xconsole is for the `xconsole' utility. To use it,
# you must invoke `xconsole' with the `-file' option:
#
# $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
# busy site..
#
daemon.*;mail.*;\
news.err;\
*.=debug;*.=info;\
*.=notice;*.=warn |/dev/xconsole
#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/
|
Next is the startup script /etc/init.d/rsyslog:
vi /etc/init.d/rsyslog
Just copy and paste the contents below:
#! /bin/sh ### BEGIN INIT INFO # Provides: syslog # Required-Start: $local_fs $time # Required-Stop: $local_fs $time # Should-Start: $network # Should-Stop: $network # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: enhanced syslogd # Description: Rsyslog is an enhanced multi-threaded syslogd. # It is quite compatible to stock sysklogd and can be # used as a drop-in replacement. ### END INIT INFO # Author: Michael Biebl <[email protected]> # # Do NOT "set -e" # PATH should only include /usr/* if it runs after the mountnfs.sh script PATH=/sbin:/usr/sbin:/bin:/usr/bin DESC="enhanced syslogd" NAME=rsyslog RSYSLOGD=rsyslogd RSYSLOGD_BIN=/usr/local/sbin/rsyslogd RSYSLOGD_OPTIONS="-m 0" RSYSLOGD_PIDFILE=/var/run/rsyslogd.pid RKLOGD=rklogd RKLOGD_BIN=/usr/local/sbin/rklogd RKLOGD_OPTIONS="-2" RKLOGD_PIDFILE=/var/run/rklogd.pid SCRIPTNAME=/etc/init.d/$NAME # Exit if the package is not installed [ -x "$RSYSLOGD_BIN" ] || exit 0 [ -x "$RKLOGD_BIN" ] || exit 0 # Read configuration variable file if it is present [ -r /etc/default/$NAME ] && . /etc/default/$NAME # Define LSB log_* functions. # Depend on lsb-base (>= 3.0-6) to ensure that this file is present. . /lib/lsb/init-functions # # Function that starts the daemon/service # do_start() { DAEMON=$1 DAEMON_ARGS=$2 PIDFILE=$3 # Return # 0 if daemon has been started # 1 if daemon was already running # 2 if daemon could not be started start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null \ || return 1 start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON -- \ $DAEMON_ARGS \ || return 2 } # # Function that stops the daemon/service # do_stop() { NAME=$1 PIDFILE=$2 # Return # 0 if daemon has been stopped # 1 if daemon was already stopped # 2 if daemon could not be stopped # other if a failure occurred start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME RETVAL="$?" #rm -f $PIDFILE return "$RETVAL" } # # Function that sends a SIGHUP to the daemon/service # do_reload() { NAME=$1 PIDFILE=$2 start-stop-daemon --stop --signal HUP --quiet --pidfile $PIDFILE --name $NAME return 0 } create_xconsole() { if [ ! -e /dev/xconsole ] then mknod -m 640 /dev/xconsole p fi } case "$1" in start) log_daemon_msg "Starting $DESC" "$RSYSLOGD" create_xconsole do_start "$RSYSLOGD_BIN" "$RSYSLOGD_OPTIONS" "$RSYSLOGD_PIDFILE" case "$?" in # 0|1) log_end_msg 0 ;; 2) log_end_msg 1 ;; esac log_progress_msg "$RKLOGD" do_start "$RKLOGD_BIN" "$RKLOGD_OPTIONS" "$RKLOGD_PIDFILE" case "$?" in 0|1) log_end_msg 0 ;; 2) log_end_msg 1 ;; esac ;; stop) log_daemon_msg "Stopping $DESC" "$RKLOGD" do_stop "$RKLOGD" "$RKLOGD_PIDFILE" case "$?" in # 0|1) log_end_msg 0 ;; 2) log_end_msg 1 ;; esac log_progress_msg "$RSYSLOGD" do_stop "$RSYSLOGD" "$RSYSLOGD_PIDFILE" case "$?" in 0|1) log_end_msg 0 ;; 2) log_end_msg 1 ;; esac ;; reload|force-reload) log_daemon_msg "Reloading $DESC" "$RSYSLOGD" do_reload "$RSYSLOGD" "$RSYSLOGD_PIDFILE" log_end_msg $? ;; restart) $0 stop $0 start ;; *) echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2 exit 3 ;; esac : |
Next we create a default configuration in /etc/default/rsyslogd:
vi /etc/default/rsyslog
Just copy and paste the contents below:
# Options to rsyslogd # -m 0 disables 'MARK' messages. # -r enables logging from remote machines # -x disables DNS lookups on messages recieved with -r # See rsyslogd(8) for more details RSYSLOGD_OPTIONS="-m 0" # Options to rklogd # -2 prints all kernel oops messages twice; once for klogd to decode, and # once for processing with 'ksymoops' # -x disables all klogd processing of oops messages entirely # See rklogd(8) for more details RKLOGD_OPTIONS="-x" |
Next we create the directory /etc/rsyslog.d - this is needed for additional configuration options (not covered in the tutorial).
mkdir /etc/rsyslog.d
Now we have all of the configuration files, but we need now to make sure rsyslog is started at boot time in the right moment. Please note that rsyslog is not compatible with klogd and sysklogd which are installed by default. These need to be disabled or de-installed. In this tutorial I will disable them.
Issue the following commands to start rsyslogd at the right time during system boot and to disable klogd and sysklogd.
ln -s /etc/init.d/rsyslog /etc/rc3.d/S10rsyslog
mv /etc/rc3.d/S10sysklogd /etc/rc3.d/_S10sysklogd
mv /etc/rc3.d/S11klogd /etc/rc3.d/_S11klogd
This concludes the build and configuration part.
Step 3: Install Apache2 And PHP5
You can skip this if you have already installed apache2 and php5.
apt-get install apache2 apache2-doc apache2-mpm-prefork apache2-utils libexpat1 ssl-cert
apt-get install libapache2-mod-php5 php5 php5-common php5-curl php5-dev php5-gd php5-idn php-pear php5-imagick php5-imap php5-json php5-mcrypt php5-memcache php5-mhash php5-ming php5-mysql php5-ps php5-pspell php5-recode php5-snmp php5-sqlite php5-tidy php5-xmlrpc php5-xsl
You will be asked the following question:
Continue installing libc-client without Maildir support? <-- Yes
Step 4: Get phpLogCon And Install phpLogCon
cd /tmp
wget http://www.phplogcon.org/Downloads-req-getit-lid-6.phtml
Next we will create a directory for phpLogCon and enable it in Apache2:
tar xvzf phplogcon-1.2.3.tar.gz
mkdir /var/www/phplogcon
cp -R phplogcon-1.2.3/* /var/www/phplogcon
Configure apache2 for phpLogcon:
vi /etc/apache2/sites-enabled/your_site_conf
Add the following alias to enable phplogcon:
Alias /phplogcon "/var/www/phplogcon"
Insert the line above before </virtualhost> or if you have already aliases defined, insert the line there.
Note: if /var/www is your server root then you don't need to create an alias.
Restart apache2 to enable the changes.
/etc/init.d/apache2 force-reload
Step 5: Configure phpLogCon And Install The Tables In The rsyslog DB
Point your browser to http://yourdomain.tld/phplogcon and follow the steps on the screen. After the installation wizard has finished, the database is populated with all of the required tables and an administrative user is created.
Delete the install directory to enable phpLogCon:
rm -R /var/www/phplogcon/install
Step 6: Start rsyslog
Issue the following commands to start rsyslog and to stop klogd and sysklogd:
/etc/init.d/sysklogd stop
/etc/init.d/klogd stop
/etc/init.d/rsyslog start
Step 7: Enjoy rsyslog
Point your browser to http://yourdomain.tld/phplogcon and log in with the username and password you specified in the installation wizard. You should now see something like this:
