Centralized Syslog Server Using syslog-NG with web Interface using php-syslog-ng
Central Log Management System is a simple web based logging system which allows logging all syslog messages from various Network Devices, Unix, Linux, Solaris and Windows Servers. This allows the visibility of logs from all these devices in one single interface.
What is Syslog-NG ?
syslog-ngŠ is the world's most flexible and scalable audit trail processing tool for organizations of any size. It provides a centralised, securely stored log of all devices on your network, whatever platform they run on. And syslog-ng also incorporates a host of powerful features, including filtering based on message content, as well as customisable data mining and analysis capabilities.
Downloads and Documentation
http://www.balabit.com/products/syslog_ng/
Syslog-Ng FAQ
http://www.campin.net/syslog-ng/faq.html
Syslog-NG Server Configuration
Install Syslog-NG in debian
#apt-get install syslog-ng
Reading Package Lists... Done
Building Dependency Tree... Done
The following packages will be REMOVED:
klogd sysklogd
The following NEW packages will be installed:
syslog-ng
0 upgraded, 1 newly installed, 2 to remove and 8 not upgraded.
Need to get 215kB of archives.
After unpacking 233kB of additional disk space will be used.
Do you want to continue? [Y/n]y
This will complete the installation process.
Installation Error with solution
Building Dependency Tree... Done
The following packages will be REMOVED:
klogd sysklogd
The following NEW packages will be installed:
syslog-ng
0 upgraded, 1 newly installed, 2 to remove and 8 not upgraded.
Need to get 215kB of archives.
After unpacking 233kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://mirror.ox.ac.uk stable/main syslog-ng 1.6.5-2.2 [215kB]
Fetched 215kB in 0s (683kB/s)
(Reading database ... 16396 files and directories currently installed.)
Removing klogd ...
Stopping kernel log daemon: klogd.
Removing sysklogd ...
Stopping system log daemon: syslogd.
Selecting previously deselected package syslog-ng.
(Reading database ... 16374 files and directories currently installed.)
Unpacking syslog-ng (from .../syslog-ng_1.6.5-2.2_i386.deb) ...
Setting up syslog-ng (1.6.5-2.2) ...
CONSOLE_LOG_LEVEL is of unaccepted value.
KERNEL_RINGBUF_SIZE is of unaccepted value.
Starting system logging: syslog-ng.
The above error is know problem with syslog-ng 1.6.5-2.2 debian package for this you need to modify the syslog-ng startup script file /etc/init.d/syslog-ng
Change the following
1) Edit the file /etc/default/syslog-ng uncomment the following lines and save the file
CONSOLE_LOG_LEVEL
KERNEL_RINGBUF_SIZE
OR
2)case "x$CONSOLE_LOG_LEVEL" in
x[0-7])
dmesg -n $CONSOLE_LOG_LEVEL
;;
*)
echo "CONSOLE_LOG_LEVEL is of unaccepted value."
;;
esac
to
case "x$CONSOLE_LOG_LEVEL" in
x[0-7])
dmesg -n $CONSOLE_LOG_LEVEL
;;
x)
;;
*)
echo "CONSOLE_LOG_LEVEL is of unaccepted value."
;;
esac
2)case "x$KERNEL_RINGBUF_SIZE" in
x[0-9]*)
dmesg -s $KERNEL_RINGBUF_SIZE
;;
*)
echo "KERNEL_RINGBUF_SIZE is of unaccepted value."
;;
esac
to
case "x$KERNEL_RINGBUF_SIZE" in
x[0-9]*)
dmesg -s $KERNEL_RINGBUF_SIZE
;;
x)
;;
*)
echo "KERNEL_RINGBUF_SIZE is of unaccepted value."
;;
esac
After changing above two changes you need to restart syslog-ng using the following command
#/etc/init.d/syslog-ng start
Syslog-NG Configuration
Default Configuration file located at /etc/syslog-ng/syslog-ng.conf file
Default syslog-ng.conf file click here
Now you need to concentrate on configuration of this file how to fit your need.I am giving example of configuring UDP as source to get the messages remotely for this you need to change the following line under Sources and options
1)# (this is equivalent to the "-r" syslogd flag)
# udp();
to
# (this is equivalent to the "-r" syslogd flag)
udp();
2)# enable or disable DNS usage
# syslog-ng blocks on DNS queries, so enabling DNS may lead to
# a Denial of Service attack
# (default is yes)
use_dns(no);
to
# enable or disable DNS usage
# syslog-ng blocks on DNS queries, so enabling DNS may lead to
# a Denial of Service attack
# (default is yes)
use_dns(yes);
dns_cache(yes);
If you want to know more sources,options,logs,filters,destination click here
Syslog-NG My-SQL Integartion with PHP-syslog-NG Web interface
Make sure the you have installed mysql server and phpmyadmin in your machine. If you want to install run the following command
#apt-get install mysql-server
#apt-get install phpmyadmin
You need to install apache Webserver click here for apache with php support installation steps.
What is PHP-syslog-NG ?
php-syslog-ng is a log monitor designed to let the user quickly and easily manage logs from many hosts.
PHP-Syslog-ng is a front-end for viewing syslog-ng messages logged to MySQL in real-time. It features customized searches based on device, time, priori ty, message, and date.
HOW IT WORKS
Syslog-ng by default collects all the messages from the various sources and pipes them to the MySQL Database. PHP-Syslog-ng which is the front end interface t hen looks and reads from the MySQL Database when searched from the web interface.
Download PHP-syslog-NG
http://www.phpwizardry.com/php-syslog-ng/phpsyslogng-2.8.tar.gz
Installing PHP-syslog-NG with Mysql support for syslog-ng
Download the PHP-syslog-NG in your web server root directory and extract this file phpsyslogng-2.8.tar.gz.Now
#cd phpsyslogng-2.8
Now you are in phpsyslogng-2.8 directory
Configure MySQL
The quickest way to do this is to use the dbsetup.sql file in the scripts directory. Just edit the file and set some passwords for the three users that are created (replace PW_HERE). The script will create a table for logs and a table for user authentication and give the three users some sensible priviliges. If you make other changes like changing the name of the database or the name of the tables then make sure you edit config.php to reflect that. After editing the dbsetup.sql file then just run it like this:
shell> mysql -uroot -p < dbsetup.sql
Configure syslog-ng
Now you need to configure syslog-ng to send the desired log messages to a pipe that can be read to send the entries to MySQL. You will need to add two entries to the syslog-ng configuration file. The configuration file is usually in /etc/syslog-ng/syslog-ng.conf.
You first need to add a new 'destination' entry. Add something like this:
destination d_mysql {
pipe("/var/log/mysql.pipe"
template("INSERT INTO logs
(host, facility, priority, level, tag, datetime, program, msg)
VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC',
'$PROGRAM', '$MSG' );\n") template-escape(yes));
};
That will take your log entries and format them into a SQL query that can be run to add it to the database.
You also need to add an entry that determines what log entries to forward to the FIFO pipe. You will usually want to forward everything to MySQL and there should already be a 'source' entry for that in your syslog-ng.conf file (usually called src or s_all). To tie that source to the destination you just created you will add something like this:
log {
source(s_all);
destination(d_mysql);
};
Setup syslog-ng to MySQL pipe
An example for a script that feeds log entries from the FIFO pipe to MySQL is included in the scripts directory. The script is called syslog2mysql.sh.
#!/bin/bash
if [ ! -e /var/log/mysql.pipe ]
then
mkfifo /var/log/mysql.pipe
fi
while [ -e /var/log/mysql.pipe ]
do
mysql -u syslogfeeder --password=PASS_HERE syslog < /var/log/mysql.pipe >/dev/null
done
If you decide to use this script then you have to replace PASS_HERE with the password for the syslogfeeder user. You will also probably want to have this script started automatically whenever you start the server. So add an entry in the inittab or start it through init.d (or whatever is appropriate on your system). But make sure you call it after MySQL has been started.
Now start the syslog2mysql.sh script:
shell> ./syslog2mysql.sh &
or if you created an init.d script:
shell> /etc/init.d/syslog2mysql start
Note:- This syslog2mysql.sh need to run always in background because using this script all the messges are pipe to mysql database
It's finally time to restart the syslog-ng daemon and start sending your logs to the database:
shell> /etc/init.d/syslog-ng restart
Edit config.php
If you are using the default database setup from the dbsetup.sql file then all you need to do is to enter the passwords for the sysloguser and syslogadmin users, set the right host and port for the database server if it is not on the same server as the web server and set the correct URL. Otherwise read through the config.php file and configure things to suit your needs. All the different options are explained in the file.
Log rotation
Log rotation should be part of most installations where you use php-syslog-ng. It is better to use log rotation than deleting rows in the main table because deleting rows can lead to performance problems. Rotating old logs out of the main table will also usually result in better performance because the tables with old logs are static and can be optimized. There is a logrotate.php script in the scripts directory. You may have to edit it to enter the correct path to your php-syslog-ng installation but after that it should be ready for use. If you enable merge tables in the config.php file then a merge table of all log tables will be created at the end of the script. The merge table will allow you to search across all tables instead of having to do searches against one table at a time. The merge table does equate to a slight performance hit on the search form because the fields are populated based on all tables instead of one particular table.
You can also specify enable the LOGRETENTION setting in config.php. If you enable this then logs older than this setting will be dropped whenever the logrotate.php is run.
If you decide to use the logrotate.php script then just add it to your crontab and have it run however frequent you want (max is currently one time per day).
Logrotate Error
When you run logrotate.php if you are getting the following error
Starting logrotate
2006-05-01 21:50:08
No DB link
then check the file /etc/php4/cli/php.ini for an entry:
;extension=mysql.so
and uncomment this entry (take away the semi-colon):
extension=mysql.so (Thanks to Richard Lucassen for this Solution)
Now you can check your installation http://ipaddress/phpsyslogng/ and login user admin and test as password.
Syslog Client Configuration
If you want to send the all the server syslog messages to syslog server you need to edit the file /etc/syslog.conf and you need to enter the following
*.* <Tab><Tab> @syslog server ipaddress
now you need to restart the syslog in client machine
#/etc/init.d/sysklogd restart - for debian
#/etc/init.d/syslog restart - for Other Linux Distributions
