Typically there are a large number of devices on the network (Firewalls, routers, switches, UNIX systems) that are generating logs and storing them locally. These fill up, rotate and eventually disappear, which can make diagnostics difficult and auditing impossible if there is only a small retention time for those logs.
Something I've used before for centralising these logs has been a small Linux installation (CentOS, RHEL or some other distribution) that runs a syslog server and stores all of those syslog entries centrally. Custom scripts can be used to archive, backup, delete or zip obsolete logs.
If you use the standard syslog server under linux, the logs get very messy and are just difficult to use, defeating the point of centralising in the first place. I've used the rsyslog package as it has enough configurability to allow you to separate logs based on the host and day that the logs were generated, perfect for viewing and archiving.
To get started, you'll need to perform an installation of Linux on something with enough space to store the logs data you expect to receive and intend to keep. If possible, create another filesystem and/or volume group for the logs to be stored in. Also, make sure that all of the devices will be able to reach the linux installation on the network.
Once you have a linux installation up and running with some basic configuration, download and install the rsyslog package.
[root@logger ~]# yum install rsyslog
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: ftp.swin.edu.au
* extras: ftp.swin.edu.au
<SNIP>
Running Transaction
Installing : rsyslog 1/1
Installed:
rsyslog.i386 0:3.22.1-3.el5_6.1
Complete!
[root@logger ~]#
Now that rsyslog is installed, you can configure the rsyslog service to log all data to the filesystem you created for syslogs, /data/logs in this example. My rsyslog.conf looks like the following:
# Provides kernel logging support (previously done by rklogd)
$ModLoad imklog # Provides support for local system logging (e.g. via logger command) $ModLoad imuxsock $ModLoad imudp $UDPServerRun 514 $template DynFile, "/data/logs/system-%HOSTNAME%-%$NOW%.log *.* ?DynFile |
The last two lines are the most important as they specify to log everything to /data/logs and include the hostname and datestamp (YYYY-MM-DD) in the filename. The two lines before that enable rsyslog to listen on the standard syslog port (UDP/514).
Now that the configuration is done, disable the native syslog service, enable rsyslog and start it up.
[root@logger etc]# chkconfig --level 2345 syslog off
[root@logger etc]# chkconfig --level 2345 rsyslog on
[root@logger etc]# chkconfig --list | grep syslog
rsyslog 0:off 1:off 2:on 3:on 4:on 5:on 6:off
syslog 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@logger etc]# service syslog stop
Shutting down kernel logger: [ OK ]
Shutting down system logger: [ OK ]
[root@logger logs]# service rsyslog start
Starting system logger: [ OK ]
[root@logger logs]#
Assuming that the firewall rules are ok on the system (inbound UDP/514 allowed), you should be able to start receiving syslog entries from remote systems. Configure the remote systems to log to the rsyslog server and there should be a single syslog file generated per system per day.
Once you start to receive logs from remote systems, you'll also need to script something to backup logs, purge and zip them as required or the filesystem will eventually fill up. You can use a combination of the 'find' command and 'gzip' to reduce space requirements e.g.
find /data/logs -mtime +5 -type f -exec gzip -9 {} \;
Similar methods can be used to purge logs or send them off to a backup server for archival to tape.
