Node Type: SyslogNg

Firewall requirements

Regular Maintenance work

In the morning have a look to the logs by running:

logwatch --logdir /var/log/remote-archive/current --range today --archive  --detail high --print --splithosts

Toy with the parameter --range.

Emergency Measures

None.

Installation

In a distributed installation is useful to install some kind of central logs server, at PSI the default system for this task is syslog-ng and we've used it at T3 but there is also rsyslog; so in our syslog-ng installation ver 2.1.4-9 retrieved by the EPEL yum repo:

• The VMWare VM t3service01 is the actual central logs host and it was installed by the Puppet profile /afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/manifests/nodes/t3syslogng.pp; have a look there.
• For security reasons t3service01 will accept logs, both TCP or UDP, just from clients hosted on 192.33.123.29/24
• For security, no SSH connections from 192.33.123.29/24, you need the Token.
• Linux servers use syslog-ng on TCP => No messages lost.
• Solaris servers still use the standard syslogd on UDP => Messages could be lost wihout notice.

Logs archive directories structure

On t3service01 you'll find:

• All logs archived below /var/log/remote-archive
• Subdirectory structure as in /var/log/remote-archive/YEAR/MONTH/DATE.
• In order to allow easy access for parsing tools, a directory /var/log/remote-archive/current exists in which the cron job /etc/cron.daily/create-log-link keeps updated a number of symbolic links to the recent log files. Basically:

[root@t3service01 puppet]# ll /var/log/remote-archive/current
total 0
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages -> /var/log/remote-archive/2012/01/15/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.1 -> /var/log/remote-archive/2012/01/14/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.10 -> /var/log/remote-archive/2012/01/05/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.2 -> /var/log/remote-archive/2012/01/13/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.3 -> /var/log/remote-archive/2012/01/12/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.4 -> /var/log/remote-archive/2012/01/11/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.5 -> /var/log/remote-archive/2012/01/10/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.6 -> /var/log/remote-archive/2012/01/09/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.7 -> /var/log/remote-archive/2012/01/08/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.8 -> /var/log/remote-archive/2012/01/07/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.9 -> /var/log/remote-archive/2012/01/06/messages

Configuration

• Central Log collector (t3service01)
  • The active configuration is kept in /etc/syslog-ng/syslog-ng.conf and /etc/sysconfig/syslog-ng, both matters, the former will be definitely different on the clients, please look their Puppet profile /afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/modules/syslog-ng/manifests/init.pp
  • The standard syslog-ng cron job /etc/cron.daily/syslog-ng has been augmented with the generation of the dynamic links.
  • There is also a cron /etc/cron.daily/create-log-link to update the link /var/log/remote-archive/current
• Linux clients
  • Like for the server, the configuration is kept in both /etc/syslog-ng/syslog-ng.conf and /etc/sysconfig/syslog-ng
• Solaris clients
  • The configuration is kept in /etc/syslog.conf
  • The configuration gets parsed by m4 when the service reads it. It is written in such a way (default) that logs get sent to loghost If loghost is defined in /etc/hosts (or elsewhere). Once you have modified /etc/hosts you need to restart the syslogd daemon by svcadm refresh svc:/system/system-log

Testing logging to the central server from clients

Use the logger shell command with a priority level that is among the filters that get routed to the central log host, like:

• logger -p daemon.notice "Test log message from df"
• logger -p user.err "Hello from this server"

How to use logwatch

Example: Execute the following line from the admin machine

ssh t3service01 logwatch --logdir /var/log/remote-archive/current --range '"between yesterday and now"' \
--archive  --detail high --print --splithosts

On the admin host there is a little utility (in the path of root) for getting such reports:

cl_logwatch.sh 
cl_logwatch.sh "-3 days"

Services

Look our Nagios.

Backups

Standard VMWare/Netapp backups performed by PSI.

-- FabioMartinelli - 2012-01-12