Firewall requirements
local port |
open to |
reason |
22/tcp |
129.129.194.77/16 |
ssh |
514/udp |
192.33.123.29/24 |
syslog-ng |
1514/tcp |
192.33.123.29/24 |
syslog-ng |
Regular Maintenance work
In the morning have a look to the logs by running:
logwatch --logdir /var/log/remote-archive/current --range today --archive --detail high --print --splithosts
Toy with the parameter
--range
.
Emergency Measures
None.
Installation
In a distributed installation is useful to install some kind of central logs server, at PSI the default system for this task is
syslog-ng and we've used it at T3 but there is also
rsyslog; so in our
syslog-ng installation ver
2.1.4-9
retrieved by the
EPEL yum repo:
- The VMWare VM
t3service01
is the actual central logs host and it was installed by the Puppet profile /afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/manifests/nodes/t3syslogng.pp
; have a look there.
- For security reasons
t3service01
will accept logs, both TCP or UDP, just from clients hosted on 192.33.123.29/24
- For security, no SSH connections from 192.33.123.29/24, you need the Token.
- Linux servers use syslog-ng on TCP => No messages lost.
- Solaris servers still use the standard syslogd on UDP => Messages could be lost wihout notice.
Logs archive directories structure
On
t3service01
you'll find:
- All logs archived below
/var/log/remote-archive
- Subdirectory structure as in
/var/log/remote-archive/YEAR/MONTH/DATE
.
- In order to allow easy access for parsing tools, a directory
/var/log/remote-archive/current
exists in which the cron job /etc/cron.daily/create-log-link
keeps updated a number of symbolic links to the recent log files. Basically:
[root@t3service01 puppet]# ll /var/log/remote-archive/current
total 0
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages -> /var/log/remote-archive/2012/01/15/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.1 -> /var/log/remote-archive/2012/01/14/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.10 -> /var/log/remote-archive/2012/01/05/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.2 -> /var/log/remote-archive/2012/01/13/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.3 -> /var/log/remote-archive/2012/01/12/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.4 -> /var/log/remote-archive/2012/01/11/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.5 -> /var/log/remote-archive/2012/01/10/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.6 -> /var/log/remote-archive/2012/01/09/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.7 -> /var/log/remote-archive/2012/01/08/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.8 -> /var/log/remote-archive/2012/01/07/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.9 -> /var/log/remote-archive/2012/01/06/messages
Configuration
- Central Log collector (t3service01)
- The active configuration is kept in
/etc/syslog-ng/syslog-ng.conf
and /etc/sysconfig/syslog-ng
, both matters, the former will be definitely different on the clients, please look their Puppet profile /afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/modules/syslog-ng/manifests/init.pp
- The standard syslog-ng cron job
/etc/cron.daily/syslog-ng
has been augmented with the generation of the dynamic links.
- There is also a cron
/etc/cron.daily/create-log-link
to update the link /var/log/remote-archive/current
- Linux clients
- Like for the server, the configuration is kept in both
/etc/syslog-ng/syslog-ng.conf
and /etc/sysconfig/syslog-ng
- Solaris clients
- The configuration is kept in
/etc/syslog.conf
- The configuration gets parsed by m4 when the service reads it. It is written in such a way (default) that logs get sent to loghost If loghost is defined in
/etc/hosts
(or elsewhere). Once you have modified /etc/hosts
you need to restart the syslogd daemon by svcadm refresh svc:/system/system-log
Testing logging to the central server from clients
Use the
logger shell command with a priority level that is among the filters that get routed to the central log host, like:
-
logger -p daemon.notice "Test log message from df"
-
logger -p user.err "Hello from this server"
How to use logwatch
Example: Execute the following line from the admin machine
ssh t3service01 logwatch --logdir /var/log/remote-archive/current --range '"between yesterday and now"' \
--archive --detail high --print --splithosts
On the admin host there is a little utility (in the path of root) for getting such reports:
cl_logwatch.sh
cl_logwatch.sh "-3 days"
Services
Look our
Nagios.
Backups
Standard VMWare/Netapp backups performed by PSI.
--
FabioMartinelli - 2012-01-12