<!-- keep this as a security measure: #uncomment if the subject should only be modifiable by the listed groups * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.CMSAdminGroup * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.CMSAdminGroup #uncomment this if you want the page only be viewable by the listed groups # * Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.CMSAdminGroup --> ---+!! Node Type: %CALC{"$SUBSTITUTE(%TOPIC%,NodeType,)"}% ---++!! Firewall requirements | *local port* | *open to* | *reason* | | 22/tcp | 129.129.194.77/16 | ssh | | 1514/tcp | 192.33.123.29/24 | syslog-ng | | 514/udp | 192.33.123.29/24 | syslog-ng | --- %TOC{title="Table of contents"}% ---+ Regular Maintenance work <!-- #List any regular activities which do not run automatically and need an administrator's action. --> In the morning have a look to the logs by running: <pre>logwatch --logdir /var/log/remote-archive/current --range today --archive --detail high --print --splithosts</pre> Toy with the parameter =--range=. ---+ Emergency Measures <!-- #List any measures that must be taken in case of some major incident, e.g. whether a mailing #list must be contacted or whether other services need to be shut down, etc. --> None. ---+ Installation <!-- #Comment here on any peculiarities of the installation, e.g. on special packages needed, special setup #procedures which are not obvious --> In a distributed installation is useful to install some kind of central logs server, at PSI the default system for this task is *syslog-ng* and we've used it at T3 but there is also *rsyslog*; so in our *syslog-ng* installation ver =2.1.4-9= retrieved by the [[http://fedoraproject.org/wiki/EPEL][EPEL yum repo]]: * The VMWare VM =t3service01= is the actual central logs host and it was installed by the Puppet profile =/afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/manifests/nodes/t3syslogng.pp=; have a look there. * For security reasons =t3service01= will accept logs, both TCP or UDP, just from clients hosted on 192.33.123.29/24 * For security, no SSH connections from 192.33.123.29/24, you need the Token. * Linux servers use *syslog-ng* on TCP => No messages lost. * Solaris servers still use the standard *syslogd* on UDP => Messages could be lost wihout notice. ---++ Logs archive directories structure On =t3service01= you'll find: * All logs archived below =/var/log/remote-archive= * Subdirectory structure as in =/var/log/remote-archive/YEAR/MONTH/DATE=. * In order to allow easy access for parsing tools, a directory =/var/log/remote-archive/current= exists in which the cron job =/etc/cron.daily/create-log-link= keeps updated a number of symbolic links to the recent log files. Basically: <pre>[root@t3service01 puppet]# ll /var/log/remote-archive/current total 0 lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages -> /var/log/remote-archive/2012/01/15/messages lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.1 -> /var/log/remote-archive/2012/01/14/messages lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.10 -> /var/log/remote-archive/2012/01/05/messages lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.2 -> /var/log/remote-archive/2012/01/13/messages lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.3 -> /var/log/remote-archive/2012/01/12/messages lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.4 -> /var/log/remote-archive/2012/01/11/messages lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.5 -> /var/log/remote-archive/2012/01/10/messages lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.6 -> /var/log/remote-archive/2012/01/09/messages lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.7 -> /var/log/remote-archive/2012/01/08/messages lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.8 -> /var/log/remote-archive/2012/01/07/messages lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.9 -> /var/log/remote-archive/2012/01/06/messages </pre> ---++ Configuration * Central Log collector (t3service01) * The active configuration is kept in =/etc/syslog-ng/syslog-ng.conf= and =/etc/sysconfig/syslog-ng=, both matters, the former will be definitely different on the clients, please look their Puppet profile =/afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/modules/syslog-ng/manifests/init.pp= * The standard *syslog-ng* cron job =/etc/cron.daily/syslog-ng= has been augmented with the generation of the dynamic links. * There is also a cron =/etc/cron.daily/create-log-link= to update the link =/var/log/remote-archive/current= * Linux clients * Like for the server, the configuration is kept in both =/etc/syslog-ng/syslog-ng.conf= and =/etc/sysconfig/syslog-ng= * Solaris clients * The configuration is kept in =/etc/syslog.conf= * The configuration gets parsed by m4 when the service reads it. It is written in such a way (default) that logs get sent to _loghost_ If _loghost_ is defined in =/etc/hosts= (or elsewhere). Once you have modified =/etc/hosts= you need to restart the syslogd daemon by =svcadm refresh svc:/system/system-log= ---++ Testing logging to the central server from clients Use the *logger* shell command with a priority level that is among the filters that get routed to the central log host, like: * =logger -p daemon.notice "Test log message from df"= * =logger -p user.err "Hello from this server"= ---++ How to use logwatch Example: Execute the following line from the admin machine <verbatim> ssh t3service01 logwatch --logdir /var/log/remote-archive/current --range '"between yesterday and now"' \ --archive --detail high --print --splithosts </verbatim> On the admin host there is a little utility (in the path of root) for getting such reports: <pre>cl_logwatch.sh cl_logwatch.sh "-3 days" </pre> ---+ Services <!-- #List all the important services, their installation, configuration and how to start and stop them --> Look our [[https://t3nagios.psi.ch/nagios/cgi-bin/status.cgi?host=t3service01][Nagios]]. ---+ Backups Standard VMWare/Netapp backups performed by PSI. -- Main.FabioMartinelli - 2012-01-12
NodeTypeForm
Hostnames
t3service01
Services
Syslog-ng 2.1.4-9 Central Logging Service
Hardware
PSI VM DMZ cluster
Install Profile
vmsyslogng
Guarantee/maintenance until
ask Peter
This topic: CmsTier3
>
WebHome
>
AdminArea
>
NodeTypeSyslogNg
Topic revision: r4 - 2012-03-27 - FabioMartinelli
Copyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki?
Send feedback