NodeTypeSyslogNg < CmsTier3

Tags: view all tags
<!-- keep this as a security measure:
   #uncomment if the subject should only be modifiable by the listed groups 
   * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.CMSAdminGroup
   * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.CMSAdminGroup
   #uncomment this if you want the page only be viewable by the listed groups
   # * Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.CMSAdminGroup
-->

---+!! Node Type: %CALC{"$SUBSTITUTE(%TOPIC%,NodeType,)"}%

---++!! Firewall requirements

| *local port* | *open to* | *reason* |
| 22/tcp  | 129.129.194.77/16 |  ssh |
| 1514/tcp | 192.33.123.29/24 |  syslog-ng |
| 514/udp | 192.33.123.29/24 |  syslog-ng |

---

%TOC{title="Table of contents"}%

---+ Regular Maintenance work

<!--
 #List any regular activities which do not run automatically and need an administrator's action.
--> In the morning have a look to the logs by running:
<pre>logwatch --logdir /var/log/remote-archive/current --range today --archive  --detail high --print --splithosts</pre>

Toy with the parameter =--range=.

---+ Emergency Measures

<!--
 #List any measures that must be taken in case of some major incident, e.g. whether a mailing
 #list must be contacted or whether other services need to be shut down, etc. 
--> None.

---+ Installation

<!--
 #Comment here on any peculiarities of the installation, e.g. on special packages needed, special setup
 #procedures which are not obvious
--> In a distributed installation is useful to install some kind of central logs server, at PSI the default system for this task is *syslog-ng* and we've used it at T3 but there is also *rsyslog*; so in our *syslog-ng* installation ver =2.1.4-9= retrieved by the [[http://fedoraproject.org/wiki/EPEL][EPEL yum repo]]:
   * The VMWare VM =t3service01= is the actual central logs host and it was installed by the Puppet profile =/afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/manifests/nodes/t3syslogng.pp=; have a look there.
   * For security reasons =t3service01= will accept logs, both TCP or UDP, just from clients hosted on 192.33.123.29/24
   * For security, no SSH connections from 192.33.123.29/24, you need the Token.
   * Linux servers use *syslog-ng* on TCP =&gt; No messages lost.
   * Solaris servers still use the standard *syslogd* on UDP =&gt; Messages could be lost wihout notice.

---++ Logs archive directories structure

On =t3service01= you'll find:
   * All logs archived below =/var/log/remote-archive=
   * Subdirectory structure as in =/var/log/remote-archive/YEAR/MONTH/DATE=.
   * In order to allow easy access for parsing tools, a directory =/var/log/remote-archive/current= exists in which the cron job =/etc/cron.daily/create-log-link= keeps updated a number of symbolic links to the recent log files. Basically:
<pre>[root@t3service01 puppet]# ll /var/log/remote-archive/current
total 0
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages -&gt; /var/log/remote-archive/2012/01/15/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.1 -&gt; /var/log/remote-archive/2012/01/14/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.10 -&gt; /var/log/remote-archive/2012/01/05/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.2 -&gt; /var/log/remote-archive/2012/01/13/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.3 -&gt; /var/log/remote-archive/2012/01/12/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.4 -&gt; /var/log/remote-archive/2012/01/11/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.5 -&gt; /var/log/remote-archive/2012/01/10/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.6 -&gt; /var/log/remote-archive/2012/01/09/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.7 -&gt; /var/log/remote-archive/2012/01/08/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.8 -&gt; /var/log/remote-archive/2012/01/07/messages
lrwxrwxrwx 1 root root 43 Jan 15 04:02 messages.9 -&gt; /var/log/remote-archive/2012/01/06/messages
</pre>
---++ Configuration
   * Central Log collector (t3service01) 
      * The active configuration is kept in =/etc/syslog-ng/syslog-ng.conf= and =/etc/sysconfig/syslog-ng=, both matters, the former will be definitely different on the clients, please look their Puppet profile =/afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/modules/syslog-ng/manifests/init.pp=
      * The standard *syslog-ng* cron job =/etc/cron.daily/syslog-ng= has been augmented with the generation of the dynamic links.
      * There is also a cron =/etc/cron.daily/create-log-link= to update the link =/var/log/remote-archive/current=
   * Linux clients 
      * Like for the server, the configuration is kept in both =/etc/syslog-ng/syslog-ng.conf= and =/etc/sysconfig/syslog-ng=
   * Solaris clients 
      * The configuration is kept in =/etc/syslog.conf=
      * The configuration gets parsed by m4 when the service reads it. It is written in such a way (default) that logs get sent to _loghost_ If _loghost_ is defined in =/etc/hosts= (or elsewhere). Once you have modified =/etc/hosts= you need to restart the syslogd daemon by =svcadm refresh svc:/system/system-log=

---++ Testing logging to the central server from clients

Use the *logger* shell command with a priority level that is among the filters that get routed to the central log host, like:
   * =logger -p daemon.notice "Test log message from df"=
   * =logger -p user.err "Hello from this server"=
---++ How to use logwatch

Example: Execute the following line from the admin machine
<verbatim>
ssh t3service01 logwatch --logdir /var/log/remote-archive/current --range '"between yesterday and now"' \
--archive  --detail high --print --splithosts
</verbatim>

On the admin host there is a little utility (in the path of root) for getting such reports:
<pre>cl_logwatch.sh 
cl_logwatch.sh "-3 days"
</pre>

---+ Services

<!--
 #List all the important services, their installation, configuration and how to start and stop them
--> Look our [[https://t3nagios.psi.ch/nagios/cgi-bin/status.cgi?host=t3service01][Nagios]].

---+ Backups

Standard VMWare/Netapp backups performed by PSI.

-- Main.FabioMartinelli - 2012-01-12
NodeTypeForm
Hostnames	t3service01
Services	Syslog-ng 2.1.4-9 Central Logging Service
Hardware	PSI VM DMZ cluster
Install Profile	vmsyslogng
Guarantee/maintenance until	ask Peter
Topic revision: r4 - 2012-03-27 - FabioMartinelli
CmsTier3
User Pages
Main Page
Policies
Physics Groups
Steering Board Meetings
Admin Pages
AdminArea
Cluster Specs