PhoenixBlog20080923x1019 < LCGTier2

Tags: view all tags

<!-- keep this as a security measure:
   * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.LCGAdminGroup
   * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.LCGAdminGroup
   #uncomment this if you want the page only be viewable by the internal people
   #* Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.LCGAdminGroup
-->

KeyWords: SysAdmin, [[Nagios]]


---+ LCG-enabled Nagios probes

An experimental [[https://ui.lcg.cscs.ch/nagios/ Nagios server]] has
been installed on host =ui.lcg.cscs.ch=, to monitor the availability
of LCG-services running on the local PHOENIX cluster.
Two types of Nagios probes are run:
   * _remote_ passive checks, gathering results from 
     [[https://lcg-sam.cern.ch:8443/sam/sam.py LCG SAM]]; 
   * _local_ active checks, comprising both simple network-level
     probes (e.g., TCP ports open, LDAP response time, etc.)
     and Grid-level probes (e.g., SRM and !GridFTP operations).
The Grid-level probes require a valid certificate to run, which the system
operators must supply; therefore, the Grid-level tests are currently run
within the [[http://glite-sv.scai.fraunhofer.de/wiki/ext/index.php?n=Main.DECHVO DECH VO]].

Anyone registered with the VOMS servers of any-one of the supported
VOs (that is: atlas, cms, dech, dteam, gear, hone, lhcb, ops) can view
the [[https://ui.lcg.cscs.ch/nagios/ Nagios server pages]].  Of
course, only local administrators can issue commands through the
Nagios web interface.

You need to have your LCG certificate loaded into the browser!
By default, Firefox keeps popping up a dialog box to confirm that you want
to send your certificate to the server; to disable this behavior,
follow this procedure:
   1. Open the _Preferences..._ item from the _Edit_ menu;
   2. Click on the _Advanced_ tab;
   3. Select item _Select one automatically_ under 
      _When server requests my personal certificate_.
   4. Click on _OK_

*Note:* In the future we may drop the web interface support and
integrate the Nagios reporting in the 
[[http://nagios.cscs.ch/ central CSCS Nagios]], 
for accessing which you will need a local CSCS account.


---++ Notes on the installation

The service configuration scripts and instructions are provided
by the [[https://savannah.cern.ch/projects/gsmonitoring/ Grid Monitoring Group]];
installation has been done as described in the [[https://twiki.cern.ch/twiki/bin/view/LCG LCG TWiki]] pages 
[[https://twiki.cern.ch/twiki/bin/view/LCG/GridMonitoringNcg GridMonitoringNcg]]
and [[https://twiki.cern.ch/twiki/bin/view/LCG/GridMonitoringNcgYaim GridMonitoringNcgYaim]].

*Note:* All the issues reported in this page have been promptly fixed by Steve Traylen.  Thanks!!

There are however a few issues stil to be ironed outl, both in the docs and in the supplied scripts;
here's a list of what we found:

   * The credential name (set with the =-k= option to =myproxy-init=)
     must be _unique_ among those uploaded to the same server: 
     a simple solution is to embed the site name into the credential name,
     that is, use =NagiosRetrieve-$SITENAME= instead of =NagiosRetrieve=.
     A [[https://twiki.cscs.ch/twiki/bin/viewfile/LCGTier2/PhoenixBlog20080923x1019?filename=use_sitename_in_myproxy_credential.diff patch]]
     to this extent has to modify two files:
      1. YAIM's =config_nagios_proxy_renew= to generate =/etc/nagios-proxy-refresh.conf=
         including the site name into the credential name;
      2. NCG's =ConfigGen/Nagios.pm= to include the credential name
         =NagiosRetrieve-$SITENAME= into =/etc/nagios/wlcg_resources.cfg=,
         (used by tests checking for credential availability).
     (A different approach could be to use the host DN as !MyProxy "username"
     (=-l= option) and store the credential under a well-known name.)

   * Retrieval of credentials by _name_ is inherently _insecure_ and
     lends itself to easy denial of service: anyone can upload a
     different proxy under that name, and have tests running with
     his/her signature, possibly disrupting the service.


   * TWiki page [[https://twiki.cern.ch/twiki/bin/view/LCG/GridMonitoringNcg GridMonitoringNcg]]:
      * In the example "myproxy-init" command line:
         * argument to "-k" option must match what is in =/etc/nagios-proxy-retrieve.conf=
           (currently it's =NagiosRetrieve=. not =NagiosRefresh= as written instead
           on the TWiki);
         * option "-s" is repeated twice;
         * =-s $MYPROXY_SERVER= would be a better choice (works also with copy+paste).
     

   * TWiki page [[https://twiki.cern.ch/twiki/bin/view/LCG/GridMonitoringNcgYaim GridMonitoringNcgYaim]]:
      * must use =-n glite-NAGIOS -n glite-UI= in the YAIM incantation
        even if the UI software has already been installed and configured;
        otherwise the UI software and environment (notably,
        ==myproxy-init== and =$MYPROXY_SERVER=) might not work.
        
     
   * In YAIM functions:
      * config_voms2htpasswd: several issues [[https://twiki.cscs.ch/twiki/bin/viewfile/LCGTier2/PhoenixBlog20080923x1019?filename=config_voms2htpasswd.diff (patch)]]:
         1. VOMS server is hard-coded (it uses the one at CERN, irrespective of what's in the =vomss://= URL); 
         2. if the VOMS server uses the old XML format, DNs are not printed; 
         3. the wrong path in installed cron job: voms2htpasswd is in =/usr/bin=, not =/bin=
      * config_nagios_proxy_renew: need to set =MYPROXY_NAME= to include site name (see above)
        [[https://twiki.cscs.ch/twiki/bin/viewfile/LCGTier2/PhoenixBlog20080923x1019?filename=config_nagios_proxy_renew.diff (patch)]];
      * config_httpd_nagios: Apache needs to be restarted every few hours to re-read the latest CRLs, but
        the cron job is too verbose; need to bracket
        [[https://twiki.cscs.ch/twiki/bin/viewfile/LCGTier2/PhoenixBlog20080923x1019?filename=config_httpd_nagios.diff (patch)]]: 
        <verbatim>
           (service httpd status && service httpd graceful) > /dev/null 2>&1     
        </verbatim>
     
   * In the generated file =/etc/nagios/wlcg.d/services.cfg=:
      * org.nagios-BDII service entry uses SRMv2 object entry DN instead
        of the top-level "mds-vo-name=resource,o=grid".  Is this intentional?
     
     
   * =/usr/sbin/nagios-proxy-refresh=:
      * error message doesn't match file location
        [[https://twiki.cscs.ch/twiki/bin/viewfile/LCGTier2/PhoenixBlog20080923x1019?rev=1;filename=nagios-proxy-refresh.diff (patch)]]:

     
     
   * The wrong LDAP DN for SRM !GlueService* objects is configured in the =org.nagios-BDII= service probe,
     causing tests to fail.  Lines 335 and 337 in
     =/usr/lib/perl5/vendor_perl/5.8.5/NCG/LocalMetricsAttrs/Active.pm= should be changed as follows 
     [[https://twiki.cscs.ch/twiki/bin/viewfile/LCGTier2/PhoenixBlog20080923x1019?filename=Active.pm.diff (patch)]]:
         $ 335:  $self->{SITEDB}->hostAttribute($hostname, "MDS_DN", "GlueServiceUniqueID=httpg://$hostname:".$self->{SITEDB}->hostAttribute($hostname,"SRM2_PORT")."/srm/managerv2,GlueSEUniqueID=".$hostname.",Mds-Vo-Name=local,O=Grid");
         $ 337:  $self->{SITEDB}->hostAttribute($hostname, "MDS_DN", "GlueServiceUniqueID=httpg://$hostname:".$self->{SITEDB}->hostAttribute($hostname,"SRM1_PORT")."/srm/managerv1,GlueSEUniqueID=".$hostname.",Mds-Vo-Name=local,O=Grid");
     

-- Main.RiccardoMurri - 18 Aug 2008


---++ Additional comments

%COMMENT{type="below"}%

Attachments

Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
diff	Active.pm.diff	r1	manage	1.6 K	2008-09-23 - 11:07	RiccardoMurri
diff	config_httpd_nagios.diff	r1	manage	0.6 K	2008-09-23 - 11:06	RiccardoMurri
diff	config_voms2htpasswd.diff	r1	manage	0.7 K	2008-09-23 - 11:06	RiccardoMurri
diff	nagios-proxy-refresh.diff	r1	manage	0.4 K	2008-09-23 - 11:05	RiccardoMurri
diff	use_sitename_in_myproxy_credential.diff	r1	manage	2.2 K	2008-09-23 - 13:08	RiccardoMurri	If SITENAME is defined, use NagiosRetrieve-$SITENAME as myproxy credential name, otherwise fall back to "NagiosRetrieve"; patches /usr/lib/perl5/vendor_perl/5.8.5/NCG/ConfigGen/Nagios.pm and /opt/glite/yaim/functions/config_nagios_proxy_renew