(r20) MeetingSwissGridOperations20143206 < LCGTier2

Tags: view all tags
<!-- keep this as a security measure:
   * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.LCGAdminGroup,Main.EgiGroup
   * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.LCGAdminGroup
   #uncomment this if you want the page only be viewable by the internal people
   #* Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.LCGAdminGroup,Main.ChippComputingBoardGroup
-->

---+ Swiss Grid Operations Meeting on 2014-03-06
   * *Date and time*: First Thursday of the month, at 14:00
   * *Place*: Vidyo (room: Swiss_Grid_Operations_Meeting, extension: 9305236)
   * *External link*: http://vidyoportal.cern.ch/flex.html?roomdirect.html&key=gDf6l4RlIAGN %RED%(NOTE it has changed since previous meetings)%ENDCOLOR%
   * *Phone gate*: From Switzerland: 0227671400 (portal) + 9305236 (extension) + # (pound sign) %RED%(NOTE it has changed since previous meetings) %ENDCOLOR%
   * *IRC chat*: irc:gridchat.cscs.ch:994#lcg (ask pw via email)

---++ Agenda

Status
   * *CSCS* (reports Miguel): 
      * Procurement: 
         * Received 16x HP WNs (40core, 128GB, 1x SATA disk, each) HP !ProLiant SL210t Gen8 1U Server (total of 8U)
         * Purchased additional storage (375TB) for the Storage Element (2x !NetApp !E5500B FC with 4TB drives + 2x IBM x3650 servers)
         * Purchased 2x IBM x3650 servers for virtualisation (128GB, 1x 10GbE, 5x400GB SAS SSD)
         * Purchased 8x HP WNs (40core, 128GB, 1x SATA disk, each) HP !ProLiant SL210t Gen8 1U Server (total of 4U). To be used preferably by Swiss users.
         * With this, the purchases of phase H are complete
      * Operations: 
         * Provisioned new GPFS nodes using [[https://github.com/puppetlabs/Razor][Razor]] and Puppet. Configuration ongoing.
         * Working on a new monitoring system.
         * Received WNs to be provisioned soon. They will be puppetized (=no more YAIM).
         * Perfsonar deployed with the help from the NGI_DE admins.
      * Issues: 
         * %Y% Thanks to ATLAS folks, spotted a mistake on the !FairShare configuration that prevented ATLAS and CMS jobs to utilise the system according to the agreed share. This has been fixed now.
         * %Y% ARC accounting successfully published to APEL testing servers. Publishing on production servers on the way (currently coordinating with APEL team) along with archived records collected by ARC CEs during previuos months.
         * %Y% Found a problem on WNs where =/tmp/slurmd= was not created by SLURM. This made the node become a job black hole not detected by the health check system.
         * %Y% Resolved intermittent issue with DHCP, during provisioning transfer of initrd could stall. Root cause was as the ethernet and infiniband fabrics share the same VLAN the DHCP server needs to know the two subnets exist as a "shared-network"
         * Some CMS jobs writing to =/tmp= (local) instead of =/tmpdir_slurm= (GPFS).
         * There seems to be still a lot of accounting data missing on the official accounting portal (related to CREAM CEs). Working on to figure out why and how to fix it.

   * *PSI* (reports Fabio): 
      * Banned a user in dCache 2.6
      * Installed the latest [[http://repository.egi.eu/mirrors/EMI/tarball/test/sl5/emi3-emi-wn/][UMD3 SL5 WN tarball]]; Daniel will test it.
      * Installed the latest UMD3 SL5 UI RPM: opened a [[https://ggus.eu/index.php?mode=ticket_info&ticket_id=101813][GGUS lcg-cp 1.16.0-2.el5 fails with "zero number of replicas"]] caused by a user moving an unusually big 18GB file: it turned out that the command =lcp-cp= got outdated and WLCG uses now [[http://rpm.pbone.net/index.php3/stat/45/idpl/25364000/numer/1/nazwa/gfal-copy][gfal-copy]]. First run of =gfal-copy= on SL5 crashed..
      * Because of the PSI computer room 'warming initiative' ( to save the $$ of cooling ) I'm beefing up the T3 HW monitoring: 
         * Learned an [[https://pypi.python.org/pypi/nagiosplugin/][Object Oriented Python Nagios API]] and created a [[https://bitbucket.org/fabio79ch/check_netapp/src][E5400/E5500 NetApp]] health check on the top of it. I'm also creating a RAID controller performance check, idea stolen from [[http://www.manageengine.com/products/opstor/monitoring-ibm-fastt.html][this company]].
         * Upgraded [[http://ftp.gnu.org/gnu/freeipmi/freeipmi-1.3.4.tar.gz][free-ipmi]], used by [[http://www.thomas-krenn.com/en/oss/ipmi-plugin/check_ipmi_sensor_v3.1.tar.gz][check_ipmi_sensor_v3.1]]
         * Installed the older [[http://www.thomas-krenn.com/en/oss/ipmi-plugin/check_ipmi_sensor_v1.3.tar.gz][check_ipmi_sensor_v1.3]] to monitor our old 6 * SUN Thumpers X4500 temperatures. [[http://www.thomas-krenn.com/en/oss/ipmi-plugin/check_ipmi_sensor_v3.1.tar.gz][check_ipmi_sensor_v3.1]] didn't work.
      * Evaluating [[http://docs.saltstack.com/][Salt Stack]]; it consists of a fast asymmetric keys protected network of Python agents communicating by the mature [[http://zeromq.org/][zeromq]] library that can: 
         * be used as a [[http://docs.saltstack.com/ref/modules/all/index.html][remote execution tool]] to replace less sophisticated tools like [[http://www.csm.ornl.gov/torc/C3/C3documentation.shtml][cexec]] or [[https://code.google.com/p/pdsh/][pdsh]] 
            * by [[http://docs.saltstack.com/ref/clients/index.html#calling-salt-from-shell-scripts][CLI]].
            * by [[http://docs.saltstack.com/ref/clients/index.html#calling-salt-from-a-python-application][Python programs]]: ouputs are handy Python dictionaries.
         * replace Puppet, by using the [[http://docs.saltstack.com/ref/states/all/index.html][salt states]].
   * *UNIBE* (reports Gianfranco): 
      * Fairly stable operations with day-to-day maintenance, except: 
         * Issues with stale files in ARC sessiondir. These are files left over by failed jobs, which end up clogging the directory. Added a weekly cron to perform a cleanup
         * Issues with large amount of job files in sessiondir. In this case, these are genuine job files not cleaned up by the users. t2k users were not yet experienced and not aware they have to regularly retrieve outputs
         * In both cases, the symptom is the culsters' infosys dropping out of the GIIS, which causes them to be invisible. Cleanup and proper service restore is painstaking, but now all seems to have stabilised.
      * Kernel panic on one lustre OSS (thumper). Disabled it on all nodes and carried on with limited pain
      * Performed crosscheck of accounting data between batch servers and EGI accounting portal from Jan 2013 to date 
         * Some months are low on the EGI portal: Alessandro trying republishing. If numbers don't change, it means that records are missing on the SGAS server, therefore they are lost.
         * Discovered that gridengine does not seem to account properly for multi-core jobs
      * DPM SE configured for xrootd protocol. Setup to join ATLAS DE FAX federation in progress (working with Geneva). CSCS final setup still pending (setup is different for DPM and dCache)
      * Obsolete Glue 2 entries in site-bdii: [[https://xgus.ggus.eu/ngi_ch/?mode=ticket_info&ticket_id=303]]
      * No progress on: 
         * Transition SGAS to Jura
         * GIIS final setup
         * Solid roadtest of VOMS server (first tests are OK)
      * Placed an order for six 36-disk servers for Lustre (intending to phase out thumpers). JBODs with 5x8 port LSI RAID controllers
      * Some new WNs procurement expected in April. Strategy is to let current ailing hardware to die (no effort to rescue, repair) and progressively replace with new nodes
   * *UNIGE* (reports Szymon): =absent=
   * 
      * Xxx
   * *UZH* (reports Sergio): 
      * Xxx
   * *Switch* (reports Alessandro): 
      * Xxx
Other topics
   * *"Swiss" Resources at CSCS* 
      * Maybe we should have a dedicated meeting (Vidyo conference is fine) discussing the plans (time-line, configuration, testing) for these non-pledged resources (especially the computing; the storage is trivial at least from the CMS side) or we can schedule it for the next monthly meeting?
      * From the CMS side we would like to prioritize the jobs of Swiss users so their jobs come first on this Swiss resources 
         * For that we have an additional group in the CMS VOMS server =/cms/chcms= that can be administrated by Christoph and Daniel.
         * For the moment Derek, Fabio, and Daniel are in this group and can provide proxy certificates with this additional role for testing purposes.
      * Are there similar plans from the ATLAS/LHCb side?
   * *Topic2*
Next meeting date:

AOB
   * Next week Pre-GDB and GDB to be held in Bologna. Miguel will attend and give small presentation on the migration to SLURM of CSCS-LCG2.

---++ Attendants
   * CSCS:
   * CMS:
   * ATLAS:
   * LHCb:
   * EGI:

---++ Action items
   * Item1
---++ A user banned in dCache 2.6

We banned a use because of the following hundreds of requests always for the same two files =&gt; Xrootd [[http://en.wikipedia.org/wiki/Denial-of-service_attack][DoS]] on the dCache pools hosting those two files.<br /> <img width="381" alt="xrootd-queued-requests.png" src="%ATTACHURLPATH%/xrootd-queued-requests.png" height="148" /><br /> Relevant dCache banning logs: <br /> 
%TWISTY%
<pre>04 Mar 2014 17:18:00 (gPlazma) [Xrootd-t3se01 Login AUTH voms] Certificate verification: Verifying certificate 'DC=ch,DC=cern,OU=computers,CN=voms.cern.ch'
04 Mar 2014 17:18:00 (gPlazma) [Xrootd-t3se01 Login MAP vorolemap] VOMS authorization successful for user with DN: /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=silveira/CN=705497/CN=Gustavo Gil Da Silveira and FQAN: /cms for user name: cmsuser.
04 Mar 2014 17:18:00 (gPlazma) [Xrootd-t3se01 Login] %RED%Login attempt failed%ENDCOLOR%; detailed explanation follows:
LOGIN FAIL
 |    in: X509 Certificate chain:
 |          |
 |          +--CN=1758990799,CN=1394689704,CN=Gustavo Gil Da Silveira,CN=705497,CN=silveira,OU=Users,OU=Organic Units,DC=cern,DC=ch [1758990799]
 |          |    |
 |          |    +--Issuer: CN=1394689704,CN=Gustavo Gil Da Silveira,CN=705497,CN=silveira,OU=Users,OU=Organic Units,DC=cern,DC=ch
 |          |    +--Validity: OK for 16 hours, 54 minutes and 51.3 seconds
 |          |    +--Algorithm: SHA-1 with RSA
 |          |    +--Public key: RSA 1024 bits
 |          |    +--Key usage: digital signature, key encipherment, data encipherment
 |          |
 |          +--CN=1394689704,CN=Gustavo Gil Da Silveira,CN=705497,CN=silveira,OU=Users,OU=Organic Units,DC=cern,DC=ch [1455016582116403591211701295546859998117066254174]
 |          |    |
 |          |    +--Issuer: CN=Gustavo Gil Da Silveira,CN=705497,CN=silveira,OU=Users,OU=Organic Units,DC=cern,DC=ch
 |          |    +--Validity: OK for 7 days, 16 hours, 22 minutes and 19.2 seconds
 |          |    +--Algorithm: SHA-1 with RSA
 |          |    +--Public key: RSA 1024 bits
 |          |    +--Attribute certificates:
 |          |    |    |
 |          |    |    +--DC=ch,DC=cern,OU=computers,CN=voms.cern.ch
 |          |    |         +--Validity: OK for 7 days, 16 hours, 22 minutes and 19.2 seconds
 |          |    |         +--Algorithm: SHA-1 with RSA
 |          |    |         +--FQANs: /cms, /cms/becms
 |          |    +--Key usage: digital signature, key encipherment, data encipherment
 |          |
 |          +--CN=Gustavo Gil Da Silveira,CN=705497,CN=silveira,OU=Users,OU=Organic Units,DC=cern,DC=ch [315385076395555361510222]
 |               |
 |               +--Issuer: CN=CERN Trusted Certification Authority,DC=cern,DC=ch
 |               +--Validity: OK for 28 days, 22 hours, 41 minutes and 41.2 seconds
 |               +--Algorithm: SHA-1 with RSA
 |               +--Public key: RSA 2048 bits
 |               +--Subject alternative names:
 |               |      otherName: 302a060a2b060104018237140203a01ca01a0c186775737461766f2e73696c7665697261406365726e2e6368
 |               |      email: gustavo.silveira@cern.ch
 |               +--Key usage: digital signature, key encipherment, SSL client, email protection, Microsoft EPS
 |        
 |
 +--AUTH OK
 |   |    added: FQANPrincipal[/cms,primary]
 |   |           FQANPrincipal[/cms/becms]
 |   |           /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=silveira/CN=705497/CN=Gustavo Gil Da Silveira
 |   |
 |   +--x509 OPTIONAL:OK =&gt; OK
 |   |      added: /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=silveira/CN=705497/CN=Gustavo Gil Da Silveira
 |   |
 |   +--voms OPTIONAL:OK =&gt; OK
 |          added: FQANPrincipal[/cms,primary]
 |                 FQANPrincipal[/cms/becms]
 |
 +--MAP OK
 |   |    added: GidPrincipal[500,primary]
 |   |           UidPrincipal[501]
 |   |           UserNamePrincipal[cmsuser]
 |   |           GroupNamePrincipal[cmsuser,primary]
 |   |
 |   +--vorolemap REQUISITE:OK =&gt; OK
 |   |      added: GroupNamePrincipal[cmsuser,primary]
 |   |
 |   +--authzdb REQUISITE:OK =&gt; OK
 |          added: GidPrincipal[500,primary]
 |                 UidPrincipal[501]
 |                 UserNamePrincipal[cmsuser]
 |
 +--ACCOUNT FAIL
 |   |
 |   +--%RED%banfile REQUISITE:FAIL (user banned) =&gt; FAIL (ends the phase)%ENDCOLOR%
 |
 +--(SESSION) skipped
 |
 +--(VALIDATION) skipped
</pre>

%ENDTWISTY%
---++ A not trivial Salt run

Here I retrieve as a Python dict all the top processes running inside my WNs. <br /> 
%TWISTY%
<pre>[root@t3admin01 ~]# %BLUE%ipython%ENDCOLOR% 
Python 2.7.6 |Anaconda 1.9.0 (64-bit)| (default, Jan 17 2014, 10:13:17) 
Type "copyright", "credits" or "license" for more information.

IPython 1.1.0 -- An enhanced Interactive Python.
?         -&gt; Introduction and overview of IPython's features.
%quickref -&gt; Quick reference.
help      -&gt; Python's own help system.
object?   -&gt; Details about 'object', use 'object??' for extra details.

In [1]: %BLUE%import salt.client%ENDCOLOR%

In [2]: %BLUE%saltclient = salt.client.LocalClient()%ENDCOLOR%  %ORANGE%# I connect to the salt master%ENDCOLOR%

In [3]: %BLUE%mydict = saltclient.cmd('nodename:%ENDCOLOR%%GREEN%t3wn*%ENDCOLOR%%BLUE%', '%ENDCOLOR%%GREEN%ps.top%ENDCOLOR%%BLUE%', expr_form='grain' )%ENDCOLOR%  %ORANGE%# I run the Python top program only on the WNs%ENDCOLOR%

%ORANGE%# For each WN, the 5 top processes%ENDCOLOR%
In [4]: %BLUE%print mydict%ENDCOLOR%
{'%GREEN%t3wn34%ENDCOLOR%': [
   {'status': 0, 'mem.vms': 1043345408, 'cmd': ['cmsRun', '/shome/cgalloni/TestSim/CMSSW_5_3_2_patch4/src/SIM/MR_M1000_SIM_cfi.py', 'maxEvents=500', 'skipEvents=228000', 'seed=3936709'], 'pid': 9985, 'cpu.user': 10463.36, 'cpu.system': 3.38, 'create_time': 1394091559.43, 'user': 'cgalloni', 'mem.rss': 725270528}, 
   {'status': 0, 'mem.vms': 1024819200, 'cmd': ['cmsRun', '/shome/cgalloni/TestSim/CMSSW_5_3_2_patch4/src/SIM/MR_M1000_SIM_cfi.py', 'maxEvents=500', 'skipEvents=228500', 'seed=7239021'], 'pid': 10243, 'cpu.user': 10431.48, 'cpu.system': 3.3, 'create_time': 1394091591.95, 'user': 'cgalloni', 'mem.rss': 703696896}, 
   {'status': 0, 'mem.vms': 1021722624, 'cmd': ['cmsRun', '/shome/cgalloni/TestSim/CMSSW_5_3_2_patch4/src/SIM/MR_M1000_SIM_cfi.py', 'maxEvents=500', 'skipEvents=224000', 'seed=2991033'], 'pid': 8030, 'cpu.user': 10838.1, 'cpu.system': 2.81, 'create_time': 1394091186.8, 'user': 'cgalloni', 'mem.rss': 702164992}, 
   {'status': 0, 'mem.vms': 1019928576, 'cmd': ['cmsRun', '/shome/cgalloni/TestSim/CMSSW_5_3_2_patch4/src/SIM/MR_M1000_SIM_cfi.py', 'maxEvents=500', 'skipEvents=234000', 'seed=9386177'], 'pid': 11313, 'cpu.user': 10170.57, 'cpu.system': 2.74, 'create_time': 1394091852.93, 'user': 'cgalloni', 'mem.rss': 703258624}, 
   {'status': 1, 'mem.vms': 4165632, 'cmd': ['mdadm', '--monitor', '--scan', '-f', '--pid-file=/var/run/mdadm/mdadm.pid'], 'pid': 20269, 'cpu.user': 0.0, 'cpu.system': 0.01, 'create_time': 1394036233.94, 'user': 'root', 'mem.rss': 401408}
                            ], 
'%GREEN%t3wn33%ENDCOLOR%':   .........
</pre>

%ENDTWISTY%