PhEDEx

Configuration

The PhEDEx configuration can be found in ~phedex/config:

DBParam.PSI: Passwords needed for accessing the central data base. These we receive encrypted from cms-phedex-admins@cern.ch. The file contains one section for every PhEDEx instance (Prod, Dev, ...)
SITECONF/T3_CH_PSI/PhEDEx/Config*: Configuration definitions for the PhEDEx instances (including load tests)
SITECONF/T3_CH_PSI/PhEDEx/storage.xml: defines the trivial file catalog mappings
SITECONF/T3_CH_PSI/PhEDEx/FileDownload*: site specific scripts called by the download agent
SITECONF/T3_CH_PSI/PhEDEx/fts.map: mapping of SRM endpoints to FTS servers (q.v. CERN Twiki)

The SITECONF area is checked in to the central CMS Gitlab repository (https://gitlab.cern.ch/SITECONF/T3_CH_PSI/).

There is a symbolic link /home/phedex/PHEDEX which points to the active PhEDEx distribution, so that the configuration files need not be changed with every update (though the link needs to be reset). Nowadays the PhEDEx SW is distributed via CVMFS, so there is no longer the need for local installations.

DB access

PhEDEx relies on a central Oracle data base at CERN. The passwords for accessing it are stored in ~/config/DBParam.PSI (q.v. configuration section above).

Agent Auth/Authz by grid certificates and how to renew them

The PhEDEx daemons require a valid grid proxy in /home/phedex/gridcert/proxy.cert to transfer files. This short-lived proxy certificate is renewed every few hours through a cron job running the myproxy-logon command (/etc/cron.d/cron_proxy). The renewal is based on a long-living proxy certificate of the phedex administrator that is stored on a myproxy server.

So, in order to have this work, the local phedex administrator needs to deposit an own long lived proxy on the myproxy server, which is used to continually renew the local certificate. The phedex user (cron job) renews the local certificate by - presenting the old but still valid proxy certificate /home/phedex/gridcert/proxy.cert of the phedex admin that is about to get renewed - presenting its own proxy certificate that is produced like a normal user certificate from the user certificate in ~/.globus/usercert.pem and ~/.globus/userkey.pem. This certificate is actually just a copy of the hostcert.pem and hostkey.pem belonging to the phedex host.

The myproxy server only allows registered entities to renew a another user's proxy certificate in that way (the host cert being allowed to renew a user proxy, as we are doing here). You need to have the DN of the host cert entered into the myproxy server's configuration (so our vobox host DN subject has to be registered with the myproxy service). I. e. you need to contact the responsible admins for the myproxy.cern.ch server if the hostname of the cmsvobox changes! Write a mail to Helpdesk@cern.ch

Checking service grid-proxy lifetime

[phedex@cms02 gridcert]$ openssl x509 -subject -dates -noout -in /home/phedex/gridcert/proxy.cert 
subject= /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=jpata/CN=727914/CN=Joosep Pata/CN=750275847/CN=1405542638/CN=1617046039/CN=1231007372
notBefore=Aug 16 12:00:05 2017 GMT
notAfter=Aug 16 12:47:05 2017 GMT

Place the long time certificate on the myproxy server

[feichtinger@t3ui01 ~]$ myproxy-init -t 168 -R 'cms02.lcg.cscs.ch' -l cscs_phedex -x -k renewable -s myproxy.cern.ch -c 1500
Your identity: /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=dfeich/CN=613756/CN=Derek Feichtinger
Enter GRID pass phrase for this identity:
Creating proxy .................................................... Done
Proxy Verify OK
Your proxy is valid until: Tue Oct 24 22:35:03 2017
A proxy valid for 1500 hours (62.5 days) for user cscs_phedex now exists on myproxy.cern.ch.

Testing whether delegation works on the vobox (for that also the local short-lived proxy.cert that you present to the server must be still valid):

[phedex@cms02 gridcert]$ myproxy-logon -s myproxy.cern.ch -v -m cms -l cscs_phedex -a /home/phedex/gridcert/proxy.cert -o /tmp/testproxy -k renewable

MyProxy v6.1 Jan 2017 PAM SASL KRB5 LDAP VOMS OCSP
Attempting to connect to 188.184.64.90:7512 
Successfully connected to myproxy.cern.ch:7512 
using trusted certificates directory /etc/grid-security/certificates
Using Proxy file (/tmp/x509up_u24024)
server name: /DC=ch/DC=cern/OU=computers/CN=px502.cern.ch
checking that server name is acceptable...
server name matches "myproxy.cern.ch"
authenticated server name is acceptable
running: voms-proxy-init -valid 11:59 -vomslife 11:59 -voms cms -cert /tmp/testproxy -key /tmp/testproxy -out /tmp/testproxy -bits 2048 -noregen -proxyver=4
Contacting voms2.cern.ch:15002 [/DC=ch/DC=cern/OU=computers/CN=voms2.cern.ch] "cms"...
Remote VOMS server contacted succesfully.


Created proxy in /tmp/testproxy.

Your proxy is valid until Wed Aug 23 22:43:17 CEST 2017
A credential has been received for user cscs_phedex in /tmp/testproxy.

--+ Setup --+ 3rd party tasks

Get, or ask to PSI to get, the X509 certificate for cms02.lcg.cscs.ch
ask px.support@cern.ch to register the host cms02.lcg.cscs.ch X509 DN for the myproxy.cern.ch service otherwise the scripts/cron_proxy.sh logic won't work
ask the CMS frontier squad to set up SNMP monitoring via GGUS for cms02.lcg.cscs.ch
ask for a GitLab SSH deploy key in order to allow Puppet to clone/pull RO https://gitlab.cern.ch/SITECONF/T3_CH_PSI/ ; see Fabio's CMS HN Thread about the GitLab SSH deploy key mechanism.