Obtaining host certificates for Grid servers

Our host certificates are issued by the SWITCH CA (using the QuoVadis service). Go here for the web interface for the ordering via certificate request files.

Email request should be sent from CERN account. The following list of CERN Authorities should be recorded to enable digital signing with user certificate:

• CERN Grid Certification Authority;
• CERN Certification Authority;
• CERN Root Certification Authority 2;

The keys and certificates are stored in a directory structure on the admin node

• /root/clusteradmin/etc/hostkeys: contains helper scripts
• /root/clusteradmin/etc/hostkeys/egieu/: contains key files, csr requests for the present EGi EU CA that is valid since 2016
• /root/clusteradmin/etc/hostkeys/switch-QuoVadis/certs-2011/: OBSOLETE. contains certificates for the years when the Swiss CA resided at QuoVadis. Do not use any more

In /root/clusteradmin/etc/hostkeys on t3admin01 there is a helper script create_keys.sh and an openssl config file openssl.cnf which will help you to get rid of most of the typing for producing the required certificate request files for new machines.

Renewing certificates

Currently renewing a certificate involves again filling out a complete registration request. I reuse the old server keys and just copy again the old CSR files into the forms. If you want to see the content of a CSR file you can run the command:

openssl req -in ./t3se02.psi.ch-csr.pem  -text

Information in relation to earlier problems with SWITCH certificates

Problem 2: Some services (e.g. myproxy) have/had problems with the certificates for PSI, since the certificates for PSI contain parentheses in the DN ( O=Paul-Scherrer-Institut (PSI)), and the services had errors in the routines that did DN string comparisons. Probably the programmers had not taken into account that parentheses are valid characters in a DN, and failed to treat them correctly in the regexp comparisons.

Problem 1: The new switch certificates that are issued by QuoVadis, no longer have the email in the dn, so we should no longer see compatibility problems with some Grid services.
The DN of the PSI certificates contains parentheses for the "(PSI)" part. This is not correctly parsed in some text matching functions (e.g. myproxy server) causing authentication failures (this is the result of a tough debugging marathon with Maarten Lithmaat in Dec of 2009)

Note: Due to a complex signing-policy configuration file, the SWITCH certificates showed problems in the past with certain non standards compliant services (quite a few). We need to test for dcache client compatibility and else get certificates from the LCG catch all CA at DOEGrids. However this entails defining a Registration Agent for PSI (see here).

First tests with dcache clients indicated that the SWITCH host certificates are ok for gridFTP and SRM protocols. gsidcap write fails due to space management problems not related to this, it seems.

Certificates for Phedex (t3cmsvobox01)

https://wiki.chipp.ch/twiki/bin/view/CmsTier3/CmsVoBox#The_host_x509_is_needed_to_regul

Certificates for LDAP service and LDAP clients

The ldap service and client certificates are special in the sense that they are based upon our own Tier-3 CA (this is a historic remnant.)

The machinery for using that CA is located on the t3admin01 node in /root/clusteradmin/etc/cluster-ca

You will find a readme there containing

Use the cluster-ca.sh script to produce the certificate.

The config files are already prepared. Just use something like
    ./cluster-ca.sh -r ./pki -s t3admin01.psi.ch

The certificates are then found under the ./pki/certs directory. You may have
to first remove any old cert and csr files for that node (e.g. if you want to
renew). The script will tell you.

                                          Derek


cluster-ca.sh help text:
-------------------------------
Synopsis:
         cluster-ca.sh [options] --rootca
         cluster-ca.sh [options] -s server.domain.org


Options:
         -r path      : Path to repository directory (/root/clusteradmin/etc/cluster-ca/test-CA)

         -s hostname  : server hostname for which to produce a cert
         -t template  : template SSL config name to use for producing server
                        request and cert files (default.cnf)
         -d           : debug mode



E.g.

For producing a Certificate for our LDAP master server:
First copy away the old cert /root/clusteradmin/etc/cluster-ca/pki/certs/t3admin01.psi.ch-cert.pem
(leave the key in place!!!), and then run
./cluster-ca.sh -r ./pki -s t3admin01.psi.ch

Producing a certificate for the LDAP service

On t3admin01 go to the following directory

cd /root/clusteradmin/etc/cluster-ca 

Ensure that the CA root certificate is still OK:

openssl x509 -in /root/clusteradmin/etc/cluster-ca/pki/rootca/rootca-cert.pem -dates -subject -noout

notBefore=Jul 28 10:25:00 2014 GMT
notAfter=Jul 27 10:25:00 2019 GMT
subject= /C=CH/ST=Aargau/L=Villigen/O=PSI/CN=PSI CMS Tier-3 Cluster Root CA

The CA root certificate is distributed to all client nodes (basically all cluster nodes) in /etc/openldap/certs/08a2f47c.0 (The filename is a hash that is obtained by using openssl x509 -noout -hash -in rootca-cert.pem)

If the CA root certificate needs to be regenerated, you must copy it to all the client nodes (and also to the LDAP server).

Now we can generate the new service certificate for LDAP.

Maybe save the old certificate first to another location

A new certificate for a server can be generated like this

./cluster-ca.sh -r ./pki -s t3ldap01.psi.ch 

This will use the CA structure under the pki subdirectory with the root CA certificate in pki/rootca to generate a new certificate for that host.

Then copy the certificate (and potentially also the key, if you changed it) to the LDAP service:

scp  /root/clusteradmin/etc/cluster-ca/pki/certs/t3ldap01.psi.ch-cert.pem t3ldap01:/etc/pki/tls/certs/slapd-cert.pem

The location of the LDAP service certificate is defined in the slapd configuration file /etc/openldap/slapd.conf

Confirm that the dates look right:

openssl x509 -in /etc/pki/tls/certs/slapd-cert.pem  -subject -dates -noout 

Restart ldap to make it reread its certificate and key:

/etc/init.d/ldap restart

from any client node (e.g. UI) test whether the certificate is active in the service by using openssl to retrieve and interpret the certificate directly from the service!

echo | openssl  s_client  -connect  t3ldap01.psi.ch:636 2 > /dev/null | openssl x509 -subject -dates -noout

Initial version: -- DerekFeichtinger - 13 Aug 2008