Service Card for Nordugrid Arc CE

It's the entry point for Nordugrid jobs, very common within Switzerland

Definition

The service consists of three different services:

• grid-manager takes care of receiving the job submission requests. The service that actually takes care of this is called A-REX
• gridftpd manages the file stagein, stageout, and also takes part on the mapping of users to local accounts.
• grid-infosys that publishes the information onto the nordugrid infrastructure

Some other useful information:

• You interact with the service from a UI with the nordugrid-client installed yum groupinstall "ARC Client"
• It runs on virtual machines, and needs to have a shared filesystem mounted at least for:
  • /home/nordugrid-atlas/sessiondir where the sessiondir is stored. It's where the temporary files between CE and WN are stored.
  • Store the ARC cache, currently in /home/nordugrid-atlas/cache. It contains data and hardlinks for the jobs that use those files.
  • Runtimedir is like the software-tags for LCG. It is currently on NFS: /experiment_software/atlas/nordugrid/runtime

Operations

Interesting information like how to deal with the service.

Client tools

• ngsub to submit a task
• ngstat to obtain the status of jobs and clusters
• ngcat to display the stdout or stderr of a running job
• ngget to retrieve the result from a finished job
• ngkill to cancel a job request
• ngclean to delete a job from a remote cluster
• ngrenew to renew user's proxy
• ngsync to synchronize the local job info with the MDS
• ngcopy to transfer files to, from and between clusters
• ngremove to remove file

Installation/Upgrade

Installation of the EMI-3 release of ARC CE is done by CFEngine using the category ARC_CE3. Please check that emi-release package is installed; after running cfagent make sure that the directory /var/run/bdii exists and belongs to user ldap.

Then there are some changes to the arc.conf file:

• changing all NULL values to *, in voms mapping.
• commenting out all log files and debug levels, save for joblog, that needs to be there.
• controldir variable needs to be in [grid-manager], not [common]. Otherwise some piece of Infosys doesn't work well.

Config

If you see something in the config file that is not listed here please add it.

Register to the Swiss level GIIS.

   [infosys/cluster/registration/ClusterToSwitzerland]
   targethostname="giis.lhep.unibe.ch"
   targetport="2135"
   targetsuffix="mds-vo-name=Switzerland,o=grid"
   regperiod="40"

Testing

From ui.lcg.cscs.ch, with a valid grid proxy (atlas, dech or dteam), the most simple yet complete test can be this:

ngsub -c arc01.lcg.cscs.ch -e '&("executable" = "env2" )("stdout" = "stdout" )("queue" = "cscs")(inputfiles=("env2" "file:///bin/env"))'
ngget gsiftp://arc01.lcg.cscs.ch:2811/jobs/6246129786993110697454

Also, you can retrieve some information about other arc sites with =ngtest -R=311069745492

When submitting a job using a dteam certificate the lcgadmin queue can be used.

Another way to test an ARC CE is to run from a UI:

$ arctest -c arc01.lcg.cscs.ch -J 1
Test submitted with jobid: gsiftp://arc01.lcg.cscs.ch:2811/jobs/gHHODmRf7zinZOuIepQ9oyOmABFKDmABFKDmRWFKDmABFKDmqRqWsn

$ arcstat gsiftp://arc01.lcg.cscs.ch:2811/jobs/gHHODmRf7zinZOuIepQ9oyOmABFKDmABFKDmRWFKDmABFKDmqRqWsn

Start/stop procedures

All three services can be individually restarted, but if you want to do it at once you can use grid-service restart.

Note: In the EMI 3 release grid-infosys no longer exists and the functionality is split between service nordugrid-arc-slapd and service nordugrid-arc-bdii. These two need to be started in this order (after a-rex has started), and stopped in reverse. There is also a new servicd nordugrid-arc-inforeg which needs to be started last. The grid-service2 script has been modified in PPARC_CE/files/opt/cscs/... as we still have older arc machines in production that use grid-infosys.

Checking logs

• ARC logs can be complex to analyze. We have developed a tool, ngtracejob, in the path, that can do it for you. Just give an arc jobid as an argument.

Manuals

• Grid Manager and Gridftpd services: http://www.nordugrid.org/documents/GM.pdf
• http://www.nordugrid.org/documents/ng-server-install.html
• http://www.nordugrid.org/documents/ng-client-install.html
• Client tools: http://wiki.nordugrid.org/index.php/NOX/Client_tutorial
• http://www.nordugrid.org/NorduGridVO/
• YUM configuration: http://download.nordugrid.org/repos.html
• arc.conf configuration template: http://svn.nordugrid.org/trac/nordugrid/browser/arc0/trunk/doc/arc.conf.template
• Notes on OPS, APEL, and how to integrate with EGI stuff: http://wiki.nordugrid.org/index.php/How_to_plug_an_ARC_site_into_EGI

Issues

Information about issues found with this service, and how to deal with them.

Max memory increased by a factor of 1.5

Note: This is no longer an issue in the current Slurm setup running in production. The below is kept for historical purposes.

ATLAS needs their hard limits to be on 3 GB of memory instead of 2. For that a hack needs to be in place for /usr/share/arc/submit-pbs-job after upgrading the software

if [ ! -z "$joboption_memory" ] ; then
#  echo "#PBS -l pvmem=${joboption_memory}mb" >> $LRMS_JOB_SCRIPT
  joboption_memory_hard=`echo  $joboption_memory \* 1.5  | bc | awk -F . '{print $1}'`
  echo "#PBS -l pvmem=${joboption_memory_hard}mb" >> $LRMS_JOB_SCRIPT
  echo "#PBS -l pmem=${joboption_memory}mb" >> $LRMS_JOB_SCRIPT
fi

SubCluster publishing PhysicalCPUs and LogicalCPUs

grid-infosys was publishing the GlueSubCluster information, but our CreamCEs are the ones supposed to publish the amount of physical and logical CPUs for GSTAT.

For that, you can edit /usr/share/arc/glue-generator.pl and change the values by hand:

GlueSubClusterPhysicalCPUs: 0
GlueSubClusterLogicalCPUs: 0 

OPS jobs getting into the cscs internal queue

ARC clients relay only on the information system (not glue, but the nordugrid part) to choose the queue where to send, and the CSCS queue was publishing all users in grid-mapfile as authorized. There are a few tricks to prevent this (because arc.conf is not enough) and one of them is this:

Oct 15 14:03 [root@arc02:arc]# pwd
/usr/share/arc
Oct 15 14:03 [root@arc02:arc]# diff -C3 ARC0ClusterInfo.pm.backup ARC0ClusterInfo.pm
*** ARC0ClusterInfo.pm.backup   2012-10-15 13:15:44.000000000 +0200
--- ARC0ClusterInfo.pm  2012-10-15 14:03:43.000000000 +0200
***************
*** 441,446 ****
--- 441,447 ----
                  while (1) {
                      return undef unless ($sn, $localid) = each %$usermap;
                      $lrms_user = $qinfo->{users}{$localid};
+                   if ($q->{'name'} eq "cscs" and $sn !~ m/Pablo Fern/) { next; }
                      last if not exists $qinfo->{acl_users};
                      last if grep { $_ eq $localid } @{$qinfo->{acl_users}};
                  } 

Publish accounting records using APEL (with a custom blah parser)

[To be updated] Publishing accounting information with APEL has never worked within Nordugrid software (they use a different acounting method). We have developed, though, a script that converts the nordugrid usage records into blahp format. This is how the script works:

• It looks into /var/spool/nordugrid/usagerecords/archive for last month's job records (one xml file per job)
• Selects only those that have a PBS entry (correctly submitted) and generates a blahp record for each job (one line per job, on a file per full day)
• At the end, you have a directory /var/spool/nordugrid/usagerecords/blahp with the right format (as CreamCE /var/log/cream/accounting)

Then, if you configure a blah parser that check that directory, and run it every day (like any other CreamCE) it will populate the APEL database correctly. The script does not have special requirements, but it needs to be modified to match the site parameters (it does some assumptions, but there may be no changes at all)

This is what is needed for the whole accounting to work:

• Get the apel_parser_fake.py script (attached to this document) and create a CRON job to run it every day.
• Install the APEL parser: get the UMD1 base/updates repository, and yum install glite-apel-pbs. You may need to work on the dependencies a bit.
  • To fix the geronimo dependency, you need to downgrade log4j: rpm -U ftp://mirror.switch.ch/pool/1/mirror/scientificlinux/5rolling/x86_64/SL/log4j-1.2.13-3jpp.2.x86_64.rpm --oldpackage; rpm -e geronimo-jms-1.1-api-1.2-13.jpp5.noarch
  • Copy the /etc/glite-apel-pbs/parser-config-yaim.xml file from a CreamCE (generated by yaim) and modify it:
  • Modify the parser-config-yaim.xml parameters to match the specific server you are configuring. Specially /var/spool/nordugrid/usagerecords/blahp should be in the BlahdLogProcessor directory section. Also, BDII port could be 2135 instead of 2170 (in the GIIS section)
  • Remove the EventsRecord section if the PBS entries are created already by another CE (to avoid duplication, but it does not hurt having it)
  • Create a cron job that runs it every day (after the apel_parser_fake.py). You could probably take it from a CreamCE too.
• Create a mysql username in the APEL server for this purpose (find instructions in the ApelServiceCard) that matches what you specified in the parser-config-yaim.xml file.
• Have ARC publishing GLUE information with a BDII, with right CPU performance information (should have the GlueCEUniqueID entry). It should be visible by APEL publisher too.
• Add your ARC server to GOCDB as an APEL entry.

• apel_parser_fake.py.txt: apel_parser_fake.py.txt

How to disable job submissions on an ARC CE

To disable the job submission it is enough to modify the arc.conf file through CFEngine:

/etc/arc.conf

[...]
[gridftpd/jobs]
path="/jobs"
plugin="jobplugin.so"
allownew="yes"                      # set this to "no"
[...]

make the CFEngine agent run and then restart the grid services:

# grid-service2 restart