T3 Downtime due to PSI yearly Compute Center Maintenance -- 05. 01. 2022 DerekFeichtinger Downtime will last from 16:00h Fri, 7. Jan until 10:00h on Mon, 10. Jan

Monitoring

*Currently monitoring page doesn't show real plots due to batch system migration and problems with monitoring node installation. Sorry for inconvenience. NL.*

Overview of PSI Tier3 fileservers Overview of PSI Tier3 services Overview of PSI Tier3 workers

Batch jobs (queuing system)

Current queue / accounting

Number of running and queued jobs:

hour day week month year

Ganglia WN page

Storage

/pnfs dir

Links:

• hosted datasets plaintext
• requests
• accounting per phys. group
• /pnfs dirs ordered by size - For a moment this monitoring is planned to be reconstructed. Sorry for inconvenience. NL.

• /pnfs current files transfers
• /pnfs searching by the dc_find tool

Show storage space graphs for

hour day week month year

/pnfs dir I/O queues

• regular I/O queue movers = dcap/gsidcap/LAN xrootd movers (heavy random IO for internal analysis) ;
• wan I/O queue movers = SRM/gridftp movers (transfers of whole files also from outside) ;
• xrootd I/O queue movers = WAN xrootd movers ;
• To check by CLI the I/O queues run from a UI watch -n 1 -d  lynx --dump --width=200 'http://t3dcachedb:2288/queueInfo' e.g. if your jobs are not progressing it might be due to a file server with too many queued movers ; in this case you can inform by email T3 admins.

/mnt/t3nfs01/data01/{shome,swshare} dirs

User Space Report

hour day week month year

Networking and File Transfers (+ PhEDEx)

Links:

• Ganglia fileserver page
• PhEDEx usage at T3 | PhEDEx usage at T2
• Last day of FTS3 jobs induced by CRAB3, transfers details | Last day of FTS3 jobs induced by CRAB3, Tot GBs
• 7d avg MBs | 24h avg MBs
• 7d quality transfer | 24h quality transfer
• PhEDEx errors
• PhEDEx links between T3_CH_PSI and other centers
• PhEDEx Doc
• PhEDEx local stats
• PhEDEx deletion request state
• PhEDEx transfer requests as XML

• $ curl  --capath /etc/grid-security/certificates -E $X509_USER_PROXY --cacert $X509_USER_PROXY https://cmsweb.cern.ch/phedex/datasvc/xml/prod/transferrequests?node=T3_CH_PSI 2>/dev/null |  xmllint --format -

• PhEDEx datasets as XML

• $ curl  --capath /etc/grid-security/certificates -E $X509_USER_PROXY --cacert $X509_USER_PROXY https://cmsweb.cern.ch/phedex/datasvc/xml/prod/blockreplicasummary?node=T3_CH_PSI 2>/dev/null |  xmllint --format - 

• PhEDEx datasets as JSON

Plotting interval:

hour day week month year

Availability reports

These tests are run by the centralized Grid monitoring services and they determine whether the T3 or the T2 are considered to be working correctly:

• Nagios : CMS | EGI1 | EG2 | PSI