PSI Tier-3 Monitoring

Availability tests

These tests are run by the centralized Grid monitoring services and they determine whether our site is considered to be working correctly for the VOs.

• EGI Operations Dashboard
• central Nagios / Nagios history
• Swiss Nagios / Swiss Nagios history
• SAM: BDII, SE, Gstat, GermanySwitzerland overview / ---++ Batch jobs (queuing system) [[http://t3mon.psi.ch/ganglia/PSIT3-custom/qstat.txt][Current queue]] */* [[http://t3mon.psi.ch/ganglia/PSIT3-custom/accounting.txt][accounting]] Number of running and queued jobs:
  hour day week month year
  
  [[http://dashb-cms-job.cern.ch/dashboard/request.py/jobsummary#site=T3_CH_PSI&sortby=user][CMS Dashboard]] ---+++ Worker nodes load monitoring [[http://t3mon.psi.ch//?c=PSI%20Tier3%20workers&m=&r=day&s=descending&hc=4][Ganglia WN page]] ---++ Storage Show space graphs for
  hour day week month year
  ---+++ Storage Element Links: * [[http://t3dcachedb.psi.ch:2288/][dCache GUI]] * List all [[https://cmsweb.cern.ch/phedex/prod/Data::Replicas?rcolumn=Name&dbs=1&node=27&node=821&nvalue=Node+bytes&rows=interesting&view=global&filter=.*][hosted datasets]] / [[https://cmsweb.cern.ch/phedex/prod/Request::View?type=xfer&nodes=T3_CH_PSI&state=pend&.submit=Submit][requests]] */* [[https://cmsweb.cern.ch/phedex/prod/Reports::SiteUsage?node=T3_CH_PSI][accounting per phys. group]] * show user/dataset [[http://t3mon.psi.ch/ganglia/PSIT3-custom/cms_space.txt][space usage]] ---+++ Home and Software areas * show detailed home area [[http://t3mon.psi.ch/ganglia/PSIT3-custom/shome-du.txt][space usage]] ---++ Networking and File Transfers (+ PhEDEx) Links: * [[http://t3mon.psi.ch//?m=network_report&r=day&s=descending&c=PSI+Tier3+fileservers&h=&sh=1&hc=4][Ganglia fileserver page]] * show [[http://t3mon.psi.ch/ganglia/PSIT3-custom/phedex-statistics.txt][phedex summary statistics]] */* [[http://cmsweb.cern.ch/phedex/prod/Components::Links?from_filter=T3_CH_PSI&andor=or&to_filter=T3_CH_PSI&Update=Update#][PhEDEx links between PSI and other centers]] */* [[http://cmsweb.cern.ch/phedex/prod/Activity::Deletions?reqfilter=*&node=T3_CH_PSI&state=any&blockfilter=.*&Update=Update#][deletion request state]] * [[http://gridse.sns.it/phedex/][Phedex summary view by S. Sarkar]] Plotting interval:
  hour day week month year
  
  dCache active movers: *regular* movers = *dcap/gsidcap* movers (heavy random IO for internal analysis), *WAN* movers = *SRM/gridftp* movers (transfers of whole files also from outside): *xrootd* movers = LAN/WAN transfers of files by [[https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookXrootdService][xrootd]] dCache queued movers: *regular* movers = *dcap/gsidcap* movers (heavy random IO for internal analysis), *WAN* movers = *SRM/gridftp* movers (transfers of whole files also from outside): *xrootd* movers = LAN/WAN transfers of files by [[https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookXrootdService][xrootd]] dCache pending requests (these are hanging transfers, almost always an error state if they persist):