T3 Downtime due to PSI yearly Compute Center Maintenance -- 05. 01. 2022 DerekFeichtinger
Downtime will last from 16:00h Fri, 7. Jan until 10:00h on Mon, 10. Jan

Monitoring

Batch jobs (queuing system)

Current queue / accounting

Number of running and queued jobs:


Ganglia WN page

Storage

/pnfs dir

Show space graphs for

Links:

/pnfs dir I/O queues

  • regular I/O queue movers = dcap/gsidcap/LAN xrootd movers (heavy random IO for internal analysis) ; MAX 100 ACTIVE movers per file server, others will get QUEUED
  • wan I/O queue movers = SRM/gridftp movers (transfers of whole files also from outside) ; MAX 2 ACTIVE movers per file server, others will get QUEUED
  • xrootd I/O queue movers = WAN xrootd movers ; MAX 2 ACTIVE movers per file server, others will get QUEUED
  • To check by CLI the I/O queues run from a UI watch -n 1 -d  lynx --dump --width=200 'http://t3dcachedb:2288/queueInfo' e.g. if your jobs are not progressing it might be due to a file server with too many queued movers ; in this case you can inform by email the T3 users ( the T3 admins will get it too )

ACTIVE movers:

QUEUED movers ( the associated I/O queue is exceeding the max amount of allowed ACTIVE movers ) :

PENDING requests (these are hanging file transfers, almost always an error state if they persist):

/mnt/t3nfs01/data01/{shome,swshare} dirs NEW

User Space Report

Networking and File Transfers (+ PhEDEx)

Links:

Plotting interval:


Availability reports

These tests are run by the centralized Grid monitoring services and they determine whether the T3 or the T2 are considered to be working correctly:
  • CMS Nagios : T3
  • German Nagios : T3
  • Gstat : T3

Computer Room Temps

private link
Edit | Attach | Watch | Print version | History: r121 | r93 < r92 < r91 < r90 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r91 - 2016-06-13 - FabioMartinelli
 
  • Edit
  • Attach
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback