Tags:
create new tag
view all tags

Tier-3 Security Update Downtime On 25.07 At 10:00 -- 13. 07. 2017 NinaLoktionova
Dear all, we expect about 2 hours interruption of services.

Monitoring

Batch jobs (queuing system)

Current queue / accounting

Number of running and queued jobs:


Ganglia WN page

Storage

Show space graphs for

/pnfs dir

Links:

/pnfs dir I/O queues

  • regular I/O queue movers = dcap/gsidcap/LAN xrootd movers (heavy random IO for internal analysis) ; MAX 100 ACTIVE movers per file server, others will get QUEUED
  • wan I/O queue movers = SRM/gridftp movers (transfers of whole files also from outside) ; MAX 2 ACTIVE movers per file server, others will get QUEUED
  • xrootd I/O queue movers = WAN xrootd movers ; MAX 2 ACTIVE movers per file server, others will get QUEUED
  • To check by CLI the I/O queues run from a UI watch -n 1 -d  lynx --dump --width=200 'http://t3dcachedb:2288/queueInfo' e.g. if your jobs are not progressing it might be due to a file server with too many queued movers ; in this case you can inform by email the T3 users ( the T3 admins will get it too )

ACTIVE movers:

QUEUED movers ( the associated I/O queue is exceeding the max amount of allowed ACTIVE movers ) :

PENDING requests (these are hanging file transfers, almost always an error state if they persist):

/mnt/t3nfs01/data01/{shome,swshare} dirs NEW

User Space Report

Networking and File Transfers (+ PhEDEx)

Links:

Plotting interval:


Availability reports

These tests are run by the centralized Grid monitoring services and they determine whether the T3 or the T2 are considered to be working correctly:

Computer Room Temps

private link
Edit | Attach | Watch | Print version | History: r109 < r108 < r107 < r106 < r105 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r109 - 2016-11-25 - FabioMartinelli
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback