Phoenix Monitoring Overview

Status plots from 2024-04-19 - 11:08 (page is refreshed every 5 minutes). Historical view here.

Note: The now values printed at the bottom of some graphs are not correct.

Phoenix is a cluster comprised of Phoenix nodes and Phoenix4 nodes. Phoenix4 is simply a new way of calling the cluster, with a newer, dedicated Slurm configuration. Over time, all nodes of Phoenix will be moved to Phoenix4 Slurm and charts.

Site Tests: Region DE (Nagios, history) / MIDMON Nagios APEL status / Site-View / MyWLCG site availability / WLCG Transfers dashboard

• Site mon:
  • SLURM top: SLTOP / SLTOP Phoenix4
  • SLURM errors: slurmerrors / slurmerrors Phoenix4
  • Reports: SLURM CPU reports / Detailed SLURM CPU reports /
• CMS: ETF / Overview / Tiny Panel / CMS hammercloud / Job status / central CMS-Frontier status / PhEDEx transfers WORLD -> CSCS / CMS Nagios for CSCS / CMS Site Readiness reports
• ATLAS: Site-view / Overview / Tiny Panel / Cloud Monitor / Panglia (Analysis , Production) / Dashboard / (Hamercloud) / Panda (Analysis , Production) / ARC (Cloud monitor, Panda) / ATLAS Nagios for CSCS
• LHCb: Overview / Tiny Panel / Dirac Site status / Dashboard / Queues length

Batch jobs

Usage by VO (Phoenix & Phoenix4)

Usage by VO (Phoenix)

Usage by CE (Phoenix)

Older charts

Usage by VO (static charts)

Usage by CE

Worker nodes Phoenix + Phoenix4

Storage Element

• dCache GUI / Ganglia dCache Headnode/dCache pools / FAX endpoint status
• ATLAS: DQ2 accounting on srm tokens / ATLAS directories at CSCS-LCG2 to download it use a tool like wget
• CMS: used space (local monitoring) / dashboard summary view / List all hosted datasets/requests / subscriptions / accounting per phys. group (also see victor tool) / CMS directories at CSCS-LCG2 to download it use a tool like wget
• LHCb: LHCb directories at CSCS-LCG2 to download it use a tool like wget

Free storage space:

The below plot is from the dcache web ui representing active movers.

Networking and File Transfers

• CMS Local PhEDEx stats / Status Prod / Debug / PhEDEx enabled data transfer links at CSCS
• ATLAS ATLAS FZK Cloud Data activity last 7 days
• FTS Channels all-CSCS / CSCS-FZK

Plotting interval:

Number of active dCache movers (transfers) on the WAN (gridftp, srm) / LAN ("regular",dcap):

Number of queued dCache movers (waiting transfers) on the WAN (gridftp, srm) / LAN ("regular",dcap):

Number of pending dCache requests:

External monitoring

• CSCS external network (shared with other projects, only visible from SWITCH network):

• EGI Nagios from FZK, last 5 days (hover mouse to see hostname)

• Power consumption (Island 1)

• Gridview current status graphs

Other monitoring websites

• GSTAT
• CSCS Internal VLAN64
• EGI Accounting
• Operations Portal - CSCS Page
• SWITCH weather map
• IRC: irc:gridchat.cscs.ch:994#lcg

CMS Monitoring Page