Phoenix Monitoring Overview

Status plots from 2024-05-18 - 07:56 (page is refreshed every 5 minutes). Historical view here.

Note: The now values printed at the bottom of some graphs are not correct.

<img src="http://ganglia.lcg.cscs.ch/ganglia//CSCS-custom/PHOENIX-workers-pie.png" width="189" height="90"/> <img src="http://ganglia.lcg.cscs.ch/ganglia//CSCS-custom/PHOENIX-fileservers-pie.png" width="189" height="90"/> <img src="http://ganglia.lcg.cscs.ch/ganglia//CSCS-custom/PHOENIX-services-pie.png" width="189" height="90"/> <img src="http://ganglia.lcg.cscs.ch/ganglia//CSCS-custom/PHOENIX-gpfs-pie.png" width="189" height="90"/> <img src="http://ganglia.lcg.cscs.ch/ganglia//CSCS-custom/PHOENIX-pps-pie.png" width="189" height="90"/>

Site Tests: Region DE (Nagios, history) / SLTOP / slurmerrors / SLURM CPU reports / Detailed SLURM CPU reports / APEL status / Site-View / MyWLCG site availability / WLCG Transfers dashboard

• CMS: Overview / Tiny Panel / CMS hammercloud / Job status / Job Robot / central CMS-Frontier status / Dashboard / WW transfers to CSCS / CMS Nagios for CSCS
• ATLAS: [[http://dashb-atlas-sum.cern.ch/dashboard/request.py/historicalsmryview-sum#view=siteavl&time[]=last12&granularity[]=default&profile=ATLAS_CRITICAL&group=All+sites&site[]=CSCS-LCG2&type=quality][Site Availability]] / Overview / Tiny Panel / Cloud Monitor / Panglia (Analysis , Production) / Dashboard / (Hamercloud) / Panda (Analysis , Production) / ARC (Cloud monitor, Panda)
• LHCb: Overview / Tiny Panel / Dirac Site status / Queues length

Batch jobs

Current Memory allocation: Current CPU allocation:

Usage by VO

Usage by CE

Node status and allocation

Worker nodes

<img src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?g=load_report&z=medium&c=PHOENIX-workers&m=&r=day&s=descending&hc=4&st=now" /> <img src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?g=cpu_report&z=medium&c=PHOENIX-workers&m=&r=day&s=descending&hc=4&st=now" /> <img src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?g=mem_report&z=medium&c=PHOENIX-workers&m=&r=day&s=descending&hc=4&st=now" />

Storage Element

Links:

• dCache GUI / Ganglia dCache Headnode/dCache pools
• CMS: used space (local monitoring) / dashboard summary view / List all hosted datasets/requests / subscriptions / accounting per phys. group (also see victor tool)
• ATLAS: DQ2 accounting on srm tokens

The below plot is from the dcache web ui representing active movers.

<img src="http://storage01.lcg.cscs.ch:2288/webadmin/poolqueueplots?2-IResourceListener-form-displayPanel-PoolQueuePlotsGridView-1-cols-2-plotlink" width="550" height="400" />

Networking and File Transfers

Links:

• CMS CMS PhEDEx downloads (detailed) / exports / Status prod / Debug / PhEDEx links
• ATLAS ATLAS FZK Cloud Data activity last 7 days
• FTS Channels all-CSCS / CSCS-FZK

<img src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?g=network_report&z=medium&c=PHOENIX-workers&m=&r=day&s=descending&hc=4&st=now" /> <img src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?g=network_report&z=medium&c=PHOENIX-fileservers&m=&r=day&s=descending&hc=4&st=now" /> <img src="http://ganglia.lcg.cscs.ch/ganglia/graph.php?g=iconnect_report&z=medium&c=PHOENIX-gpfs&m=&r=day&s=descending&hc=4&st=now" /> <img src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?g=network_report&z=medium&c=PHOENIX-services&m=&r=day&s=descending&hc=4&st=now" />

Number of active dCache movers (transfers) on the WAN (gridftp, srm) / LAN ("regular",dcap):

Number of queued dCache movers (waiting transfers) on the WAN (gridftp, srm) / LAN ("regular",dcap):

Number of pending dCache requests:

External monitoring

• CSCS external network (shared with other projects, only visible from SWITCH network):

• EGI Nagios from FZK, last 5 days (hover mouse to see hostname)

• Power consumption (Island 1)

• Gridview current status graphs

Other links

• EGI Discussion Forum

• A/R official metrics
• GSTAT
• CSCS Internal VLAN64
• EGI Accounting
• Operations Portal - Dashboard
• SMSCG Swiss sites
• SWITCH weather map
• IRC: irc:gridchat.cscs.ch:994#lcg
• GGUS tickets this year