<!-- keep this as a security measure: * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.LCGAdminGroup * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.LCGAdminGroup * Set GANGLIA3BASE = http://ganglia.lcg.cscs.ch/ganglia3/ #uncomment this if you want the page only be viewable by the internal people #* Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.LCGAdminGroup --> %CALC{"$SET(jobint,$IF($EXACT(%URLPARAM{"jobint"}%,),day,%URLPARAM{"jobint"}%))"}% ---+ LHConCRAYMonOverview %TOC% ---++ External links * SLTOP: http://ganglia.lcg.cscs.ch/ganglia/sltop_lhconcray.html * Common ETF view: https://etf-11.cern.ch/etf/check_mk/view.py?view_name=searchhost&host_regex=arc04.lcg.cscs.ch&filled_in=filter# (not working) * ATLAS: * Last 12h of PanDA jobs in CSCS-LCG2-HPC: https://bigpanda.cern.ch/jobs/?computingsite=CSCS-LCG2-HPC&hours=12&display_limit=100 * Last 12h of PanDA jobs in CSCS-LCG2-HPC_MCORE: https://bigpanda.cern.ch/jobs/?computingsite=CSCS-LCG2-HPC_MCORE&hours=12&display_limit=100 * Last 12h of PanDA jobs in ANALY_CSCS-LCG2-HPC: https://bigpanda.cern.ch/jobs/?computingsite=ANALY_CSCS-LCG2-HPC&hours=12&display_limit=100 * PanDA CSCS-LCG2-HPC: https://bigpanda.cern.ch/site/CSCS-LCG2-HPC * PanDA CSCS-LCG2-HPC_MCORE: https://bigpanda.cern.ch/site/CSCS-LCG2-HPC_MCORE * PanDA ANALY_CSCS-HPC: https://bigpanda.cern.ch/site/ANALY_CSCS-LCG2-HPC * HammerCloud full site last 7 days: http://hammercloud.cern.ch/hc/app/atlas/siteoverview/?site=CSCS-LCG2&startTime=&endTime=&templateType=isGolden * ETF: https://etf-atlas-prod.cern.ch/etf/check_mk/index.py?start_url=%2Fetf%2Fcheck_mk%2Fview.py%3Fview_name%3Dhost%26host%3Darc04.lcg.cscs.ch%26site%3D * CMS: * Glidein configuration: http://glidein-itb.grid.iu.edu/factory/monitor/factoryEntryStatusNow.html?entry=CMSHTPC_T2_CH_CSCS_arc04 * HammerCloud (requires CMS account): http://hammercloud.cern.ch/hc/app/cms/ * ETF: https://etf-cms-prod.cern.ch/etf/check_mk/index.py?start_url=%2Fetf%2Fcheck_mk%2Fview.py%3Fview_name%3Dhost%26host%3Darc04.lcg.cscs.ch%26site%3D * Site Status Board: http://dashb-ssb.cern.ch/dashboard/request.py/siteview#currentView=Site+Readiness&search_0=CSCS * Site Readiness: http://dashb-ssb.cern.ch/dashboard/request.py/siteview#currentView=default&search_0=T2_CH_CSCS * LHCb: * ETF: https://etf-lhcb-prod.cern.ch/etf/check_mk/index.py?start_url=%2Fetf%2Fcheck_mk%2Fview.py%3Fview_name%3Dhost%26host%3Darc04.lcg.cscs.ch%26site%3D ---++ Plots Note, the value for *current* may be incorrect. Ignore it. <br>Status plots from %SERVERTIME% (page is refreshed every 5 minutes). <form name="formJobint" action="%TOPICURL%?#TEST" method=GET> <select name="jobint" onchange="formJobint.submit()"> <option %CALC{"$IF($EXACT($GET(jobint),hour),selected,)"}%>hour</option> <option %CALC{"$IF($EXACT($GET(jobint),day),selected,)"}%>day</option> <option %CALC{"$IF($EXACT($GET(jobint),week),selected,)"}%>week</option> <option %CALC{"$IF($EXACT($GET(jobint),month),selected,)"}%>month</option> <option %CALC{"$IF($EXACT($GET(jobint),year),selected,)"}%>year</option> </select> <input type="hidden" name="freestorageint" value=%CALC{"$GET(freestorageint)"}%> <input type="hidden" name="netwint" value=%CALC{"$GET(netwint)"}%> </form> ---+++ System totals (Grafana) ---++++Utilization Note, CPU utilization chart is based on 72 CPUs/node. At the moment only 68 CPUs are allocatable, which means that 93% is the relative 100% in this case. <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/grafana_utilization_%CALC{"$GET(jobint)"}%.png"> ---++++CPU statistics <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/grafana_cpu_%CALC{"$GET(jobint)"}%.png"> ---++++ Queue status <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/grafana_queue_%CALC{"$GET(jobint)"}%.png"> ---+++ VO totals (Grafana) ---++++ CPUs allocated per VO <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/grafana_cpu_alloc_per_vo_%CALC{"$GET(jobint)"}%.png"> ---++++ Running jobs per VO <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/grafana_job_running_per_vo_%CALC{"$GET(jobint)"}%.png"> ---++++ Pending jobs per VO <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/grafana_job_pending_per_vo_%CALC{"$GET(jobint)"}%.png"> ---+++ Node performance (Ganglia) ---++++ Servers <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_arc_ce_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_dvs_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_dws_%CALC{"$GET(jobint)"}%.png"> ---++++ Compute nodes <br> Starting on *2017-09-06T08:44* any job launching on =nid0[1333-1357]= should run within DataWarp, since =RUNTIME_LOCAL_SCRATCH_DIR= is set to =/var/opt/cray/dws/lhc_scratch=.<br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01333_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01333_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01333_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01333_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01334_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01334_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01334_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01334_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01335_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01335_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01335_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01335_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01336_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01336_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01336_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01336_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01337_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01337_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01337_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01337_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01338_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01338_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01338_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01338_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01339_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01339_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01339_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01339_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01340_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01340_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01340_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01340_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01341_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01341_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01341_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01341_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01342_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01342_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01342_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01342_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01343_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01343_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01343_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01343_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01344_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01344_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01344_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01344_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01345_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01345_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01345_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01345_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01346_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01346_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01346_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01346_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01347_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01347_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01347_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01347_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01348_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01348_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01348_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01348_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01349_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01349_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01349_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01349_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01350_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01350_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01350_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01350_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01351_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01351_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01351_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01351_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01352_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01352_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01352_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01352_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01353_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01353_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01353_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01353_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01354_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01354_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01354_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01354_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01355_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01355_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01355_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01355_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01356_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01356_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01356_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01356_%CALC{"$GET(jobint)"}%.png"><br> <img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_mem_report_nid01357_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_cpu_report_nid01357_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_load_report_nid01357_%CALC{"$GET(jobint)"}%.png"><img src="http://ganglia.lcg.cscs.ch/ganglia3/lhconcray/ganglia_network_report_nid01357_%CALC{"$GET(jobint)"}%.png"><br> ---++ How this page is generated Plots for this page are generated using a cron job on ganglia that fetches data as needed: <verbatim># crontab -l # every 10min we get all LHConCRAY charts! 1,10,20,30,40,50 * * * * /root/lhconcray/lhconcray_download_static_images.sh ___PATH___ &> /var/log/lhconcray_download_static_images.log </verbatim> The script looks like this: <verbatim>#!/bin/bash size=medium per="hour day week month year" #per="hour" nod=$(echo nid013{33..57}) curl_opts="-g -s" grafana_theme="theme=light" destdir=$1 grafana_size='width=700&height=350' echo "Starting on `date`" echo "==========================" echo "Downloading Grafana charts" echo "==========================" for period in ${per}; do echo -n "Period: ${period} " utc_epoch_now=$(( `date -u +%s -d "now"` * 1000 )) utc_epoch_per=$(( `date -u +%s -d "last ${period}"` * 1000 )) curl ${curl_opts} "http://___GRAFANA_SERVER___/render/dashboard-solo/db/daint?refresh=1m&panelId=33&orgId=1&from=${utc_epoch_per}&to=${utc_epoch_now}&${grafana_size}&tz=UTC%2B02%3A00&${grafana_theme}" -o "${destdir}/grafana_utilization_${period}.png" echo -n "." curl ${curl_opts} "http://___GRAFANA_SERVER___/render/dashboard-solo/db/daint?refresh=1m&panelId=36&orgId=1&from=${utc_epoch_per}&to=${utc_epoch_now}&${grafana_size}&tz=UTC%2B02%3A00&${grafana_theme}" -o "${destdir}/grafana_cpu_${period}.png" echo -n "." curl ${curl_opts} "http://___GRAFANA_SERVER___/render/dashboard-solo/db/daint?refresh=1m&panelId=35&orgId=1&from=${utc_epoch_per}&to=${utc_epoch_now}&${grafana_size}&tz=UTC%2B02%3A00&${grafana_theme}" -o "${destdir}/grafana_queue_${period}.png" echo -n "." curl ${curl_opts} "http://___GRAFANA_SERVER___/render/dashboard-solo/db/cscs-slurm-top?panelId=3&orgId=1&var-cluster=daint&var-feature=wlcg*&var-account=atlas&var-account=cms&var-account=lhcb&from=${utc_epoch_per}&to=${utc_epoch_now}&${grafana_size}&tz=UTC%2B02%3A0&${grafana_theme}" -o "${destdir}/grafana_job_running_per_vo_${period}.png" echo -n "." curl ${curl_opts} "http://___GRAFANA_SERVER___/render/dashboard-solo/db/cscs-slurm-top?panelId=36&orgId=1&var-cluster=daint&var-feature=wlcg*&var-account=atlas&var-account=cms&var-account=lhcb&from=${utc_epoch_per}&to=${utc_epoch_now}&${grafana_size}&tz=UTC%2B02%3A0&${grafana_theme}" -o "${destdir}/grafana_job_pending_per_vo_${period}.png" echo -n "." curl ${curl_opts} "http://___GRAFANA_SERVER___/render/dashboard-solo/db/cscs-slurm-top?panelId=56&orgId=1&var-cluster=daint&var-feature=wlcg*&var-account=atlas&var-account=cms&var-account=lhcb&from=${utc_epoch_per}&to=${utc_epoch_now}&${grafana_size}&tz=UTC%2B02%3A0&${grafana_theme}" -o "${destdir}/grafana_job_pending_per_vo_${period}.png" echo -n "." echo done echo "==========================" echo "Downloading Ganglia charts" echo "==========================" echo "*** Downloading ARC, DVS and DWS load charts" for period in ${per}; do echo -n "Period: ${period} " curl ${curl_opts} "http://___GANGLIA_SERVER___/ganglia4/graph.php?hreg[]=arcds1%7Carc04%7Carc05&mreg[]=load_one&z=${size}>ype=stack&vl=load&title=ARC-CE+Load&aggregate=1&r=${period}" -o "${destdir}/ganglia_load_arc_ce_${period}.png" curl ${curl_opts} "http://___GANGLIA_SERVER___/ganglia4/graph.php?hreg%5B%5D=daintdvs(11%7C13%7C14%7C15%7C16)&mreg%5B%5D=load_one&z=${size}>ype=stack&vl=load&title=DVS+Load&aggregate=1&r=${period}" -o "${destdir}/ganglia_load_dvs_${period}.png" curl ${curl_opts} "http://___GANGLIA_SERVER___/ganglia4/graph.php?hreg[]=daintdws1%24%7Cdaintdws5%24%7Cdaintdws9%24%7Cdaintdws12%24&mreg[]=load_one&z=${size}>ype=stack&vl=load&title=DWS+Load&aggregate=1&r=${period}" -o "${destdir}/ganglia_load_dws_${period}.png" echo -n "." echo done echo Done echo "*** Downloading CN charts" for period in ${per}; do echo -n "Period: ${period} " for node in ${nod}; do curl ${curl_opts} "http://___GANGLIA_SERVER___/ganglia4/graph.php?g=mem_report&z=${size}&h=${node}&c=DAINT&r=${period}" -o "${destdir}/ganglia_mem_report_${node}_${period}.png" curl ${curl_opts} "http://___GANGLIA_SERVER___/ganglia4/graph.php?g=cpu_report&z=${size}&h=${node}&c=DAINT&r=${period}" -o "${destdir}/ganglia_cpu_report_${node}_${period}.png" curl ${curl_opts} "http://___GANGLIA_SERVER___/ganglia4/graph.php?g=load_report&z=${size}&h=${node}&c=DAINT&r=${period}" -o "${destdir}/ganglia_load_report_${node}_${period}.png" curl ${curl_opts} "http://___GANGLIA_SERVER___/ganglia4/graph.php?g=network_report&z=${size}&h=${node}&c=DAINT&r=${period}" -o "${destdir}/ganglia_network_report_${node}_${period}.png" echo -n "." done echo done echo Done echo "Finished on `date`" </verbatim> <br> -- Main.MiguelGila - 2017-07-21
This topic: LCGTier2
>
WebHome
>
PhoenixMonOverview
>
LHConCRAYMonitoring
Topic revision: r9 - 2017-09-06 - MiguelGila
Copyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki?
Send feedback