(r225) PhoenixMonOverview < LCGTier2

Tags: view all tags
%INCLUDE{%SYSTEMWEB%.WebChangesRightBox}%
<!-- keep this as a security measure:
   * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.LCGAdminGroup
   * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.LCGAdminGroup
   * Set GANGLIA3BASE = http://ganglia.lcg.cscs.ch/ganglia3/

   #uncomment this if you want the page only be viewable by the internal people
   # * Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.LCGAdminGroup
-->

%CALC{"$SET(jobint,$IF($EXACT(%URLPARAM{"jobint"}%,),day,%URLPARAM{"jobint"}%))"}% 
%CALC{"$SET(freestorageint,$IF($EXACT(%URLPARAM{"freestorageint"}%,),week,%URLPARAM{"freestorageint"}%))"}% 
%CALC{"$SET(netwint,$IF($EXACT(%URLPARAM{"netwint"}%,),day,%URLPARAM{"netwint"}%))"}% 

---+ !!Phoenix Monitoring Overview
%TOC%

Status plots from %SERVERTIME% (page is refreshed every 5 minutes). Historical view <a href="http://ganglia.lcg.cscs.ch/MonOverviewHistory"> here.</a> 

%ICON{warning}% Note: The _now values_ printed at the bottom of some graphs are not correct. 


*Site Tests*: Region DE ([[https://ngi-de-nagios.gridka.de/nagios/cgi-bin/status.cgi?hostgroup=site-CSCS-LCG2&style=detail][Nagios]], [[https://ngi-de-nagios.gridka.de/myegi/history/][history]]) */* [[https://midmon.egi.eu/nagios/cgi-bin/status.cgi?hostgroup=site-CSCS-LCG2&style=detail][MIDMON Nagios]] */* [[http://ganglia.lcg.cscs.ch/ganglia/sltop.html][SLTOP]] */* [[http://ganglia.lcg.cscs.ch/ganglia/slurmerrors.html][slurmerrors]] */* [[http://ganglia.lcg.cscs.ch/ganglia/phoenix_slurm_report.txt][SLURM CPU reports]] */* [[http://ganglia.lcg.cscs.ch/ganglia/SLURM_REPORTS/][Detailed SLURM CPU reports]]  */* [[http://goc-accounting.grid-support.ac.uk/rss/CSCS-LCG2_Pub.html][APEL status]] */* [[http://dashb-siteview.cern.ch/templates/siteview/index.html?site=CSCS-LCG2][Site-View]] */* [[http://grid-monitoring.cern.ch/mywlcg/ss/?vo=37&profile=28&filters-value-Regions=&filters-value-Sites=103&filters-value-Flavours=&filters-value-Hostnames=&production=1&preproduction=1&monitored=1&iDisplayLength=10&dateorperiod=pd&period=pW&startdate=15-01-2014&enddate=22-01-2014][MyWLCG site availability]] */* [[http://dashb-wlcg-transfers.cern.ch/ui/#][WLCG Transfers dashboard]]

   * *CMS:* [[http://dashb-ssb.cern.ch/dashboard/request.py/siteview?site=T2_CH_CSCS][Overview]] */* [[http://dashb-nagios-cms-dev.cern.ch/dashboard/request.py/latestresultssmrytable?siteSelect3=T2&serviceTypeSelect3=vo&sites=T2_CH_CSCS&services=CE&services=CREAMCE&services=SRMv2&tests=10523&tests=10467&tests=10480&tests=10492&tests=10512&tests=10506&tests=3336&tests=10469&tests=10479&tests=10500&tests=10505&tests=10522&tests=10466&tests=3309&tests=10496&tests=10517&tests=10472&tests=10515&exitStatus=all][Tiny Panel]] */* [[http://hammercloud.cern.ch/hc/app/cms/][CMS hammercloud]] */* [[http://dashb-cms-job.cern.ch/dashboard/request.py/jobsummary#site=T2_CH_CSCS][Job status]] */* [[https://cmsdoc.cern.ch/cms/LCG/SiteComm/JobRobot/html/][Job Robot]] */* [[http://frontier.cern.ch/squidstats/nonproxycms_summary.html][central CMS-Frontier status]] */* [[http://dashb-cms-sum.cern.ch/dashboard/request.py/latestresultssmry-sum#profile=CMS_CRITICAL_FULL&group=AllGroups&site%5B%5D=T2_CH_CSCS&flavour%5B%5D=All+Service+Flavours&metric%5B%5D=All+Metrics&status%5B%5D=All+Exit+Status][Dashboard]] */* [[https://cmsweb.cern.ch/phedex/debug/Activity::QualityPlots?graph=quality_all&entity=src&src_filter=&dest_filter=CSCS&no_mss=true&period=l7d&upto=][WW transfers to CSCS]] */* [[https://sam-cms-prod.cern.ch/nagios/cgi-bin/status.cgi?hostgroup=site-CSCS-LCG2][CMS Nagios for CSCS]] */* [[http://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/][CMS Site Readiness reports]]
   * *ATLAS:* [[http://dashb-atlas-sum.cern.ch/dashboard/request.py/historicalsmryview-sum#view=siteavl&time=last12&granularity=default&profile=ATLAS_CRITICAL&group=All+sites&site=CSCS-LCG2&type=quality][Site-view]] */* [[http://dashb-atlas-ssb.cern.ch/dashboard/request.py/siteview?site=CSCS-LCG2][Overview]] */* [[http://dashb-atlas-sam.cern.ch/dashboard/request.py/latestresultssmry?siteSelect3=400&serviceTypeSelect3=0&sites=CSCS-LCG2&services=CE&services=CREAMCE&services=FTS&services=LFC_C&services=LFC_L&services=SRMv2&services=VOBOX&services=gRB&tests=74541&tests=74543&tests=37878&tests=74732&tests=74734&tests=37947&tests=74569&tests=74571&tests=74567&exitStatus=all&table=true][Tiny Panel]] */* [[http://happyface-goegrid.gwdg.de/cloudmon/CloudMon.html][Cloud Monitor]] */* Panglia ([[http://gridinfo.triumf.ca/panglia/sites/site_detail.php?SITE=ANALY_CSCS][Analysis]]  , [[http://gridinfo.triumf.ca/panglia/sites/site_detail.php?SITE=CSCS-LCG2][Production]]) */* [[http://dashb-atlas-data.cern.ch/dashboard/request.py/site?name=DE&statsInterval=4][Dashboard]] */*  ([[http://hammercloud.cern.ch/hc/app/atlas/siteoverview/?site=CSCS-LCG2&startTime=&endTime=&templateType=isGolden ][Hamercloud]]) */* Panda ([[http://panda.cern.ch/server/pandamon/query?job=*&type=&hours=12&jobsetID=any&jobStatus=&site=ANALY_CSCS&cplot=yes&plot=yes&processingType=gangarobot&cplot=yes][Analysis]] , [[http://panda.cern.ch/server/pandamon/query?job=*&type=&hours=12&jobsetID=any&jobStatus=&site=&cplot=yes&plot=yes&processingType=gangarobot-pft&computingSite=CSCS-LCG2][Production]]) */* ARC ([[http://www.nordugrid.org/monitor/loadmon.php][Cloud monitor]], [[http://panda.cern.ch/server/pandamon/query?jobsummary=site&site=ARC-T2][Panda]]) */* [[https://sam-atlas-prod.cern.ch/nagios/cgi-bin/status.cgi?hostgroup=site-CSCS-LCG2&style=detail][ATLAS Nagios for CSCS]]
   * *LHCb:* [[http://dashb-lhcb-ssb.cern.ch/dashboard/request.py/siteview?site=LCG.CSCS.ch][Overview]] */* [[http://dashb-lhcb-sam.cern.ch/dashboard/request.py/latestresultssmry?siteSelect3=500&serviceTypeSelect3=0&sites=LCG.CSCS.ch&services=CE&services=CREAMCE&services=FTS&services=LFC_C&services=LFC_L&services=RB&services=SRMv2&services=VOBOX&services=gRB&tests=398&tests=404&tests=405&tests=406&tests=403&tests=407&tests=37624&tests=399&tests=2&tests=5&tests=7&tests=14&tests=25&tests=37732&tests=37700&tests=37703&tests=37710&tests=37715&tests=37760&tests=51&tests=50&tests=37638&tests=37553&tests=37554&tests=37555&tests=37636&tests=37637&tests=37556&tests=37557&tests=37643&tests=37399&exitStatus=all&table=true][Tiny Panel]] */* [[http://lhcbweb.pic.es/DIRAC/LHCb-Production/visitor/jobs/SiteSummary/display][Dirac Site status]] */* [[http://dashb-lhcb-ssb.cern.ch/dashboard/request.py/sitehistory?site=LCG.CSCS.ch#currentView=By+tier&time=168&start_date=&end_date=&values=false&spline=false&white=false][Dashboard]] */* [[http://lhcbproject.web.cern.ch/lhcbproject/Operations/queues.html][Queues length]]

---++ Batch jobs 
*Current Memory allocation:* <img alt="" src="http://ganglia.lcg.cscs.ch/ganglia/PHOENIX-custom/percentage_mem_used.png" /> *Current CPU allocation:* <img alt="" src="http://ganglia.lcg.cscs.ch/ganglia/PHOENIX-custom/percentage_cpu_used.png" /> <br>

<form name="formJobint" action="%TOPICURL%?#Batch_jobs_on_the_CE" method=GET>
<select name="jobint" onchange="formJobint.submit()">
<option %CALC{"$IF($EXACT($GET(jobint),hour),selected,)"}%>hour</option>
<option %CALC{"$IF($EXACT($GET(jobint),day),selected,)"}%>day</option>
<option %CALC{"$IF($EXACT($GET(jobint),week),selected,)"}%>week</option>
<option %CALC{"$IF($EXACT($GET(jobint),month),selected,)"}%>month</option>
<option %CALC{"$IF($EXACT($GET(jobint),year),selected,)"}%>year</option>
</select>
<input type="hidden" name="freestorageint" value=%CALC{"$GET(freestorageint)"}%>
<input type="hidden" name="netwint" value=%CALC{"$GET(netwint)"}%>
</form>

---+++ Usage by VO 

<br>
<img 
src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?r=%CALC{"$GET(jobint)"}%&z=xsmall&hreg[]=slurm1&mreg[]=slurm_cpu_alloc_partition_%28atlas%2A%7Ccms%7Clhcb%29&gtype=stack&vl=ncpus&title=CPU+allocation+per+VO&glegend=show&aggregate=1" />
<br>
<img 
src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?r=%CALC{"$GET(jobint)"}%&z=xsmall&hreg[]=slurm1&mreg[]=Running_%28ATLAS%7CCMS%7CLHCB%29&gtype=stack&vl=njobs&title=Running+jobs+per+VO&glegend=show&aggregate=1" />
<br>
<img 
src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?r=%CALC{"$GET(jobint)"}%&z=xsmall&hreg[]=slurm1&mreg[]=Pending_%28ATLAS%7CCMS%7CLHCB%29&gtype=stack&vl=njobs&title=Pending+jobs+per+VO&glegend=show&aggregate=1" />

---+++ Usage by CE 

<br>
<img 
src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?r=%CALC{"$GET(jobint)"}%&z=xsmall&hreg[]=slurm1&mreg[]=running_%28arc01%7Carc02%7Ccream01%7Ccream02%7Ccream03%7Ccream04%29&gtype=stack&vl=njobs&title=Running+jobs+per+CE&glegend=show&aggregate=1" />
<br>
<img 
src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?r=%CALC{"$GET(jobint)"}%&z=xsmall&hreg[]=slurm1&mreg[]=pending_%28arc01%7Carc02%7Ccream01%7Ccream02%7Ccream03%7Ccream04%29&gtype=stack&vl=njobs&title=Pending+jobs+per+CE&glegend=show&aggregate=1" />
<br>
<img 
src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?r=%CALC{"$GET(jobint)"}%&z=xsmall&hreg[]=slurm1&mreg[]=running_%28arc01%7Carc02%7Ccream01%7Ccream02%7Ccream03%7Ccream04%29_cores&gtype=stack&vl=ncpus&title=Allocated+cpus+per+CE&glegend=show&aggregate=1" />
<br>
<img 
src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?r=%CALC{"$GET(jobint)"}%&z=xsmall&hreg[]=slurm1&mreg[]=pending_%28arc01%7Carc02%7Ccream01%7Ccream02%7Ccream03%7Ccream04%29_cores&gtype=stack&vl=ncpus&title=Pending+cpus+per+CE&glegend=show&aggregate=1" />
<br>

---+++ Node status and allocation
<img src="http://ganglia.lcg.cscs.ch/ganglia/PHOENIX-custom/prod_nodestatus-%CALC{"$GET(jobint)"}%.gif" /><br>
<img src="http://ganglia.lcg.cscs.ch/ganglia/PHOENIX-custom/prod_cpu_alloc-%CALC{"$GET(jobint)"}%.gif" />
<img src="http://ganglia.lcg.cscs.ch/ganglia/PHOENIX-custom/prod_mem_alloc-%CALC{"$GET(jobint)"}%.gif" />

---+++ Older charts
---++++ Usage by VO (static charts)
<img src="http://ganglia.lcg.cscs.ch/ganglia/PHOENIX-custom/prod_vojobs-running-%CALC{"$GET(jobint)"}%.gif" />
<img src="http://ganglia.lcg.cscs.ch/ganglia/PHOENIX-custom/prod_vojobs-pending-%CALC{"$GET(jobint)"}%.gif" />
<img src="http://ganglia.lcg.cscs.ch/ganglia/PHOENIX-custom/prod_vojobs-TotalFailed-%CALC{"$GET(jobint)"}%.gif" />

---++++ Usage by CE
<img src="http://ganglia.lcg.cscs.ch/ganglia/PHOENIX-custom/prod_cejobs-running-%CALC{"$GET(jobint)"}%.gif" />
<img src="http://ganglia.lcg.cscs.ch/ganglia/PHOENIX-custom/prod_cejobs-pending-%CALC{"$GET(jobint)"}%.gif" />
<img src="http://ganglia.lcg.cscs.ch/ganglia/PHOENIX-custom/prod_cejobs-totalfailed-%CALC{"$GET(jobint)"}%.gif" />

---++ Worker nodes
<img
src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?g=load_report&z=medium&c=PHOENIX-workers&m=&r=%CALC{"$GET(jobint)"}%&s=descending&hc=4&st=now" />
<img
src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?g=cpu_report&z=medium&c=PHOENIX-workers&m=&r=%CALC{"$GET(jobint)"}%&s=descending&hc=4&st=now" />
<img
src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?g=mem_report&z=medium&c=PHOENIX-workers&m=&r=%CALC{"$GET(jobint)"}%&s=descending&hc=4&st=now" />

---++ Storage Element 
Links:
   * [[http://%SEHOST%:2288/][dCache GUI]] */* [[http://ganglia.lcg.cscs.ch/ganglia3/?c=PHOENIX-services&h=%SEHOST%&m=&r=day&s=descending&hc=4][Ganglia dCache Headnode]]/[[http://ganglia.lcg.cscs.ch/ganglia3/?c=PHOENIX-fileservers&m=network_report&r=day&s=descending&hc=4][dCache pools]] */* [[http://dashb-atlas-ssb.cern.ch/dashboard/request.py/sitehistory?site=CSCS-LCG2#currentView=FAX+endpoints][FAX endpoint status]]
   * *ATLAS:* [[http://bourricot.cern.ch/dq2/accounting/allspacetokens_view/CSCS-LCG2/30/][DQ2 accounting on srm tokens]] */* [[http://ganglia.lcg.cscs.ch/ganglia/files_atlas.html][ATLAS directories at CSCS-LCG2]] to download it use a tool like =wget=
   * *CMS:* [[http://ganglia.lcg.cscs.ch/ganglia/cms_sespace.txt][used space (local monitoring)]] */* [[http://dashb-ssb.cern.ch/dashboard/request.py/siteview?view=storage][dashboard summary view]] */* List all [[https://cmsweb.cern.ch/phedex/prod/Data::Replicas?view=global&rcolumn=Name&nvalue=Node+bytes&rows=interesting&dbs=7&dbs=11&dbs=14&dbs=15&dbs=10&dbs=16&dbs=12&dbs=2&dbs=1&dbs=3&dbs=9&dbs=13&dbs=6&dbs=4&dbs=5&dbs=22&dbs=8&node=27&filter=.*][hosted datasets]]/[[https://cmsweb.cern.ch/phedex/prod/Request::View?type=xfer&nodes=T2_CH_CSCS&state=pend&.submit=Submit][requests]] */* [[https://cmsweb.cern.ch/phedex/prod/Data::Subscriptions?filter=.%2A;node=27][subscriptions]] */*  [[https://cmsweb.cern.ch/phedex/prod/Reports::SiteUsage?node=T2_CH_CSCS][accounting per phys. group]] (also see [[https://cmsweb.cern.ch/victor/association_view/T2_CH_CSCS/AnalysisOps][victor tool]]) */* [[http://ganglia.lcg.cscs.ch/ganglia/files_cms.html][CMS directories at CSCS-LCG2]] to download it use a tool like =wget=
   * *LHCb:* [[http://ganglia.lcg.cscs.ch/ganglia/files_lhcb.html][LHCb directories at CSCS-LCG2]] to download it use a tool like =wget=

Free storage space: <form name="formFreestorageint" action="%TOPICURL%?#Storage_Element" method=GET>
<select name="freestorageint" onchange="formFreestorageint.submit()">
<option %CALC{"$IF($EXACT($GET(freestorageint),hour),selected,)"}%>hour</option>
<option %CALC{"$IF($EXACT($GET(freestorageint),day),selected,)"}%>day</option>
<option %CALC{"$IF($EXACT($GET(freestorageint),week),selected,)"}%>week</option>
<option %CALC{"$IF($EXACT($GET(freestorageint),month),selected,)"}%>month</option>
<option %CALC{"$IF($EXACT($GET(freestorageint),year),selected,)"}%>year</option>
</select>
<input type="hidden" name="jobint" value=%CALC{"$GET(jobint)"}%>
<input type="hidden" name="netwint" value=%CALC{"$GET(netwint)"}%>
</form>

<img 
src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?c=PHOENIX-services&h=%SEHOST%&m=storage01_free_atlas&r=%CALC{"$GET(freestorageint)"}%&z=medium&jr=&js=&vl=GB&st=now" />
<img 
src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?c=PHOENIX-services&h=%SEHOST%&m=storage01_free_cms&r=%CALC{"$GET(freestorageint)"}%&z=medium&jr=&js=&vl=GB&st=now" />
<img 
src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?c=PHOENIX-services&h=%SEHOST%&m=storage01_free_lhcb&r=%CALC{"$GET(freestorageint)"}%&z=medium&jr=&js=&vl=GB&st=now" /><br>
<img 
src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?hreg[]=%28storage01%29&mreg[]=storage01_used_%2A&z=medium&gtype=stack&vl=GB&title=Used+space+by+VO&glegend=show&aggregate=1&r=%CALC{"$GET(freestorageint)"}%" />

The below plot is from the dcache web ui representing active movers.

<img
src="http://storage01.lcg.cscs.ch:2288/webadmin/poolqueueplots?2-IResourceListener-form-displayPanel-PoolQueuePlotsGridView-1-cols-2-plotlink" width="550" height="400" />

---++ Networking and File Transfers
Links:
   * *CMS* [[http://ganglia.lcg.cscs.ch/ganglia/phedex/][Local PhEDEx stats]] */* Status [[http://cmsweb.cern.ch/phedex/prod/Components::Status][Prod]] / [[http://cmsweb.cern.ch/phedex/Debug/Components::Status][Debug]] */* [[http://cmsweb.cern.ch/phedex/prod/Components::Links?from_filter=T2_CH_CSCS&andor=or&to_filter=T2_CH_CSCS&Update=Update#][PhEDEx enabled data transfer links at CSCS]]
   * *ATLAS*  [[http://dashb-atlas-data.cern.ch/dashboard/request.py/site?statsInterval=168&name=FZK][ATLAS FZK Cloud Data activity last 7 days]]
   * *FTS Channels* [[http://ftm-kit.gridka.de/ftsmonitor/ftschannel.php?channel=STAR-CSCS&vo=all][all-CSCS]] */* [[http://ftm-kit.gridka.de/ftsmonitor/ftschannel.php?channel=CSCS-FZK&vo=all][CSCS-FZK]]

Plotting interval: <form name="formNetwint" action="%TOPICURL%?#Networking_and_File_Transfers" method=GET>
<select name="netwint" onchange="formNetwint.submit()">
<option %CALC{"$IF($EXACT($GET(netwint),hour),selected,)"}%>hour</option>
<option %CALC{"$IF($EXACT($GET(netwint),day),selected,)"}%>day</option>
<option %CALC{"$IF($EXACT($GET(netwint),week),selected,)"}%>week</option>
<option %CALC{"$IF($EXACT($GET(netwint),month),selected,)"}%>month</option>
<option %CALC{"$IF($EXACT($GET(netwint),year),selected,)"}%>year</option>
</select>
<input type="hidden" name="jobint" value=%CALC{"$GET(jobint)"}%>
<input type="hidden" name="freestorageint" value=%CALC{"$GET(freestorageint)"}%>
</form>

%CALC{"$IF($EXACT($GET(netwint),hour),$SET(netwint_sec, 3600),)"}%
%CALC{"$IF($EXACT($GET(netwint),day),$SET(netwint_sec, 86400),)"}%
%CALC{"$IF($EXACT($GET(netwint),week),$SET(netwint_sec, 604800),)"}%
%CALC{"$IF($EXACT($GET(netwint),month),$SET(netwint_sec, 2419200),)"}%
%CALC{"$IF($EXACT($GET(netwint),year),$SET(netwint_sec, 31536000),)"}%


<br><img
src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?g=network_report&z=medium&c=PHOENIX-workers&m=&r=%CALC{"$GET(netwint)"}%&s=descending&hc=4&st=now" />
<img
src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?g=network_report&z=medium&c=PHOENIX-fileservers&m=&r=%CALC{"$GET(netwint)"}%&s=descending&hc=4&st=now" />
<img
src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?g=network_report&z=medium&c=PHOENIX-gpfs2&m=&r=%CALC{"$GET(netwint)"}%&s=descending&hc=4&st=now" />
<img
src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?g=network_report&z=medium&c=PHOENIX-services&m=&r=%CALC{"$GET(netwint)"}%&s=descending&hc=4&st=now" />

Number of active dCache movers (transfers) on the WAN (gridftp, srm) / LAN ("regular",dcap):<br>
<img src="%GANGLIABASE%/CSCS-custom/storage01_movers_wan-%CALC{"$GET(netwint)"}%.gif" />
<img src="%GANGLIABASE%/CSCS-custom/storage01_movers_regular-%CALC{"$GET(netwint)"}%.gif" />

Number of queued dCache movers (waiting transfers) on the WAN (gridftp, srm) / LAN ("regular",dcap):<br>
<img src="%GANGLIABASE%/CSCS-custom/storage01_moversQ_wan-%CALC{"$GET(netwint)"}%.gif" />
<img src="%GANGLIABASE%/CSCS-custom/storage01_moversQ_regular-%CALC{"$GET(netwint)"}%.gif" />

Number of pending dCache requests:<br>
<img  src="http://ganglia.lcg.cscs.ch/ganglia3/graph.php?c=PHOENIX-services&h=storage01.lcg.cscs.ch&m=storage01_pending_requests&r=%CALC{"$GET(netwint)"}%&z=medium&st=now" />


---++ External monitoring
   * CSCS external network (shared with other projects, only visible from SWITCH network):
<img src="https://www.switch.ch/network/operation/statistics/cricket/cms-mini-graph.cgi?type=png;target=/Overview/cscs-tot;inst=0;dslist=ifInOctets,ifOutOctets;range=%CALC{"$GET(netwint_sec)"}%" />

   * EGI Nagios from FZK, last 5 days (hover mouse to see hostname)
%CALC{"$SET(SECONDS,$EVAL(3600 * 24 * 5))"}% 
%CALC{"$SET(PERCENT_DAY,$INT($EVAL(($TIME() - $TODAY()) / 864 )))"}% 
%CALC{"$SET(NAGIOS_EXT,https://ngi-de-nagios.gridka.de/myegi/ss/)"}% 
<table style="border-collapse:collapse; border-spacing:0px; margin:0px; padding:0px;"><tr>
<td width="%CALC{"$SUM(99, -$GET(PERCENT_DAY))"}%" align="center" style="border-right: 1px solid; margin:0px; padding:0px;"></td>
<td width="99" align="center" style="border-right: 1px solid; margin:0px; padding:0px;">%CALC{"$FORMATGMTIME($EVAL($TODAY() - 3600 * 24 * 4), $day)"}%</td>
<td width="99" align="center" style="border-right: 1px solid; margin:0px; padding:0px;">%CALC{"$FORMATGMTIME($EVAL($TODAY() - 3600 * 24 * 3), $day)"}%</td>
<td width="99" align="center" style="border-right: 1px solid; margin:0px; padding:0px;">%CALC{"$FORMATGMTIME($EVAL($TODAY() - 3600 * 24 * 2), $day)"}%</td>
<td width="99" align="center" style="border-right: 1px solid; margin:0px; padding:0px;">%CALC{"$FORMATGMTIME($EVAL($TODAY() - 3600 * 24), $day)"}%</td>
<td width="%CALC{"$GET(PERCENT_DAY)"}%" align="center" style="border-right: none; margin:0px; padding:0px; ">%CALC{"$FORMATGMTIME($TODAY(), $day)"}%</td>
</tr></table>
\
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/274/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/" title="cream01" width="500" height="3"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/274/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=17&view=-1" title="cream01" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/274/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=18&view=-1" title="cream01" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/274/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=20&view=-1" title="cream01" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/274/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=21&view=-1" title="cream01" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/274/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=22&view=-1" title="cream01" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/274/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=23&view=-1" title="cream01" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/274/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=30&view=-1" title="cream01" width="500" height="1"/></a><br>
\
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/275/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/" title="cream02" style="margin-top:2px" width="500" height="3"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/275/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=17&view=-1" title="cream02" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/275/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=18&view=-1" title="cream02" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/275/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=20&view=-1" title="cream02" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/275/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=21&view=-1" title="cream02" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/275/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=22&view=-1" title="cream02" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/275/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=23&view=-1" title="cream02" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/275/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=30&view=-1" title="cream02" width="500" height="1"/></a><br>
\
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/273/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/" title="cream03" style="margin-top:2px" width="500" height="3"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/273/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=17&view=-1" title="cream03" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/273/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=18&view=-1" title="cream03" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/273/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=20&view=-1" title="cream03" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/273/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=21&view=-1" title="cream03" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/273/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=22&view=-1" title="cream03" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/273/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=23&view=-1" title="cream03" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/273/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=30&view=-1" title="cream03" width="500" height="1"/></a><br>
\
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/288/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/" title="storage02" style="margin-top:2px" width="500" height="3"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/288/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=52&view=-1" title="storage02" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/288/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=54&view=-1" title="storage02" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/288/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=55&view=-1" title="storage02" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/288/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=56&view=-1" title="storage02" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/288/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=57&view=-1" title="storage02" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/288/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=58&view=-1" title="storage02" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/288/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=59&view=-1" title="storage02" width="500" height="1"/></a><br>
<a href="%CALC{"$GET(NAGIOS_EXT)"}%"><img src="%CALC{"$GET(NAGIOS_EXT)"}%graph/288/3/%CALC{"$SUM($TIME(),  -$GET(SECONDS))"}%/%CALC{"$TIME()"}%/?metric_id=60&view=-1" title="storage02" width="500" height="1"/></a><br>
\

   * Power consumption (Island 1)
<img src="%GANGLIABASE%/CSCS-custom/power-consumption-%CALC{"$GET(netwint)"}%.gif" />

   * *Gridview* current status graphs
<img src="http://gridview.cern.ch/GRIDVIEW/ace/bin/ace_graphs.php?GraphName=CREAM-CE&Information=SiteDetail&Profile=6&TestVO=-1&DurationOption=current&LComponent=-2&Hostname=&TestID=-1&Hour1=0&StartDay=-1&StartMonth=-1&StartYear=-1&&RelOrAvail=AverageStatus&OnlyCritical=ON&Report=0&LSiteGroup[]=103" />
<img src="http://gridview.cern.ch/GRIDVIEW/ace/bin/ace_graphs.php?GraphName=SRMv2&Information=SiteDetail&Profile=6&TestVO=-1&DurationOption=current&LComponent=-2&Hostname=&TestID=-1&Hour1=0&StartDay=-1&StartMonth=-1&StartYear=-1&RelOrAvail=AverageStatus&OnlyCritical=ON&Report=0&LSiteGroup[]=103" />

<iframe src="http://dashb-siteview.cern.ch/templates/siteview/onlymap.html?site=CSCS-LCG2" width="715" height="390" scrolling="no"><p>Your browser does not support iframes.</p></iframe>

---++ Other monitoring websites
   * [[http://gstat2.grid.sinica.edu.tw/gstat/site/CSCS-LCG2/][GSTAT]]
   * [[http://grono.cscs.ch/switchmap/vlans/vlan64.html][CSCS Internal VLAN64]]
   * [[https://accounting.egi.eu/egi.php?ExecutingSite=CSCS-LCG2][EGI Accounting]]
   * [[http://operations-portal.in2p3.fr/availability/siteAvailabilities/details/true/site/CSCS-LCG2][Operations Portal - CSCS Page]]
   * [[http://www.switch.ch/network/operation/weathermap/][SWITCH weather map]]
   * IRC: irc:gridchat.cscs.ch:994#lcg

---++ CMS Monitoring vs CSCS services
   * [[https://twiki.cern.ch/twiki/bin/view/CMSPublic/WaitingRoomMorgueAndSiteReadiness][Site Readiness logic]]
   * [[http://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html][Site Readiness reports]] ( search for CSCS ) 
   * [[https://twiki.cern.ch/twiki/bin/view/CMSPublic/UsableSitesForAnalysis][Site Readiness logic for CRAB3]]
   * Site Readiness logic for CRAB3 - files to be checked
      * https://cmst1.web.cern.ch/CMST1/SST/analysis/usableSites.txt
      * https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/ASCii/UsableSites.txt
   * [[https://sam-cms-prod.cern.ch/nagios/cgi-bin/status.cgi?hostgroup=site-CSCS-LCG2&style=overview][Nagios checks ( SAM Tests ) vs CSCS]]
   * [[https://ggus.eu/?mode=ticket_search&cms_site=T2_CH_CSCS&timeframe=any&status=open&search_submit=GO%21][Tickets vs CSCS]]
   * [[https://cmsweb.cern.ch/phedex/prod/Activity::ErrorInfo?tofilter=T2_CH_CSCS&fromfilter=.*&report_code=.*&xfer_code=.*&to_pfn=.*&from_pfn=.*&log_detail=.*&log_validate=.*&.submit=Update#][PhEDEx errors at CSCS]]