Difference: CmsVoBox (1 vs. 50)

Revision 502017-01-10 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"
Changed:
<
<
>
>
Contact hn-cms-t2@cern.ch for support.
 

Services

/home/phedex/config/T3_CH_PSI/PhEDEx/tools/init.d/phedex_* status

<--/twistyPlugin twikiMakeVisibleInline-->
Line: 1045 to 754
 Tue Dec 6 22:14:34 CET 2016 fileremove
<--/twistyPlugin-->
Changed:
<
<

/home/phedex/config/T3_CH_PSI/PhEDEx/tools/scripts/phedex-list-agents.sh slow but more details

>
>

/home/phedex/config/T3_CH_PSI/PhEDEx/tools/scripts/phedex-list-agents.sh slower but providing more details

 
<--/twistyPlugin twikiMakeVisibleInline-->
2016-12-09 12:52:41: ShowAgents[11056]: (re)connecting to database

Line: 1222 to 932
 udp 0 0 t3cmsvobox01.psi.ch:34702 t3ossec.psi.c:fujitsu-dtcns ESTABLISHED 6131/ossec-agentd
<--/twistyPlugin-->
Added:
>
>
 

Checking each CMS pool by Nagios through both the t3se01:SRM and t3dcachedb:Xrootd dCache doors

Changed:
<
<
By t3cmsvobox , in turn contacted by t3nagios , we retrieve a file from each CMS pool through both t3se01:SRM and t3dcachedb:Xrootd https://t3nagios.psi.ch/check_mk/index.py?start_url=%2Fcheck_mk%2Fview.py%3Fview_name%3Dhost%26host%3Dt3cmsvobox01%26site%3D
>
>
By t3cmsvobox , in turn contacted by t3nagios , we retrieve a file from each CMS pool through both t3se01:SRM and t3dcachedb:Xrootd :
  1. https://t3nagios.psi.ch/nagios/cgi-bin/status.cgi?servicegroup=SRM+T3+Tests&style=detail&&servicestatustypes=2&hoststatustypes=15&serviceprops=0&hostprops=0
  2. https://t3nagios.psi.ch/nagios/cgi-bin/status.cgi?servicegroup=ROOT+T3+Tests&style=detail&&servicestatustypes=2&hoststatustypes=15&serviceprops=0&hostprops=0
  In both the cases the test files retrieved are :
[martinelli_f@t3ui12 ~]$ find /pnfs/psi.ch/cms/t3-nagios/ | grep M | sort

Line: 1240 to 952
 
[root@t3dcachedb03 ~]# psql -U nagios -d chimera -c " select path,ipnfsid,pools from v_pnfs where path like '%1MB-test-file_pool_%' ; " 
                            path                             |               ipnfsid                |               pools                

Changed:
<
<

--------------------------------------+------------------------------------

>
>

--------------------------------------+---------------

  /pnfs/psi.ch/dteam/t3-nagios/1MB-test-file_pool_t3fs09_ops | 0000BCDA4B329DA94D64AAAFE7C0C7501E5C | t3fs09_ops /pnfs/psi.ch/dteam/t3-nagios/1MB-test-file_pool_t3fs08_ops | 0000358B14867ED5402184C2C22F81EFC861 | t3fs08_ops /pnfs/psi.ch/dteam/t3-nagios/1MB-test-file_pool_t3fs07_ops | 0000409BB804C95944A38DBE8220B416A8A3 | t3fs07_ops
Changed:
<
<
/pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_9 | 0000B58A7FA17778439F8F6F47C5CBBED5E7 | t3fs03_cms t3fs11_cms t3fs14_cms_9 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_8 | 00001A2FD52D31DB4CCAB99C8B8336522339 | t3fs09_cms t3fs11_cms t3fs14_cms_8 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_7 | 000018AA61C1E30F43709F0D9FE3B9CD65D1 | t3fs03_cms t3fs14_cms_7 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_6 | 0000E88C6CBB2D5A4365B11BE2EDD1554366 | t3fs02_cms t3fs14_cms_6 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_5 | 000200000000000006300738 | t3fs10_cms t3fs14_cms_5 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_4 | 0002000000000000052EF198 | t3fs03_cms t3fs14_cms_4 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_3 | 0002000000000000052EF168 | t3fs03_cms t3fs14_cms_3 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_2 | 0002000000000000052EF138 | t3fs07_cms t3fs14_cms_2 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_11 | 00003616229002194F439925DA3C7F1CFA02 | t3fs10_cms t3fs14_cms_11 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_10 | 0000B3D6A96EF961473AACB05F80CF9D6892 | t3fs07_cms t3fs14_cms_10 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_1 | 0002000000000000052EF108 | t3fs02_cms t3fs11_cms t3fs14_cms_1 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_0 | 0000A6470E0458354BD99D6C2DD27B196DCC | t3fs08_cms t3fs14_cms_0 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms | 0002000000000000052EF0D8 | t3fs03_cms t3fs04_cms t3fs14_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_9 | 00004783F9158A5941B284342FF4A8EDE126 | t3fs08_cms t3fs13_cms_9 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_8 | 0000132841305C27434891574015FD2CF923 | t3fs09_cms t3fs13_cms_8 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_7 | 00003FC27733ACBA4A809677419256FE22F9 | t3fs02_cms t3fs11_cms t3fs13_cms_7 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_6 | 0002000000000000072F8630 | t3fs07_cms t3fs11_cms t3fs13_cms_6 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_5 | 0002000000000000052EF0A8 | t3fs03_cms t3fs13_cms_5 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_4 | 0002000000000000052EF078 | t3fs10_cms t3fs11_cms t3fs13_cms_4 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_3 | 0002000000000000052EF048 | t3fs10_cms t3fs13_cms_3 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_2 | 0002000000000000052EF018 | t3fs02_cms t3fs13_cms_2 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_11 | 00000DB49D5B69EB4C568834BD162C3DA8E7 | t3fs09_cms t3fs13_cms_11 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_10 | 0000073FF4F754BB4AB1B4599F412811BDA2 | t3fs10_cms t3fs13_cms_10 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_1 | 00000CB9E97140F940CD973C319045B43FDA | t3fs04_cms t3fs11_cms t3fs13_cms_1 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_0 | 00005560491A76DE49DBA142D3BE3CFE38D5 | t3fs02_cms t3fs11_cms t3fs13_cms_0 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms | 0002000000000000052EEFB8 | t3fs07_cms t3fs11_cms t3fs13_cms
>
>
/pnfs/psi.ch/cms/trivcat/store/user/martinelli_f/1MB-test-file_pool_t3fs14_cms_11 | 00009E6424128A5F4F7AA7A24E0E13B778E1 | t3fs13_cms_7 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3nfs02_cms_1 | 00004E4DF3282B1F49A38994C7D968E288DA | t3nfs02_cms_1 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3nfs02_cms | 0000DD327FC27102417ABDBDF4CA1638E92A | t3nfs02_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_9 | 0000B58A7FA17778439F8F6F47C5CBBED5E7 | t3fs14_cms_9 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_8 | 00001A2FD52D31DB4CCAB99C8B8336522339 | t3fs14_cms_8 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_7 | 000018AA61C1E30F43709F0D9FE3B9CD65D1 | t3fs14_cms_7 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_6 | 0000E88C6CBB2D5A4365B11BE2EDD1554366 | t3fs14_cms_6 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_5 | 000200000000000006300738 | t3fs14_cms_5 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_4 | 0002000000000000052EF198 | t3fs14_cms_4 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_3 | 0002000000000000052EF168 | t3fs14_cms_3 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_2 | 0002000000000000052EF138 | t3fs14_cms_2 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_11 | 00003616229002194F439925DA3C7F1CFA02 | t3fs14_cms_11 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_10 | 0000B3D6A96EF961473AACB05F80CF9D6892 | t3fs14_cms_10 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_1 | 0002000000000000052EF108 | t3fs14_cms_1 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_0 | 0000A6470E0458354BD99D6C2DD27B196DCC | t3fs14_cms_0 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms | 0002000000000000052EF0D8 | t3fs14_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_9 | 00004783F9158A5941B284342FF4A8EDE126 | t3fs13_cms_9 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_8 | 0000132841305C27434891574015FD2CF923 | t3fs13_cms_8 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_7 | 00003FC27733ACBA4A809677419256FE22F9 | t3fs13_cms_7 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_6 | 0002000000000000072F8630 | t3fs13_cms_6 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_5 | 0002000000000000052EF0A8 | t3fs13_cms_5 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_4 | 0002000000000000052EF078 | t3fs13_cms_4 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_3 | 0002000000000000052EF048 | t3fs13_cms_3 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_2 | 0002000000000000052EF018 | t3fs13_cms_2 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_11 | 00000DB49D5B69EB4C568834BD162C3DA8E7 | t3fs13_cms_11 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_10 | 0000073FF4F754BB4AB1B4599F412811BDA2 | t3fs13_cms_10 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_1 | 00000CB9E97140F940CD973C319045B43FDA | t3fs13_cms_1 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_0 | 00005560491A76DE49DBA142D3BE3CFE38D5 | t3fs13_cms_0 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms | 0000ADB314586EFA40369C76D1348C3C001B | t3fs13_cms
  /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs11_cms | 00009E4A9774085C4799B5C9C827DA03406F | t3fs11_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs10_cms | 000005D1DD24CA14448694E5C46A8AA8E91F | t3fs10_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs09_cms | 0000479ED8FDDC374BC68827AEDF1C146686 | t3fs09_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs08_cms | 00003A989AB6D1074D738594B1D01E2D03DE | t3fs08_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs07_cms | 0000119DDCFD0C5F42B89769BC9C104A997F | t3fs07_cms
Changed:
<
<
/pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs04_cms_1 | 0002000000000000063D8C68 | t3fs04_cms_1 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs04_cms | 00020000000000000395B300 | t3fs04_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs03_cms | 000200000000000006391F88 | t3fs03_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs02_cms | 00020000000000000330BF10 | t3fs02_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs01_cms | 00020000000000000330BF90 | t3fs01_cms
>
>
(37 rows)
  </>
<--/twistyPlugin-->
Changed:
<
<

Pitfalls in dcache-srmclient-2.10.7-1 ( currently the latest dcache-srmclient )

>
>

Pitfalls in dcache-srmclient-2.10.7-1 ( currently the latest dcache-srmclient ) outdated since PhEDEx uses gfal-copy nowadays

 Strangely PhEDEx has a strong dependency on dcache-srmclient ; by strong we mean that you can't use equivalent SRM tools like lcg-cp or gfal-copy ; in its latest version, Fabio noticed that :
srmcp as in dcache-srmclient-2.2.4-2.el6.x86_64 had, by default, -delegate=true
srmcp as in dcache-srmclient-2.10.7-1.noarch has now, by default, -delegate=false 
Line: 1320 to 1031
 

Backups

Changed:
<
<
OS snapshots are nightly taken by the PSI VMWare Team ( contact Peter Huesser ) + we have LinuxBackupsByLegato to recover a single file.
>
>
OS snapshots are nightly taken by the PSI VMWare Team ( contact Peter Huesser or Daniel Webster ) + we can use LinuxBackupsByLegato in order to recover live a single file.
 
Changed:
<
<
You might also want to exploit t3nfs02:/data01/backups
>
>
You might also want to exploit the dedicated FS t3nfs02:/data01/backups to take occasionally a full backup.
 
META FORM name="NodeTypeForm"
FORM FIELD Hostnames Hostnames t3cmsvobox ( t3cmsvobox01 )

Revision 492016-12-09 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Services

Changed:
<
<

/home/phedex/phedex_start.sh

To be manually invoked after a server restart !

/home/phedex/phedex_stop.sh

/home/phedex/phedex_status.sh

>
>

/home/phedex/config/T3_CH_PSI/PhEDEx/tools/init.d/phedex_* status

<--/twistyPlugin twikiMakeVisibleInline-->
[phedex@t3cmsvobox01 ~]$ /home/phedex/config/T3_CH_PSI/PhEDEx/tools/init.d/phedex_Prod status 
blockverify (14243) 	[UP]
download-remove (14310) 	[UP]
download-t1 (14377) 	[UP]
download-t2 (14464) 	[UP]
exp-pfn (14545) 	[UP]
Watchdog (14644) 	[UP]
WatchdogLite (14664) 	[UP]

[phedex@t3cmsvobox01 ~]$ /home/phedex/config/T3_CH_PSI/PhEDEx/tools/init.d/phedex_Dev status 
blockverify (13764) 	[UP]
download-remove (13831) 	[UP]
download-t1 (13898) 	[UP]
download-t2 (13985) 	[UP]
exp-pfn (14066) 	[UP]
Watchdog (14165) 	[UP]
WatchdogLite (14185) 	[UP]

[phedex@t3cmsvobox01 ~]$ /home/phedex/config/T3_CH_PSI/PhEDEx/tools/init.d/phedex_Debug status 
blockverify (14766) 	[UP]
download-remove (14833) 	[UP]
download-t1 (14916) 	[UP]
download-t2 (15019) 	[UP]
exp-pfn (15166) 	[UP]
Watchdog (15285) 	[UP]
WatchdogLite (15305) 	[UP]
<--/twistyPlugin-->

/home/phedex/config/T3_CH_PSI/PhEDEx/tools/init.d/phedex_* stop

/home/phedex/config/T3_CH_PSI/PhEDEx/tools/init.d/phedex_* start

ps aux --forest | grep phedex

 
<--
 #List all the important services, their installation, configuration and how to start and stop them
-->
Added:
>
>
<--/twistyPlugin twikiMakeVisibleInline-->
 

Changed:
<
<
current phedex status perl /home/phedex/PHEDEX/4.1.7/Toolkit/Transfer/FileDownload -state /home/phedex/state/Debug/incoming/download/ -log /home/phedex/log/Debug/download -verbose -db/home/phed perl /home/phedex/PHEDEX/4.1.7/Toolkit/Transfer/FileExport -state /home/phedex/state/Debug/incoming/fileexport/ -log /home/phedex/log/Debug/fileexport -db/home/phedex/conf perl /home/phedex/PHEDEX/4.1.7/Toolkit/Transfer/FileRemove -state /home/phedex/state/Debug/incoming/fileremove/ -log /home/phedex/log/Debug/fileremove -node T3_CH_PSI -db/ perl /home/phedex/PHEDEX/4.1.7/Toolkit/Verify/BlockDownloadVerify -state /home/phedex/state/Debug/incoming/blockverify/ -log /home/phedex/log/Debug/blockverify --db/home/p perl /home/phedex/PHEDEX/4.1.7/Utilities/AgentFactory.pl -state /home/phedex/state/Debug/incoming/watchdog/ -log /home/phedex/log/Debug/watchdog -db/home/phedex/config/DBP perl /home/phedex/PHEDEX/4.1.7/Utilities/AgentFactoryLite.pl -state /home/phedex/state/Debug/incoming/WatchdogLite/ -log /home/phedex/log/Debug/WatchdogLite -nodeT3_CH_PSI perl /home/phedex/PHEDEX/4.1.7/Toolkit/Transfer/FileDownload -state /home/phedex/state/Dev/incoming/download/ -log /home/phedex/log/Dev/download -verbose -db/home/phedex/c perl /home/phedex/PHEDEX/4.1.7/Toolkit/Transfer/FileExport -state /home/phedex/state/Dev/incoming/fileexport/ -log /home/phedex/log/Dev/fileexport -db/home/phedex/config/D perl /home/phedex/PHEDEX/4.1.7/Toolkit/Transfer/FileRemove -state /home/phedex/state/Dev/incoming/fileremove/ -log /home/phedex/log/Dev/fileremove -node T3_CH_PSI -db/home perl /home/phedex/PHEDEX/4.1.7/Toolkit/Verify/BlockDownloadVerify -state /home/phedex/state/Dev/incoming/blockverify/ -log /home/phedex/log/Dev/blockverify --db/home/phede perl /home/phedex/PHEDEX/4.1.7/Utilities/AgentFactory.pl -state /home/phedex/state/Dev/incoming/watchdog/ -log /home/phedex/log/Dev/watchdog -db/home/phedex/config/DBParam perl /home/phedex/PHEDEX/4.1.7/Utilities/AgentFactoryLite.pl -state /home/phedex/state/Dev/incoming/WatchdogLite/ -log /home/phedex/log/Dev/WatchdogLite -node T3_CH_PSI-ag perl /home/phedex/PHEDEX/4.1.7/Toolkit/Transfer/FileDownload -state /home/phedex/state/Prod/incoming/download/ -log /home/phedex/log/Prod/download -verbose -db/home/phedex perl /home/phedex/PHEDEX/4.1.7/Toolkit/Transfer/FileExport -state /home/phedex/state/Prod/incoming/fileexport/ -log /home/phedex/log/Prod/fileexport -db/home/phedex/config perl /home/phedex/PHEDEX/4.1.7/Toolkit/Transfer/FileRemove -state /home/phedex/state/Prod/incoming/fileremove/ -log /home/phedex/log/Prod/fileremove -node T3_CH_PSI -db/ho perl /home/phedex/PHEDEX/4.1.7/Toolkit/Verify/BlockDownloadVerify -state /home/phedex/state/Prod/incoming/blockverify/ -log /home/phedex/log/Prod/blockverify --db/home/phe perl /home/phedex/PHEDEX/4.1.7/Utilities/AgentFactory.pl -state /home/phedex/state/Prod/incoming/watchdog/ -log /home/phedex/log/Prod/watchdog -db/home/phedex/config/DBPar perl /home/phedex/PHEDEX/4.1.7/Utilities/AgentFactoryLite.pl -state /home/phedex/state/Prod/incoming/WatchdogLite/ -log /home/phedex/log/Prod/WatchdogLite -node T3_CH_PSI-

bash └─bash └─pstree -uh phedex -la

TRANSFER LOGs ================================================================= source /home/phedex/PHEDEX/4.1.7/etc/profile.d/env.sh /home/phedex/PHEDEX/4.1.7/Utilities/InspectPhedexLog -es "-1 hours" -d /home/phedex/log/Prod/download /home/phedex/log/Debug/download /home/phedex/PHEDEX/4.1.7/Utilities/InspectPhedexLog -es "-1 days" -d /home/phedex/log/Prod/download /home/phedex/log/Debug/download

How to update ========= https://twiki.cern.ch/twiki/bin/view/CMSPublic/PhedexAdminDocsInstallation#Updating_Software

How to test /home/phedex/config/SITECONF/T3_CH_PSI/PhEDEx/storage.xml ================================================================= source /home/phedex/PHEDEX/4.1.7/etc/profile.d/env.sh /home/phedex/PHEDEX/4.1.7/Utilities/TestCatalogue -c /home/phedex/config/SITECONF/T3_CH_PSI/PhEDEx/storage.xml -p srmv2 -L /store/data/file

StorageConsistencyCheck example =========================== source /home/phedex/PHEDEX/4.1.7/etc/profile.d/env.sh /home/phedex/PHEDEX/4.1.7/Utilities/StorageConsistencyCheck -db /home/phedex/config/DBParam.PSI:Prod/PSI -lfnlist /home/phedex/PSI.lfnlist.txt -node T3_CH_PSI /home/phedex/PHEDEX/4.1.7/Utilities/StorageConsistencyCheck -db /home/phedex/config/DBParam.PSI:Prod/PSI -lfnlist /home/phedex/CSCS.lfnlist.txt -node T2_CH_CSCS [root@t3dcachedb03 ~]# psql -U nagios -d chimera -c " select path from v_pnfs where path like '/pnfs/psi.ch/cms%' ; " -t -q -o ./PSI.txt <-------- to get the LFN

PSI agents as perceived from the CERN DBs ===================================== source /home/phedex/PHEDEX/4.1.7/etc/profile.d/env.sh /home/phedex/PHEDEX/4.1.7/Utilities/ShowAgents -db /home/phedex/config/DBParam.PSI:Prod/PSI -node T3_CH_PSI https://cmsweb.cern.ch/phedex/datasvc/xml/prod/agents?node=T3_CH_PSI https://cmsweb.cern.ch/phedex/datasvc/json/prod/agents?node=T3_CH_PSI

>
>
[phedex@t3cmsvobox01 ~]$ ps aux --forest | grep phedex phedex 13764 0.0 0.2 330716 24040 ? S Dec06 1:22 perl /home/phedex/PHEDEX/Toolkit/Verify/BlockDownloadVerify -state /home/phedex/agents/Dev_T3_CH_PSI/state/blockverify/ -log /home/phedex/agents/Dev_T3_CH_PSI/logs/blockverify -db /home/phedex/config/DBParam.PSI:Dev/PSI -nodes T3_CH_PSI -namespace gfal phedex 13831 0.0 0.2 331044 21648 ? S Dec06 1:53 perl /home/phedex/PHEDEX/Toolkit/Transfer/FileRemove -state /home/phedex/agents/Dev_T3_CH_PSI/state/download-remove/ -log /home/phedex/agents/Dev_T3_CH_PSI/logs/download-remove -db /home/phedex/config/DBParam.PSI:Dev/PSI -nodes T3_CH_PSI -delete /home/phedex/config/T3_CH_PSI/PhEDEx/FileDownloadGFALDelete -protocol srmv2 phedex 13898 0.0 0.3 341048 26388 ? S Dec06 2:11 perl /home/phedex/PHEDEX/Toolkit/Transfer/FileDownload -state /home/phedex/agents/Dev_T3_CH_PSI/state/download-t1/ -log /home/phedex/agents/Dev_T3_CH_PSI/logs/download-t1 -db /home/phedex/config/DBParam.PSI:Dev/PSI -nodes T3_CH_PSI -accept T1% -delete /home/phedex/config/T3_CH_PSI/PhEDEx/FileDownloadGFALDelete -validate /home/phedex/config/T3_CH_PSI/PhEDEx/FileDownloadGFALVerify -verbose -backend FTS3 -service https://fts3.cern.ch:8446 -protocols srmv2 -batch-files 20 -max-active-files 4 phedex 13985 0.0 0.3 341108 26192 ? S Dec06 2:11 perl /home/phedex/PHEDEX/Toolkit/Transfer/FileDownload -state /home/phedex/agents/Dev_T3_CH_PSI/state/download-t2/ -log /home/phedex/agents/Dev_T3_CH_PSI/logs/download-t2 -db /home/phedex/config/DBParam.PSI:Dev/PSI -nodes T3_CH_PSI -accept T2% -delete /home/phedex/config/T3_CH_PSI/PhEDEx/FileDownloadGFALDelete -validate /home/phedex/config/T3_CH_PSI/PhEDEx/FileDownloadGFALVerify -verbose -backend FTS3 -service https://fts3.cern.ch:8446 -protocols srmv2 -batch-files 20 -max-active-files 4 phedex 14066 0.0 0.2 329640 20544 ? S Dec06 1:11 perl /home/phedex/PHEDEX/Toolkit/Transfer/FileExport -state /home/phedex/agents/Dev_T3_CH_PSI/state/exp-pfn/ -log /home/phedex/agents/Dev_T3_CH_PSI/logs/exp-pfn -db /home/phedex/config/DBParam.PSI:Dev/PSI -nodes T3_CH_PSI -storagemap /home/phedex/config/T3_CH_PSI/PhEDEx/storage.xml -protocols srmv2 phedex 14165 0.0 0.2 330632 21380 ? S Dec06 1:16 perl /home/phedex/PHEDEX/Utilities/AgentFactory.pl -state /home/phedex/agents/Dev_T3_CH_PSI/state/Watchdog/ -log /home/phedex/agents/Dev_T3_CH_PSI/logs/Watchdog -db /home/phedex/config/DBParam.PSI:Dev/PSI -node T3_CH_PSI -agent_list exp-pfn -agent_list download-t1 -agent_list download-t2 -agent_list download-remove -agent_list blockverify phedex 14185 0.0 0.2 159644 17136 ? S Dec06 1:27 perl /home/phedex/PHEDEX/Utilities/AgentFactoryLite.pl -state /home/phedex/agents/Dev_T3_CH_PSI/state/WatchdogLite/ -log /home/phedex/agents/Dev_T3_CH_PSI/logs/WatchdogLite -node T3_CH_PSI -agent_list watchdog phedex 14243 0.0 0.3 330716 24976 ? S Dec06 1:21 perl /home/phedex/PHEDEX/Toolkit/Verify/BlockDownloadVerify -state /home/phedex/agents/Prod_T3_CH_PSI/state/blockverify/ -log /home/phedex/agents/Prod_T3_CH_PSI/logs/blockverify -db /home/phedex/config/DBParam.PSI:Prod/PSI -nodes T3_CH_PSI -namespace gfal phedex 14310 0.0 0.3 331044 24208 ? S Dec06 1:54 perl /home/phedex/PHEDEX/Toolkit/Transfer/FileRemove -state /home/phedex/agents/Prod_T3_CH_PSI/state/download-remove/ -log /home/phedex/agents/Prod_T3_CH_PSI/logs/download-remove -db /home/phedex/config/DBParam.PSI:Prod/PSI -nodes T3_CH_PSI -delete /home/phedex/config/T3_CH_PSI/PhEDEx/FileDownloadGFALDelete -protocol srmv2 phedex 14377 0.3 0.5 353016 41764 ? S Dec06 12:35 perl /home/phedex/PHEDEX/Toolkit/Transfer/FileDownload -state /home/phedex/agents/Prod_T3_CH_PSI/state/download-t1/ -log /home/phedex/agents/Prod_T3_CH_PSI/logs/download-t1 -db /home/phedex/config/DBParam.PSI:Prod/PSI -nodes T3_CH_PSI -accept T1% -delete /home/phedex/config/T3_CH_PSI/PhEDEx/FileDownloadGFALDelete -validate /home/phedex/config/T3_CH_PSI/PhEDEx/FileDownloadGFALVerify -verbose -backend FTS3 -service https://fts3.cern.ch:8446 -protocols srmv2 -batch-files 20 -max-active-files 4 phedex 14464 0.0 0.4 345828 35928 ? S Dec06 3:25 perl /home/phedex/PHEDEX/Toolkit/Transfer/FileDownload -state /home/phedex/agents/Prod_T3_CH_PSI/state/download-t2/ -log /home/phedex/agents/Prod_T3_CH_PSI/logs/download-t2 -db /home/phedex/config/DBParam.PSI:Prod/PSI -nodes T3_CH_PSI -accept T2% -delete /home/phedex/config/T3_CH_PSI/PhEDEx/FileDownloadGFALDelete -validate /home/phedex/config/T3_CH_PSI/PhEDEx/FileDownloadGFALVerify -verbose -backend FTS3 -service https://fts3.cern.ch:8446 -protocols srmv2 -batch-files 20 -max-active-files 4 phedex 14545 0.0 0.2 329640 22760 ? S Dec06 1:12 perl /home/phedex/PHEDEX/Toolkit/Transfer/FileExport -state /home/phedex/agents/Prod_T3_CH_PSI/state/exp-pfn/ -log /home/phedex/agents/Prod_T3_CH_PSI/logs/exp-pfn -db /home/phedex/config/DBParam.PSI:Prod/PSI -nodes T3_CH_PSI -storagemap /home/phedex/config/T3_CH_PSI/PhEDEx/storage.xml -protocols srmv2 phedex 14644 0.0 0.2 330632 22368 ? S Dec06 1:16 perl /home/phedex/PHEDEX/Utilities/AgentFactory.pl -state /home/phedex/agents/Prod_T3_CH_PSI/state/Watchdog/ -log /home/phedex/agents/Prod_T3_CH_PSI/logs/Watchdog -db /home/phedex/config/DBParam.PSI:Prod/PSI -node T3_CH_PSI -agent_list exp-pfn -agent_list download-t1 -agent_list download-t2 -agent_list download-remove -agent_list blockverify phedex 14664 0.0 0.2 159644 17144 ? S Dec06 1:27 perl /home/phedex/PHEDEX/Utilities/AgentFactoryLite.pl -state /home/phedex/agents/Prod_T3_CH_PSI/state/WatchdogLite/ -log /home/phedex/agents/Prod_T3_CH_PSI/logs/WatchdogLite -node T3_CH_PSI -agent_list watchdog phedex 14766 0.0 0.3 330716 27044 ? S Dec06 1:21 perl /home/phedex/PHEDEX/Toolkit/Verify/BlockDownloadVerify -state /home/phedex/agents/Debug_T3_CH_PSI/state/blockverify/ -log /home/phedex/agents/Debug_T3_CH_PSI/logs/blockverify -db /home/phedex/config/DBParam.PSI:Debug/PSI -nodes T3_CH_PSI -namespace gfal phedex 14833 0.0 0.3 331044 27524 ? S Dec06 1:57 perl /home/phedex/PHEDEX/Toolkit/Transfer/FileRemove -state /home/phedex/agents/Debug_T3_CH_PSI/state/download-remove/ -log /home/phedex/agents/Debug_T3_CH_PSI/logs/download-remove -db /home/phedex/config/DBParam.PSI:Debug/PSI -nodes T3_CH_PSI -delete /home/phedex/config/T3_CH_PSI/PhEDEx/FileDownloadGFALDelete -protocol srmv2 phedex 14916 0.0 0.3 340916 30984 ? S Dec06 2:10 perl /home/phedex/PHEDEX/Toolkit/Transfer/FileDownload -state /home/phedex/agents/Debug_T3_CH_PSI/state/download-t1/ -log /home/phedex/agents/Debug_T3_CH_PSI/logs/download-t1 -db /home/phedex/config/DBParam.PSI:Debug/PSI -nodes T3_CH_PSI -accept T1% -delete /home/phedex/config/T3_CH_PSI/PhEDEx/FileDownloadGFALDelete -validate /home/phedex/config/T3_CH_PSI/PhEDEx/FileDownloadGFALVerify -verbose -backend FTS3 -service https://fts3.cern.ch:8446 -protocols srmv2 -batch-files 20 -max-active-files 4 phedex 15019 0.0 0.3 341100 31128 ? S Dec06 2:10 perl /home/phedex/PHEDEX/Toolkit/Transfer/FileDownload -state /home/phedex/agents/Debug_T3_CH_PSI/state/download-t2/ -log /home/phedex/agents/Debug_T3_CH_PSI/logs/download-t2 -db /home/phedex/config/DBParam.PSI:Debug/PSI -nodes T3_CH_PSI -accept T2% -delete /home/phedex/config/T3_CH_PSI/PhEDEx/FileDownloadGFALDelete -validate /home/phedex/config/T3_CH_PSI/PhEDEx/FileDownloadGFALVerify -verbose -backend FTS3 -service https://fts3.cern.ch:8446 -protocols srmv2 -batch-files 20 -max-active-files 4 phedex 15166 0.0 0.2 329640 22756 ? S Dec06 1:12 perl /home/phedex/PHEDEX/Toolkit/Transfer/FileExport -state /home/phedex/agents/Debug_T3_CH_PSI/state/exp-pfn/ -log /home/phedex/agents/Debug_T3_CH_PSI/logs/exp-pfn -db /home/phedex/config/DBParam.PSI:Debug/PSI -nodes T3_CH_PSI -storagemap /home/phedex/config/T3_CH_PSI/PhEDEx/storage.xml -protocols srmv2 phedex 15285 0.0 0.2 330628 22368 ? S Dec06 1:16 perl /home/phedex/PHEDEX/Utilities/AgentFactory.pl -state /home/phedex/agents/Debug_T3_CH_PSI/state/Watchdog/ -log /home/phedex/agents/Debug_T3_CH_PSI/logs/Watchdog -db /home/phedex/config/DBParam.PSI:Debug/PSI -node T3_CH_PSI -agent_list exp-pfn -agent_list download-t1 -agent_list download-t2 -agent_list download-remove -agent_list blockverify phedex 15305 0.0 0.2 159644 17176 ? S Dec06 1:28 perl /home/phedex/PHEDEX/Utilities/AgentFactoryLite.pl -state /home/phedex/agents/Debug_T3_CH_PSI/state/WatchdogLite/ -log /home/phedex/agents/Debug_T3_CH_PSI/logs/WatchdogLite -node T3_CH_PSI -agent_list watchdog
<--/twistyPlugin-->

/home/phedex/config/T3_CH_PSI/PhEDEx/tools/scripts/phedex-list-agents2.sh fast

<--/twistyPlugin twikiMakeVisibleInline-->
[phedex@t3cmsvobox01 ~]$ cd /home/phedex/config/T3_CH_PSI/PhEDEx/tools/scripts/
[phedex@t3cmsvobox01 scripts]$ ./phedex-list-agents2.sh  
Fri Dec 9 13:05:25 CET 2016 exp-pfn
Fri Dec 9 13:10:13 CET 2016 mgmt-blockverifyinjector
Fri Dec 9 13:35:57 CET 2016 Watchdog
Fri Dec 9 13:37:54 CET 2016 download-t1
Fri Dec 9 13:41:40 CET 2016 download-remove
Fri Dec 9 13:41:55 CET 2016 download-t2
Fri Dec 9 13:46:49 CET 2016 mgmt-router
Fri Dec 9 13:48:25 CET 2016 mgmt-pump
Fri Dec 9 13:50:55 CET 2016 blockverify
Fri Dec 9 13:51:38 CET 2016 mgmt-issue
Tue Dec 6 21:50:42 CET 2016 fileexport
Tue Dec 6 21:56:28 CET 2016 download
Tue Dec 6 21:57:16 CET 2016 watchdog
Tue Dec 6 22:14:34 CET 2016 fileremove
<--/twistyPlugin-->

/home/phedex/config/T3_CH_PSI/PhEDEx/tools/scripts/phedex-list-agents.sh slow but more details

<--/twistyPlugin twikiMakeVisibleInline-->
2016-12-09 12:52:41: ShowAgents[11056]: (re)connecting to database

 
Deleted:
<
<
PSI Prod Datasets ===================================== https://cmsweb.cern.ch/phedex/prod/Data::Replicas?view=global&rcolumn=Name&nvalue=Node+files&rows=interesting&dbs=21&dbs=1&dbs=41&node=821&filter=.*

PSI Prod Errors ===================================== https://cmsweb.cern.ch/phedex/prod/Activity::ErrorInfo?from_pfn=.%2A;fromfilter=.%2A;log_detail=.%2A;log_validate=.%2A;xfer_code=.%2A;tofilter=T3_CH_PSI;to_pfn=.%2A;report_code=.%2A https://cmsweb.cern.ch/phedex/datasvc/xml/prod/errorlog?to=T3_CH_PSI

HOW TO IDENTIFY THE DATASETS TO BE DELETED AT PSI AND CSCS ====================================================== https://wiki.chipp.ch/twiki/bin/view/CmsTier3/CmsVoBox#Dataset_cleaning <-- HowTo Doc source /home/phedex/PHEDEX/4.1.7/etc/profile.d/env.sh cd svn-sandbox/phedex/DB-query-tools/ source /home/phedex/PHEDEX/4.1.7/etc/profile.d/env.sh ./ListSiteDataInfo.pl -w -t --db ~/config/DBParam.PSI:Prod/PSI -s "%CSCS%" | grep "eleted" echo ./ListSiteDataInfo.pl -w -t --db ~/config/DBParam.PSI:Prod/PSI -s "%CSCS%" | grep -vE "Dutta|Fanfani|Kress|Magini|Wuerthwein|Belforte|Spinoso|Ajit|DataOps|eleted|StoreResults|Argiro|Klute|Cremonesi|Jean-Roch Vlimant|vocms[0-9]+|cmsgwms-submit[0-9]+|IntelROCCS|retention time: 2016|Retention date: 2016" echo ./ListSiteDataInfo.pl -w -t --db ~/config/DBParam.PSI:Prod/PSI -s "%PSI%" | grep -Ev "retention time: 2016|Retention date: 2016" https://wiki.chipp.ch/twiki/bin/view/CmsTier3/DataSetCleaningQuery <-- where to publish the ListSiteDataInfo.pl outputs

 
Changed:
<
<
be aware that if a data sets appeared, disappeared and appeared again ListSiteDataInfo.pl as it's written today will report the 1st old occurrence, so old user and old retention time
>
>
============================================================================
node_name
T3_CH_PSI agent_name: BlockDownloadVerify agent_label: blockverify
host_name
t3cmsvobox01 process_id: 14243 release: PHEDEX_4_2_1 status_update: 2016-12-09 12:50:55 UTC (1481287855.45748) log_update: 2016-12-09 12:50:55 UTC (1481287855.51542) last_update: 0h01 ago
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - log_reason: AGENT RECONNECTED log_process: phedex 14243 0.0 0.3 330716 27056 ? S Dec06 1:21 perl /home/phedex/PHEDEX/Toolkit/Verify/BlockDownloadVerify -state /home/phedex/agents/Prod_T3_CH_PSI/state/blockverify/ -log /home/phedex/agents/Prod_T3_CH_PSI/logs/blockverify -db /home/phedex/config/DBParam.PSI:Prod/PSI -nodes T3_CH_PSI -namespace gfal ============================================================================
node_name
T3_CH_PSI agent_name: FileDownload agent_label: download
host_name
t3cmsvobox01 process_id: 8057 release: PHEDEX_4_2_1 status_update: 2016-12-06 20:56:28 UTC (1481057788.04664) log_update: 2016-12-06 20:56:28 UTC (1481057788.13163) last_update: 2d15h56 ago
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - log_reason: AGENT RECONNECTED log_process: phedex 8057 0.5 0.5 349608 46116 ? S Nov24 97:38 perl /home/phedex/PHEDEX/4.1.7/Toolkit/Transfer/FileDownload -state /home/phedex/state/Prod/incoming/download/ -log /home/phedex/log/Prod/download -verbose -db /home/phedex/config/DBParam.PSI:Prod/PSI -nodes T3_CH_PSI -delete /home/phedex/config/SITECONF/T3_CH_PSI/PhEDEx/FileDownloadDelete -validate /home/phedex/config/SITECONF/T3_CH_PSI/PhEDEx/FileDownloadSRMVerify -backend SRM -protocols srmv2,srm -command srmcp,-delegate=true,-pushmode=false,-debug=true,-retry_num=2,-protocols=gsiftp,-srm_protocol_version=2,-streams_num=1,-globus_tcp_port_range=20000:25000 -ignore FNAL -timeout 9999 -batch-files 10 -jobs 3 ============================================================================
node_name
T3_CH_PSI agent_name: FileDownload agent_label: download-t1
host_name
t3cmsvobox01 process_id: 14377 release: PHEDEX_4_2_1 status_update: 2016-12-09 12:37:54 UTC (1481287074.03753) log_update: 2016-12-09 12:37:54 UTC (1481287074.08362) last_update: 0h14 ago
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - log_reason: AGENT RECONNECTED log_process: phedex 14377 0.3 0.5 353016 45084 ? S Dec06 12:34 perl /home/phedex/PHEDEX/Toolkit/Transfer/FileDownload -state /home/phedex/agents/Prod_T3_CH_PSI/state/download-t1/ -log /home/phedex/agents/Prod_T3_CH_PSI/logs/download-t1 -db /home/phedex/config/DBParam.PSI:Prod/PSI -nodes T3_CH_PSI -accept T1% -delete /home/phedex/config/T3_CH_PSI/PhEDEx/FileDownloadGFALDelete -validate /home/phedex/config/T3_CH_PSI/PhEDEx/FileDownloadGFALVerify -verbose -backend FTS3 -service https://fts3.cern.ch:8446 -protocols srmv2 -batch-files 20 -max-active-files 4 ============================================================================
node_name
T3_CH_PSI agent_name: FileDownload agent_label: download-t2
host_name
t3cmsvobox01 process_id: 14464 release: PHEDEX_4_2_1 status_update: 2016-12-09 12:41:55 UTC (1481287315.78247) log_update: 2016-12-09 12:41:55 UTC (1481287315.8289) last_update: 0h10 ago
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - log_reason: AGENT RECONNECTED log_process: phedex 14464 0.0 0.4 345828 38080 ? S Dec06 3:24 perl /home/phedex/PHEDEX/Toolkit/Transfer/FileDownload -state /home/phedex/agents/Prod_T3_CH_PSI/state/download-t2/ -log /home/phedex/agents/Prod_T3_CH_PSI/logs/download-t2 -db /home/phedex/config/DBParam.PSI:Prod/PSI -nodes T3_CH_PSI -accept T2% -delete /home/phedex/config/T3_CH_PSI/PhEDEx/FileDownloadGFALDelete -validate /home/phedex/config/T3_CH_PSI/PhEDEx/FileDownloadGFALVerify -verbose -backend FTS3 -service https://fts3.cern.ch:8446 -protocols srmv2 -batch-files 20 -max-active-files 4 ============================================================================
node_name
T3_CH_PSI agent_name: FileExport agent_label: exp-pfn
host_name
t3cmsvobox01 process_id: 14545 release: PHEDEX_4_2_1 status_update: 2016-12-09 12:05:25 UTC (1481285125.79724) log_update: 2016-12-09 12:05:25 UTC (1481285125.84701) last_update: 0h47 ago
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - log_reason: AGENT RECONNECTED log_process: phedex 14545 0.0 0.3 329640 25932 ? S Dec06 1:12 perl /home/phedex/PHEDEX/Toolkit/Transfer/FileExport -state /home/phedex/agents/Prod_T3_CH_PSI/state/exp-pfn/ -log /home/phedex/agents/Prod_T3_CH_PSI/logs/exp-pfn -db /home/phedex/config/DBParam.PSI:Prod/PSI -nodes T3_CH_PSI -storagemap /home/phedex/config/T3_CH_PSI/PhEDEx/storage.xml -protocols srmv2 ============================================================================
node_name
T3_CH_PSI agent_name: FileExport agent_label: fileexport
host_name
t3cmsvobox01 process_id: 8127 release: PHEDEX_4_2_1 status_update: 2016-12-06 20:50:42 UTC (1481057442.2398) log_update: 2016-12-06 20:50:42 UTC (1481057442.2888) last_update: 2d16h01 ago
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - log_reason: AGENT RECONNECTED log_process: phedex 8127 0.0 0.3 329412 25788 ? S Nov24 4:58 perl /home/phedex/PHEDEX/4.1.7/Toolkit/Transfer/FileExport -state /home/phedex/state/Prod/incoming/fileexport/ -log /home/phedex/log/Prod/fileexport -db /home/phedex/config/DBParam.PSI:Prod/PSI -nodes T3_CH_PSI -storagemap /home/phedex/config/SITECONF/T3_CH_PSI/PhEDEx/storage.xml -protocols srmv2,srm ============================================================================
node_name
T3_CH_PSI agent_name: FileRemove agent_label: download-remove
host_name
t3cmsvobox01 process_id: 14310 release: PHEDEX_4_2_1 status_update: 2016-12-09 12:41:40 UTC (1481287300.95789) log_update: 2016-12-09 12:41:41 UTC (1481287301.00236) last_update: 0h11 ago
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - log_reason: AGENT RECONNECTED log_process: phedex 14310 0.0 0.3 331044 27468 ? S Dec06 1:54 perl /home/phedex/PHEDEX/Toolkit/Transfer/FileRemove -state /home/phedex/agents/Prod_T3_CH_PSI/state/download-remove/ -log /home/phedex/agents/Prod_T3_CH_PSI/logs/download-remove -db /home/phedex/config/DBParam.PSI:Prod/PSI -nodes T3_CH_PSI -delete /home/phedex/config/T3_CH_PSI/PhEDEx/FileDownloadGFALDelete -protocol srmv2 ============================================================================
node_name
T3_CH_PSI agent_name: FileRemove agent_label: fileremove
host_name
t3cmsvobox01 process_id: 8222 release: PHEDEX_4_2_1 status_update: 2016-12-06 21:14:34 UTC (1481058874.03595) log_update: 2016-12-06 21:14:34 UTC (1481058874.08392) last_update: 2d15h38 ago
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - log_reason: AGENT RECONNECTED log_process: phedex 8222 0.1 0.5 350316 46680 ? S Nov24 20:23 perl /home/phedex/PHEDEX/4.1.7/Toolkit/Transfer/FileRemove -state /home/phedex/state/Prod/incoming/fileremove/ -log /home/phedex/log/Prod/fileremove -node T3_CH_PSI -db /home/phedex/config/DBParam.PSI:Prod/PSI -protocol srmv2 -delete /home/phedex/config/SITECONF/T3_CH_PSI/PhEDEx/FileDownloadDelete -jobs 50 -timeout 600 ============================================================================
node_name
T3_CH_PSI agent_name: Watchdog agent_label: Watchdog
host_name
t3cmsvobox01 process_id: 14644 release: status_update: 2016-12-09 12:35:57 UTC (1481286957.32296) log_update: 2016-12-09 12:35:57 UTC (1481286957.36874) last_update: 0h16 ago
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - log_reason: AGENT RECONNECTED log_process: phedex 14644 0.0 0.3 330632 25648 ? S Dec06 1:16 perl /home/phedex/PHEDEX/Utilities/AgentFactory.pl -state /home/phedex/agents/Prod_T3_CH_PSI/state/Watchdog/ -log /home/phedex/agents/Prod_T3_CH_PSI/logs/Watchdog -db /home/phedex/config/DBParam.PSI:Prod/PSI -node T3_CH_PSI -agent_list exp-pfn -agent_list download-t1 -agent_list download-t2 -agent_list download-remove -agent_list blockverify ============================================================================
node_name
T3_CH_PSI agent_name: Watchdog agent_label: watchdog
host_name
t3cmsvobox01 process_id: 8445 release: status_update: 2016-12-06 20:57:16 UTC (1481057836.51531) log_update: 2016-12-06 20:57:16 UTC (1481057836.56134) last_update: 2d15h55 ago
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - log_reason: AGENT RECONNECTED log_process: phedex 8445 0.0 0.3 330400 25456 ? S Nov24 5:19 perl /home/phedex/PHEDEX/4.1.7/Utilities/AgentFactory.pl -state /home/phedex/state/Prod/incoming/watchdog/ -log /home/phedex/log/Prod/watchdog -db /home/phedex/config/DBParam.PSI:Prod/PSI -node T3_CH_PSI -config /home/phedex/config/SITECONF/T3_CH_PSI/PhEDEx/Config.Prod -agent_list download -agent_list fileexport -agent_list fileremove -agent_list blockverify 2016-12-09 12:55:28: ShowAgents[11056]: disconnected from database
 
Changed:
<
<
<--/twistyPlugin-->

netstat -tp

>
>
See also: https://cmsweb.cern.ch/phedex/datasvc/xml/prod/agents?node=T3_CH_PSI https://cmsweb.cern.ch/phedex/datasvc/json/prod/agents?node=T3_CH_PSI </>
<--/twistyPlugin-->

netstat -tup

 
<--/twistyPlugin twikiMakeVisibleInline-->

Deleted:
<
<
[root@t3cmsvobox01 git]# netstat -tp
 Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
Changed:
<
<
tcp 0 0 t3cmsvobox01.psi.ch:5666 t3nagios.psi.ch:42759 TIME_WAIT - tcp 0 0 t3cmsvobox01.psi.ch:44718 t3service01.p:fujitsu-dtcns ESTABLISHED 1154/syslog-ng tcp 0 0 t3cmsvobox01.psi.ch:33474 t3ldap01.psi.ch:ldaps ESTABLISHED 21373/nslcd tcp 0 0 t3cmsvobox01.psi.ch:36140 itrac50078-v.cern.ch:10121 TIME_WAIT - tcp 0 0 t3cmsvobox01.psi.ch:49704 t3dcachedb03.psi.ch:rootd ESTABLISHED 2084/xrdcp tcp 0 0 t3cmsvobox01.psi.ch:49657 t3dcachedb03.psi.ch:rootd ESTABLISHED 1810/xrdcp tcp 0 0 t3cmsvobox01.psi.ch:vsinet t3fs06.psi.ch:nfs ESTABLISHED - tcp 0 0 t3cmsvobox01.psi.ch:36185 itrac50078-v.cern.ch:10121 ESTABLISHED 30705/perl tcp 0 0 t3cmsvobox01.psi.ch:35926 itrac50078-v.cern.ch:10121 ESTABLISHED 31158/perl tcp 0 0 t3cmsvobox01.psi.ch:36344 itrac50078-v.cern.ch:10121 ESTABLISHED 29777/perl tcp 0 0 t3cmsvobox01.psi.ch:39625 t3se01.psi.ch:pcsync-https TIME_WAIT - tcp 1 0 t3cmsvobox01.psi.ch:57092 t3frontier01.psi.ch:squid CLOSE_WAIT 29416/cvmfs2 tcp 0 0 t3cmsvobox01.psi.ch:36198 itrac50078-v.cern.ch:10121 ESTABLISHED 30198/perl tcp 0 0 t3cmsvobox01.psi.ch:36191 itrac50078-v.cern.ch:10121 ESTABLISHED 29695/perl tcp 1 0 t3cmsvobox01.psi.ch:5666 t3nagios.psi.ch:42436 CLOSE_WAIT 2057/nrpe tcp 0 0 t3cmsvobox01.psi.ch:44651 topbdii04.cern.ch:eyetv TIME_WAIT - tcp 0 0 t3cmsvobox01.psi.ch:40025 t3admin01.psi.ch:4505 ESTABLISHED 21525/python2.6 tcp 0 0 t3cmsvobox01.psi.ch:36196 itrac50078-v.cern.ch:10121 ESTABLISHED 29898/perl tcp 0 0 t3cmsvobox01.psi.ch:57901 topbdii03.cern.ch:eyetv TIME_WAIT - tcp 1 0 t3cmsvobox01.psi.ch:56951 t3frontier01.psi.ch:squid CLOSE_WAIT 29416/cvmfs2 tcp 0 0 t3cmsvobox01.psi.ch:33462 t3ldap01.psi.ch:ldaps ESTABLISHED 21373/nslcd tcp 1 0 t3cmsvobox01.psi.ch:5666 t3nagios.psi.ch:41657 CLOSE_WAIT 1783/nrpe tcp 0 0 t3cmsvobox01.psi.ch:ssh t3admin01.psi.ch:56676 ESTABLISHED 1571/sshd tcp 0 0 t3cmsvobox01.psi.ch:33448 t3ldap01.psi.ch:ldaps ESTABLISHED 21373/nslcd tcp 0 0 t3cmsvobox01.psi.ch:36194 itrac50078-v.cern.ch:10121 ESTABLISHED 30280/perl tcp 0 0 t3cmsvobox01.psi.ch:36075 itrac50078-v.cern.ch:10121 ESTABLISHED 30401/perl tcp 0 0 t3cmsvobox01.psi.ch:54268 topbdii05.cern.ch:eyetv TIME_WAIT - tcp 0 0 t3cmsvobox01.psi.ch:36182 itrac50078-v.cern.ch:10121 ESTABLISHED 30992/perl tcp 0 0 t3cmsvobox01.psi.ch:36098 itrac50078-v.cern.ch:10121 ESTABLISHED 30540/perl tcp 0 0 t3cmsvobox01.psi.ch:33453 t3ldap01.psi.ch:ldaps ESTABLISHED 21373/nslcd tcp 0 0 t3cmsvobox01.psi.ch:33461 t3ldap01.psi.ch:ldaps ESTABLISHED 21373/nslcd tcp 0 0 t3cmsvobox01.psi.ch:56928 topbdii02.cern.ch:eyetv TIME_WAIT - tcp 0 0 t3cmsvobox01.psi.ch:39623 t3se01.psi.ch:pcsync-https TIME_WAIT - tcp 1 0 t3cmsvobox01.psi.ch:56952 t3frontier01.psi.ch:squid CLOSE_WAIT 29416/cvmfs2 tcp 0 0 t3cmsvobox01.psi.ch:56555 t3fs13.psi.ch:gsiftp TIME_WAIT - tcp 0 0 t3cmsvobox01.psi.ch:52096 t3admin01.psi.ch:4506 ESTABLISHED 21525/python2.6
>
>
tcp 0 0 t3cmsvobox01.psi.ch:57184 itrac50063-v.cern.ch:10121 ESTABLISHED 14833/perl tcp 0 0 t3cmsvobox01.psi.ch:ssh t3admin01.psi.ch:52419 ESTABLISHED 1055/sshd tcp 0 0 t3cmsvobox01.psi.ch:bacnet itrac50011-v.cern.ch:10121 ESTABLISHED 14243/perl tcp 0 0 t3cmsvobox01.psi.ch:58600 t3admin01.psi.ch:4505 ESTABLISHED 6089/python2.6 tcp 0 0 t3cmsvobox01.psi.ch:47870 itrac50011-v.cern.ch:10121 ESTABLISHED 11097/perl tcp 0 0 t3cmsvobox01.psi.ch:46866 itrac50011-v.cern.ch:10121 ESTABLISHED 14377/perl tcp 0 0 t3cmsvobox01.psi.ch:57228 itrac50063-v.cern.ch:10121 ESTABLISHED 13831/perl tcp 0 0 t3cmsvobox01.psi.ch:ssh Fabios-MBP.psi.ch:49951 ESTABLISHED 11102/sshd tcp 0 0 t3cmsvobox01.psi.ch:42984 t3ldap01.psi.ch:ldaps ESTABLISHED 1116/nslcd tcp 0 0 t3cmsvobox01.psi.ch:57252 itrac50063-v.cern.ch:10121 ESTABLISHED 15285/perl tcp 0 0 t3cmsvobox01.psi.ch:817 t3nfs01.psi.ch:nfs ESTABLISHED - tcp 0 0 t3cmsvobox01.psi.ch:43985 t3service01.p:fujitsu-dtcns ESTABLISHED 1131/syslog-ng tcp 0 0 t3cmsvobox01.psi.ch:57256 itrac50063-v.cern.ch:10121 ESTABLISHED 13764/perl tcp 0 0 t3cmsvobox01.psi.ch:45624 t3ldap01.psi.ch:ldaps ESTABLISHED 1116/nslcd tcp 0 0 t3cmsvobox01.psi.ch:46862 itrac50011-v.cern.ch:10121 ESTABLISHED 14644/perl tcp 1 0 t3cmsvobox01.psi.ch:45726 t3frontier01.psi.ch:squid CLOSE_WAIT 12339/cvmfs2 tcp 0 0 t3cmsvobox01.psi.ch:57262 itrac50063-v.cern.ch:10121 ESTABLISHED 14766/perl tcp 0 0 t3cmsvobox01.psi.ch:56330 itrac50063-v.cern.ch:10121 ESTABLISHED 14165/perl tcp 0 0 t3cmsvobox01.psi.ch:42978 t3ldap01.psi.ch:ldaps ESTABLISHED 1116/nslcd tcp 0 0 t3cmsvobox01.psi.ch:42982 t3ldap01.psi.ch:ldaps ESTABLISHED 1116/nslcd tcp 1 0 t3cmsvobox01.psi.ch:39324 t3frontier01.psi.ch:squid CLOSE_WAIT 12339/cvmfs2 tcp 0 0 t3cmsvobox01.psi.ch:47852 itrac50011-v.cern.ch:10121 ESTABLISHED 14310/perl tcp 0 0 t3cmsvobox01.psi.ch:ssh Fabios-MBP.psi.ch:64149 ESTABLISHED 7543/sshd tcp 1 0 t3cmsvobox01.psi.ch:39322 t3frontier01.psi.ch:squid CLOSE_WAIT 12339/cvmfs2 tcp 0 0 t3cmsvobox01.psi.ch:45614 t3ldap01.psi.ch:ldaps ESTABLISHED 1116/nslcd udp 0 0 t3cmsvobox01.psi.ch:51950 t3mon01.psi.ch:8649 ESTABLISHED 5997/gmond udp 0 0 t3cmsvobox01.psi.ch:34702 t3ossec.psi.c:fujitsu-dtcns ESTABLISHED 6131/ossec-agentd
 
<--/twistyPlugin-->

Checking each CMS pool by Nagios through both the t3se01:SRM and t3dcachedb:Xrootd dCache doors

Line: 540 to 1283
  </>
<--/twistyPlugin-->
Added:
>
>

Pitfalls in dcache-srmclient-2.10.7-1 ( currently the latest dcache-srmclient )

Strangely PhEDEx has a strong dependency on dcache-srmclient ; by strong we mean that you can't use equivalent SRM tools like lcg-cp or gfal-copy ; in its latest version, Fabio noticed that :
srmcp as in dcache-srmclient-2.2.4-2.el6.x86_64 had, by default, -delegate=true
srmcp as in dcache-srmclient-2.10.7-1.noarch has now, by default, -delegate=false 
Paul Millar ( a primary dCache Dev ) commented in this way :
srmcp tries to avoid the wall-clock time and CPU overhead of delegation if that delegation isn't necessary. 
Unfortunately, there is a bug: the copyjobfile ( used by PhEDEx ) option is not consulted when determining 
whether third-party transfers are involved. The consequence is that all such transfers are considered 
second-party and no delegation is done.
This bug badly affects PhEDEx ; due to it a working PhEDEx/dcache-srmclient-2.2.4-2 configuration will stop to work by simply migrating to PhEDEx/dcache-srmclient-2.10.7-1.noarch and you'll get ( cryptic ) errors like :
21 Apr 2015 07:11:13 (SRM-t3se01) [192.33.123.205:52205 VI8:439841:srm2:copy:-2098574001]
  failed to connect to srm://storage01.lcg.cscs.ch:8443/srm/managerv2?SFN=/pnfs/lcg.cscs.ch/cms/trivcat/store/mc/RunIIWinter15GS/RSGravToWW_kMpl01_M-2000_TuneCUETP8M1_13TeV-pythia8/GEN-SIM/MCRUN2_71_V1-v1/30000/AACEC97E-11B0-E411-9245-001E68862A32.root
     credential remaining lifetime is less then a minute
Fabio fixed this by explicitly requesting -delegate=true to bypass the current copyjob bug :
[root@t3cmsvobox01 PhEDEx]# grep -Hn srmcp /home/phedex/config/SITECONF/T3_CH_PSI/PhEDEx/ConfigPart* | grep -v \#
/home/phedex/config/SITECONF/T3_CH_PSI/PhEDEx/ConfigPart.DebugServices:13:   -command     srmcp,-delegate=true,-pushmode=true,-debug=true,-retry_num=2,-protocols=gsiftp,-srm_protocol_version=2,-streams_num=1,-globus_tcp_port_range=20000:25000
/home/phedex/config/SITECONF/T3_CH_PSI/PhEDEx/ConfigPart.Standard:13:   -command     srmcp,-delegate=true,-pushmode=true,-debug=true,-retry_num=2,-protocols=gsiftp,-srm_protocol_version=2,-streams_num=1,-globus_tcp_port_range=20000:25000
Fabio noticed another bug again in dcache-srmclient-2.10.7-1 where the default proxy location /tmp/x509up_u`id -u` is considered even if we explicitly specify the option -x509_user_proxy to use a different path :
Dear Paul and dCache colleagues, I believe I've found another bug in dcache-srmclient-2.10.7-1.noarch
$ srmls -debug=false -x509_user_proxy=/home/phedex/gridcert/proxy.cert -retry_num=0 'srm://t3se01.psi.ch:8443/srm/managerv2?SFN=/pnfs/psi.ch/cms/trivcat/store/mc/RunIIWinter15GS/RSGravToWWToLNQQ_kMpl01_M-4000_TuneCUETP8M1_13TeV-pythia8/GEN-SIM/MCRUN2_71_V1-v1/10000/2898A22B-62B0-E411-B1D4-002590D600EE.root'
srm client error:
java.lang.IllegalArgumentException: Multiple entries with same key:
x509_user_proxy=/home/phedex/gridcert/proxy.cert and
x509_user_proxy=/tmp/x509up_u205 
Fabio fixed it by tweaking the following PhEDEx scripts :
[root@t3cmsvobox01 PhEDEx]# grep -Hn export /home/phedex/config/SITECONF/T3_CH_PSI/PhEDEx/FileDownload* --color
/home/phedex/config/SITECONF/T3_CH_PSI/PhEDEx/FileDownloadDelete:14:   export X509_USER_PROXY=/home/phedex/gridcert/proxy.cert && srmrm -retry_num=0 "$pfn";
/home/phedex/config/SITECONF/T3_CH_PSI/PhEDEx/FileDownloadSRMVerify:31:    *managerv2* ) echo $(export X509_USER_PROXY=/home/phedex/gridcert/proxy.cert && srmls -debug=false -retry_num=0 "$path" 2>/dev/null| grep $file | cut -d\  -f3);;
/home/phedex/config/SITECONF/T3_CH_PSI/PhEDEx/FileDownloadSRMVerify:44:        fields=($(export X509_USER_PROXY=/home/phedex/gridcert/proxy.cert && srmls -l -debug=false -retry_num=0 "$pfn" 2>/dev/null| grep Checksum))
/home/phedex/config/SITECONF/T3_CH_PSI/PhEDEx/FileDownloadSRMVerify:116:        *managerv2*) export X509_USER_PROXY=/home/phedex/gridcert/proxy.cert && srmrm -retry_num=0 "$pfn";;
 

Backups

OS snapshots are nightly taken by the PSI VMWare Team ( contact Peter Huesser ) + we have LinuxBackupsByLegato to recover a single file.

Added:
>
>
You might also want to exploit t3nfs02:/data01/backups
 
META FORM name="NodeTypeForm"
FORM FIELD Hostnames Hostnames t3cmsvobox ( t3cmsvobox01 )
Changed:
<
<
FORM FIELD Services Services PhEDEx 4.1.7
>
>
FORM FIELD Services Services PhEDEx 4.2.1
 
FORM FIELD Hardware Hardware PSI DMZ VMWare cluster
FORM FIELD Install Profile InstallProfile vobox
FORM FIELD Guarantee/maintenance until Guaranteemaintenanceuntil VMWare PSI Cluster

Revision 482016-11-08 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 472016-09-08 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 462016-06-14 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 452016-05-31 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 442016-02-28 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Changed:
<
<
-bash-4.1$ pstree -uh phedex -la perl /home/phedex/PHEDEX/Toolkit/Transfer/FileDownload -state /home/phedex/state/Debug/incoming/download/ -log /home/phedex/log/Debug/download -verbose -db /home/phedex/config/DBParam.PSI:Debug/PSI -nodesT3_CH perl /home/phedex/PHEDEX/Toolkit/Transfer/FileExport -state /home/phedex/state/Debug/incoming/fileexport/ -log /home/phedex/log/Debug/fileexport -db /home/phedex/config/DBParam.PSI:Debug/PSI -nodes T3_CH_PSI-s perl /home/phedex/PHEDEX/Toolkit/Transfer/FileRemove -state /home/phedex/state/Debug/incoming/fileremove/ -log /home/phedex/log/Debug/fileremove -node T3_CH_PSI -db /home/phedex/config/DBParam.PSI:Debug/PSI-pr perl /home/phedex/PHEDEX/Toolkit/Verify/BlockDownloadVerify -state /home/phedex/state/Debug/incoming/blockverify/ -log /home/phedex/log/Debug/blockverify --db /home/phedex/config/DBParam.PSI:Debug/PSI --nodesT perl /home/phedex/PHEDEX/Utilities/AgentFactory.pl -state /home/phedex/state/Debug/incoming/watchdog/ -log /home/phedex/log/Debug/watchdog -db /home/phedex/config/DBParam.PSI:Debug/PSI -node T3_CH_PSI -config/ perl /home/phedex/PHEDEX/Utilities/AgentFactoryLite.pl -state /home/phedex/state/Debug/incoming/WatchdogLite/ -log /home/phedex/log/Debug/WatchdogLite -node T3_CH_PSI -agent_list watchdog

perl /home/phedex/PHEDEX/Toolkit/Transfer/FileDownload -state /home/phedex/state/Dev/incoming/download/ -log /home/phedex/log/Dev/download -verbose -db /home/phedex/config/DBParam.PSI:Dev/PSI -nodes T3_CH_PSI- perl /home/phedex/PHEDEX/Toolkit/Transfer/FileExport -state /home/phedex/state/Dev/incoming/fileexport/ -log /home/phedex/log/Dev/fileexport -db /home/phedex/config/DBParam.PSI:Dev/PSI -nodes T3_CH_PSI-storage perl /home/phedex/PHEDEX/Toolkit/Transfer/FileRemove -state /home/phedex/state/Dev/incoming/fileremove/ -log /home/phedex/log/Dev/fileremove -node T3_CH_PSI -db /home/phedex/config/DBParam.PSI:Dev/PSI-protocol perl /home/phedex/PHEDEX/Toolkit/Verify/BlockDownloadVerify -state /home/phedex/state/Dev/incoming/blockverify/ -log /home/phedex/log/Dev/blockverify --db /home/phedex/config/DBParam.PSI:Dev/PSI --nodesT3_CH_P perl /home/phedex/PHEDEX/Utilities/AgentFactory.pl -state /home/phedex/state/Dev/incoming/watchdog/ -log /home/phedex/log/Dev/watchdog -db /home/phedex/config/DBParam.PSI:Dev/PSI -node T3_CH_PSI -config/home/p perl /home/phedex/PHEDEX/Utilities/AgentFactoryLite.pl -state /home/phedex/state/Dev/incoming/WatchdogLite/ -log /home/phedex/log/Dev/WatchdogLite -node T3_CH_PSI -agent_list watchdog

perl /home/phedex/PHEDEX/Toolkit/Transfer/FileDownload -state /home/phedex/state/Prod/incoming/download/ -log /home/phedex/log/Prod/download -verbose -db /home/phedex/config/DBParam.PSI:Prod/PSI -nodesT3_CH_PS perl /home/phedex/PHEDEX/Toolkit/Transfer/FileExport -state /home/phedex/state/Prod/incoming/fileexport/ -log /home/phedex/log/Prod/fileexport -db /home/phedex/config/DBParam.PSI:Prod/PSI -nodes T3_CH_PSI-stor perl /home/phedex/PHEDEX/Toolkit/Transfer/FileRemove -state /home/phedex/state/Prod/incoming/fileremove/ -log /home/phedex/log/Prod/fileremove -node T3_CH_PSI -db /home/phedex/config/DBParam.PSI:Prod/PSI-proto perl /home/phedex/PHEDEX/Toolkit/Verify/BlockDownloadVerify -state /home/phedex/state/Prod/incoming/blockverify/ -log /home/phedex/log/Prod/blockverify --db /home/phedex/config/DBParam.PSI:Prod/PSI --nodesT3_C perl /home/phedex/PHEDEX/Utilities/AgentFactory.pl -state /home/phedex/state/Prod/incoming/watchdog/ -log /home/phedex/log/Prod/watchdog -db /home/phedex/config/DBParam.PSI:Prod/PSI -node T3_CH_PSI -config/hom perl /home/phedex/PHEDEX/Utilities/AgentFactoryLite.pl -state /home/phedex/state/Prod/incoming/WatchdogLite/ -log /home/phedex/log/Prod/WatchdogLite -node T3_CH_PSI -agent_list watchdog

>
>
current phedex status perl /home/phedex/PHEDEX/4.1.7/Toolkit/Transfer/FileDownload -state /home/phedex/state/Debug/incoming/download/ -log /home/phedex/log/Debug/download -verbose -db/home/phed perl /home/phedex/PHEDEX/4.1.7/Toolkit/Transfer/FileExport -state /home/phedex/state/Debug/incoming/fileexport/ -log /home/phedex/log/Debug/fileexport -db/home/phedex/conf perl /home/phedex/PHEDEX/4.1.7/Toolkit/Transfer/FileRemove -state /home/phedex/state/Debug/incoming/fileremove/ -log /home/phedex/log/Debug/fileremove -node T3_CH_PSI -db/ perl /home/phedex/PHEDEX/4.1.7/Toolkit/Verify/BlockDownloadVerify -state /home/phedex/state/Debug/incoming/blockverify/ -log /home/phedex/log/Debug/blockverify --db/home/p perl /home/phedex/PHEDEX/4.1.7/Utilities/AgentFactory.pl -state /home/phedex/state/Debug/incoming/watchdog/ -log /home/phedex/log/Debug/watchdog -db/home/phedex/config/DBP perl /home/phedex/PHEDEX/4.1.7/Utilities/AgentFactoryLite.pl -state /home/phedex/state/Debug/incoming/WatchdogLite/ -log /home/phedex/log/Debug/WatchdogLite -nodeT3_CH_PSI perl /home/phedex/PHEDEX/4.1.7/Toolkit/Transfer/FileDownload -state /home/phedex/state/Dev/incoming/download/ -log /home/phedex/log/Dev/download -verbose -db/home/phedex/c perl /home/phedex/PHEDEX/4.1.7/Toolkit/Transfer/FileExport -state /home/phedex/state/Dev/incoming/fileexport/ -log /home/phedex/log/Dev/fileexport -db/home/phedex/config/D perl /home/phedex/PHEDEX/4.1.7/Toolkit/Transfer/FileRemove -state /home/phedex/state/Dev/incoming/fileremove/ -log /home/phedex/log/Dev/fileremove -node T3_CH_PSI -db/home perl /home/phedex/PHEDEX/4.1.7/Toolkit/Verify/BlockDownloadVerify -state /home/phedex/state/Dev/incoming/blockverify/ -log /home/phedex/log/Dev/blockverify --db/home/phede perl /home/phedex/PHEDEX/4.1.7/Utilities/AgentFactory.pl -state /home/phedex/state/Dev/incoming/watchdog/ -log /home/phedex/log/Dev/watchdog -db/home/phedex/config/DBParam perl /home/phedex/PHEDEX/4.1.7/Utilities/AgentFactoryLite.pl -state /home/phedex/state/Dev/incoming/WatchdogLite/ -log /home/phedex/log/Dev/WatchdogLite -node T3_CH_PSI-ag perl /home/phedex/PHEDEX/4.1.7/Toolkit/Transfer/FileDownload -state /home/phedex/state/Prod/incoming/download/ -log /home/phedex/log/Prod/download -verbose -db/home/phedex perl /home/phedex/PHEDEX/4.1.7/Toolkit/Transfer/FileExport -state /home/phedex/state/Prod/incoming/fileexport/ -log /home/phedex/log/Prod/fileexport -db/home/phedex/config perl /home/phedex/PHEDEX/4.1.7/Toolkit/Transfer/FileRemove -state /home/phedex/state/Prod/incoming/fileremove/ -log /home/phedex/log/Prod/fileremove -node T3_CH_PSI -db/ho perl /home/phedex/PHEDEX/4.1.7/Toolkit/Verify/BlockDownloadVerify -state /home/phedex/state/Prod/incoming/blockverify/ -log /home/phedex/log/Prod/blockverify --db/home/phe perl /home/phedex/PHEDEX/4.1.7/Utilities/AgentFactory.pl -state /home/phedex/state/Prod/incoming/watchdog/ -log /home/phedex/log/Prod/watchdog -db/home/phedex/config/DBPar perl /home/phedex/PHEDEX/4.1.7/Utilities/AgentFactoryLite.pl -state /home/phedex/state/Prod/incoming/WatchdogLite/ -log /home/phedex/log/Prod/WatchdogLite -node T3_CH_PSI-
 bash
Added:
>
>
└─bash
  └─pstree -uh phedex -la
Added:
>
>
TRANSFER LOGs ================================================================= source /home/phedex/PHEDEX/4.1.7/etc/profile.d/env.sh /home/phedex/PHEDEX/4.1.7/Utilities/InspectPhedexLog -es "-1 hours" -d /home/phedex/log/Prod/download /home/phedex/log/Debug/download /home/phedex/PHEDEX/4.1.7/Utilities/InspectPhedexLog -es "-1 days" -d /home/phedex/log/Prod/download /home/phedex/log/Debug/download

How to update ========= https://twiki.cern.ch/twiki/bin/view/CMSPublic/PhedexAdminDocsInstallation#Updating_Software

How to test /home/phedex/config/SITECONF/T3_CH_PSI/PhEDEx/storage.xml ================================================================= source /home/phedex/PHEDEX/4.1.7/etc/profile.d/env.sh /home/phedex/PHEDEX/4.1.7/Utilities/TestCatalogue -c /home/phedex/config/SITECONF/T3_CH_PSI/PhEDEx/storage.xml -p srmv2 -L /store/data/file

StorageConsistencyCheck example =========================== source /home/phedex/PHEDEX/4.1.7/etc/profile.d/env.sh /home/phedex/PHEDEX/4.1.7/Utilities/StorageConsistencyCheck -db /home/phedex/config/DBParam.PSI:Prod/PSI -lfnlist /home/phedex/PSI.lfnlist.txt -node T3_CH_PSI /home/phedex/PHEDEX/4.1.7/Utilities/StorageConsistencyCheck -db /home/phedex/config/DBParam.PSI:Prod/PSI -lfnlist /home/phedex/CSCS.lfnlist.txt -node T2_CH_CSCS [root@t3dcachedb03 ~]# psql -U nagios -d chimera -c " select path from v_pnfs where path like '/pnfs/psi.ch/cms%' ; " -t -q -o ./PSI.txt <-------- to get the LFN

PSI agents as perceived from the CERN DBs ===================================== source /home/phedex/PHEDEX/4.1.7/etc/profile.d/env.sh /home/phedex/PHEDEX/4.1.7/Utilities/ShowAgents -db /home/phedex/config/DBParam.PSI:Prod/PSI -node T3_CH_PSI https://cmsweb.cern.ch/phedex/datasvc/xml/prod/agents?node=T3_CH_PSI https://cmsweb.cern.ch/phedex/datasvc/json/prod/agents?node=T3_CH_PSI

PSI Prod Datasets ===================================== https://cmsweb.cern.ch/phedex/prod/Data::Replicas?view=global&rcolumn=Name&nvalue=Node+files&rows=interesting&dbs=21&dbs=1&dbs=41&node=821&filter=.*

PSI Prod Errors ===================================== https://cmsweb.cern.ch/phedex/prod/Activity::ErrorInfo?from_pfn=.%2A;fromfilter=.%2A;log_detail=.%2A;log_validate=.%2A;xfer_code=.%2A;tofilter=T3_CH_PSI;to_pfn=.%2A;report_code=.%2A https://cmsweb.cern.ch/phedex/datasvc/xml/prod/errorlog?to=T3_CH_PSI

HOW TO IDENTIFY THE DATASETS TO BE DELETED AT PSI AND CSCS ====================================================== https://wiki.chipp.ch/twiki/bin/view/CmsTier3/CmsVoBox#Dataset_cleaning <-- HowTo Doc source /home/phedex/PHEDEX/4.1.7/etc/profile.d/env.sh cd svn-sandbox/phedex/DB-query-tools/ source /home/phedex/PHEDEX/4.1.7/etc/profile.d/env.sh ./ListSiteDataInfo.pl -w -t --db ~/config/DBParam.PSI:Prod/PSI -s "%CSCS%" | grep "eleted" echo ./ListSiteDataInfo.pl -w -t --db ~/config/DBParam.PSI:Prod/PSI -s "%CSCS%" | grep -vE "Dutta|Fanfani|Kress|Magini|Wuerthwein|Belforte|Spinoso|Ajit|DataOps|eleted|StoreResults|Argiro|Klute|Cremonesi|Jean-Roch Vlimant|vocms[0-9]+|cmsgwms-submit[0-9]+|IntelROCCS|retention time: 2016|Retention date: 2016" echo ./ListSiteDataInfo.pl -w -t --db ~/config/DBParam.PSI:Prod/PSI -s "%PSI%" | grep -Ev "retention time: 2016|Retention date: 2016" https://wiki.chipp.ch/twiki/bin/view/CmsTier3/DataSetCleaningQuery <-- where to publish the ListSiteDataInfo.pl outputs

be aware that if a data sets appeared, disappeared and appeared again ListSiteDataInfo.pl as it's written today will report the 1st old occurrence, so old user and old retention time

 
<--/twistyPlugin-->

netstat -tp

[root@t3cmsvobox01 git]# netstat -tp 
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   

Changed:
<
<
tcp 0 0 t3cmsvobox01.psi.ch:42088 itrac50063-v.cern.ch:10121 ESTABLISHED 23470/perl tcp 0 0 t3cmsvobox01.psi.ch:34154 itrac50063-v.cern.ch:10121 ESTABLISHED 22967/perl tcp 0 0 t3cmsvobox01.psi.ch:38659 t3ldap01.psi.ch:ldaps ESTABLISHED 5581/nslcd tcp 0 0 t3cmsvobox01.psi.ch:38662 t3ldap01.psi.ch:ldaps ESTABLISHED 5581/nslcd tcp 0 0 t3cmsvobox01.psi.ch:43269 itrac50063-v.cern.ch:10121 ESTABLISHED 23973/perl tcp 0 0 t3cmsvobox01.psi.ch:41053 itrac50063-v.cern.ch:10121 ESTABLISHED 22846/perl tcp 0 0 t3cmsvobox01.psi.ch:43886 t3admin01.psi.ch:4505 ESTABLISHED 9353/python2.6 <-- salt minion tcp 0 0 t3cmsvobox01.psi.ch:40990 itrac50063-v.cern.ch:10121 ESTABLISHED 23267/perl tcp 0 0 t3cmsvobox01.psi.ch:51930 t3service01.p:fujitsu-dtcns ESTABLISHED 1224/syslog-ng tcp 1 0 t3cmsvobox01.psi.ch:39198 t3frontier01.psi.ch:squid CLOSE_WAIT 2530/cvmfs2 tcp 0 0 t3cmsvobox01.psi.ch:41978 itrac50063-v.cern.ch:10121 ESTABLISHED 23770/perl tcp 1 0 t3cmsvobox01.psi.ch:55127 t3frontier01.psi.ch:squid CLOSE_WAIT 2530/cvmfs2 tcp 0 0 t3cmsvobox01.psi.ch:38663 t3ldap01.psi.ch:ldaps ESTABLISHED 5581/nslcd tcp 0 0 t3cmsvobox01.psi.ch:733 t3fs06.psi.ch:nfs ESTABLISHED - tcp 0 0 t3cmsvobox01.psi.ch:41150 itrac50063-v.cern.ch:10121 ESTABLISHED 23852/perl tcp 0 0 t3cmsvobox01.psi.ch:42399 itrac50063-v.cern.ch:10121 ESTABLISHED 22764/perl tcp 0 0 t3cmsvobox01.psi.ch:38660 t3ldap01.psi.ch:ldaps ESTABLISHED 5581/nslcd tcp 0 0 t3cmsvobox01.psi.ch:41061 t3admin01.psi.ch:4506 ESTABLISHED 9353/python2.6 <-- salt minion tcp 0 0 t3cmsvobox01.psi.ch:38674 t3ldap01.psi.ch:ldaps ESTABLISHED 5581/nslcd tcp 0 0 t3cmsvobox01.psi.ch:41821 itrac50063-v.cern.ch:10121 ESTABLISHED 23349/perl
>
>
tcp 0 0 t3cmsvobox01.psi.ch:5666 t3nagios.psi.ch:42759 TIME_WAIT - tcp 0 0 t3cmsvobox01.psi.ch:44718 t3service01.p:fujitsu-dtcns ESTABLISHED 1154/syslog-ng tcp 0 0 t3cmsvobox01.psi.ch:33474 t3ldap01.psi.ch:ldaps ESTABLISHED 21373/nslcd tcp 0 0 t3cmsvobox01.psi.ch:36140 itrac50078-v.cern.ch:10121 TIME_WAIT - tcp 0 0 t3cmsvobox01.psi.ch:49704 t3dcachedb03.psi.ch:rootd ESTABLISHED 2084/xrdcp tcp 0 0 t3cmsvobox01.psi.ch:49657 t3dcachedb03.psi.ch:rootd ESTABLISHED 1810/xrdcp tcp 0 0 t3cmsvobox01.psi.ch:vsinet t3fs06.psi.ch:nfs ESTABLISHED - tcp 0 0 t3cmsvobox01.psi.ch:36185 itrac50078-v.cern.ch:10121 ESTABLISHED 30705/perl tcp 0 0 t3cmsvobox01.psi.ch:35926 itrac50078-v.cern.ch:10121 ESTABLISHED 31158/perl tcp 0 0 t3cmsvobox01.psi.ch:36344 itrac50078-v.cern.ch:10121 ESTABLISHED 29777/perl tcp 0 0 t3cmsvobox01.psi.ch:39625 t3se01.psi.ch:pcsync-https TIME_WAIT - tcp 1 0 t3cmsvobox01.psi.ch:57092 t3frontier01.psi.ch:squid CLOSE_WAIT 29416/cvmfs2 tcp 0 0 t3cmsvobox01.psi.ch:36198 itrac50078-v.cern.ch:10121 ESTABLISHED 30198/perl tcp 0 0 t3cmsvobox01.psi.ch:36191 itrac50078-v.cern.ch:10121 ESTABLISHED 29695/perl tcp 1 0 t3cmsvobox01.psi.ch:5666 t3nagios.psi.ch:42436 CLOSE_WAIT 2057/nrpe tcp 0 0 t3cmsvobox01.psi.ch:44651 topbdii04.cern.ch:eyetv TIME_WAIT - tcp 0 0 t3cmsvobox01.psi.ch:40025 t3admin01.psi.ch:4505 ESTABLISHED 21525/python2.6 tcp 0 0 t3cmsvobox01.psi.ch:36196 itrac50078-v.cern.ch:10121 ESTABLISHED 29898/perl tcp 0 0 t3cmsvobox01.psi.ch:57901 topbdii03.cern.ch:eyetv TIME_WAIT - tcp 1 0 t3cmsvobox01.psi.ch:56951 t3frontier01.psi.ch:squid CLOSE_WAIT 29416/cvmfs2 tcp 0 0 t3cmsvobox01.psi.ch:33462 t3ldap01.psi.ch:ldaps ESTABLISHED 21373/nslcd tcp 1 0 t3cmsvobox01.psi.ch:5666 t3nagios.psi.ch:41657 CLOSE_WAIT 1783/nrpe tcp 0 0 t3cmsvobox01.psi.ch:ssh t3admin01.psi.ch:56676 ESTABLISHED 1571/sshd tcp 0 0 t3cmsvobox01.psi.ch:33448 t3ldap01.psi.ch:ldaps ESTABLISHED 21373/nslcd tcp 0 0 t3cmsvobox01.psi.ch:36194 itrac50078-v.cern.ch:10121 ESTABLISHED 30280/perl tcp 0 0 t3cmsvobox01.psi.ch:36075 itrac50078-v.cern.ch:10121 ESTABLISHED 30401/perl tcp 0 0 t3cmsvobox01.psi.ch:54268 topbdii05.cern.ch:eyetv TIME_WAIT - tcp 0 0 t3cmsvobox01.psi.ch:36182 itrac50078-v.cern.ch:10121 ESTABLISHED 30992/perl tcp 0 0 t3cmsvobox01.psi.ch:36098 itrac50078-v.cern.ch:10121 ESTABLISHED 30540/perl tcp 0 0 t3cmsvobox01.psi.ch:33453 t3ldap01.psi.ch:ldaps ESTABLISHED 21373/nslcd tcp 0 0 t3cmsvobox01.psi.ch:33461 t3ldap01.psi.ch:ldaps ESTABLISHED 21373/nslcd tcp 0 0 t3cmsvobox01.psi.ch:56928 topbdii02.cern.ch:eyetv TIME_WAIT - tcp 0 0 t3cmsvobox01.psi.ch:39623 t3se01.psi.ch:pcsync-https TIME_WAIT - tcp 1 0 t3cmsvobox01.psi.ch:56952 t3frontier01.psi.ch:squid CLOSE_WAIT 29416/cvmfs2 tcp 0 0 t3cmsvobox01.psi.ch:56555 t3fs13.psi.ch:gsiftp TIME_WAIT - tcp 0 0 t3cmsvobox01.psi.ch:52096 t3admin01.psi.ch:4506 ESTABLISHED 21525/python2.6
 
Changed:
<
<

Checking each CMS pool by Nagios through both t3se01:SRM and t3dcachedb:Xrootd

>
>

Checking each CMS pool by Nagios through both the t3se01:SRM and t3dcachedb:Xrootd dCache doors

 
Changed:
<
<
By t3cmsvobox in turn contacted by t3nagios we retrieve a file from each CMS pool through both t3se01:SRM and t3dcachedb:Xrootd
>
>
By t3cmsvobox , in turn contacted by t3nagios , we retrieve a file from each CMS pool through both t3se01:SRM and t3dcachedb:Xrootd
 https://t3nagios.psi.ch/check_mk/index.py?start_url=%2Fcheck_mk%2Fview.py%3Fview_name%3Dhost%26host%3Dt3cmsvobox01%26site%3D

In both the cases the test files retrieved are :

Revision 432016-01-28 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 422016-01-28 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 412016-01-21 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 402016-01-21 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 392015-11-19 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 382015-11-16 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Services

Changed:
<
<

/home/phedex/phedex_start.sh

To be manually invoked following a server restart !

/home/phedex/phedex_stop.sh

/home/phedex/phedex_status.sh

>
>

/home/phedex/phedex_start.sh

To be manually invoked after a server restart !

/home/phedex/phedex_stop.sh

/home/phedex/phedex_status.sh

 
<--
 #List all the important services, their installation, configuration and how to start and stop them
-->
Line: 360 to 360
  └─pstree -uh phedex -la

</>

<--/twistyPlugin-->
Changed:
<
<

netstat -tp

>
>

netstat -tp

 
[root@t3cmsvobox01 git]# netstat -tp 
Active Internet connections (w/o servers)

Line: 386 to 386
 tcp 0 0 t3cmsvobox01.psi.ch:38674 t3ldap01.psi.ch:ldaps ESTABLISHED 5581/nslcd tcp 0 0 t3cmsvobox01.psi.ch:41821 itrac50063-v.cern.ch:10121 ESTABLISHED 23349/perl
Changed:
<
<

Nagios checks each CMS pool through both t3se01:SRM and t3dcachedb:Xrootd

>
>

Checking each CMS pool by Nagios through both t3se01:SRM and t3dcachedb:Xrootd

 
Changed:
<
<
By t3cmsvobox in turn contacted by t3nagios we read each CMS pool through both t3se01 SRM and t3dcachedb Xrootd
>
>
By t3cmsvobox in turn contacted by t3nagios we retrieve a file from each CMS pool through both t3se01:SRM and t3dcachedb:Xrootd
 https://t3nagios.psi.ch/check_mk/index.py?start_url=%2Fcheck_mk%2Fview.py%3Fview_name%3Dhost%26host%3Dt3cmsvobox01%26site%3D
Changed:
<
<
In both the cases the tests files are :
>
>
In both the cases the test files retrieved are :
 
[martinelli_f@t3ui12 ~]$ find /pnfs/psi.ch/cms/t3-nagios/ | grep M | sort
/pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs01_cms
/pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs02_cms

Line: 399 to 399
 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_9
Changed:
<
<
The related dCache files have to be obviously placed on the right CMS pool or the Nagios tests will be wrong! To easily check where are really placed run:

More... Close

<--/twistyPlugin twikiMakeVisibleInline-->
>
>
The related dCache files have to be obviously placed on the right CMS pool otherwise the Nagios tests will be wrong ! To easily check where they are really placed run this SQL code ( in this example some test files are erroneously available in the wrong pool ! that was due to a bad migration cache command ) More... Close
<--/twistyPlugin twikiMakeVisibleInline-->
 

Changed:
<
<
$ find /pnfs/psi.ch/cms/t3-nagios/ | grep M | sort | dc_get_ID_from_pnfsnamelist.sh | dc_get_cacheinfo_from_IDlist.sh | xargs -iG echo [\'G\'],| sed s/' '/\',\'/| xargs -iG echo G \\ [00020000000000000330BF90,t3fs01_cms], [00020000000000000330BF10,t3fs02_cms], [000200000000000006391F88,t3fs03_cms], [00020000000000000395B300,t3fs04_cms], [0002000000000000063D8C68,t3fs04_cms_1], [0000119DDCFD0C5F42B89769BC9C104A997F,t3fs07_cms], [00003A989AB6D1074D738594B1D01E2D03DE,t3fs08_cms], [0000479ED8FDDC374BC68827AEDF1C146686,t3fs09_cms], [000005D1DD24CA14448694E5C46A8AA8E91F,t3fs10_cms], [00009E4A9774085C4799B5C9C827DA03406F,t3fs11_cms], [0002000000000000052EEFB8,t3fs13_cms], [00005560491A76DE49DBA142D3BE3CFE38D5,t3fs13_cms_0], [00000CB9E97140F940CD973C319045B43FDA,t3fs13_cms_1], [0000073FF4F754BB4AB1B4599F412811BDA2,t3fs13_cms_10], [00000DB49D5B69EB4C568834BD162C3DA8E7,t3fs13_cms_11], [0002000000000000052EF018,t3fs13_cms_2], [0002000000000000052EF048,t3fs13_cms_3], [0002000000000000052EF078,t3fs13_cms_4], [0002000000000000052EF0A8,t3fs13_cms_5], [0002000000000000072F8630,t3fs13_cms_6], [00003FC27733ACBA4A809677419256FE22F9,t3fs13_cms_7], [0000132841305C27434891574015FD2CF923,t3fs13_cms_8], [00004783F9158A5941B284342FF4A8EDE126,t3fs13_cms_9], [0002000000000000052EF0D8,t3fs14_cms], [0000A6470E0458354BD99D6C2DD27B196DCC,t3fs14_cms_0], [0002000000000000052EF108,t3fs14_cms_1], [0000B3D6A96EF961473AACB05F80CF9D6892,t3fs14_cms_10], [00003616229002194F439925DA3C7F1CFA02,t3fs14_cms_11], [0002000000000000052EF138,t3fs14_cms_2], [0002000000000000052EF168,t3fs14_cms_3], [0002000000000000052EF198,t3fs14_cms_4], [000200000000000006300738,t3fs14_cms_5], [0000E88C6CBB2D5A4365B11BE2EDD1554366,t3fs14_cms_6], [000018AA61C1E30F43709F0D9FE3B9CD65D1,t3fs14_cms_7], [00001A2FD52D31DB4CCAB99C8B8336522339,t3fs14_cms_8], [0000B58A7FA17778439F8F6F47C5CBBED5E7,t3fs14_cms_9],
>
>
[root@t3dcachedb03 ~]# psql -U nagios -d chimera -c " select path,ipnfsid,pools from v_pnfs where path like '%1MB-test-file_pool_%' ; " path | ipnfsid | pools

--------------------------------------+------------------------------------

/pnfs/psi.ch/dteam/t3-nagios/1MB-test-file_pool_t3fs09_ops | 0000BCDA4B329DA94D64AAAFE7C0C7501E5C | t3fs09_ops /pnfs/psi.ch/dteam/t3-nagios/1MB-test-file_pool_t3fs08_ops | 0000358B14867ED5402184C2C22F81EFC861 | t3fs08_ops /pnfs/psi.ch/dteam/t3-nagios/1MB-test-file_pool_t3fs07_ops | 0000409BB804C95944A38DBE8220B416A8A3 | t3fs07_ops /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_9 | 0000B58A7FA17778439F8F6F47C5CBBED5E7 | t3fs03_cms t3fs11_cms t3fs14_cms_9 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_8 | 00001A2FD52D31DB4CCAB99C8B8336522339 | t3fs09_cms t3fs11_cms t3fs14_cms_8 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_7 | 000018AA61C1E30F43709F0D9FE3B9CD65D1 | t3fs03_cms t3fs14_cms_7 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_6 | 0000E88C6CBB2D5A4365B11BE2EDD1554366 | t3fs02_cms t3fs14_cms_6 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_5 | 000200000000000006300738 | t3fs10_cms t3fs14_cms_5 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_4 | 0002000000000000052EF198 | t3fs03_cms t3fs14_cms_4 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_3 | 0002000000000000052EF168 | t3fs03_cms t3fs14_cms_3 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_2 | 0002000000000000052EF138 | t3fs07_cms t3fs14_cms_2 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_11 | 00003616229002194F439925DA3C7F1CFA02 | t3fs14_cms_11 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_10 | 0000B3D6A96EF961473AACB05F80CF9D6892 | t3fs14_cms_10 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_1 | 0002000000000000052EF108 | t3fs02_cms t3fs11_cms t3fs14_cms_1 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_0 | 0000A6470E0458354BD99D6C2DD27B196DCC | t3fs08_cms t3fs14_cms_0 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms | 0002000000000000052EF0D8 | t3fs03_cms t3fs04_cms t3fs14_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_9 | 00004783F9158A5941B284342FF4A8EDE126 | t3fs08_cms t3fs13_cms_9 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_8 | 0000132841305C27434891574015FD2CF923 | t3fs09_cms t3fs13_cms_8 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_7 | 00003FC27733ACBA4A809677419256FE22F9 | t3fs02_cms t3fs11_cms t3fs13_cms_7 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_6 | 0002000000000000072F8630 | t3fs07_cms t3fs11_cms t3fs13_cms_6 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_5 | 0002000000000000052EF0A8 | t3fs03_cms t3fs13_cms_5 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_4 | 0002000000000000052EF078 | t3fs10_cms t3fs11_cms t3fs13_cms_4 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_3 | 0002000000000000052EF048 | t3fs10_cms t3fs13_cms_3 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_2 | 0002000000000000052EF018 | t3fs02_cms t3fs13_cms_2 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_11 | 00000DB49D5B69EB4C568834BD162C3DA8E7 | t3fs13_cms_11 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_10 | 0000073FF4F754BB4AB1B4599F412811BDA2 | t3fs13_cms_10 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_1 | 00000CB9E97140F940CD973C319045B43FDA | t3fs04_cms t3fs11_cms t3fs13_cms_1 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_0 | 00005560491A76DE49DBA142D3BE3CFE38D5 | t3fs02_cms t3fs11_cms t3fs13_cms_0 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms | 0002000000000000052EEFB8 | t3fs07_cms t3fs11_cms t3fs13_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs11_cms | 00009E4A9774085C4799B5C9C827DA03406F | t3fs11_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs10_cms | 000005D1DD24CA14448694E5C46A8AA8E91F | t3fs10_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs09_cms | 0000479ED8FDDC374BC68827AEDF1C146686 | t3fs09_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs08_cms | 00003A989AB6D1074D738594B1D01E2D03DE | t3fs08_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs07_cms | 0000119DDCFD0C5F42B89769BC9C104A997F | t3fs07_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs04_cms_1 | 0002000000000000063D8C68 | t3fs04_cms_1 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs04_cms | 00020000000000000395B300 | t3fs04_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs03_cms | 000200000000000006391F88 | t3fs03_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs02_cms | 00020000000000000330BF10 | t3fs02_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs01_cms | 00020000000000000330BF90 | t3fs01_cms
 
<--/twistyPlugin-->

Revision 372015-06-04 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 362015-06-01 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 352015-05-28 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"
Deleted:
<
<

Installation

Look the Puppet file /afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/manifests/nodes/SL6_vobox.pp

<--
 #Comment here on any peculiarities of the installation, e.g. on special packages needed, special setup
 #procedures which are not obvious
-->

add the following package to run our custom "accounting"-scripts:

yum install perl-XML-Twig 
 

Services

Changed:
<
<
>
>

/home/phedex/phedex_start.sh

To be manually invoked following a server restart !

/home/phedex/phedex_stop.sh

/home/phedex/phedex_status.sh

 
<--
 #List all the important services, their installation, configuration and how to start and stop them
-->
Changed:
<
<

PhEDEx

Refer to the description on the Tier-2 VOBox.

There is one important difference: while we use FTS channels for the transfers to the Tier-2 we use the SRM backend for transfers to the Tier-3, because we do not have a FTS channel for PSI. This issue is linked to registering PSI as a regular grid site, which until recently was not possible, since we only support a Grid SE, but no a CE.

>
>
-bash-4.1$ pstree -uh phedex -la
perl /home/phedex/PHEDEX/Toolkit/Transfer/FileDownload -state /home/phedex/state/Debug/incoming/download/ -log /home/phedex/log/Debug/download -verbose -db /home/phedex/config/DBParam.PSI:Debug/PSI -nodesT3_CH
perl /home/phedex/PHEDEX/Toolkit/Transfer/FileExport -state /home/phedex/state/Debug/incoming/fileexport/ -log /home/phedex/log/Debug/fileexport -db /home/phedex/config/DBParam.PSI:Debug/PSI -nodes T3_CH_PSI-s
perl /home/phedex/PHEDEX/Toolkit/Transfer/FileRemove -state /home/phedex/state/Debug/incoming/fileremove/ -log /home/phedex/log/Debug/fileremove -node T3_CH_PSI -db /home/phedex/config/DBParam.PSI:Debug/PSI-pr
perl /home/phedex/PHEDEX/Toolkit/Verify/BlockDownloadVerify -state /home/phedex/state/Debug/incoming/blockverify/ -log /home/phedex/log/Debug/blockverify --db /home/phedex/config/DBParam.PSI:Debug/PSI --nodesT
perl /home/phedex/PHEDEX/Utilities/AgentFactory.pl -state /home/phedex/state/Debug/incoming/watchdog/ -log /home/phedex/log/Debug/watchdog -db /home/phedex/config/DBParam.PSI:Debug/PSI -node T3_CH_PSI -config/
perl /home/phedex/PHEDEX/Utilities/AgentFactoryLite.pl -state /home/phedex/state/Debug/incoming/WatchdogLite/ -log /home/phedex/log/Debug/WatchdogLite -node T3_CH_PSI -agent_list watchdog

perl /home/phedex/PHEDEX/Toolkit/Transfer/FileDownload -state /home/phedex/state/Dev/incoming/download/ -log /home/phedex/log/Dev/download -verbose -db /home/phedex/config/DBParam.PSI:Dev/PSI -nodes T3_CH_PSI-
perl /home/phedex/PHEDEX/Toolkit/Transfer/FileExport -state /home/phedex/state/Dev/incoming/fileexport/ -log /home/phedex/log/Dev/fileexport -db /home/phedex/config/DBParam.PSI:Dev/PSI -nodes T3_CH_PSI-storage
perl /home/phedex/PHEDEX/Toolkit/Transfer/FileRemove -state /home/phedex/state/Dev/incoming/fileremove/ -log /home/phedex/log/Dev/fileremove -node T3_CH_PSI -db /home/phedex/config/DBParam.PSI:Dev/PSI-protocol
perl /home/phedex/PHEDEX/Toolkit/Verify/BlockDownloadVerify -state /home/phedex/state/Dev/incoming/blockverify/ -log /home/phedex/log/Dev/blockverify --db /home/phedex/config/DBParam.PSI:Dev/PSI --nodesT3_CH_P
perl /home/phedex/PHEDEX/Utilities/AgentFactory.pl -state /home/phedex/state/Dev/incoming/watchdog/ -log /home/phedex/log/Dev/watchdog -db /home/phedex/config/DBParam.PSI:Dev/PSI -node T3_CH_PSI -config/home/p
perl /home/phedex/PHEDEX/Utilities/AgentFactoryLite.pl -state /home/phedex/state/Dev/incoming/WatchdogLite/ -log /home/phedex/log/Dev/WatchdogLite -node T3_CH_PSI -agent_list watchdog

perl /home/phedex/PHEDEX/Toolkit/Transfer/FileDownload -state /home/phedex/state/Prod/incoming/download/ -log /home/phedex/log/Prod/download -verbose -db /home/phedex/config/DBParam.PSI:Prod/PSI -nodesT3_CH_PS
perl /home/phedex/PHEDEX/Toolkit/Transfer/FileExport -state /home/phedex/state/Prod/incoming/fileexport/ -log /home/phedex/log/Prod/fileexport -db /home/phedex/config/DBParam.PSI:Prod/PSI -nodes T3_CH_PSI-stor
perl /home/phedex/PHEDEX/Toolkit/Transfer/FileRemove -state /home/phedex/state/Prod/incoming/fileremove/ -log /home/phedex/log/Prod/fileremove -node T3_CH_PSI -db /home/phedex/config/DBParam.PSI:Prod/PSI-proto
perl /home/phedex/PHEDEX/Toolkit/Verify/BlockDownloadVerify -state /home/phedex/state/Prod/incoming/blockverify/ -log /home/phedex/log/Prod/blockverify --db /home/phedex/config/DBParam.PSI:Prod/PSI --nodesT3_C
perl /home/phedex/PHEDEX/Utilities/AgentFactory.pl -state /home/phedex/state/Prod/incoming/watchdog/ -log /home/phedex/log/Prod/watchdog -db /home/phedex/config/DBParam.PSI:Prod/PSI -node T3_CH_PSI -config/hom
perl /home/phedex/PHEDEX/Utilities/AgentFactoryLite.pl -state /home/phedex/state/Prod/incoming/WatchdogLite/ -log /home/phedex/log/Prod/WatchdogLite -node T3_CH_PSI -agent_list watchdog
bash
  └─pstree -uh phedex -la

 
Changed:
<
<
Thus there is no fts.map file in the configuration area for the PhEDEx services.

Nagios checking each T3 pool by both t3se01 SRM and t3dcachedb Xrootd protocols

>
>
<--/twistyPlugin-->

netstat -tp

[root@t3cmsvobox01 git]# netstat -tp 
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 t3cmsvobox01.psi.ch:42088   itrac50063-v.cern.ch:10121  ESTABLISHED 23470/perl          
tcp        0      0 t3cmsvobox01.psi.ch:34154   itrac50063-v.cern.ch:10121  ESTABLISHED 22967/perl          
tcp        0      0 t3cmsvobox01.psi.ch:38659   t3ldap01.psi.ch:ldaps       ESTABLISHED 5581/nslcd                    
tcp        0      0 t3cmsvobox01.psi.ch:38662   t3ldap01.psi.ch:ldaps       ESTABLISHED 5581/nslcd          
tcp        0      0 t3cmsvobox01.psi.ch:43269   itrac50063-v.cern.ch:10121  ESTABLISHED 23973/perl          
tcp        0      0 t3cmsvobox01.psi.ch:41053   itrac50063-v.cern.ch:10121  ESTABLISHED 22846/perl          
tcp        0      0 t3cmsvobox01.psi.ch:43886   t3admin01.psi.ch:4505       ESTABLISHED 9353/python2.6      <-- salt minion
tcp        0      0 t3cmsvobox01.psi.ch:40990   itrac50063-v.cern.ch:10121  ESTABLISHED 23267/perl          
tcp        0      0 t3cmsvobox01.psi.ch:51930   t3service01.p:fujitsu-dtcns ESTABLISHED 1224/syslog-ng      
tcp        1      0 t3cmsvobox01.psi.ch:39198   t3frontier01.psi.ch:squid   CLOSE_WAIT  2530/cvmfs2         
tcp        0      0 t3cmsvobox01.psi.ch:41978   itrac50063-v.cern.ch:10121  ESTABLISHED 23770/perl          
tcp        1      0 t3cmsvobox01.psi.ch:55127   t3frontier01.psi.ch:squid   CLOSE_WAIT  2530/cvmfs2         
tcp        0      0 t3cmsvobox01.psi.ch:38663   t3ldap01.psi.ch:ldaps       ESTABLISHED 5581/nslcd          
tcp        0      0 t3cmsvobox01.psi.ch:733     t3fs06.psi.ch:nfs           ESTABLISHED -                   
tcp        0      0 t3cmsvobox01.psi.ch:41150   itrac50063-v.cern.ch:10121  ESTABLISHED 23852/perl          
tcp        0      0 t3cmsvobox01.psi.ch:42399   itrac50063-v.cern.ch:10121  ESTABLISHED 22764/perl          
tcp        0      0 t3cmsvobox01.psi.ch:38660   t3ldap01.psi.ch:ldaps       ESTABLISHED 5581/nslcd          
tcp        0      0 t3cmsvobox01.psi.ch:41061   t3admin01.psi.ch:4506       ESTABLISHED 9353/python2.6      <-- salt minion      
tcp        0      0 t3cmsvobox01.psi.ch:38674   t3ldap01.psi.ch:ldaps       ESTABLISHED 5581/nslcd          
tcp        0      0 t3cmsvobox01.psi.ch:41821   itrac50063-v.cern.ch:10121  ESTABLISHED 23349/perl  

Nagios checks each T3 pool by t3se01:SRM and t3dcachedb:Xrootd

 
Changed:
<
<
By t3cmsvobox we check each T3 pool twice, by t3se01 SRM and by t3dcachedb Xrootd ; look https://t3nagios.psi.ch/nagios/cgi-bin/status.cgi?navbarsearch=1&host=t3cmsvobox
>
>
By t3cmsvobox + t3nagios we read each T3 pool twice, by t3se01 SRM and by t3dcachedb Xrootd ; look https://t3nagios.psi.ch/check_mk/index.py?start_url=%2Fcheck_mk%2Fview.py%3Fview_name%3Dhost%26host%3Dt3cmsvobox01%26site%3D
 
Changed:
<
<
In both the cases the file names retrieved by the tests are :
>
>
In both the cases the probe files read by the tests are :
 
[martinelli_f@t3ui02 ~]$ find /pnfs/psi.ch/cms/t3-nagios/ | grep M | sort
/pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs01_cms
/pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs02_cms

Changed:
<
<
/pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs03_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs04_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs04_cms_1 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs07_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs08_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs09_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs10_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs11_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_0 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_1 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_10 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_11 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_2 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_3 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_4 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_5 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_6 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_7 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_8 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs13_cms_9 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_0 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_1 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_10 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_11 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_2 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_3 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_4 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_5 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_6 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_7 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_8
>
>
...
 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_9
Changed:
<
<
The related dCache files have to be placed in the right pool or the Nagios tests will be obviously wrong! You can easily check on which pool are stored the dCache files by :
>
>
The related dCache files have to be placed on the right pool or the Nagios tests will be obviously wrong! You can easily check on which pool are stored the dCache probe files by :
 
$ find /pnfs/psi.ch/cms/t3-nagios/ | grep M | sort | dc_get_ID_from_pnfsnamelist.sh | dc_get_cacheinfo_from_IDlist.sh | xargs -iG echo [\'G\'],| sed s/' '/\',\'/| xargs -iG echo G \\

Deleted:
<
<
 [00020000000000000330BF90,t3fs01_cms], [00020000000000000330BF10,t3fs02_cms], [000200000000000006391F88,t3fs03_cms],

Revision 342015-05-27 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 332015-05-27 - DanielMeister

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 322015-05-27 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 312015-05-27 - DanielMeister

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 302015-05-27 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 292015-05-27 - DanielMeister

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"
Deleted:
<
<
 

Installation

Changed:
<
<
Look the Puppet file /afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/manifests/nodes/SL6_vobox.pp
Line: 225 to 215
  Thus there is no fts.map file in the configuration area for the PhEDEx services.

Nagios checking each T3 pool by both t3se01 SRM and t3dcachedb Xrootd protocols

Added:
>
>
 By t3cmsvobox we check each T3 pool twice, by t3se01 SRM and by t3dcachedb Xrootd ; look https://t3nagios.psi.ch/nagios/cgi-bin/status.cgi?navbarsearch=1&host=t3cmsvobox

In both the cases the file names retrieved by the tests are :

Changed:
<
<
[martinelli_f@t3ui02 ~]$ find /pnfs/psi.ch/cms/t3-nagios/ | grep M | sort

>
>
[martinelli_f@t3ui02 ~]$ find /pnfs/psi.ch/cms/t3-nagios/ | grep M | sort

 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs01_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs02_cms /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs03_cms
Line: 267 to 257
 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_8 /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_9
Added:
>
>
 The related dCache files have to be placed in the right pool or the Nagios tests will be obviously wrong! You can easily check on which pool are stored the dCache files by :
Changed:
<
<
$ find /pnfs/psi.ch/cms/t3-nagios/ | grep M | sort | dc_get_ID_from_pnfsnamelist.sh | dc_get_cacheinfo_from_IDlist.sh | xargs -iG echo [\'G\'],| sed s/' '/\',\'/| xargs -iG echo G \\

>
>
$ find /pnfs/psi.ch/cms/t3-nagios/ | grep M | sort | dc_get_ID_from_pnfsnamelist.sh | dc_get_cacheinfo_from_IDlist.sh | xargs -iG echo [\'G\'],| sed s/' '/\',\'/| xargs -iG echo G \\

  [00020000000000000330BF90,t3fs01_cms], [00020000000000000330BF10,t3fs02_cms],

Revision 282015-05-27 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 272015-05-27 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 262015-04-23 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 252015-04-14 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 242015-03-24 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 232014-08-26 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 222014-06-23 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 212014-04-14 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 202014-04-11 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 192014-04-10 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 182014-01-10 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 172013-11-20 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 162013-09-18 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 152013-06-17 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 142012-11-25 - DanielMeister

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"
Added:
>
>
add the following package to run our custom "accounting"-scripts:
 
Added:
>
>
yum install perl-XML-Twig 
 

Services

Revision 132012-11-19 - DanielMeister

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 122012-11-12 - DanielMeister

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 112012-11-06 - FabioMartinelli

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 102012-10-19 - DanielMeister

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 92012-07-11 - LeonardoSala

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 82010-06-17 - DerekFeichtinger

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 72010-01-07 - DerekFeichtinger

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 62009-12-22 - DerekFeichtinger

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 52009-11-18 - DerekFeichtinger

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Node Type: CmsVoBox

Revision 42009-11-02 - DerekFeichtinger

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 32009-06-26 - DerekFeichtinger

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"

Revision 22009-01-21 - DerekFeichtinger

Line: 1 to 1
 
META TOPICPARENT name="AdminArea"
Added:
>
>
 

Services

<--
 #List all the important services, their installation, configuration and how to start and stop them
-->
Added:
>
>

PhEDEx

Refer to the description on the Tier-2 VOBox.

There is one important difference: While we use FTS channels for the transfers to the Tier-2, we use the SRM backend for transfers to the Tier-3, because we do not have a FTS channel for PSI. This issue is linked to registering PSI as a regular grid site, which until recently was not possible, since we only sport a Grid SE, but no CE.

So, there is no fts.map file in the configuration area for the PhEDEx services.

 

Backups

Revision 12009-01-19 - DerekFeichtinger

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="AdminArea"
<-- keep this as a security measure:
   #uncomment if the subject should only be modifiable by the listed groups  
  • Set ALLOWTOPICCHANGE = TWikiAdminGroup,Main.CMSAdminGroup
  • Set ALLOWTOPICRENAME = TWikiAdminGroup,Main.CMSAdminGroup #uncomment this if you want the page only be viewable by the listed groups # * Set ALLOWTOPICVIEW = TWikiAdminGroup,Main.CMSAdminGroup
-->

Node Type: CmsVoBox

Firewall requirements

local port open to reason
<-- Example line
 #| 22/tcp      | * |  Example entry for ssh |
-->


Table of contents

Regular Maintenance work

Renew myproxy certificate for PhEDEx transfers

 
voms-proxy-init -voms cms
myproxyserver=myproxy.cern.ch
myproxy-init -s $myproxyserver -l psi_phedex -x      -R "/C=CH/O=Paul-Scherrer-Institut (PSI)/OU=AIT/CN=t3cmsvobox.psi.ch/Email=cms-tier3@psi.ch" -c 720
scp ~/.x509up_u$(id -u) phedex@t3ui01:gridcert/proxy.cert
#  for testing, you can try
myproxy-info -s $myproxyserver -l psi_phedex

As the phedex user do

chmod 600 ~/gridcert/proxy.cert

You should test whether the renewal of the certificate works for the phedex user: unset X509_USER_PROXY # make sure that the service credentials from ~/.globus are used!

voms-proxy-init  # initializes the service proxy cert that is allowed to retrieve the user cert
myproxyserver=myproxy.cern.ch
myproxy-get-delegation -s $myproxyserver -v -l psi_phedex             -a /home/phedex/gridcert/proxy.cert -o /tmp/gagatest

export X509_USER_PROXY=/tmp/gagatest
srm-get-metadata srm://t3se01.psi.ch:8443/srm/managerv1?SFN=/pnfs/psi.ch/cms
rm /tmp/gagatest

Emergency Measures

<--
 #List any measures that must be taken in case of some major incident, e.g. whether a mailing
 #list must be contacted or whether other services need to be shut down, etc. 
-->

Installation

<--
 #Comment here on any peculiarities of the installation, e.g. on special packages needed, special setup
 #procedures which are not obvious
-->

Services

<--
 #List all the important services, their installation, configuration and how to start and stop them
-->

Backups

-- DerekFeichtinger - 19 Jan 2009

META FORM name="NodeTypeForm"
FORM FIELD Hostnames Hostnames t3ui01
FORM FIELD Services Services PhEDEx
FORM FIELD Hardware Hardware SUN X4150, 1*Xeon E5410, 8GB RAM, 2*146 GB SAS disk
FORM FIELD Install Profile InstallProfile none
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback