Tags:
tag this topic
create new tag
view all tags
Site logs for CMS PSI Tier3
CMSTier3Log77
06. 07. 2016 dCache new SRM limits got enforced
CMSTier3Log76
15. 06. 2016 Major upgrades performed on t3nfs02
CMSTier3Log75
2016 Derek's handover - 1st meeting
CMSTier3Log74
09. 06. 2016 dCache 2.15 stuck on t3se01
CMSTier3Log73
15. 04. 2016 Old /swshare dirs to be removed
CMSTier3Log71
03. 05. 2015 Son of Grid Engine 8.1.8 cpuset error and fix
CMSTier3Log69
30. 04. 2015 17TB of files read last time before 01-01-2012
CMSTier3Log70
24. 03. 2015 ./check_http_json.py
CMSTier3Log68
13. 03. 2015 t3wn[30-40] RAM errors
EDCA RAM errors in one server
EDAC RAM analysis
CMSTier3Log67
20. 11. 2014 t3fs06 RPC error
Nagios warning
RPC check
NFS check
Mounting ZFS filesystems: (1/8)cannot mount '/swshare2': directory is not empty
zfs list ( to check /swshare2 )
zfs properties of /swshare2
Solution
CMSTier3Log66
11. 10. 2014 T3 VMs rebooted
CMSTier3Log65
29. 09. 2014 CMS dcaps average MB/s at T3
CMSTier3Log63
19. 03. 2014 t3fs05 unresponsive
Symptoms
Solution
Lessons Learned
CMSTier3Log62
10. 03. 2014 Lost Sensor n.60 on each SUN Thor fileserver
/opt/nagios/check_ipmi_sensor Nagios invocation
/usr/sbin/ipmi-sensors direct invocation
CMSTier3Log61
04. 03. 2014 t3vmui01 lcg-cp 18GB file Not OK / OK
GGUS Ticket and solution
Not OK Site T3_CH_PSI
Not OK Site T2_CH_CSCS
OK Site T3_CH_PSI
OK Site T2_CH_CSCS
CMSTier3Log60
14. 02. 2014 Missing Billing DB and Billing logs entries about t3fs14 files
CMSTier3Log59
28. 01. 2014 A Python Pandas + Postgresql example
CMSTier3Log58
14. 01. 2014 dCache 2.6.19 pool - (Too many open files) error
CMSTier3Log57
20. 12. 2013 Installing Puppet 2.7.21 on each T3 Solaris server ver. 10/13
CMSTier3Log56
10. 12. 2013 dCache Authentication failed: Certificates does not conform to algorithm constraints
CMSTier3Log55
27. 10. 2013 t3fs05 swap full
CMSTier3Log54
17. 09. 2013 t3fs07 hangs because of a broken disk
CMSTier3Log53
10. 09. 2013 SGI IS5500SP broken drawer
CMSTier3Log52
05. 09. 2013 Evacuating t3fs13_cms_1 to get it from 22TB to 17TB
CMSTier3Log51
05. 09. 2013 Solaris t3fs* disks catalogue
Totals
t3fs07
t3fs08
t3fs09
t3fs10
t3fs11
CMSTier3Log50
04. 09. 2013 Introduced CERN/REDHAT devtools-1.1
CMSTier3Log49
22. 08. 2013 Updated dcap binaries on SL5 UIs and WNs
CMSTier3Log48
22. 08. 2013 Swapping our first broken disk in the E5460 enclosure
CMSTier3Log43
21. 08. 2013 t3fs13 dCache pools found unresponsive
CMSTier3Log47
11. 06. 2013 Constantly more than 250 PG DB connections
CMSTier3Log46
04. 06. 2013 t3dcachedb03 again frozen
probable cause
Fabio's e-mail vs Peter ( VMWare Manager )
t3se01-Domain-srm.log relevant logs
CMSTier3Log45
31.05.2013 VOMS Server Issue
Problem
Solution
CMSTier3Log44
16. 05. 2013 t3dcachedb03 frozen
CMSTier3Log42
CMSTier3Log41
04. 03. 2013 SRM Error: Already have 1 record(s) with pnfsPath
Other instances of the same issue
CMSTier3Log38
03. 01. 2013 dCache file deletion
File Deletion Test
Conclusion
Update from 2013-02-11
CMSTier3Log37
28. 12. 2012 t3fs07,t3fs08 went down
CMSTier3Log36
26. 12. 2012 t3fs14 reboot on Dec 25th
CMSTier3Log35
19. 12. 2012 Largest directories via Chimera
New version
Outdated since May 2014
CMSTier3Log34
13. 12. 2012 SMARTd configuration change requested after an Hitachi 1TB swap
CMSTier3Log32
Problems with myproxy renewal from PSI vobox and CSCS vobox
04. 02. 2016
04. 12. 2012
CMSTier3Log31
29. 11. 2012 146GB disks to be absorbed by UIs
CMSTier3Log30
16. 11. 2012 Restarted pnfsd
CMSTier3Log26
30. 07. 2012 Dealing with the fallout from the new memory limits
Comparing crab and non-crab jobs
Looking at the number of crab jobs below and above a 3GB threshold
CMSTier3Log25
26. 07. 2012 Enforcing flexible memory limits on SGE
Change proposal
Some logs collected during the change
CMSTier3Log22
Tests between t3fs13 and t3fs14 to test the pure 10GbE connectivity
Tests between t3fs13, t3fs14 and 10+10 1Gbit/s clients
CMSTier3Log21
22. 02. 2012 Qlogic FW update on t3fs13,14 + SGI case about RDAC
FW update
SGI Case 2904405
CMSTier3Log17
25. 10. 2011 Workernode to fileserver throughput. About 8MB/s per job
CMSTier3Log16
15. 01. 2011 Three breakdowns of t3ui01 in 2 days
CMSTier3Log15
06. 07. 2010 Worker nodes reading from two Thors at about 800 MB/s
Mail from Lukas Baeni describing his jobs
Monitoring graphs
CMSTier3Log14
28. 06. 2010 Bandwidth measurement between CSCS and PSI Tier-3
CMSTier3Log13
11. 06. 2010 Second breakdown of t3ui01 in 7 days
CMSTier3Log12
01. 06. 2010 Implementing the ZFS incremental snapshot backup
The naive send/recv approach has terrible performance
Establishing a baseline throughput by having receiver dump the stream to /dev/null
Doing full snapshot transfers with mbuffer + zfs send/recv
Incremental snapshot transfer
CMSTier3Log11
migration commands
some monitoring information
Possible Disk problem on t3fs07
Checking the migration by hand
CMSTier3Log10
10. 03. 2010 Testing the SEs
File server setup
Choosing the cfg
Setting up the test
Cfg
Results
CMSTier3Log9
04. 02. 2010 SunBlade X620 Burning tests (wn10-19)
Test bed
How To Run
CMSTier3Log8
28. 01. 2010 Test of data transfer latency at PSI
CMSTier3Log7
29. 06. 2009
29.06.2009 Home was nearly full. Phedex instances was down
CMSTier3Log6
25. 05. 2009 OS patching and reconfiguration of some X4500s
t3fs05
t3fs01 unkillable java process problem and patching
CMSTier3Log5
25. 02. 2009 Worker Node to Fileserver write throughput
13. 05. 2009 PhEDEx download throughput example
CMSTier3Log4
25. 11. 2008 Thumper Fileserver t3fs03 problems
15. 12. 2008 Again t3fs03 problems - OS patching
CMSTier3Log3
Log: CRAB+SGE
Task: Merging modified code to CRAB 2.4.1
CMSTier3Log2
16. 10. 2008 Out of memory problem impacting t3wn06
CMSTier3Log1
06. 10. 2008 Test user feedback
Feedback from FredericRonga:
Feedback from ChristinaEggel:
CMSTier3Log0
25. 09. 2008 Testing of PhEDEx service for the PSI Tier-3
Error Mode: retrieval of "from" TURL failed
Transfers from CSCS
--
DerekFeichtinger
- 25 Sep 2008
E
dit
|
A
ttach
|
Watch
|
P
rint version
|
H
istory
: r2
<
r1
|
B
acklinks
|
R
aw View
|
Ra
w
edit
|
M
ore topic actions
Topic revision: r2 - 2008-09-25
-
DerekFeichtinger
CmsTier3
Log In
CmsTier3 Web
Create New Topic
Index
Search
Changes
Notifications
Statistics
Preferences
User Pages
Main Page
Policies
Monitoring Storage Space
Monitoring Slurm Usage
Physics Groups
Steering Board Meetings
Admin Pages
AdminArea
Cluster Specs
Home
Site map
CmsTier3 web
LCGTier2 web
PhaseC web
Main web
Sandbox web
TWiki web
CmsTier3 Web
Create New Topic
Index
Search
Changes
Notifications
RSS Feed
Statistics
Preferences
View
Raw View
Print version
Find backlinks
History
More topic actions
Edit
Raw edit
Attach file or image
Edit topic preference settings
Set new parent
More topic actions
Account
Log In
E
dit
A
ttach
Copyright © 2008-2025 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki?
Send feedback