Tags:
dcache
1
tag this topic
create new tag
view all tags
<!-- keep this as a security measure: #uncomment if the subject should only be modifiable by the listed groups * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.CMSAdminGroup * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.CMSAdminGroup #uncomment this if you want the page only be viewable by the listed groups # * Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.CMSAdminGroup --> %TOC% %ICON{arrowleft}% Go to [[CMSTier3LogXX][previous page]] / [[CMSTier3LogXX][next page]] of Tier3 site log %M% ---+ 14. 01. 2014 dCache 2.6.19 pool - (Too many open files) error %TWISTY% <pre> 13 Jan 2014 21:47:57 (t3fs13_cms_1) [door:GFTP-t3se01-<unknown>-828@t3se01-Domain-gridftp:1389646013190 GFTP-t3se01-<unknown>-828 PoolAcceptFile 0000908132F5C10A47138311C55D0D65B9E3] Transfer failed in post-processing: File size mismatch (expected=449396194, actual=0) 13 Jan 2014 21:52:38 (t3fs13_cms_10) [door:GFTP-t3se01-<unknown>-832@t3se01-Domain-gridftp:1389646168695 GFTP-t3se01-<unknown>-832 PoolAcceptFile 00004B3E7FC1C63E48D4A96542481803E980] Marking pool entry as BROKEN 13 Jan 2014 21:52:38 (t3fs13_cms_10) [door:GFTP-t3se01-<unknown>-832@t3se01-Domain-gridftp:1389646168695 GFTP-t3se01-<unknown>-832 PoolAcceptFile 00004B3E7FC1C63E48D4A96542481803E980] Transfer failed in post-processing: File size mismatch (expected=429375613, actual=0) 13 Jan 2014 22:02:00 (t3fs13_cms_11) [door:GFTP-t3se01-<unknown>-839@t3se01-Domain-gridftp:1389646857277 GFTP-t3se01-<unknown>-839 PoolAcceptFile 00008E02111C4F9545AD9FA073FE1C97D5B6] Marking pool entry as BROKEN 13 Jan 2014 22:02:01 (t3fs13_cms_11) [door:GFTP-t3se01-<unknown>-839@t3se01-Domain-gridftp:1389646857277 GFTP-t3se01-<unknown>-839 PoolAcceptFile 00008E02111C4F9545AD9FA073FE1C97D5B6] Transfer failed in post-processing: File size mismatch (expected=440561453, actual=0) 13 Jan 2014 22:21:54 (t3fs13_cms) [door:GFTP-t3se01-<unknown>-852@t3se01-Domain-gridftp:1389648050261 GFTP-t3se01-<unknown>-852 PoolAcceptFile 00009A55DDCAFCF34471A2EB02FE6B1E7FC5] Marking pool entry as BROKEN 13 Jan 2014 22:21:54 (t3fs13_cms) [door:GFTP-t3se01-<unknown>-852@t3se01-Domain-gridftp:1389648050261 GFTP-t3se01-<unknown>-852 PoolAcceptFile 00009A55DDCAFCF34471A2EB02FE6B1E7FC5] Transfer failed in post-processing: File size mismatch (expected=447047203, actual=0) 13 Jan 2014 22:25:11 (t3fs13_cms_11) [door:GFTP-t3se01-<unknown>-854@t3se01-Domain-gridftp:1389648121173 GFTP-t3se01-<unknown>-854 PoolAcceptFile 00008DFFA57177A24524A43AF52A8EFEC32E] Marking pool entry as BROKEN 13 Jan 2014 22:25:11 (t3fs13_cms_11) [door:GFTP-t3se01-<unknown>-854@t3se01-Domain-gridftp:1389648121173 GFTP-t3se01-<unknown>-854 PoolAcceptFile 00008DFFA57177A24524A43AF52A8EFEC32E] Transfer failed in post-processing: File size mismatch (expected=439879762, actual=0) 13 Jan 2014 22:26:25 (t3fs13_cms_11) [door:GFTP-t3se01-<unknown>-855@t3se01-Domain-gridftp:1389648321543 GFTP-t3se01-<unknown>-855 PoolAcceptFile 00000A3B69CB1F97498EA53E7B451DB8BEFF] Marking pool entry as BROKEN 13 Jan 2014 22:26:25 (t3fs13_cms_11) [door:GFTP-t3se01-<unknown>-855@t3se01-Domain-gridftp:1389648321543 GFTP-t3se01-<unknown>-855 PoolAcceptFile 00000A3B69CB1F97498EA53E7B451DB8BEFF] Transfer failed in post-processing: File size mismatch (expected=431818865, actual=0) 13 Jan 2014 22:30:26 (t3fs13_cms_11) [door:GFTP-t3se01-<unknown>-860@t3se01-Domain-gridftp:1389648562428 GFTP-t3se01-<unknown>-860 PoolAcceptFile 0000BCA3BABFCCE944A6AD6564821E5705D3] Marking pool entry as BROKEN 13 Jan 2014 22:30:26 (t3fs13_cms_11) [door:GFTP-t3se01-<unknown>-860@t3se01-Domain-gridftp:1389648562428 GFTP-t3se01-<unknown>-860 PoolAcceptFile 0000BCA3BABFCCE944A6AD6564821E5705D3] Transfer failed in post-processing: File size mismatch (expected=441029035, actual=0) 13 Jan 2014 22:31:48 (t3fs13_cms_11) [door:GFTP-t3se01-<unknown>-862@t3se01-Domain-gridftp:1389648644806 GFTP-t3se01-<unknown>-862 PoolAcceptFile 0000AACE0718CD2947AC8A337A9660E2BFA5] Marking pool entry as BROKEN 13 Jan 2014 22:31:48 (t3fs13_cms_11) [door:GFTP-t3se01-<unknown>-862@t3se01-Domain-gridftp:1389648644806 GFTP-t3se01-<unknown>-862 PoolAcceptFile 0000AACE0718CD2947AC8A337A9660E2BFA5] Transfer failed in post-processing: File size mismatch (expected=448837082, actual=0) 13 Jan 2014 22:33:32 (t3fs13_cms_11) [door:GFTP-t3se01-<unknown>-863@t3se01-Domain-gridftp:1389648748351 GFTP-t3se01-<unknown>-863 PoolAcceptFile 0000490A01CD9DF54E4988E312175FE72AED] Marking pool entry as BROKEN 13 Jan 2014 22:33:32 (t3fs13_cms_11) [door:GFTP-t3se01-<unknown>-863@t3se01-Domain-gridftp:1389648748351 GFTP-t3se01-<unknown>-863 PoolAcceptFile 0000490A01CD9DF54E4988E312175FE72AED] Transfer failed in post-processing: File size mismatch (expected=449403392, actual=0) 13 Jan 2014 23:51:34 (t3fs13_cms_7) [door:GFTP-t3se01-<unknown>-878@t3se01-Domain-gridftp:1389653430454 GFTP-t3se01-<unknown>-878 PoolAcceptFile 0000ED99AEBB03054B6DA3EDB74156128F6F] Marking pool entry as BROKEN 13 Jan 2014 23:51:34 (t3fs13_cms_7) [door:GFTP-t3se01-<unknown>-878@t3se01-Domain-gridftp:1389653430454 GFTP-t3se01-<unknown>-878 PoolAcceptFile 0000ED99AEBB03054B6DA3EDB74156128F6F] Transfer failed in post-processing: File size mismatch (expected=609450440, actual=0) 13 Jan 2014 23:57:05 (t3fs13_cms_11) [door:GFTP-t3se01-<unknown>-882@t3se01-Domain-gridftp:1389653761378 GFTP-t3se01-<unknown>-882 PoolAcceptFile 00000E3D503F13984F96B89FF63E9C6B0E47] Marking pool entry as BROKEN 13 Jan 2014 23:57:05 (t3fs13_cms_11) [door:GFTP-t3se01-<unknown>-882@t3se01-Domain-gridftp:1389653761378 GFTP-t3se01-<unknown>-882 PoolAcceptFile 00000E3D503F13984F96B89FF63E9C6B0E47] Transfer failed in post-processing: File size mismatch (expected=608483932, actual=0) 14 Jan 2014 13:33:59 (t3fs13_cms_9) [DCap-t3se01-<unknown>-14638 PoolDeliverFile 0000203E41A81062435D8FA8E4C376739E73] Problem in command block : RequestBlock [Size=0 Code=0 Buffer=java.nio.HeapByteBuffer[pos=0 lim=4 cap=16384] java.io.IOException: Connection reset by peer 14 Jan 2014 13:33:59 (t3fs13_cms_9) [DCap-t3se01-<unknown>-14638 PoolDeliverFile 0000203E41A81062435D8FA8E4C376739E73] Transfer failed: java.io.IOException: Connection reset by peer 14 Jan 2014 13:59:00 (t3fs13_cms_9) [DCap-t3se01-<unknown>-14478 PoolDeliverFile 00007E804B4380294466B052E0D6BDB0F454] Fault occured in repository: File could not be opened [/mnt/data10/t3fs13_cms/pool/data/00007E804B4380294466B052E0D6BDB0F454 (Too many open files)]; please check the file system diskCacheV111.util.DiskErrorCacheException: File could not be opened [/mnt/data10/t3fs13_cms/pool/data/00007E804B4380294466B052E0D6BDB0F454 (Too many open files)]; please check the file system at org.dcache.pool.movers.AbstractMover.openChannel(AbstractMover.java:244) ~[dcache-core-2.6.19.jar:2.6.19] at org.dcache.pool.classic.MoverProtocolTransferService$MoverTask.run(MoverProtocolTransferService.java:154) ~[dcache-core-2.6.19.jar:2.6.19] at org.dcache.util.CDCExecutorServiceDecorator$1.run(CDCExecutorServiceDecorator.java:106) [dcache-core-2.6.19.jar:2.6.19] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_21] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_21] at java.lang.Thread.run(Thread.java:722) [na:1.7.0_21] Caused by: java.io.FileNotFoundException: /mnt/data10/t3fs13_cms/pool/data/00007E804B4380294466B052E0D6BDB0F454 (Too many open files) at java.io.RandomAccessFile.open(Native Method) ~[na:1.7.0_21] at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) ~[na:1.7.0_21] at org.dcache.pool.repository.FileRepositoryChannel.<init>(FileRepositoryChannel.java:85) ~[dcache-core-2.6.19.jar:2.6.19] at org.dcache.pool.movers.AbstractMover.openChannel(AbstractMover.java:242) ~[dcache-core-2.6.19.jar:2.6.19] ... 5 common frames omitted 14 Jan 2014 13:59:00 (t3fs13_cms_9) [DCap-t3se01-<unknown>-14478 PoolDeliverFile 00007E804B4380294466B052E0D6BDB0F454] Pool mode changed to disabled(fetch,store,stage,p2p-client,p2p-server): Pool disabled: File could not be opened [/mnt/data10/t3fs13_cms/pool/data/00007E804B4380294466B052E0D6BDB0F454 (Too many open files)]; please check the file system 14 Jan 2014 13:59:00 (t3fs13_cms_9) [DCap-t3se01-<unknown>-14478 PoolDeliverFile 00007E804B4380294466B052E0D6BDB0F454] Transfer failed due to a disk error: CacheException(rc=204;msg=File could not be opened [/mnt/data10/t3fs13_cms/pool/data/00007E804B4380294466B052E0D6BDB0F454 (Too many open files)]; please check the file system) 14 Jan 2014 13:59:05 (t3fs13_cms_9) [PinManager PoolSetSticky 00006B85783E25084626B927566FAF45A2DE] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server) 14 Jan 2014 13:59:05 (t3fs13_cms_9) [PinManager PoolSetSticky 0000B96B6B377C4C4E55AAEFECAABD821619] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server) 14 Jan 2014 13:59:06 (t3fs13_cms_9) [PinManager PoolSetSticky 0000923F9DDC894E4BC3B34191D56E8CC6B3] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server) 14 Jan 2014 13:59:06 (t3fs13_cms_9) [PinManager PoolSetSticky 0000CB8B7BA85CD04FA687B4B253FF04A2B5] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server) 14 Jan 2014 13:59:07 (t3fs13_cms_9) [PinManager PoolSetSticky 00001F460570B5144D99884B7D1705867663] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server) 14 Jan 2014 13:59:07 (t3fs13_cms_9) [PinManager PoolSetSticky 0000B868D5B152C845B8A9E695775AA94000] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server) 14 Jan 2014 13:59:08 (t3fs13_cms_9) [PinManager PoolSetSticky 0000AE24148B817E4F5F959BD0F57A2063F4] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server) 14 Jan 2014 13:59:08 (t3fs13_cms_9) [PinManager PoolSetSticky 00006432E31C39454FAD982BA3538942E3A2] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server) 14 Jan 2014 13:59:09 (t3fs13_cms_9) [PinManager PoolSetSticky 00003E4D06C716FE4CF8A0E208E746FDE1AE] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server) 14 Jan 2014 13:59:09 (t3fs13_cms_9) [PinManager PoolSetSticky 0000C3F2E412CCD34D1EA4ABC79942E8A4BE] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server) 14 Jan 2014 13:59:10 (t3fs13_cms_9) [PinManager PoolSetSticky 0000BCD4E5F9F0354D1EABF792C052578F74] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server) 14 Jan 2014 13:59:11 (t3fs13_cms_9) [PinManager PoolSetSticky 0000C54A5ADE7D784990BCAC878EF1C5BEC5] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server) 14 Jan 2014 13:59:55 (t3fs13_cms_3) [DCap-t3se01-<unknown>-14572 PoolDeliverFile 0000190EB94007744E69A1486339B38A6EAA] Fault occured in repository: File could not be opened [/mnt/data03/t3fs13_cms/pool/data/0000190EB94007744E69A1486339B38A6EAA (%RED%Too many open files)]; please check the file system%ENDCOLOR% diskCacheV111.util.DiskErrorCacheException: File could not be opened [/mnt/data03/t3fs13_cms/pool/data/0000190EB94007744E69A1486339B38A6EAA (Too many open files)]; please check the file system at org.dcache.pool.movers.AbstractMover.openChannel(AbstractMover.java:244) ~[dcache-core-2.6.19.jar:2.6.19] at org.dcache.pool.classic.MoverProtocolTransferService$MoverTask.run(MoverProtocolTransferService.java:154) ~[dcache-core-2.6.19.jar:2.6.19] at org.dcache.util.CDCExecutorServiceDecorator$1.run(CDCExecutorServiceDecorator.java:106) [dcache-core-2.6.19.jar:2.6.19] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_21] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_21] at java.lang.Thread.run(Thread.java:722) [na:1.7.0_21] Caused by: java.io.FileNotFoundException: /mnt/data03/t3fs13_cms/pool/data/0000190EB94007744E69A1486339B38A6EAA (Too many open files) at java.io.RandomAccessFile.open(Native Method) ~[na:1.7.0_21] at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) ~[na:1.7.0_21] at org.dcache.pool.repository.FileRepositoryChannel.<init>(FileRepositoryChannel.java:85) ~[dcache-core-2.6.19.jar:2.6.19] at org.dcache.pool.movers.AbstractMover.openChannel(AbstractMover.java:242) ~[dcache-core-2.6.19.jar:2.6.19] ... 5 common frames omitted 14 Jan 2014 13:59:55 (t3fs13_cms_3) [DCap-t3se01-<unknown>-14572 PoolDeliverFile 0000190EB94007744E69A1486339B38A6EAA] Pool mode changed to disabled(fetch,store,stage,p2p-client,p2p-server): Pool disabled: File could not be opened [/mnt/data03/t3fs13_cms/pool/data/0000190EB94007744E69A1486339B38A6EAA (Too many open files)]; please check the file system 14 Jan 2014 13:59:55 (t3fs13_cms_3) [DCap-t3se01-<unknown>-14572 PoolDeliverFile 0000190EB94007744E69A1486339B38A6EAA] Transfer failed due to a disk error: CacheException(rc=204;msg=File could not be opened [/mnt/data03/t3fs13_cms/pool/data/0000190EB94007744E69A1486339B38A6EAA (Too many open files)]; please check the file system) 14 Jan 2014 14:01:08 (t3fs13_cms_6) [DCap-t3se01-<unknown>-14492 PoolDeliverFile 0000D5B4380906A54A7D93A9727930E89512] Transfer failed: java.net.SocketException: Too many open files 14 Jan 2014 14:01:29 (t3fs13_cms_1) [DCap-t3se01-<unknown>-14627 PoolDeliverFile 00001A29CDFDE2BB4681A588FDF785F83550] Transfer failed: java.net.SocketException: Too many open files 14 Jan 2014 14:01:29 (t3fs13_cms_11) [DCap-t3se01-<unknown>-14483 PoolDeliverFile 0000202915F40ED14432ABCF253625CF3E3D] Transfer failed: java.net.SocketException: Too many open files 14 Jan 2014 14:01:39 (t3fs13_cms_2) [DCap-t3se01-<unknown>-14626 PoolDeliverFile 0000AAF00445FCA24216B7F16734EAC26EAA] Transfer failed: java.net.SocketException: Too many open files 14 Jan 2014 14:01:39 (t3fs13_cms_1) [DCap-t3se01-<unknown>-14470 PoolDeliverFile 0000CCE6991A92FD4262A4A1AB86598BAB41] Transfer failed: java.net.SocketException: Too many open files 14 Jan 2014 14:01:39 (t3fs13_cms_5) [DCap-t3se01-<unknown>-14692 PoolDeliverFile 000033B540B82F9F4A25959AFF93BE0B9C72] Transfer failed: java.net.SocketException: Too many open files 14 Jan 2014 14:01:43 (t3fs13_cms_8) [DCap-t3se01-<unknown>-14592 PoolDeliverFile 000013D8FDE247314E9B9288297621369137] Transfer failed: java.net.SocketException: Too many open files 14 Jan 2014 14:01:43 (t3fs13_cms_1) [DCap-t3se01-<unknown>-14566 PoolDeliverFile 0000BE11186F44FF4984A4C862F8F651D9C3] Transfer failed: java.net.SocketException: Too many open files 14 Jan 2014 14:01:43 (t3fs13_cms_5) [DCap-t3se01-<unknown>-14512 PoolDeliverFile 0000DD10977C1D764450B721B92400E342ED] Transfer failed: java.net.SocketException: Too many open files 14 Jan 2014 14:01:48 (t3fs13_cms_11) [DCap-t3se01-<unknown>-14624 PoolDeliverFile 0000913B7064ECF64CC18CE3C930C70A4A6F] Transfer failed: java.net.SocketException: Too many open files 14 Jan 2014 14:01:48 (t3fs13_ops_8) [] Fault occured in repository: I/O test failed 14 Jan 2014 14:01:48 (t3fs13_ops_8) [] Pool mode changed to disabled(fetch,store,stage,p2p-client,p2p-server): Pool disabled: I/O test failed 14 Jan 2014 14:01:55 (t3fs13_cms_1) [DCap-t3se01-<unknown>-14703 PoolDeliverFile 00009DE9D18FCD3F4E2C8822FAF90A2D7355] Transfer failed: java.net.SocketException: Too many open files 14 Jan 2014 14:01:56 (t3fs13_cms_1) [DCap-t3se01-<unknown>-14507 PoolDeliverFile 00001590C6E040914E81B818C45DE8282E24] Transfer failed: java.net.SocketException: Too many open files 14 Jan 2014 14:02:18 (t3fs13_cms_7) [DCap-t3se01-<unknown>-14531 PoolDeliverFile 00007FA4A54FB47044E1980EA13916507190] Transfer failed: java.net.SocketException: Too many open files 14 Jan 2014 14:02:19 (t3fs13_cms_1) [DCap-t3se01-<unknown>-14471 PoolDeliverFile 00003E2C77FD5A5B473E99FA4569A7A4462B] Transfer failed: java.net.SocketException: Too many open files 14 Jan 2014 14:02:20 (t3fs13_cms_6) [DCap-t3se01-<unknown>-14705 PoolDeliverFile 00007D3C4A8E5A914CF693206E12EEAF0449] Transfer failed: java.net.SocketException: Too many open files 14 Jan 2014 14:02:20 (t3fs13_cms_1) [DCap-t3se01-<unknown>-14473 PoolDeliverFile 0000C6D0ECACB2324E97A628EAF4B671C36D] Transfer failed: java.net.SocketException: Too many open files 14 Jan 2014 14:02:36 (t3fs13_cms_11) [DCap-t3se01-<unknown>-14530 PoolDeliverFile 000081AB3BE4ADAE4C9D9EDF2C8C8A673AF8] Transfer failed: java.net.SocketException: Too many open files 14 Jan 2014 14:02:37 (t3fs13_cms_7) [DCap-t3se01-<unknown>-14610 PoolDeliverFile 0000DE15D921A9E84240A5445DAC11F878C8] Transfer failed: java.net.SocketException: Too many open files </pre> %ENDTWISTY% </br> -- Main.FabioMartinelli - 2014-01-14 ---------------- %ICON{arrowleft}% Go to [[CMSTier3LogXX][previous page]] / [[CMSTier3LogXX][next page]] of Tier3 site log %M%
E
dit
|
A
ttach
|
Watch
|
P
rint version
|
H
istory
: r1
|
B
acklinks
|
V
iew topic
|
Ra
w
edit
|
M
ore topic actions
Topic revision: r1 - 2014-01-14
-
FabioMartinelli
CmsTier3
Log In
CmsTier3 Web
Create New Topic
Index
Search
Changes
Notifications
Statistics
Preferences
User Pages
Main Page
Policies
Monitoring Storage Space
Monitoring Slurm Usage
Physics Groups
Steering Board Meetings
Admin Pages
AdminArea
Cluster Specs
Home
Site map
CmsTier3 web
LCGTier2 web
PhaseC web
Main web
Sandbox web
TWiki web
CmsTier3 Web
Create New Topic
Index
Search
Changes
Notifications
RSS Feed
Statistics
Preferences
P
View
Raw View
Print version
Find backlinks
History
More topic actions
Edit
Raw edit
Attach file or image
Edit topic preference settings
Set new parent
More topic actions
Account
Log In
E
dit
A
ttach
Copyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki?
Send feedback