14. 01. 2014 dCache 2.6.19 pool - (Too many open files) error

13 Jan 2014 21:47:57 (t3fs13_cms_1) [door:GFTP-t3se01--828@t3se01-Domain-gridftp:1389646013190 GFTP-t3se01--828 PoolAcceptFile 0000908132F5C10A47138311C55D0D65B9E3] Transfer failed in post-processing: File size mismatch (expected=449396194, actual=0)
13 Jan 2014 21:52:38 (t3fs13_cms_10) [door:GFTP-t3se01--832@t3se01-Domain-gridftp:1389646168695 GFTP-t3se01--832 PoolAcceptFile 00004B3E7FC1C63E48D4A96542481803E980] Marking pool entry as BROKEN
13 Jan 2014 21:52:38 (t3fs13_cms_10) [door:GFTP-t3se01--832@t3se01-Domain-gridftp:1389646168695 GFTP-t3se01--832 PoolAcceptFile 00004B3E7FC1C63E48D4A96542481803E980] Transfer failed in post-processing: File size mismatch (expected=429375613, actual=0)
13 Jan 2014 22:02:00 (t3fs13_cms_11) [door:GFTP-t3se01--839@t3se01-Domain-gridftp:1389646857277 GFTP-t3se01--839 PoolAcceptFile 00008E02111C4F9545AD9FA073FE1C97D5B6] Marking pool entry as BROKEN
13 Jan 2014 22:02:01 (t3fs13_cms_11) [door:GFTP-t3se01--839@t3se01-Domain-gridftp:1389646857277 GFTP-t3se01--839 PoolAcceptFile 00008E02111C4F9545AD9FA073FE1C97D5B6] Transfer failed in post-processing: File size mismatch (expected=440561453, actual=0)
13 Jan 2014 22:21:54 (t3fs13_cms) [door:GFTP-t3se01--852@t3se01-Domain-gridftp:1389648050261 GFTP-t3se01--852 PoolAcceptFile 00009A55DDCAFCF34471A2EB02FE6B1E7FC5] Marking pool entry as BROKEN
13 Jan 2014 22:21:54 (t3fs13_cms) [door:GFTP-t3se01--852@t3se01-Domain-gridftp:1389648050261 GFTP-t3se01--852 PoolAcceptFile 00009A55DDCAFCF34471A2EB02FE6B1E7FC5] Transfer failed in post-processing: File size mismatch (expected=447047203, actual=0)
13 Jan 2014 22:25:11 (t3fs13_cms_11) [door:GFTP-t3se01--854@t3se01-Domain-gridftp:1389648121173 GFTP-t3se01--854 PoolAcceptFile 00008DFFA57177A24524A43AF52A8EFEC32E] Marking pool entry as BROKEN
13 Jan 2014 22:25:11 (t3fs13_cms_11) [door:GFTP-t3se01--854@t3se01-Domain-gridftp:1389648121173 GFTP-t3se01--854 PoolAcceptFile 00008DFFA57177A24524A43AF52A8EFEC32E] Transfer failed in post-processing: File size mismatch (expected=439879762, actual=0)
13 Jan 2014 22:26:25 (t3fs13_cms_11) [door:GFTP-t3se01--855@t3se01-Domain-gridftp:1389648321543 GFTP-t3se01--855 PoolAcceptFile 00000A3B69CB1F97498EA53E7B451DB8BEFF] Marking pool entry as BROKEN
13 Jan 2014 22:26:25 (t3fs13_cms_11) [door:GFTP-t3se01--855@t3se01-Domain-gridftp:1389648321543 GFTP-t3se01--855 PoolAcceptFile 00000A3B69CB1F97498EA53E7B451DB8BEFF] Transfer failed in post-processing: File size mismatch (expected=431818865, actual=0)
13 Jan 2014 22:30:26 (t3fs13_cms_11) [door:GFTP-t3se01--860@t3se01-Domain-gridftp:1389648562428 GFTP-t3se01--860 PoolAcceptFile 0000BCA3BABFCCE944A6AD6564821E5705D3] Marking pool entry as BROKEN
13 Jan 2014 22:30:26 (t3fs13_cms_11) [door:GFTP-t3se01--860@t3se01-Domain-gridftp:1389648562428 GFTP-t3se01--860 PoolAcceptFile 0000BCA3BABFCCE944A6AD6564821E5705D3] Transfer failed in post-processing: File size mismatch (expected=441029035, actual=0)
13 Jan 2014 22:31:48 (t3fs13_cms_11) [door:GFTP-t3se01--862@t3se01-Domain-gridftp:1389648644806 GFTP-t3se01--862 PoolAcceptFile 0000AACE0718CD2947AC8A337A9660E2BFA5] Marking pool entry as BROKEN
13 Jan 2014 22:31:48 (t3fs13_cms_11) [door:GFTP-t3se01--862@t3se01-Domain-gridftp:1389648644806 GFTP-t3se01--862 PoolAcceptFile 0000AACE0718CD2947AC8A337A9660E2BFA5] Transfer failed in post-processing: File size mismatch (expected=448837082, actual=0)
13 Jan 2014 22:33:32 (t3fs13_cms_11) [door:GFTP-t3se01--863@t3se01-Domain-gridftp:1389648748351 GFTP-t3se01--863 PoolAcceptFile 0000490A01CD9DF54E4988E312175FE72AED] Marking pool entry as BROKEN
13 Jan 2014 22:33:32 (t3fs13_cms_11) [door:GFTP-t3se01--863@t3se01-Domain-gridftp:1389648748351 GFTP-t3se01--863 PoolAcceptFile 0000490A01CD9DF54E4988E312175FE72AED] Transfer failed in post-processing: File size mismatch (expected=449403392, actual=0)
13 Jan 2014 23:51:34 (t3fs13_cms_7) [door:GFTP-t3se01--878@t3se01-Domain-gridftp:1389653430454 GFTP-t3se01--878 PoolAcceptFile 0000ED99AEBB03054B6DA3EDB74156128F6F] Marking pool entry as BROKEN
13 Jan 2014 23:51:34 (t3fs13_cms_7) [door:GFTP-t3se01--878@t3se01-Domain-gridftp:1389653430454 GFTP-t3se01--878 PoolAcceptFile 0000ED99AEBB03054B6DA3EDB74156128F6F] Transfer failed in post-processing: File size mismatch (expected=609450440, actual=0)
13 Jan 2014 23:57:05 (t3fs13_cms_11) [door:GFTP-t3se01--882@t3se01-Domain-gridftp:1389653761378 GFTP-t3se01--882 PoolAcceptFile 00000E3D503F13984F96B89FF63E9C6B0E47] Marking pool entry as BROKEN
13 Jan 2014 23:57:05 (t3fs13_cms_11) [door:GFTP-t3se01--882@t3se01-Domain-gridftp:1389653761378 GFTP-t3se01--882 PoolAcceptFile 00000E3D503F13984F96B89FF63E9C6B0E47] Transfer failed in post-processing: File size mismatch (expected=608483932, actual=0)
14 Jan 2014 13:33:59 (t3fs13_cms_9) [DCap-t3se01--14638 PoolDeliverFile 0000203E41A81062435D8FA8E4C376739E73] Problem in command block : RequestBlock [Size=0 Code=0 Buffer=java.nio.HeapByteBuffer[pos=0 lim=4 cap=16384] java.io.IOException: Connection reset by peer
14 Jan 2014 13:33:59 (t3fs13_cms_9) [DCap-t3se01--14638 PoolDeliverFile 0000203E41A81062435D8FA8E4C376739E73] Transfer failed: java.io.IOException: Connection reset by peer
14 Jan 2014 13:59:00 (t3fs13_cms_9) [DCap-t3se01--14478 PoolDeliverFile 00007E804B4380294466B052E0D6BDB0F454] Fault occured in repository: File could not be opened  [/mnt/data10/t3fs13_cms/pool/data/00007E804B4380294466B052E0D6BDB0F454 (Too many open files)]; please check the file system
diskCacheV111.util.DiskErrorCacheException: File could not be opened  [/mnt/data10/t3fs13_cms/pool/data/00007E804B4380294466B052E0D6BDB0F454 (Too many open files)]; please check the file system
	at org.dcache.pool.movers.AbstractMover.openChannel(AbstractMover.java:244) ~[dcache-core-2.6.19.jar:2.6.19]
	at org.dcache.pool.classic.MoverProtocolTransferService$MoverTask.run(MoverProtocolTransferService.java:154) ~[dcache-core-2.6.19.jar:2.6.19]
	at org.dcache.util.CDCExecutorServiceDecorator$1.run(CDCExecutorServiceDecorator.java:106) [dcache-core-2.6.19.jar:2.6.19]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_21]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_21]
	at java.lang.Thread.run(Thread.java:722) [na:1.7.0_21]
Caused by: java.io.FileNotFoundException: /mnt/data10/t3fs13_cms/pool/data/00007E804B4380294466B052E0D6BDB0F454 (Too many open files)
	at java.io.RandomAccessFile.open(Native Method) ~[na:1.7.0_21]
	at java.io.RandomAccessFile.(RandomAccessFile.java:233) ~[na:1.7.0_21]
	at org.dcache.pool.repository.FileRepositoryChannel.(FileRepositoryChannel.java:85) ~[dcache-core-2.6.19.jar:2.6.19]
	at org.dcache.pool.movers.AbstractMover.openChannel(AbstractMover.java:242) ~[dcache-core-2.6.19.jar:2.6.19]
	... 5 common frames omitted
14 Jan 2014 13:59:00 (t3fs13_cms_9) [DCap-t3se01--14478 PoolDeliverFile 00007E804B4380294466B052E0D6BDB0F454] Pool mode changed to disabled(fetch,store,stage,p2p-client,p2p-server): Pool disabled: File could not be opened  [/mnt/data10/t3fs13_cms/pool/data/00007E804B4380294466B052E0D6BDB0F454 (Too many open files)]; please check the file system
14 Jan 2014 13:59:00 (t3fs13_cms_9) [DCap-t3se01--14478 PoolDeliverFile 00007E804B4380294466B052E0D6BDB0F454] Transfer failed due to a disk error: CacheException(rc=204;msg=File could not be opened  [/mnt/data10/t3fs13_cms/pool/data/00007E804B4380294466B052E0D6BDB0F454 (Too many open files)]; please check the file system)
14 Jan 2014 13:59:05 (t3fs13_cms_9) [PinManager PoolSetSticky 00006B85783E25084626B927566FAF45A2DE] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server)
14 Jan 2014 13:59:05 (t3fs13_cms_9) [PinManager PoolSetSticky 0000B96B6B377C4C4E55AAEFECAABD821619] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server)
14 Jan 2014 13:59:06 (t3fs13_cms_9) [PinManager PoolSetSticky 0000923F9DDC894E4BC3B34191D56E8CC6B3] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server)
14 Jan 2014 13:59:06 (t3fs13_cms_9) [PinManager PoolSetSticky 0000CB8B7BA85CD04FA687B4B253FF04A2B5] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server)
14 Jan 2014 13:59:07 (t3fs13_cms_9) [PinManager PoolSetSticky 00001F460570B5144D99884B7D1705867663] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server)
14 Jan 2014 13:59:07 (t3fs13_cms_9) [PinManager PoolSetSticky 0000B868D5B152C845B8A9E695775AA94000] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server)
14 Jan 2014 13:59:08 (t3fs13_cms_9) [PinManager PoolSetSticky 0000AE24148B817E4F5F959BD0F57A2063F4] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server)
14 Jan 2014 13:59:08 (t3fs13_cms_9) [PinManager PoolSetSticky 00006432E31C39454FAD982BA3538942E3A2] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server)
14 Jan 2014 13:59:09 (t3fs13_cms_9) [PinManager PoolSetSticky 00003E4D06C716FE4CF8A0E208E746FDE1AE] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server)
14 Jan 2014 13:59:09 (t3fs13_cms_9) [PinManager PoolSetSticky 0000C3F2E412CCD34D1EA4ABC79942E8A4BE] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server)
14 Jan 2014 13:59:10 (t3fs13_cms_9) [PinManager PoolSetSticky 0000BCD4E5F9F0354D1EABF792C052578F74] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server)
14 Jan 2014 13:59:11 (t3fs13_cms_9) [PinManager PoolSetSticky 0000C54A5ADE7D784990BCAC878EF1C5BEC5] PoolSetStickyMessage request rejected due to disabled(fetch,store,stage,p2p-client,p2p-server)
14 Jan 2014 13:59:55 (t3fs13_cms_3) [DCap-t3se01--14572 PoolDeliverFile 0000190EB94007744E69A1486339B38A6EAA] Fault occured in repository: File could not be opened  [/mnt/data03/t3fs13_cms/pool/data/0000190EB94007744E69A1486339B38A6EAA (Too many open files)]; please check the file system
diskCacheV111.util.DiskErrorCacheException: File could not be opened  [/mnt/data03/t3fs13_cms/pool/data/0000190EB94007744E69A1486339B38A6EAA (Too many open files)]; please check the file system
	at org.dcache.pool.movers.AbstractMover.openChannel(AbstractMover.java:244) ~[dcache-core-2.6.19.jar:2.6.19]
	at org.dcache.pool.classic.MoverProtocolTransferService$MoverTask.run(MoverProtocolTransferService.java:154) ~[dcache-core-2.6.19.jar:2.6.19]
	at org.dcache.util.CDCExecutorServiceDecorator$1.run(CDCExecutorServiceDecorator.java:106) [dcache-core-2.6.19.jar:2.6.19]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_21]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_21]
	at java.lang.Thread.run(Thread.java:722) [na:1.7.0_21]
Caused by: java.io.FileNotFoundException: /mnt/data03/t3fs13_cms/pool/data/0000190EB94007744E69A1486339B38A6EAA (Too many open files)
	at java.io.RandomAccessFile.open(Native Method) ~[na:1.7.0_21]
	at java.io.RandomAccessFile.(RandomAccessFile.java:233) ~[na:1.7.0_21]
	at org.dcache.pool.repository.FileRepositoryChannel.(FileRepositoryChannel.java:85) ~[dcache-core-2.6.19.jar:2.6.19]
	at org.dcache.pool.movers.AbstractMover.openChannel(AbstractMover.java:242) ~[dcache-core-2.6.19.jar:2.6.19]
	... 5 common frames omitted
14 Jan 2014 13:59:55 (t3fs13_cms_3) [DCap-t3se01--14572 PoolDeliverFile 0000190EB94007744E69A1486339B38A6EAA] Pool mode changed to disabled(fetch,store,stage,p2p-client,p2p-server): Pool disabled: File could not be opened  [/mnt/data03/t3fs13_cms/pool/data/0000190EB94007744E69A1486339B38A6EAA (Too many open files)]; please check the file system
14 Jan 2014 13:59:55 (t3fs13_cms_3) [DCap-t3se01--14572 PoolDeliverFile 0000190EB94007744E69A1486339B38A6EAA] Transfer failed due to a disk error: CacheException(rc=204;msg=File could not be opened  [/mnt/data03/t3fs13_cms/pool/data/0000190EB94007744E69A1486339B38A6EAA (Too many open files)]; please check the file system)
14 Jan 2014 14:01:08 (t3fs13_cms_6) [DCap-t3se01--14492 PoolDeliverFile 0000D5B4380906A54A7D93A9727930E89512] Transfer failed: java.net.SocketException: Too many open files
14 Jan 2014 14:01:29 (t3fs13_cms_1) [DCap-t3se01--14627 PoolDeliverFile 00001A29CDFDE2BB4681A588FDF785F83550] Transfer failed: java.net.SocketException: Too many open files
14 Jan 2014 14:01:29 (t3fs13_cms_11) [DCap-t3se01--14483 PoolDeliverFile 0000202915F40ED14432ABCF253625CF3E3D] Transfer failed: java.net.SocketException: Too many open files
14 Jan 2014 14:01:39 (t3fs13_cms_2) [DCap-t3se01--14626 PoolDeliverFile 0000AAF00445FCA24216B7F16734EAC26EAA] Transfer failed: java.net.SocketException: Too many open files
14 Jan 2014 14:01:39 (t3fs13_cms_1) [DCap-t3se01--14470 PoolDeliverFile 0000CCE6991A92FD4262A4A1AB86598BAB41] Transfer failed: java.net.SocketException: Too many open files
14 Jan 2014 14:01:39 (t3fs13_cms_5) [DCap-t3se01--14692 PoolDeliverFile 000033B540B82F9F4A25959AFF93BE0B9C72] Transfer failed: java.net.SocketException: Too many open files
14 Jan 2014 14:01:43 (t3fs13_cms_8) [DCap-t3se01--14592 PoolDeliverFile 000013D8FDE247314E9B9288297621369137] Transfer failed: java.net.SocketException: Too many open files
14 Jan 2014 14:01:43 (t3fs13_cms_1) [DCap-t3se01--14566 PoolDeliverFile 0000BE11186F44FF4984A4C862F8F651D9C3] Transfer failed: java.net.SocketException: Too many open files
14 Jan 2014 14:01:43 (t3fs13_cms_5) [DCap-t3se01--14512 PoolDeliverFile 0000DD10977C1D764450B721B92400E342ED] Transfer failed: java.net.SocketException: Too many open files
14 Jan 2014 14:01:48 (t3fs13_cms_11) [DCap-t3se01--14624 PoolDeliverFile 0000913B7064ECF64CC18CE3C930C70A4A6F] Transfer failed: java.net.SocketException: Too many open files
14 Jan 2014 14:01:48 (t3fs13_ops_8) [] Fault occured in repository: I/O test failed
14 Jan 2014 14:01:48 (t3fs13_ops_8) [] Pool mode changed to disabled(fetch,store,stage,p2p-client,p2p-server): Pool disabled: I/O test failed
14 Jan 2014 14:01:55 (t3fs13_cms_1) [DCap-t3se01--14703 PoolDeliverFile 00009DE9D18FCD3F4E2C8822FAF90A2D7355] Transfer failed: java.net.SocketException: Too many open files
14 Jan 2014 14:01:56 (t3fs13_cms_1) [DCap-t3se01--14507 PoolDeliverFile 00001590C6E040914E81B818C45DE8282E24] Transfer failed: java.net.SocketException: Too many open files
14 Jan 2014 14:02:18 (t3fs13_cms_7) [DCap-t3se01--14531 PoolDeliverFile 00007FA4A54FB47044E1980EA13916507190] Transfer failed: java.net.SocketException: Too many open files
14 Jan 2014 14:02:19 (t3fs13_cms_1) [DCap-t3se01--14471 PoolDeliverFile 00003E2C77FD5A5B473E99FA4569A7A4462B] Transfer failed: java.net.SocketException: Too many open files
14 Jan 2014 14:02:20 (t3fs13_cms_6) [DCap-t3se01--14705 PoolDeliverFile 00007D3C4A8E5A914CF693206E12EEAF0449] Transfer failed: java.net.SocketException: Too many open files
14 Jan 2014 14:02:20 (t3fs13_cms_1) [DCap-t3se01--14473 PoolDeliverFile 0000C6D0ECACB2324E97A628EAF4B671C36D] Transfer failed: java.net.SocketException: Too many open files
14 Jan 2014 14:02:36 (t3fs13_cms_11) [DCap-t3se01--14530 PoolDeliverFile 000081AB3BE4ADAE4C9D9EDF2C8C8A673AF8] Transfer failed: java.net.SocketException: Too many open files
14 Jan 2014 14:02:37 (t3fs13_cms_7) [DCap-t3se01--14610 PoolDeliverFile 0000DE15D921A9E84240A5445DAC11F878C8] Transfer failed: java.net.SocketException: Too many open files