Tags:
create new tag
view all tags

CMS Tier-3 Upgrade Planning Page

dCache 2.6 upgrade

Summary

2.2 to 2.6 upgrade

Details

Planned / Status

Planned StatusSorted ascending
Linux pools upgrade to 2.6 + JDK 1.7 DONE but removed the xrootd monitor plugin
UIs UMD3 upgrade DONE
UIs latest dCache SRM RPM DONE
UIs latest voms clients DONE
t3bdii UMD3 upgrade DONE
t3se01 bdii UMD3 upgrade DONE
t3se01 PG 9 upgrade DONE
t3dcachedb03 PG 9 upgrade DONE
t3se01 2.6 upgrade DONE
t3dcachedb03 2.6 upgrade DONE
t3ldap01 New Primary Groups DONE
t3dcachedb03 gPlazma2 aware of the new Primary Groups DONE
t3dcachedb03 huge chgrp -R vs /pnfs/psi.ch/cms/trivcat/store/user DONE
Solaris pools upgrade to 2.6 + JDK 1.7 DONE
Daniel discovered that the dCache xrootd plugin on fs13,14 causes issues Removed RPM on fs13,14

Error evidence: 

[root@t3fs14 meta]# cat /mnt/data10/t3fs14_cms/pool/meta/je.info.0
131108 16:52:45:165 INFO [/mnt/data10/t3fs14_cms/pool/meta] Chose lowest utilized file for cleaning. fileChosen: 0xa totalUtilization: 37 bestFileUtilization: 8 lnSizeCorrectionFactor: NaN isProbe: false
131108 16:52:48:530 INFO [/mnt/data10/t3fs14_cms/pool/meta] CleanerRun 1 ends on file 0xa probe=false invokedFromDaemon=true finished=true fileDeleted=false nEntriesRead=47022 nINsObsolete=7 nINsCleaned=703 nINsDead=558 nINsMigrated=145 nBINDeltasObsolete=0 nBINDeltasCleaned=0 nBINDeltasDead=0 nBINDeltasMigrated=0 nLNsObsolete=13897 nLNsCleaned=32414 nLNsDead=21441 nLNsMigrated=10973 nLNsMarked=0 nLNQueueHits=5292 nLNsLocked=0 logSummary= inSummary= estFileSummary= recalcFileSummary= lnSizeCorrection=NaN newLnSizeCorrection=NaN estimatedUtilization=8 correctedUtilization=8 recalcUtilization=77
131108 16:52:48:530 INFO [/mnt/data10/t3fs14_cms/pool/meta] Chose lowest utilized file for cleaning. fileChosen: 0xc totalUtilization: 40 bestFileUtilization: 13 lnSizeCorrectionFactor: NaN isProbe: false
131108 16:52:48:814 INFO [/mnt/data10/t3fs14_cms/pool/meta] CleanerRun 2 ends on file 0xc probe=false invokedFromDaemon=true finished=true fileDeleted=false nEntriesRead=42335 nINsObsolete=72 nINsCleaned=913 nINsDead=775 nINsMigrated=138 nBINDeltasObsolete=0 nBINDeltasCleaned=0 nBINDeltasDead=0 nBINDeltasMigrated=0 nLNsObsolete=31517 nLNsCleaned=9831 nLNsDead=5833 nLNsMigrated=3998 nLNsMarked=0 nLNQueueHits=209 nLNsLocked=0 logSummary= inSummary= estFileSummary= recalcFileSummary= lnSizeCorrection=NaN newLnSizeCorrection=0.1390739 estimatedUtilization=13 correctedUtilization=13 recalcUtilization=53
131108 16:52:48:815 INFO [/mnt/data10/t3fs14_cms/pool/meta] Chose lowest utilized file for cleaning. fileChosen: 0x6 totalUtilization: 48 bestFileUtilization: 29 lnSizeCorrectionFactor: 0.1390739 isProbe: false
131108 16:52:49:698 INFO [/mnt/data10/t3fs14_cms/pool/meta] CleanerRun 3 ends on file 0x6 probe=false invokedFromDaemon=true finished=true fileDeleted=false nEntriesRead=26144 nINsObsolete=1208 nINsCleaned=7 nINsDead=7 nINsMigrated=0 nBINDeltasObsolete=555 nBINDeltasCleaned=0 nBINDeltasDead=0 nBINDeltasMigrated=0 nLNsObsolete=14013 nLNsCleaned=10359 nLNsDead=10215 nLNsMigrated=144 nLNsMarked=0 nLNQueueHits=2 nLNsLocked=0 logSummary= inSummary= estFileSummary= recalcFileSummary= lnSizeCorrection=0.1390739 newLnSizeCorrection=0.111519866 estimatedUtilization=28 correctedUtilization=29 recalcUtilization=29
131108 17:05:47:854 INFO [/mnt/data10/t3fs14_cms/pool/meta] Chose lowest utilized file for cleaning. fileChosen: 0x9 totalUtilization: 49 bestFileUtilization: 30 lnSizeCorrectionFactor: 0.111519866 isProbe: false
131108 17:05:48:528 SEVERE [/mnt/data10/t3fs14_cms/pool/meta] 
com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.73) /mnt/data10/t3fs14_cms/pool/meta fetchTarget of 0xe/0x10f443 parent IN=80629 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0xe/0x43b417 lastLoggedVersion=0xe/0x43b417 parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.
	at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:196)
	at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1512)
	at com.sleepycat.je.log.LogManager.serialLogInternal(LogManager.java:490)
	at com.sleepycat.je.log.SyncedLogManager.serialLog(SyncedLogManager.java:42)
	at com.sleepycat.je.log.LogManager.multiLog(LogManager.java:395)
	at com.sleepycat.je.log.LogManager.log(LogManager.java:335)
	at com.sleepycat.je.tree.LN.logInternal(LN.java:547)
	at com.sleepycat.je.tree.LN.log(LN.java:402)
	at com.sleepycat.je.cleaner.FileProcessor.processFoundLN(FileProcessor.java:1081)
	at com.sleepycat.je.cleaner.FileProcessor.processLN(FileProcessor.java:857)
	at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:674)
	at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:289)
	at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:148)
	at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:163)
	at java.lang.Thread.run(Thread.java:722)
Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.73) /mnt/data10/t3fs14_cms/pool/meta fetchTarget of 0xe/0x10f443 parent IN=80629 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0xe/0x43b417 lastLoggedVersion=0xe/0x43b417 parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.
	at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1429)
	at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1251)
	at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2261)
	at com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1466)
	at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1593)
	at com.sleepycat.je.Cursor.retrieveNextAllowPhantoms(Cursor.java:2924)
	at com.sleepycat.je.Cursor.retrieveNextNoDups(Cursor.java:2801)
	at com.sleepycat.je.Cursor.retrieveNext(Cursor.java:2775)
	at com.sleepycat.je.Cursor.getNextNoDup(Cursor.java:1244)
	at com.sleepycat.util.keyrange.RangeCursor.doGetNextNoDup(RangeCursor.java:920)
	at com.sleepycat.util.keyrange.RangeCursor.getNextNoDup(RangeCursor.java:475)
	at com.sleepycat.collections.DataCursor.getNextNoDup(DataCursor.java:456)
	at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:355)
	at org.dcache.pool.repository.meta.db.BerkeleyDBMetaDataRepository.list(BerkeleyDBMetaDataRepository.java:107)
	at org.dcache.pool.repository.ConsistentStore.list(ConsistentStore.java:111)
	at org.dcache.pool.repository.MetaDataCache.(MetaDataCache.java:47)
	at org.dcache.pool.repository.v5.CacheRepositoryV5.init(CacheRepositoryV5.java:319)
	at org.dcache.pool.classic.PoolV4$1.run(PoolV4.java:437)
Caused by: java.io.FileNotFoundException: /mnt/data10/t3fs14_cms/pool/meta/0000000e.jdb (Too many open files)
	at java.io.RandomAccessFile.open(Native Method)
	at java.io.RandomAccessFile.(RandomAccessFile.java:233)
	at java.io.RandomAccessFile.(RandomAccessFile.java:118)
	at com.sleepycat.je.log.FileManager$DefaultRandomAccessFile.(FileManager.java:3209)
	at com.sleepycat.je.log.FileManager$6.createFile(FileManager.java:3237)
	at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:1280)
	at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:1189)
	at com.sleepycat.je.log.LogManager.getLogSource(LogManager.java:1141)
	at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:845)
	at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:809)
	at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1412)
	... 17 more

131108 17:05:48:529 INFO [/mnt/data10/t3fs14_cms/pool/meta] CleanerRun 1 ends on file 0x9 probe=false invokedFromDaemon=true finished=false fileDeleted=false nEntriesRead=0 nINsObsolete=198 nINsCleaned=1220 nINsDead=1220 nINsMigrated=0 nBINDeltasObsolete=0 nBINDeltasCleaned=0 nBINDeltasDead=0 nBINDeltasMigrated=0 nLNsObsolete=7129 nLNsCleaned=5729 nLNsDead=279 nLNsMigrated=5449 nLNsMarked=0 nLNQueueHits=519 nLNsLocked=0 logSummary= inSummary= estFileSummary= recalcFileSummary= lnSizeCorrection=0.111519866 newLnSizeCorrection=0.111519866 estimatedUtilization=26 correctedUtilization=30 recalcUtilization=74
...

dCache logs: 

08 Nov 2013 17:05:45 (t3fs14_cms_5) [] Pool mode changed to disabled(fetch,store,stage,p2p-client,p2p-server): Initializing

08 Nov 2013 17:05:45 (t3fs14_cms_5) [] Reading inventory from [data=/mnt/data05/t3fs14_cms/pool/data;meta=/mnt/data05/t3fs14_cms/pool/meta]

08 Nov 2013 17:05:46 (t3fs14_cms_6) [] Pool mode changed to disabled(fetch,store,stage,p2p-client,p2p-server): Initializing

08 Nov 2013 17:05:46 (t3fs14_cms_6) [] Reading inventory from [data=/mnt/data07/t3fs14_cms/pool/data;meta=/mnt/data07/t3fs14_cms/pool/meta]

08 Nov 2013 17:05:46 (t3fs14_cms_7) [] Pool mode changed to disabled(fetch,store,stage,p2p-client,p2p-server): Initializing

08 Nov 2013 17:05:47 (t3fs14_cms_7) [] Reading inventory from [data=/mnt/data08/t3fs14_cms/pool/data;meta=/mnt/data08/t3fs14_cms/pool/meta]

08 Nov 2013 17:05:47 (t3fs14_cms_8) [] Pool mode changed to disabled(fetch,store,stage,p2p-client,p2p-server): Initializing

08 Nov 2013 17:05:47 (t3fs14_cms_8) [] Reading inventory from [data=/mnt/data09/t3fs14_cms/pool/data;meta=/mnt/data09/t3fs14_cms/pool/meta]

08 Nov 2013 17:05:48 (t3fs14_cms_9) [] Pool mode changed to disabled(fetch,store,stage,p2p-client,p2p-server): Initializing

08 Nov 2013 17:05:48 (t3fs14_cms_9) [] Reading inventory from [data=/mnt/data10/t3fs14_cms/pool/data;meta=/mnt/data10/t3fs14_cms/pool/meta]

08 Nov 2013 17:05:48 (t3fs14_cms_9) [] Pool restart required due to Berkeley DB failure: (JE 5.0.73) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.73) /mnt/data10/t3fs14_cms/pool/meta fetchTarget of 0xe/0x10f443 parent IN=80629 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0xe/0x43b417 lastLoggedVersion=0xe/0x43b417 parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.

Nov 08, 2013 5:05:48 PM com.sleepycat.je.utilint.DaemonThread run

SEVERE:  caught exception, com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.73) /mnt/data10/t3fs14_cms/pool/meta fetchTarget of 0xe/0x10f443 parent IN=80629 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0xe/0x43b417 lastLoggedVersion=0xe/0x43b417 parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed. Exiting

com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.73) /mnt/data10/t3fs14_cms/pool/meta fetchTarget of 0xe/0x10f443 parent IN=80629 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0xe/0x43b417 lastLoggedVersion=0xe/0x43b417 parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.

        at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:196)

        at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1512)

        at com.sleepycat.je.log.LogManager.serialLogInternal(LogManager.java:490)

        at com.sleepycat.je.log.SyncedLogManager.serialLog(SyncedLogManager.java:42)

        at com.sleepycat.je.log.LogManager.multiLog(LogManager.java:395)

        at com.sleepycat.je.log.LogManager.log(LogManager.java:335)

        at com.sleepycat.je.tree.LN.logInternal(LN.java:547)

        at com.sleepycat.je.tree.LN.log(LN.java:402)

        at com.sleepycat.je.cleaner.FileProcessor.processFoundLN(FileProcessor.java:1081)

        at com.sleepycat.je.cleaner.FileProcessor.processLN(FileProcessor.java:857)

        at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:674)

        at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:289)

        at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:148)

        at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:163)

        at java.lang.Thread.run(Thread.java:722)

Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.73) /mnt/data10/t3fs14_cms/pool/meta fetchTarget of 0xe/0x10f443 parent IN=80629 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0xe/0x43b417 lastLoggedVersion=0xe/0x43b417 parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.

        at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1429)

        at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1251)

        at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2261)

        at com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1466)

        at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1593)

        at com.sleepycat.je.Cursor.retrieveNextAllowPhantoms(Cursor.java:2924)

        at com.sleepycat.je.Cursor.retrieveNextNoDups(Cursor.java:2801)

        at com.sleepycat.je.Cursor.retrieveNext(Cursor.java:2775)

        at com.sleepycat.je.Cursor.getNextNoDup(Cursor.java:1244)

        at com.sleepycat.util.keyrange.RangeCursor.doGetNextNoDup(RangeCursor.java:920)

        at com.sleepycat.util.keyrange.RangeCursor.getNextNoDup(RangeCursor.java:475)

        at com.sleepycat.collections.DataCursor.getNextNoDup(DataCursor.java:456)

        at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:355)

        at org.dcache.pool.repository.meta.db.BerkeleyDBMetaDataRepository.list(BerkeleyDBMetaDataRepository.java:107)

        at org.dcache.pool.repository.ConsistentStore.list(ConsistentStore.java:111)

        at org.dcache.pool.repository.MetaDataCache.(MetaDataCache.java:47)

        at org.dcache.pool.repository.v5.CacheRepositoryV5.init(CacheRepositoryV5.java:319)

        at org.dcache.pool.classic.PoolV4$1.run(PoolV4.java:437)

Caused by: java.io.FileNotFoundException: /mnt/data10/t3fs14_cms/pool/meta/0000000e.jdb (Too many open files)

        at java.io.RandomAccessFile.open(Native Method)

        at java.io.RandomAccessFile.(RandomAccessFile.java:233)

        at java.io.RandomAccessFile.(RandomAccessFile.java:118)

        at com.sleepycat.je.log.FileManager$DefaultRandomAccessFile.(FileManager.java:3209)

        at com.sleepycat.je.log.FileManager$6.createFile(FileManager.java:3237)

        at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:1280)

        at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:1189)

        at com.sleepycat.je.log.LogManager.getLogSource(LogManager.java:1141)

        at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:845)

        at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:809)

        at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1412)

        ... 17 more

(666) file:/usr/share/dcache/services/pool.batch: line 95: (3) java.lang.RuntimeException: dmg.util.CommandThrowableException: (3) Failed to create bean 'pool' : Too many open files from ac_create_$_2_3

08 Nov 2013 17:05:48 (t3fs14_cms_9) [] Repository reported a problem. Please report this to support@dcache.org.

com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.73) /mnt/data10/t3fs14_cms/pool/meta fetchTarget of 0xe/0x10f443 parent IN=80629 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0xe/0x43b417 lastLoggedVersion=0xe/0x43b417 parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.

        at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:196) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1512) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.dbi.CursorImpl.checkEnv(CursorImpl.java:2838) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.Cursor.checkEnv(Cursor.java:4048) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.Cursor.close(Cursor.java:515) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.collections.CurrentTransaction.closeCursor(CurrentTransaction.java:484) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.collections.DataCursor.close(DataCursor.java:215) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.collections.StoredContainer.closeCursor(StoredContainer.java:422) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:376) ~[je-5.0.73.jar:5.0.73]

        at org.dcache.pool.repository.meta.db.BerkeleyDBMetaDataRepository.list(BerkeleyDBMetaDataRepository.java:107) ~[dcache-core-2.6.12.jar:2.6.12]

        at org.dcache.pool.repository.ConsistentStore.list(ConsistentStore.java:111) ~[dcache-core-2.6.12.jar:2.6.12]

        at org.dcache.pool.repository.MetaDataCache.(MetaDataCache.java:47) ~[dcache-core-2.6.12.jar:2.6.12]

        at org.dcache.pool.repository.v5.CacheRepositoryV5.init(CacheRepositoryV5.java:319) ~[dcache-core-2.6.12.jar:2.6.12]

        at org.dcache.pool.classic.PoolV4$1.run(PoolV4.java:437) ~[dcache-core-2.6.12.jar:2.6.12]

Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.73) /mnt/data10/t3fs14_cms/pool/meta fetchTarget of 0xe/0x10f443 parent IN=80629 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0xe/0x43b417 lastLoggedVersion=0xe/0x43b417 parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.

        at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1429) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1251) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2261) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1466) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1593) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.Cursor.retrieveNextAllowPhantoms(Cursor.java:2924) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.Cursor.retrieveNextNoDups(Cursor.java:2801) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.Cursor.retrieveNext(Cursor.java:2775) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.Cursor.getNextNoDup(Cursor.java:1244) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.util.keyrange.RangeCursor.doGetNextNoDup(RangeCursor.java:920) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.util.keyrange.RangeCursor.getNextNoDup(RangeCursor.java:475) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.collections.DataCursor.getNextNoDup(DataCursor.java:456) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:355) ~[je-5.0.73.jar:5.0.73]

        ... 5 common frames omitted

Caused by: java.io.FileNotFoundException: /mnt/data10/t3fs14_cms/pool/meta/0000000e.jdb (Too many open files)

        at java.io.RandomAccessFile.open(Native Method) ~[na:1.7.0_21]

        at java.io.RandomAccessFile.(RandomAccessFile.java:233) ~[na:1.7.0_21]

        at java.io.RandomAccessFile.(RandomAccessFile.java:118) ~[na:1.7.0_21]

        at com.sleepycat.je.log.FileManager$DefaultRandomAccessFile.(FileManager.java:3209) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.log.FileManager$6.createFile(FileManager.java:3237) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:1280) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:1189) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.log.LogManager.getLogSource(LogManager.java:1141) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:845) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:809) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1412) ~[je-5.0.73.jar:5.0.73]

        ... 17 common frames omitted

08 Nov 2013 17:05:48 (System) [] Failure at startup: (666) file:/usr/share/dcache/services/pool.batch: line 95: (3) java.lang.RuntimeException: dmg.util.CommandThrowableException: (3) Failed to create bean 'pool' : Too many open files from ac_create_$_2_3

08 Nov 2013 17:05:48 (t3fs14_cms_9) [] Pool not enabled t3fs14_cms_9

08 Nov 2013 17:05:48 (t3fs14_cms_9) [] Pool mode changed to disabled(fetch,store,stage,p2p-client,p2p-server,dead): Init failed: (JE 5.0.73) Envir
UpgradePlanningForm
Title dCache 2.6 upgrade
Summary 2.2 to 2.6 upgrade
Target Date 08. 11. 2013
Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r3 - 2013-11-11 - FabioMartinelli
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback