Tags:
create new tag
view all tags

CMS Tier-3 Upgrade Planning Page

dCache 2.6 upgrade

Summary

2.2 to 2.6 upgrade

Details

Planned / Status

Planned Status
UIs UMD3 upgrade DONE
UIs latest dCache SRM RPM DONE
UIs latest voms clients DONE
t3bdii UMD3 upgrade DONE
t3se01 bdii UMD3 upgrade DONE
t3se01 PG 9 upgrade DONE
t3dcachedb03 PG 9 upgrade DONE
t3se01 2.6 upgrade DONE
t3dcachedb03 2.6 upgrade DONE
t3ldap01 New Primary Groups DONE
t3dcachedb03 gPlazma2 aware of the new Primary Groups DONE
t3dcachedb03 huge chgrp -R vs /pnfs/psi.ch/cms/trivcat/store/user DONE
Solaris pools upgrade to 2.6 + JDK 1.7 DONE
Linux pools upgrade to 2.6 + JDK 1.7 DONE but removed the xrootd monitor plugin
Daniel discovered that the dCache xrootd plugin on fs13,14 causes issues Removed RPM on fs13,14

Error evidence: 

[root@t3fs14 meta]# cat /mnt/data10/t3fs14_cms/pool/meta/je.info.0
131108 16:52:45:165 INFO [/mnt/data10/t3fs14_cms/pool/meta] Chose lowest utilized file for cleaning. fileChosen: 0xa totalUtilization: 37 bestFileUtilization: 8 lnSizeCorrectionFactor: NaN isProbe: false
131108 16:52:48:530 INFO [/mnt/data10/t3fs14_cms/pool/meta] CleanerRun 1 ends on file 0xa probe=false invokedFromDaemon=true finished=true fileDeleted=false nEntriesRead=47022 nINsObsolete=7 nINsCleaned=703 nINsDead=558 nINsMigrated=145 nBINDeltasObsolete=0 nBINDeltasCleaned=0 nBINDeltasDead=0 nBINDeltasMigrated=0 nLNsObsolete=13897 nLNsCleaned=32414 nLNsDead=21441 nLNsMigrated=10973 nLNsMarked=0 nLNQueueHits=5292 nLNsLocked=0 logSummary= inSummary= estFileSummary= recalcFileSummary= lnSizeCorrection=NaN newLnSizeCorrection=NaN estimatedUtilization=8 correctedUtilization=8 recalcUtilization=77
131108 16:52:48:530 INFO [/mnt/data10/t3fs14_cms/pool/meta] Chose lowest utilized file for cleaning. fileChosen: 0xc totalUtilization: 40 bestFileUtilization: 13 lnSizeCorrectionFactor: NaN isProbe: false
131108 16:52:48:814 INFO [/mnt/data10/t3fs14_cms/pool/meta] CleanerRun 2 ends on file 0xc probe=false invokedFromDaemon=true finished=true fileDeleted=false nEntriesRead=42335 nINsObsolete=72 nINsCleaned=913 nINsDead=775 nINsMigrated=138 nBINDeltasObsolete=0 nBINDeltasCleaned=0 nBINDeltasDead=0 nBINDeltasMigrated=0 nLNsObsolete=31517 nLNsCleaned=9831 nLNsDead=5833 nLNsMigrated=3998 nLNsMarked=0 nLNQueueHits=209 nLNsLocked=0 logSummary= inSummary= estFileSummary= recalcFileSummary= lnSizeCorrection=NaN newLnSizeCorrection=0.1390739 estimatedUtilization=13 correctedUtilization=13 recalcUtilization=53
131108 16:52:48:815 INFO [/mnt/data10/t3fs14_cms/pool/meta] Chose lowest utilized file for cleaning. fileChosen: 0x6 totalUtilization: 48 bestFileUtilization: 29 lnSizeCorrectionFactor: 0.1390739 isProbe: false
131108 16:52:49:698 INFO [/mnt/data10/t3fs14_cms/pool/meta] CleanerRun 3 ends on file 0x6 probe=false invokedFromDaemon=true finished=true fileDeleted=false nEntriesRead=26144 nINsObsolete=1208 nINsCleaned=7 nINsDead=7 nINsMigrated=0 nBINDeltasObsolete=555 nBINDeltasCleaned=0 nBINDeltasDead=0 nBINDeltasMigrated=0 nLNsObsolete=14013 nLNsCleaned=10359 nLNsDead=10215 nLNsMigrated=144 nLNsMarked=0 nLNQueueHits=2 nLNsLocked=0 logSummary= inSummary= estFileSummary= recalcFileSummary= lnSizeCorrection=0.1390739 newLnSizeCorrection=0.111519866 estimatedUtilization=28 correctedUtilization=29 recalcUtilization=29
131108 17:05:47:854 INFO [/mnt/data10/t3fs14_cms/pool/meta] Chose lowest utilized file for cleaning. fileChosen: 0x9 totalUtilization: 49 bestFileUtilization: 30 lnSizeCorrectionFactor: 0.111519866 isProbe: false
131108 17:05:48:528 SEVERE [/mnt/data10/t3fs14_cms/pool/meta] 
com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.73) /mnt/data10/t3fs14_cms/pool/meta fetchTarget of 0xe/0x10f443 parent IN=80629 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0xe/0x43b417 lastLoggedVersion=0xe/0x43b417 parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.
	at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:196)
	at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1512)
	at com.sleepycat.je.log.LogManager.serialLogInternal(LogManager.java:490)
	at com.sleepycat.je.log.SyncedLogManager.serialLog(SyncedLogManager.java:42)
	at com.sleepycat.je.log.LogManager.multiLog(LogManager.java:395)
	at com.sleepycat.je.log.LogManager.log(LogManager.java:335)
	at com.sleepycat.je.tree.LN.logInternal(LN.java:547)
	at com.sleepycat.je.tree.LN.log(LN.java:402)
	at com.sleepycat.je.cleaner.FileProcessor.processFoundLN(FileProcessor.java:1081)
	at com.sleepycat.je.cleaner.FileProcessor.processLN(FileProcessor.java:857)
	at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:674)
	at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:289)
	at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:148)
	at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:163)
	at java.lang.Thread.run(Thread.java:722)
Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.73) /mnt/data10/t3fs14_cms/pool/meta fetchTarget of 0xe/0x10f443 parent IN=80629 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0xe/0x43b417 lastLoggedVersion=0xe/0x43b417 parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.
	at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1429)
	at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1251)
	at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2261)
	at com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1466)
	at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1593)
	at com.sleepycat.je.Cursor.retrieveNextAllowPhantoms(Cursor.java:2924)
	at com.sleepycat.je.Cursor.retrieveNextNoDups(Cursor.java:2801)
	at com.sleepycat.je.Cursor.retrieveNext(Cursor.java:2775)
	at com.sleepycat.je.Cursor.getNextNoDup(Cursor.java:1244)
	at com.sleepycat.util.keyrange.RangeCursor.doGetNextNoDup(RangeCursor.java:920)
	at com.sleepycat.util.keyrange.RangeCursor.getNextNoDup(RangeCursor.java:475)
	at com.sleepycat.collections.DataCursor.getNextNoDup(DataCursor.java:456)
	at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:355)
	at org.dcache.pool.repository.meta.db.BerkeleyDBMetaDataRepository.list(BerkeleyDBMetaDataRepository.java:107)
	at org.dcache.pool.repository.ConsistentStore.list(ConsistentStore.java:111)
	at org.dcache.pool.repository.MetaDataCache.(MetaDataCache.java:47)
	at org.dcache.pool.repository.v5.CacheRepositoryV5.init(CacheRepositoryV5.java:319)
	at org.dcache.pool.classic.PoolV4$1.run(PoolV4.java:437)
Caused by: java.io.FileNotFoundException: /mnt/data10/t3fs14_cms/pool/meta/0000000e.jdb (Too many open files)
	at java.io.RandomAccessFile.open(Native Method)
	at java.io.RandomAccessFile.(RandomAccessFile.java:233)
	at java.io.RandomAccessFile.(RandomAccessFile.java:118)
	at com.sleepycat.je.log.FileManager$DefaultRandomAccessFile.(FileManager.java:3209)
	at com.sleepycat.je.log.FileManager$6.createFile(FileManager.java:3237)
	at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:1280)
	at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:1189)
	at com.sleepycat.je.log.LogManager.getLogSource(LogManager.java:1141)
	at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:845)
	at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:809)
	at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1412)
	... 17 more

131108 17:05:48:529 INFO [/mnt/data10/t3fs14_cms/pool/meta] CleanerRun 1 ends on file 0x9 probe=false invokedFromDaemon=true finished=false fileDeleted=false nEntriesRead=0 nINsObsolete=198 nINsCleaned=1220 nINsDead=1220 nINsMigrated=0 nBINDeltasObsolete=0 nBINDeltasCleaned=0 nBINDeltasDead=0 nBINDeltasMigrated=0 nLNsObsolete=7129 nLNsCleaned=5729 nLNsDead=279 nLNsMigrated=5449 nLNsMarked=0 nLNQueueHits=519 nLNsLocked=0 logSummary= inSummary= estFileSummary= recalcFileSummary= lnSizeCorrection=0.111519866 newLnSizeCorrection=0.111519866 estimatedUtilization=26 correctedUtilization=30 recalcUtilization=74
...

dCache logs: 

08 Nov 2013 17:05:45 (t3fs14_cms_5) [] Pool mode changed to disabled(fetch,store,stage,p2p-client,p2p-server): Initializing

08 Nov 2013 17:05:45 (t3fs14_cms_5) [] Reading inventory from [data=/mnt/data05/t3fs14_cms/pool/data;meta=/mnt/data05/t3fs14_cms/pool/meta]

08 Nov 2013 17:05:46 (t3fs14_cms_6) [] Pool mode changed to disabled(fetch,store,stage,p2p-client,p2p-server): Initializing

08 Nov 2013 17:05:46 (t3fs14_cms_6) [] Reading inventory from [data=/mnt/data07/t3fs14_cms/pool/data;meta=/mnt/data07/t3fs14_cms/pool/meta]

08 Nov 2013 17:05:46 (t3fs14_cms_7) [] Pool mode changed to disabled(fetch,store,stage,p2p-client,p2p-server): Initializing

08 Nov 2013 17:05:47 (t3fs14_cms_7) [] Reading inventory from [data=/mnt/data08/t3fs14_cms/pool/data;meta=/mnt/data08/t3fs14_cms/pool/meta]

08 Nov 2013 17:05:47 (t3fs14_cms_8) [] Pool mode changed to disabled(fetch,store,stage,p2p-client,p2p-server): Initializing

08 Nov 2013 17:05:47 (t3fs14_cms_8) [] Reading inventory from [data=/mnt/data09/t3fs14_cms/pool/data;meta=/mnt/data09/t3fs14_cms/pool/meta]

08 Nov 2013 17:05:48 (t3fs14_cms_9) [] Pool mode changed to disabled(fetch,store,stage,p2p-client,p2p-server): Initializing

08 Nov 2013 17:05:48 (t3fs14_cms_9) [] Reading inventory from [data=/mnt/data10/t3fs14_cms/pool/data;meta=/mnt/data10/t3fs14_cms/pool/meta]

08 Nov 2013 17:05:48 (t3fs14_cms_9) [] Pool restart required due to Berkeley DB failure: (JE 5.0.73) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.73) /mnt/data10/t3fs14_cms/pool/meta fetchTarget of 0xe/0x10f443 parent IN=80629 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0xe/0x43b417 lastLoggedVersion=0xe/0x43b417 parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.

Nov 08, 2013 5:05:48 PM com.sleepycat.je.utilint.DaemonThread run

SEVERE:  caught exception, com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.73) /mnt/data10/t3fs14_cms/pool/meta fetchTarget of 0xe/0x10f443 parent IN=80629 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0xe/0x43b417 lastLoggedVersion=0xe/0x43b417 parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed. Exiting

com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.73) /mnt/data10/t3fs14_cms/pool/meta fetchTarget of 0xe/0x10f443 parent IN=80629 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0xe/0x43b417 lastLoggedVersion=0xe/0x43b417 parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.

        at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:196)

        at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1512)

        at com.sleepycat.je.log.LogManager.serialLogInternal(LogManager.java:490)

        at com.sleepycat.je.log.SyncedLogManager.serialLog(SyncedLogManager.java:42)

        at com.sleepycat.je.log.LogManager.multiLog(LogManager.java:395)

        at com.sleepycat.je.log.LogManager.log(LogManager.java:335)

        at com.sleepycat.je.tree.LN.logInternal(LN.java:547)

        at com.sleepycat.je.tree.LN.log(LN.java:402)

        at com.sleepycat.je.cleaner.FileProcessor.processFoundLN(FileProcessor.java:1081)

        at com.sleepycat.je.cleaner.FileProcessor.processLN(FileProcessor.java:857)

        at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:674)

        at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:289)

        at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:148)

        at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:163)

        at java.lang.Thread.run(Thread.java:722)

Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.73) /mnt/data10/t3fs14_cms/pool/meta fetchTarget of 0xe/0x10f443 parent IN=80629 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0xe/0x43b417 lastLoggedVersion=0xe/0x43b417 parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.

        at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1429)

        at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1251)

        at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2261)

        at com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1466)

        at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1593)

        at com.sleepycat.je.Cursor.retrieveNextAllowPhantoms(Cursor.java:2924)

        at com.sleepycat.je.Cursor.retrieveNextNoDups(Cursor.java:2801)

        at com.sleepycat.je.Cursor.retrieveNext(Cursor.java:2775)

        at com.sleepycat.je.Cursor.getNextNoDup(Cursor.java:1244)

        at com.sleepycat.util.keyrange.RangeCursor.doGetNextNoDup(RangeCursor.java:920)

        at com.sleepycat.util.keyrange.RangeCursor.getNextNoDup(RangeCursor.java:475)

        at com.sleepycat.collections.DataCursor.getNextNoDup(DataCursor.java:456)

        at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:355)

        at org.dcache.pool.repository.meta.db.BerkeleyDBMetaDataRepository.list(BerkeleyDBMetaDataRepository.java:107)

        at org.dcache.pool.repository.ConsistentStore.list(ConsistentStore.java:111)

        at org.dcache.pool.repository.MetaDataCache.(MetaDataCache.java:47)

        at org.dcache.pool.repository.v5.CacheRepositoryV5.init(CacheRepositoryV5.java:319)

        at org.dcache.pool.classic.PoolV4$1.run(PoolV4.java:437)

Caused by: java.io.FileNotFoundException: /mnt/data10/t3fs14_cms/pool/meta/0000000e.jdb (Too many open files)

        at java.io.RandomAccessFile.open(Native Method)

        at java.io.RandomAccessFile.(RandomAccessFile.java:233)

        at java.io.RandomAccessFile.(RandomAccessFile.java:118)

        at com.sleepycat.je.log.FileManager$DefaultRandomAccessFile.(FileManager.java:3209)

        at com.sleepycat.je.log.FileManager$6.createFile(FileManager.java:3237)

        at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:1280)

        at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:1189)

        at com.sleepycat.je.log.LogManager.getLogSource(LogManager.java:1141)

        at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:845)

        at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:809)

        at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1412)

        ... 17 more

(666) file:/usr/share/dcache/services/pool.batch: line 95: (3) java.lang.RuntimeException: dmg.util.CommandThrowableException: (3) Failed to create bean 'pool' : Too many open files from ac_create_$_2_3

08 Nov 2013 17:05:48 (t3fs14_cms_9) [] Repository reported a problem. Please report this to support@dcache.org.

com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.73) /mnt/data10/t3fs14_cms/pool/meta fetchTarget of 0xe/0x10f443 parent IN=80629 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0xe/0x43b417 lastLoggedVersion=0xe/0x43b417 parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.

        at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:196) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1512) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.dbi.CursorImpl.checkEnv(CursorImpl.java:2838) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.Cursor.checkEnv(Cursor.java:4048) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.Cursor.close(Cursor.java:515) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.collections.CurrentTransaction.closeCursor(CurrentTransaction.java:484) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.collections.DataCursor.close(DataCursor.java:215) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.collections.StoredContainer.closeCursor(StoredContainer.java:422) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:376) ~[je-5.0.73.jar:5.0.73]

        at org.dcache.pool.repository.meta.db.BerkeleyDBMetaDataRepository.list(BerkeleyDBMetaDataRepository.java:107) ~[dcache-core-2.6.12.jar:2.6.12]

        at org.dcache.pool.repository.ConsistentStore.list(ConsistentStore.java:111) ~[dcache-core-2.6.12.jar:2.6.12]

        at org.dcache.pool.repository.MetaDataCache.(MetaDataCache.java:47) ~[dcache-core-2.6.12.jar:2.6.12]

        at org.dcache.pool.repository.v5.CacheRepositoryV5.init(CacheRepositoryV5.java:319) ~[dcache-core-2.6.12.jar:2.6.12]

        at org.dcache.pool.classic.PoolV4$1.run(PoolV4.java:437) ~[dcache-core-2.6.12.jar:2.6.12]

Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.73) /mnt/data10/t3fs14_cms/pool/meta fetchTarget of 0xe/0x10f443 parent IN=80629 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0xe/0x43b417 lastLoggedVersion=0xe/0x43b417 parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.

        at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1429) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1251) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2261) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1466) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1593) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.Cursor.retrieveNextAllowPhantoms(Cursor.java:2924) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.Cursor.retrieveNextNoDups(Cursor.java:2801) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.Cursor.retrieveNext(Cursor.java:2775) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.Cursor.getNextNoDup(Cursor.java:1244) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.util.keyrange.RangeCursor.doGetNextNoDup(RangeCursor.java:920) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.util.keyrange.RangeCursor.getNextNoDup(RangeCursor.java:475) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.collections.DataCursor.getNextNoDup(DataCursor.java:456) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.collections.BlockIterator.hasNext(BlockIterator.java:355) ~[je-5.0.73.jar:5.0.73]

        ... 5 common frames omitted

Caused by: java.io.FileNotFoundException: /mnt/data10/t3fs14_cms/pool/meta/0000000e.jdb (Too many open files)

        at java.io.RandomAccessFile.open(Native Method) ~[na:1.7.0_21]

        at java.io.RandomAccessFile.(RandomAccessFile.java:233) ~[na:1.7.0_21]

        at java.io.RandomAccessFile.(RandomAccessFile.java:118) ~[na:1.7.0_21]

        at com.sleepycat.je.log.FileManager$DefaultRandomAccessFile.(FileManager.java:3209) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.log.FileManager$6.createFile(FileManager.java:3237) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:1280) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:1189) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.log.LogManager.getLogSource(LogManager.java:1141) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:845) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:809) ~[je-5.0.73.jar:5.0.73]

        at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1412) ~[je-5.0.73.jar:5.0.73]

        ... 17 common frames omitted

08 Nov 2013 17:05:48 (System) [] Failure at startup: (666) file:/usr/share/dcache/services/pool.batch: line 95: (3) java.lang.RuntimeException: dmg.util.CommandThrowableException: (3) Failed to create bean 'pool' : Too many open files from ac_create_$_2_3

08 Nov 2013 17:05:48 (t3fs14_cms_9) [] Pool not enabled t3fs14_cms_9

08 Nov 2013 17:05:48 (t3fs14_cms_9) [] Pool mode changed to disabled(fetch,store,stage,p2p-client,p2p-server,dead): Init failed: (JE 5.0.73) Envir
UpgradePlanningForm
Title dCache 2.6 upgrade
Summary 2.2 to 2.6 upgrade
Target Date 08. 11. 2013
Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r3 - 2013-11-11 - FabioMartinelli
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback