IssueNewWNsKernelPanic < CmsTier3

Tags: view all tags
<!-- keep this as a security measure:
   #uncomment if the subject should only be modifiable by the listed groups 
   # * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.CMSAdminGroup
   # * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.CMSAdminGroup
   #uncomment this if you want the page only be viewable by the listed groups
   # * Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.CMSAdminGroup
-->

---+!! %TOPIC%

%TOC%

---++ Symptoms

Summary: %FORMFIELD{"Symptom summary"}%

---++ Occurrences

At what times did this problem occur (used to estimate frequency):

| 2012-07-19/22 |

---++ Observations

<!--
   #collect here the information which may help to better understand the state of the system or services, e.g.
   #log excerpts, strace output, etc.
   #this also may help to identify the problem if similar conditions arise again
-->

=Out of memory: Killed process ... events, all generated by the same UID=536, old nodes survived while nodes in the range t3wn[30-40] affected became unresponsive ( maybe kernel panic ).<br /> 
%TWISTY{
mode="div"
showlink="Show..."
hidelink="Hide"
class="twikiHelp"
}%
<pre>[root@t3service01 07]# grep --color Killed  *  -B 20 -R 
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Node 0 DMA32: 8*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41992kB
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Node 0 Normal: 14*4kB 1*8kB 1*16kB 1*32kB 0*64kB 0*128kB 0*256kB 2*512kB 1*1024kB 1*2048kB 1*4096kB = 8304kB
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Node 0 HighMem: empty
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Node 1 DMA: empty
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Node 1 DMA32: empty
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Node 1 Normal: 0*4kB 1*8kB 1*16kB 3*32kB 5*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9912kB
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Node 1 HighMem: empty
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: 1471 pagecache pages
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Swap cache: add 49421983, delete 49421817, find 23303824/26862469, race 0+104
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Free swap  = 0kB
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Total swap = 11157132kB
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Free swap:            0kB
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: 6815744 pages of RAM
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: 650272 reserved pages
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: 6592 pages shared
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: 256 pages swap cached
19/messages:Jul 19 23:03:53 t3wn24.psi.ch kernel: Out of memory: Killed process 22821, UID 536, (cmsRun).
--
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Node 0 DMA32: 1*4kB 0*8kB 0*16kB 2*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41988kB
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Node 0 Normal: 1*4kB 0*8kB 6*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8260kB
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Node 0 HighMem: empty
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Node 1 DMA: empty
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Node 1 DMA32: empty
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Node 1 Normal: 1*4kB 15*8kB 8*16kB 3*32kB 2*64kB 0*128kB 3*256kB 1*512kB 0*1024kB 0*2048kB 2*4096kB = 9948kB
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Node 1 HighMem: empty
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: 1628 pagecache pages
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Swap cache: add 41449855, delete 41449654, find 19710027/21874517, race 0+121
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Free swap  = 0kB
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Total swap = 11157132kB
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Free swap:            0kB
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: 6815744 pages of RAM
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: 650272 reserved pages
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: 6840 pages shared
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: 251 pages swap cached
20/messages:Jul 20 02:33:54 t3wn27.psi.ch kernel: Out of memory: Killed process 12595, UID 536, (cmsRun).
--
20/messages-Jul 20 03:51:45 t3wn37 kernel: lowmem_reserve[]: 0 0 0 0
20/messages-Jul 20 03:51:45 t3wn37 kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
20/messages-Jul 20 03:51:45 t3wn37 kernel: lowmem_reserve[]: 0 0 0 0
20/messages-Jul 20 03:51:45 t3wn37 kernel: Node 0 DMA: 2*4kB 2*8kB 2*16kB 2*32kB 2*64kB 3*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9848kB
20/messages-Jul 20 03:51:45 t3wn37 kernel: Node 0 DMA32: 3*4kB 2*8kB 0*16kB 2*32kB 1*64kB 0*128kB 3*256kB 0*512kB 0*1024kB 0*2048kB 13*4096kB = 54172kB
20/messages-Jul 20 03:51:45 t3wn37 kernel: Node 0 Normal: 3*4kB 5*8kB 64*16kB 25*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 1*4096kB = 7636kB
20/messages-Jul 20 03:51:45 t3wn37 kernel: Node 0 HighMem: empty
20/messages-Jul 20 03:51:45 t3wn37 kernel: Node 1 DMA: empty
20/messages-Jul 20 03:51:45 t3wn37 kernel: Node 1 DMA32: empty
20/messages-Jul 20 03:51:45 t3wn37 kernel: Node 1 Normal: 1*4kB 0*8kB 1*16kB 3*32kB 1*64kB 37*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 3*4096kB = 18740kB
20/messages-Jul 20 03:51:45 t3wn37 kernel: Node 1 HighMem: empty
20/messages-Jul 20 03:51:45 t3wn37 kernel: 1807 pagecache pages
20/messages-Jul 20 03:51:45 t3wn37 kernel: Swap cache: add 8575777, delete 8578450, find 225232/302624, race 0+7
20/messages-Jul 20 03:51:45 t3wn37 kernel: Free swap  = 0kB
20/messages-Jul 20 03:51:45 t3wn37 kernel: Total swap = 24579320kB
20/messages-Jul 20 03:51:45 t3wn37 kernel: Free swap:            0kB
20/messages-Jul 20 03:51:45 t3wn37 kernel: 12845056 pages of RAM
20/messages-Jul 20 03:51:45 t3wn37 kernel: 491588 reserved pages
20/messages-Jul 20 03:51:45 t3wn37 kernel: 15920 pages shared
20/messages-Jul 20 03:51:45 t3wn37 kernel: 247 pages swap cached
20/messages:Jul 20 03:51:45 t3wn37 kernel: Out of memory: Killed process 9131, UID 536, (cmsRun).
--
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Node 0 DMA32: 0*4kB 1*8kB 2*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41992kB
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Node 0 Normal: 7*4kB 15*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8308kB
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Node 0 HighMem: empty
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Node 1 DMA: empty
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Node 1 DMA32: empty
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Node 1 Normal: 10*4kB 4*8kB 0*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9896kB
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Node 1 HighMem: empty
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: 1464 pagecache pages
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Swap cache: add 21457018, delete 21456862, find 14417084/15447073, race 0+59
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Free swap  = 0kB
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Total swap = 11157132kB
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Free swap:            0kB
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: 6815744 pages of RAM
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: 650272 reserved pages
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: 6598 pages shared
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: 251 pages swap cached
20/messages:Jul 20 23:14:57 t3wn13.psi.ch kernel: Out of memory: Killed process 10257, UID 536, (cmsRun).
--
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Node 0 DMA32: 1*4kB 0*8kB 2*16kB 3*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41988kB
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Node 0 Normal: 9*4kB 3*8kB 1*16kB 3*32kB 3*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8300kB
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Node 0 HighMem: empty
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Node 1 DMA: empty
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Node 1 DMA32: empty
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Node 1 Normal: 3*4kB 7*8kB 6*16kB 5*32kB 0*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9924kB
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Node 1 HighMem: empty
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: 1468 pagecache pages
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Swap cache: add 67384526, delete 67384564, find 25383944/29622322, race 0+133
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Free swap  = 0kB
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Total swap = 11157132kB
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Free swap:            0kB
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: 6815744 pages of RAM
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: 650272 reserved pages
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: 7344 pages shared
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: 245 pages swap cached
20/messages:Jul 20 23:21:28 t3wn22.psi.ch kernel: Out of memory: Killed process 11005, UID 536, (cmsRun).
--
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Node 0 DMA32: 0*4kB 2*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41984kB
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Node 0 Normal: 1*4kB 1*8kB 1*16kB 1*32kB 2*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8252kB
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Node 0 HighMem: empty
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Node 1 DMA: empty
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Node 1 DMA32: empty
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Node 1 Normal: 17*4kB 1*8kB 1*16kB 1*32kB 0*64kB 0*128kB 4*256kB 5*512kB 0*1024kB 1*2048kB 1*4096kB = 9852kB
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Node 1 HighMem: empty
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: 1553 pagecache pages
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Swap cache: add 21827802, delete 21827646, find 14420483/15451327, race 0+59
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Free swap  = 0kB
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Total swap = 11157132kB
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Free swap:            0kB
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: 6815744 pages of RAM
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: 650272 reserved pages
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: 7387 pages shared
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: 251 pages swap cached
20/messages:Jul 20 23:58:39 t3wn13.psi.ch kernel: Out of memory: Killed process 1549, UID 536, (cmsRun).
--
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Node 0 DMA32: 4*4kB 4*8kB 5*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 42016kB
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Node 0 Normal: 11*4kB 1*8kB 4*16kB 0*32kB 2*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8308kB
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Node 1 Normal: 13*4kB 11*8kB 2*16kB 2*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9964kB
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: 1492 pagecache pages
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Swap cache: add 49965510, delete 49965322, find 22930536/25999153, race 0+132
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: 6600 pages shared
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: 260 pages swap cached
21/messages:Jul 21 00:09:30 t3wn16.psi.ch kernel: Out of memory: Killed process 24321, UID 536, (cmsRun).
--
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Node 0 DMA32: 8*4kB 0*8kB 2*16kB 3*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 42016kB
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Node 0 Normal: 25*4kB 4*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8308kB
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Node 1 Normal: 1*4kB 1*8kB 2*16kB 4*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9964kB
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: 1682 pagecache pages
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Swap cache: add 17035951, delete 17035747, find 11221659/11357884, race 0+113
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: 7726 pages shared
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: 250 pages swap cached
21/messages:Jul 21 00:14:43 t3wn12.psi.ch kernel: Out of memory: Killed process 2677, UID 536, (cmsRun).
--
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Node 0 DMA32: 3*4kB 0*8kB 1*16kB 3*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41980kB
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Node 0 Normal: 5*4kB 2*8kB 4*16kB 2*32kB 1*64kB 1*128kB 1*256kB 1*512kB 3*1024kB 0*2048kB 1*4096kB = 8292kB
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Node 1 Normal: 3*4kB 0*8kB 0*16kB 1*32kB 2*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9900kB
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: 1470 pagecache pages
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Swap cache: add 45503174, delete 45503208, find 19500707/23162145, race 9+363
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: 6544 pages shared
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: 248 pages swap cached
21/messages:Jul 21 00:39:20 t3wn23.psi.ch kernel: Out of memory: Killed process 26870, UID 536, (cmsRun).
--
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Node 0 DMA32: 11*4kB 5*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41972kB
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Node 0 Normal: 2*4kB 0*8kB 0*16kB 4*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8264kB
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Node 1 Normal: 0*4kB 0*8kB 8*16kB 4*32kB 6*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9856kB
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: 1444 pagecache pages
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Swap cache: add 37698580, delete 37698415, find 19903054/22239623, race 0+91
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: 6618 pages shared
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: 249 pages swap cached
21/messages:Jul 21 00:44:52 t3wn19.psi.ch kernel: Out of memory: Killed process 7629, UID 536, (cmsRun).
--
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Node 0 DMA32: 8*4kB 2*8kB 2*16kB 2*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 42000kB
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Node 0 Normal: 8*4kB 1*8kB 1*16kB 1*32kB 8*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 8280kB
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Node 1 Normal: 1*4kB 1*8kB 3*16kB 3*32kB 1*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9948kB
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: 1625 pagecache pages
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Swap cache: add 50943674, delete 50943494, find 20737417/24071920, race 0+66
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: 6928 pages shared
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: 250 pages swap cached
21/messages:Jul 21 01:17:17 t3wn28.psi.ch kernel: Out of memory: Killed process 13393, UID 536, (cmsRun).
--
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Node 0 DMA32: 2*4kB 1*8kB 1*16kB 0*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 42016kB
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Node 0 Normal: 0*4kB 1*8kB 2*16kB 1*32kB 0*64kB 2*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8264kB
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Node 1 Normal: 1*4kB 0*8kB 4*16kB 0*32kB 0*64kB 1*128kB 0*256kB 7*512kB 0*1024kB 1*2048kB 1*4096kB = 9924kB
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: 1483 pagecache pages
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Swap cache: add 22476388, delete 22476232, find 14424336/15455293, race 0+59
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: 6564 pages shared
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: 251 pages swap cached
21/messages:Jul 21 01:19:10 t3wn13.psi.ch kernel: Out of memory: Killed process 3442, UID 536, (cmsRun).
--
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Node 0 DMA32: 3*4kB 1*8kB 3*16kB 2*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41988kB
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Node 0 Normal: 6*4kB 2*8kB 2*16kB 1*32kB 0*64kB 2*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8296kB
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Node 1 Normal: 8*4kB 9*8kB 0*16kB 1*32kB 1*64kB 28*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 1*4096kB = 9928kB
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: 1521 pagecache pages
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Swap cache: add 50751008, delete 50750820, find 22933780/26002611, race 0+142
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: 7572 pages shared
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: 260 pages swap cached
21/messages:Jul 21 01:21:39 t3wn16.psi.ch kernel: Out of memory: Killed process 25931, UID 536, (cmsRun).
--
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Node 0 DMA32: 3*4kB 0*8kB 4*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41996kB
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Node 0 Normal: 13*4kB 1*8kB 3*16kB 0*32kB 0*64kB 4*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8300kB
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Node 1 Normal: 18*4kB 6*8kB 2*16kB 0*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9944kB
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: 1469 pagecache pages
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Swap cache: add 53473730, delete 53473757, find 21178488/24498105, race 1+139
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: 6476 pages shared
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: 251 pages swap cached
21/messages:Jul 21 01:48:37 t3wn26.psi.ch kernel: Out of memory: Killed process 3957, UID 536, (cmsRun).
--
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Node 0 DMA32: 4*4kB 2*8kB 1*16kB 3*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 42000kB
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Node 0 Normal: 1*4kB 0*8kB 2*16kB 3*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8260kB
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Node 1 Normal: 2*4kB 9*8kB 0*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9968kB
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: 1469 pagecache pages
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Swap cache: add 68665185, delete 68665223, find 25416751/29660627, race 0+135
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: 7085 pages shared
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: 245 pages swap cached
21/messages:Jul 21 01:51:56 t3wn22.psi.ch kernel: Out of memory: Killed process 4619, UID 536, (cmsRun).
--
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 01:53:03 t3bdii02.psi.ch sendmail[18705]: q6KNr1ai018705: to=root, ctladdr=root (0/0), delay=00:00:02, xdelay=00:00:00, mailer=relay, pri=30691, relay=psquad.psi.ch. [192.33.120.33], dsn=2.0.0, stat=Sent (ok:  Message 55623335 accepted)
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Node 0 DMA32: 13*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 42012kB
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Node 0 Normal: 0*4kB 1*8kB 0*16kB 2*32kB 0*64kB 4*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8264kB
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Node 1 Normal: 6*4kB 3*8kB 0*16kB 0*32kB 1*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9968kB
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: 1475 pagecache pages
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Swap cache: add 40024629, delete 40024468, find 20645340/22915715, race 0+123
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: 7328 pages shared
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: 248 pages swap cached
21/messages:Jul 21 01:53:02 t3wn21.psi.ch kernel: Out of memory: Killed process 20497, UID 536, (cmsRun).
--
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Node 0 DMA32: 6*4kB 4*8kB 2*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41976kB
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Node 0 Normal: 1*4kB 3*8kB 3*16kB 1*32kB 2*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8300kB
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Node 1 Normal: 1*4kB 0*8kB 4*16kB 5*32kB 2*64kB 1*128kB 1*256kB 4*512kB 1*1024kB 1*2048kB 1*4096kB = 9956kB
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: 1472 pagecache pages
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Swap cache: add 38708341, delete 38708176, find 19931323/22274932, race 0+91
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: 6503 pages shared
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: 249 pages swap cached
21/messages:Jul 21 02:07:12 t3wn19.psi.ch kernel: Out of memory: Killed process 14435, UID 536, (cmsRun).
--
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Node 0 DMA32: 0*4kB 16*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 42016kB
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Node 0 Normal: 5*4kB 4*8kB 6*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8308kB
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Node 1 Normal: 1*4kB 0*8kB 6*16kB 1*32kB 3*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9924kB
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: 1661 pagecache pages
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Swap cache: add 45907862, delete 45907430, find 21772877/23912683, race 0+45
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: 7264 pages shared
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: 472 pages swap cached
21/messages:Jul 21 17:40:18 t3wn14.psi.ch kernel: Out of memory: Killed process 29665, UID 536, (cmsRun).
--
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Node 0 DMA32: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41960kB
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Node 0 Normal: 0*4kB 14*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8272kB
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Node 1 Normal: 2*4kB 1*8kB 2*16kB 4*32kB 1*64kB 0*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9968kB
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: 1563 pagecache pages
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Swap cache: add 60609033, delete 60608868, find 23789215/27435897, race 0+110
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: 7084 pages shared
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: 256 pages swap cached
21/messages:Jul 21 21:52:09 t3wn24.psi.ch kernel: Out of memory: Killed process 15671, UID 536, (cmsRun).
--
21/messages-Jul 21 22:09:51 t3wn35 kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 22:09:51 t3wn35 kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 22:09:51 t3wn35 kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 22:09:51 t3wn35 kernel: Node 0 DMA: 2*4kB 2*8kB 2*16kB 2*32kB 2*64kB 3*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9848kB
21/messages-Jul 21 22:09:51 t3wn35 kernel: Node 0 DMA32: 1*4kB 1*8kB 0*16kB 1*32kB 2*64kB 4*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 13*4096kB = 54188kB
21/messages-Jul 21 22:09:51 t3wn35 kernel: Node 0 Normal: 0*4kB 9*8kB 2*16kB 58*32kB 4*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 1*4096kB = 7592kB
21/messages-Jul 21 22:09:51 t3wn35 kernel: Node 0 HighMem: empty
21/messages-Jul 21 22:09:51 t3wn35 kernel: Node 1 DMA: empty
21/messages-Jul 21 22:09:51 t3wn35 kernel: Node 1 DMA32: empty
21/messages-Jul 21 22:09:51 t3wn35 kernel: Node 1 Normal: 2*4kB 0*8kB 4*16kB 4*32kB 2*64kB 2*128kB 17*256kB 1*512kB 1*1024kB 0*2048kB 3*4096kB = 18760kB
21/messages-Jul 21 22:09:51 t3wn35 kernel: Node 1 HighMem: empty
21/messages-Jul 21 22:09:51 t3wn35 kernel: 1758 pagecache pages
21/messages-Jul 21 22:09:51 t3wn35 kernel: Swap cache: add 17053988, delete 17057516, find 953040/1173587, race 0+7
21/messages-Jul 21 22:09:51 t3wn35 kernel: Free swap  = 0kB
21/messages-Jul 21 22:09:51 t3wn35 kernel: Total swap = 24579320kB
21/messages-Jul 21 22:09:51 t3wn35 kernel: Free swap:            0kB
21/messages-Jul 21 22:09:51 t3wn35 kernel: 12845056 pages of RAM
21/messages-Jul 21 22:09:51 t3wn35 kernel: 491588 reserved pages
21/messages-Jul 21 22:09:51 t3wn35 kernel: 15887 pages shared
21/messages-Jul 21 22:09:51 t3wn35 kernel: 249 pages swap cached
21/messages:Jul 21 22:09:51 t3wn35 kernel: Out of memory: Killed process 10085, UID 536, (cmsRun).
--
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Node 0 DMA32: 1*4kB 0*8kB 0*16kB 2*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41988kB
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Node 0 Normal: 7*4kB 1*8kB 4*16kB 0*32kB 0*64kB 0*128kB 2*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8292kB
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Node 1 Normal: 24*4kB 1*8kB 0*16kB 3*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9928kB
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: 1714 pagecache pages
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Swap cache: add 61553373, delete 61553209, find 23792826/27439699, race 0+110
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: 7552 pages shared
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: 256 pages swap cached
21/messages:Jul 21 23:12:06 t3wn24.psi.ch kernel: Out of memory: Killed process 17887, UID 536, (cmsRun).</pre>

%ENDTWISTY%

---++ Solution or Workaround

---++ Monitoring for this condition

<!--
   #how can this condition be recognized automatically, if at all?
-->

-- Main.FabioMartinelli - 2012-07-23
IssueForm
Affected Service	The whole OS
Symptom summary	OOM Killer is invoked but sometimes it's not enoguh to make survive the OS.
Reason Understood	yes
Solution Exists	yes
Obsolete	no
Topic revision: r1 - 2012-07-23 - FabioMartinelli
CmsTier3
User Pages
Main Page
Policies
Physics Groups
Steering Board Meetings
Admin Pages
AdminArea
Cluster Specs