IssueNewWNsKernelPanic

[root@t3service01 07]# grep --color Killed  *  -B 20 -R 
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Node 0 DMA32: 8*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41992kB
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Node 0 Normal: 14*4kB 1*8kB 1*16kB 1*32kB 0*64kB 0*128kB 0*256kB 2*512kB 1*1024kB 1*2048kB 1*4096kB = 8304kB
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Node 0 HighMem: empty
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Node 1 DMA: empty
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Node 1 DMA32: empty
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Node 1 Normal: 0*4kB 1*8kB 1*16kB 3*32kB 5*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9912kB
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Node 1 HighMem: empty
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: 1471 pagecache pages
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Swap cache: add 49421983, delete 49421817, find 23303824/26862469, race 0+104
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Free swap  = 0kB
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Total swap = 11157132kB
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: Free swap:            0kB
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: 6815744 pages of RAM
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: 650272 reserved pages
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: 6592 pages shared
19/messages-Jul 19 23:03:53 t3wn24.psi.ch kernel: 256 pages swap cached
19/messages:Jul 19 23:03:53 t3wn24.psi.ch kernel: Out of memory: Killed process 22821, UID 536, (cmsRun).
--
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Node 0 DMA32: 1*4kB 0*8kB 0*16kB 2*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41988kB
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Node 0 Normal: 1*4kB 0*8kB 6*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8260kB
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Node 0 HighMem: empty
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Node 1 DMA: empty
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Node 1 DMA32: empty
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Node 1 Normal: 1*4kB 15*8kB 8*16kB 3*32kB 2*64kB 0*128kB 3*256kB 1*512kB 0*1024kB 0*2048kB 2*4096kB = 9948kB
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Node 1 HighMem: empty
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: 1628 pagecache pages
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Swap cache: add 41449855, delete 41449654, find 19710027/21874517, race 0+121
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Free swap  = 0kB
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Total swap = 11157132kB
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: Free swap:            0kB
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: 6815744 pages of RAM
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: 650272 reserved pages
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: 6840 pages shared
20/messages-Jul 20 02:33:54 t3wn27.psi.ch kernel: 251 pages swap cached
20/messages:Jul 20 02:33:54 t3wn27.psi.ch kernel: Out of memory: Killed process 12595, UID 536, (cmsRun).
--
20/messages-Jul 20 03:51:45 t3wn37 kernel: lowmem_reserve[]: 0 0 0 0
20/messages-Jul 20 03:51:45 t3wn37 kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
20/messages-Jul 20 03:51:45 t3wn37 kernel: lowmem_reserve[]: 0 0 0 0
20/messages-Jul 20 03:51:45 t3wn37 kernel: Node 0 DMA: 2*4kB 2*8kB 2*16kB 2*32kB 2*64kB 3*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9848kB
20/messages-Jul 20 03:51:45 t3wn37 kernel: Node 0 DMA32: 3*4kB 2*8kB 0*16kB 2*32kB 1*64kB 0*128kB 3*256kB 0*512kB 0*1024kB 0*2048kB 13*4096kB = 54172kB
20/messages-Jul 20 03:51:45 t3wn37 kernel: Node 0 Normal: 3*4kB 5*8kB 64*16kB 25*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 1*4096kB = 7636kB
20/messages-Jul 20 03:51:45 t3wn37 kernel: Node 0 HighMem: empty
20/messages-Jul 20 03:51:45 t3wn37 kernel: Node 1 DMA: empty
20/messages-Jul 20 03:51:45 t3wn37 kernel: Node 1 DMA32: empty
20/messages-Jul 20 03:51:45 t3wn37 kernel: Node 1 Normal: 1*4kB 0*8kB 1*16kB 3*32kB 1*64kB 37*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 3*4096kB = 18740kB
20/messages-Jul 20 03:51:45 t3wn37 kernel: Node 1 HighMem: empty
20/messages-Jul 20 03:51:45 t3wn37 kernel: 1807 pagecache pages
20/messages-Jul 20 03:51:45 t3wn37 kernel: Swap cache: add 8575777, delete 8578450, find 225232/302624, race 0+7
20/messages-Jul 20 03:51:45 t3wn37 kernel: Free swap  = 0kB
20/messages-Jul 20 03:51:45 t3wn37 kernel: Total swap = 24579320kB
20/messages-Jul 20 03:51:45 t3wn37 kernel: Free swap:            0kB
20/messages-Jul 20 03:51:45 t3wn37 kernel: 12845056 pages of RAM
20/messages-Jul 20 03:51:45 t3wn37 kernel: 491588 reserved pages
20/messages-Jul 20 03:51:45 t3wn37 kernel: 15920 pages shared
20/messages-Jul 20 03:51:45 t3wn37 kernel: 247 pages swap cached
20/messages:Jul 20 03:51:45 t3wn37 kernel: Out of memory: Killed process 9131, UID 536, (cmsRun).
--
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Node 0 DMA32: 0*4kB 1*8kB 2*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41992kB
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Node 0 Normal: 7*4kB 15*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8308kB
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Node 0 HighMem: empty
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Node 1 DMA: empty
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Node 1 DMA32: empty
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Node 1 Normal: 10*4kB 4*8kB 0*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9896kB
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Node 1 HighMem: empty
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: 1464 pagecache pages
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Swap cache: add 21457018, delete 21456862, find 14417084/15447073, race 0+59
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Free swap  = 0kB
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Total swap = 11157132kB
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: Free swap:            0kB
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: 6815744 pages of RAM
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: 650272 reserved pages
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: 6598 pages shared
20/messages-Jul 20 23:14:57 t3wn13.psi.ch kernel: 251 pages swap cached
20/messages:Jul 20 23:14:57 t3wn13.psi.ch kernel: Out of memory: Killed process 10257, UID 536, (cmsRun).
--
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Node 0 DMA32: 1*4kB 0*8kB 2*16kB 3*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41988kB
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Node 0 Normal: 9*4kB 3*8kB 1*16kB 3*32kB 3*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8300kB
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Node 0 HighMem: empty
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Node 1 DMA: empty
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Node 1 DMA32: empty
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Node 1 Normal: 3*4kB 7*8kB 6*16kB 5*32kB 0*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9924kB
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Node 1 HighMem: empty
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: 1468 pagecache pages
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Swap cache: add 67384526, delete 67384564, find 25383944/29622322, race 0+133
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Free swap  = 0kB
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Total swap = 11157132kB
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: Free swap:            0kB
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: 6815744 pages of RAM
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: 650272 reserved pages
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: 7344 pages shared
20/messages-Jul 20 23:21:28 t3wn22.psi.ch kernel: 245 pages swap cached
20/messages:Jul 20 23:21:28 t3wn22.psi.ch kernel: Out of memory: Killed process 11005, UID 536, (cmsRun).
--
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Node 0 DMA32: 0*4kB 2*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41984kB
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Node 0 Normal: 1*4kB 1*8kB 1*16kB 1*32kB 2*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8252kB
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Node 0 HighMem: empty
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Node 1 DMA: empty
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Node 1 DMA32: empty
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Node 1 Normal: 17*4kB 1*8kB 1*16kB 1*32kB 0*64kB 0*128kB 4*256kB 5*512kB 0*1024kB 1*2048kB 1*4096kB = 9852kB
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Node 1 HighMem: empty
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: 1553 pagecache pages
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Swap cache: add 21827802, delete 21827646, find 14420483/15451327, race 0+59
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Free swap  = 0kB
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Total swap = 11157132kB
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: Free swap:            0kB
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: 6815744 pages of RAM
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: 650272 reserved pages
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: 7387 pages shared
20/messages-Jul 20 23:58:39 t3wn13.psi.ch kernel: 251 pages swap cached
20/messages:Jul 20 23:58:39 t3wn13.psi.ch kernel: Out of memory: Killed process 1549, UID 536, (cmsRun).
--
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Node 0 DMA32: 4*4kB 4*8kB 5*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 42016kB
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Node 0 Normal: 11*4kB 1*8kB 4*16kB 0*32kB 2*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8308kB
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Node 1 Normal: 13*4kB 11*8kB 2*16kB 2*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9964kB
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: 1492 pagecache pages
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Swap cache: add 49965510, delete 49965322, find 22930536/25999153, race 0+132
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: 6600 pages shared
21/messages-Jul 21 00:09:30 t3wn16.psi.ch kernel: 260 pages swap cached
21/messages:Jul 21 00:09:30 t3wn16.psi.ch kernel: Out of memory: Killed process 24321, UID 536, (cmsRun).
--
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Node 0 DMA32: 8*4kB 0*8kB 2*16kB 3*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 42016kB
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Node 0 Normal: 25*4kB 4*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8308kB
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Node 1 Normal: 1*4kB 1*8kB 2*16kB 4*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9964kB
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: 1682 pagecache pages
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Swap cache: add 17035951, delete 17035747, find 11221659/11357884, race 0+113
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: 7726 pages shared
21/messages-Jul 21 00:14:43 t3wn12.psi.ch kernel: 250 pages swap cached
21/messages:Jul 21 00:14:43 t3wn12.psi.ch kernel: Out of memory: Killed process 2677, UID 536, (cmsRun).
--
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Node 0 DMA32: 3*4kB 0*8kB 1*16kB 3*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41980kB
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Node 0 Normal: 5*4kB 2*8kB 4*16kB 2*32kB 1*64kB 1*128kB 1*256kB 1*512kB 3*1024kB 0*2048kB 1*4096kB = 8292kB
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Node 1 Normal: 3*4kB 0*8kB 0*16kB 1*32kB 2*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9900kB
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: 1470 pagecache pages
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Swap cache: add 45503174, delete 45503208, find 19500707/23162145, race 9+363
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: 6544 pages shared
21/messages-Jul 21 00:39:20 t3wn23.psi.ch kernel: 248 pages swap cached
21/messages:Jul 21 00:39:20 t3wn23.psi.ch kernel: Out of memory: Killed process 26870, UID 536, (cmsRun).
--
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Node 0 DMA32: 11*4kB 5*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41972kB
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Node 0 Normal: 2*4kB 0*8kB 0*16kB 4*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8264kB
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Node 1 Normal: 0*4kB 0*8kB 8*16kB 4*32kB 6*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9856kB
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: 1444 pagecache pages
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Swap cache: add 37698580, delete 37698415, find 19903054/22239623, race 0+91
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: 6618 pages shared
21/messages-Jul 21 00:44:52 t3wn19.psi.ch kernel: 249 pages swap cached
21/messages:Jul 21 00:44:52 t3wn19.psi.ch kernel: Out of memory: Killed process 7629, UID 536, (cmsRun).
--
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Node 0 DMA32: 8*4kB 2*8kB 2*16kB 2*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 42000kB
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Node 0 Normal: 8*4kB 1*8kB 1*16kB 1*32kB 8*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 8280kB
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Node 1 Normal: 1*4kB 1*8kB 3*16kB 3*32kB 1*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9948kB
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: 1625 pagecache pages
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Swap cache: add 50943674, delete 50943494, find 20737417/24071920, race 0+66
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: 6928 pages shared
21/messages-Jul 21 01:17:17 t3wn28.psi.ch kernel: 250 pages swap cached
21/messages:Jul 21 01:17:17 t3wn28.psi.ch kernel: Out of memory: Killed process 13393, UID 536, (cmsRun).
--
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Node 0 DMA32: 2*4kB 1*8kB 1*16kB 0*32kB 0*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 42016kB
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Node 0 Normal: 0*4kB 1*8kB 2*16kB 1*32kB 0*64kB 2*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8264kB
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Node 1 Normal: 1*4kB 0*8kB 4*16kB 0*32kB 0*64kB 1*128kB 0*256kB 7*512kB 0*1024kB 1*2048kB 1*4096kB = 9924kB
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: 1483 pagecache pages
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Swap cache: add 22476388, delete 22476232, find 14424336/15455293, race 0+59
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: 6564 pages shared
21/messages-Jul 21 01:19:10 t3wn13.psi.ch kernel: 251 pages swap cached
21/messages:Jul 21 01:19:10 t3wn13.psi.ch kernel: Out of memory: Killed process 3442, UID 536, (cmsRun).
--
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Node 0 DMA32: 3*4kB 1*8kB 3*16kB 2*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41988kB
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Node 0 Normal: 6*4kB 2*8kB 2*16kB 1*32kB 0*64kB 2*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8296kB
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Node 1 Normal: 8*4kB 9*8kB 0*16kB 1*32kB 1*64kB 28*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 1*4096kB = 9928kB
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: 1521 pagecache pages
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Swap cache: add 50751008, delete 50750820, find 22933780/26002611, race 0+142
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: 7572 pages shared
21/messages-Jul 21 01:21:39 t3wn16.psi.ch kernel: 260 pages swap cached
21/messages:Jul 21 01:21:39 t3wn16.psi.ch kernel: Out of memory: Killed process 25931, UID 536, (cmsRun).
--
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Node 0 DMA32: 3*4kB 0*8kB 4*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41996kB
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Node 0 Normal: 13*4kB 1*8kB 3*16kB 0*32kB 0*64kB 4*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8300kB
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Node 1 Normal: 18*4kB 6*8kB 2*16kB 0*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9944kB
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: 1469 pagecache pages
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Swap cache: add 53473730, delete 53473757, find 21178488/24498105, race 1+139
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: 6476 pages shared
21/messages-Jul 21 01:48:37 t3wn26.psi.ch kernel: 251 pages swap cached
21/messages:Jul 21 01:48:37 t3wn26.psi.ch kernel: Out of memory: Killed process 3957, UID 536, (cmsRun).
--
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Node 0 DMA32: 4*4kB 2*8kB 1*16kB 3*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 42000kB
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Node 0 Normal: 1*4kB 0*8kB 2*16kB 3*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8260kB
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Node 1 Normal: 2*4kB 9*8kB 0*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9968kB
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: 1469 pagecache pages
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Swap cache: add 68665185, delete 68665223, find 25416751/29660627, race 0+135
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: 7085 pages shared
21/messages-Jul 21 01:51:56 t3wn22.psi.ch kernel: 245 pages swap cached
21/messages:Jul 21 01:51:56 t3wn22.psi.ch kernel: Out of memory: Killed process 4619, UID 536, (cmsRun).
--
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 01:53:03 t3bdii02.psi.ch sendmail[18705]: q6KNr1ai018705: to=root, ctladdr=root (0/0), delay=00:00:02, xdelay=00:00:00, mailer=relay, pri=30691, relay=psquad.psi.ch. [192.33.120.33], dsn=2.0.0, stat=Sent (ok:  Message 55623335 accepted)
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Node 0 DMA32: 13*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 42012kB
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Node 0 Normal: 0*4kB 1*8kB 0*16kB 2*32kB 0*64kB 4*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8264kB
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Node 1 Normal: 6*4kB 3*8kB 0*16kB 0*32kB 1*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9968kB
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: 1475 pagecache pages
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Swap cache: add 40024629, delete 40024468, find 20645340/22915715, race 0+123
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: 7328 pages shared
21/messages-Jul 21 01:53:02 t3wn21.psi.ch kernel: 248 pages swap cached
21/messages:Jul 21 01:53:02 t3wn21.psi.ch kernel: Out of memory: Killed process 20497, UID 536, (cmsRun).
--
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Node 0 DMA32: 6*4kB 4*8kB 2*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41976kB
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Node 0 Normal: 1*4kB 3*8kB 3*16kB 1*32kB 2*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8300kB
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Node 1 Normal: 1*4kB 0*8kB 4*16kB 5*32kB 2*64kB 1*128kB 1*256kB 4*512kB 1*1024kB 1*2048kB 1*4096kB = 9956kB
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: 1472 pagecache pages
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Swap cache: add 38708341, delete 38708176, find 19931323/22274932, race 0+91
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: 6503 pages shared
21/messages-Jul 21 02:07:12 t3wn19.psi.ch kernel: 249 pages swap cached
21/messages:Jul 21 02:07:12 t3wn19.psi.ch kernel: Out of memory: Killed process 14435, UID 536, (cmsRun).
--
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Node 0 DMA32: 0*4kB 16*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 42016kB
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Node 0 Normal: 5*4kB 4*8kB 6*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8308kB
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Node 1 Normal: 1*4kB 0*8kB 6*16kB 1*32kB 3*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9924kB
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: 1661 pagecache pages
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Swap cache: add 45907862, delete 45907430, find 21772877/23912683, race 0+45
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: 7264 pages shared
21/messages-Jul 21 17:40:18 t3wn14.psi.ch kernel: 472 pages swap cached
21/messages:Jul 21 17:40:18 t3wn14.psi.ch kernel: Out of memory: Killed process 29665, UID 536, (cmsRun).
--
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Node 0 DMA32: 0*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41960kB
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Node 0 Normal: 0*4kB 14*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8272kB
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Node 1 Normal: 2*4kB 1*8kB 2*16kB 4*32kB 1*64kB 0*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9968kB
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: 1563 pagecache pages
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Swap cache: add 60609033, delete 60608868, find 23789215/27435897, race 0+110
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: 7084 pages shared
21/messages-Jul 21 21:52:09 t3wn24.psi.ch kernel: 256 pages swap cached
21/messages:Jul 21 21:52:09 t3wn24.psi.ch kernel: Out of memory: Killed process 15671, UID 536, (cmsRun).
--
21/messages-Jul 21 22:09:51 t3wn35 kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 22:09:51 t3wn35 kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 22:09:51 t3wn35 kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 22:09:51 t3wn35 kernel: Node 0 DMA: 2*4kB 2*8kB 2*16kB 2*32kB 2*64kB 3*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9848kB
21/messages-Jul 21 22:09:51 t3wn35 kernel: Node 0 DMA32: 1*4kB 1*8kB 0*16kB 1*32kB 2*64kB 4*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 13*4096kB = 54188kB
21/messages-Jul 21 22:09:51 t3wn35 kernel: Node 0 Normal: 0*4kB 9*8kB 2*16kB 58*32kB 4*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 1*4096kB = 7592kB
21/messages-Jul 21 22:09:51 t3wn35 kernel: Node 0 HighMem: empty
21/messages-Jul 21 22:09:51 t3wn35 kernel: Node 1 DMA: empty
21/messages-Jul 21 22:09:51 t3wn35 kernel: Node 1 DMA32: empty
21/messages-Jul 21 22:09:51 t3wn35 kernel: Node 1 Normal: 2*4kB 0*8kB 4*16kB 4*32kB 2*64kB 2*128kB 17*256kB 1*512kB 1*1024kB 0*2048kB 3*4096kB = 18760kB
21/messages-Jul 21 22:09:51 t3wn35 kernel: Node 1 HighMem: empty
21/messages-Jul 21 22:09:51 t3wn35 kernel: 1758 pagecache pages
21/messages-Jul 21 22:09:51 t3wn35 kernel: Swap cache: add 17053988, delete 17057516, find 953040/1173587, race 0+7
21/messages-Jul 21 22:09:51 t3wn35 kernel: Free swap  = 0kB
21/messages-Jul 21 22:09:51 t3wn35 kernel: Total swap = 24579320kB
21/messages-Jul 21 22:09:51 t3wn35 kernel: Free swap:            0kB
21/messages-Jul 21 22:09:51 t3wn35 kernel: 12845056 pages of RAM
21/messages-Jul 21 22:09:51 t3wn35 kernel: 491588 reserved pages
21/messages-Jul 21 22:09:51 t3wn35 kernel: 15887 pages shared
21/messages-Jul 21 22:09:51 t3wn35 kernel: 249 pages swap cached
21/messages:Jul 21 22:09:51 t3wn35 kernel: Out of memory: Killed process 10085, UID 536, (cmsRun).
--
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Node 1 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: lowmem_reserve[]: 0 0 0 0
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Node 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10024kB
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Node 0 DMA32: 1*4kB 0*8kB 0*16kB 2*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 10*4096kB = 41988kB
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Node 0 Normal: 7*4kB 1*8kB 4*16kB 0*32kB 0*64kB 0*128kB 2*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8292kB
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Node 0 HighMem: empty
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Node 1 DMA: empty
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Node 1 DMA32: empty
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Node 1 Normal: 24*4kB 1*8kB 0*16kB 3*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9928kB
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Node 1 HighMem: empty
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: 1714 pagecache pages
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Swap cache: add 61553373, delete 61553209, find 23792826/27439699, race 0+110
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Free swap  = 0kB
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Total swap = 11157132kB
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: Free swap:            0kB
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: 6815744 pages of RAM
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: 650272 reserved pages
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: 7552 pages shared
21/messages-Jul 21 23:12:06 t3wn24.psi.ch kernel: 256 pages swap cached
21/messages:Jul 21 23:12:06 t3wn24.psi.ch kernel: Out of memory: Killed process 17887, UID 536, (cmsRun).