gistfile1.txt

Jan 12 15:01:48 lb02 kernel: [88879.477199] Free swap  = 917496kB
Jan 12 15:01:48 lb02 kernel: [88879.477201] Total swap = 917496kB
Jan 12 15:01:48 lb02 kernel: [88879.481930] 447488 pages RAM
Jan 12 15:01:48 lb02 kernel: [88879.481933] 263682 pages HighMem
Jan 12 15:01:48 lb02 kernel: [88879.481935] 10460 pages reserved
Jan 12 15:01:48 lb02 kernel: [88879.481936] 11249 pages shared
Jan 12 15:01:48 lb02 kernel: [88879.481937] 329651 pages non-shared
Jan 12 15:01:48 lb02 kernel: [88879.502144] rsyslogd invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0
Jan 12 15:01:48 lb02 kernel: [88879.502149] rsyslogd cpuset=/ mems_allowed=0
Jan 12 15:01:48 lb02 kernel: [88879.502153] Pid: 23973, comm: rsyslogd Not tainted 2.6.32-309-ec2 #18-Ubuntu
Jan 12 15:01:48 lb02 kernel: [88879.502155] Call Trace:
Jan 12 15:01:48 lb02 kernel: [88879.502165]  [<c0199648>] oom_kill_process+0xb8/0x1e0
Jan 12 15:01:48 lb02 kernel: [88879.502168]  [<c01997c1>] __out_of_memory+0x51/0xa0
Jan 12 15:01:48 lb02 kernel: [88879.502171]  [<c0199862>] out_of_memory+0x52/0xa0
Jan 12 15:01:48 lb02 kernel: [88879.502175]  [<c019bf77>] __alloc_pages_slowpath+0x407/0x4a0
Jan 12 15:01:48 lb02 kernel: [88879.502178]  [<c019c15d>] __alloc_pages_nodemask+0x14d/0x160
Jan 12 15:01:48 lb02 kernel: [88879.502181]  [<c019c187>] __get_free_pages+0x17/0x30
Jan 12 15:01:48 lb02 kernel: [88879.502186]  [<c012f858>] dup_task_struct+0x38/0x120
Jan 12 15:01:48 lb02 kernel: [88879.502189]  [<c0130938>] copy_process+0x88/0xc10
Jan 12 15:01:48 lb02 kernel: [88879.502193]  [<c014c780>] ? autoremove_wake_function+0x0/0x40
Jan 12 15:01:48 lb02 kernel: [88879.502196]  [<c013153e>] do_fork+0x7e/0x390
Jan 12 15:01:48 lb02 kernel: [88879.502200]  [<c0155ab6>] ? getnstimeofday+0x56/0x110
Jan 12 15:01:48 lb02 kernel: [88879.502203]  [<c01027e4>] sys_clone+0x34/0x40
Jan 12 15:01:48 lb02 kernel: [88879.502206]  [<c01047e0>] syscall_call+0x7/0xb
Jan 12 15:01:48 lb02 kernel: [88879.502208] Mem-Info:
Jan 12 15:01:48 lb02 kernel: [88879.502210] DMA per-cpu:
Jan 12 15:01:48 lb02 kernel: [88879.502212] CPU    0: hi:    0, btch:   1 usd:   0
Jan 12 15:01:48 lb02 kernel: [88879.502214] CPU    1: hi:    0, btch:   1 usd:   0
Jan 12 15:01:48 lb02 kernel: [88879.502215] Normal per-cpu:
Jan 12 15:01:48 lb02 kernel: [88879.502217] CPU    0: hi:  155, btch:  38 usd:   0
Jan 12 15:01:48 lb02 kernel: [88879.502219] CPU    1: hi:  155, btch:  38 usd:   0
Jan 12 15:01:48 lb02 kernel: [88879.502220] HighMem per-cpu:
Jan 12 15:01:48 lb02 kernel: [88879.502221] CPU    0: hi:  155, btch:  38 usd: 128
Jan 12 15:01:48 lb02 kernel: [88879.502223] CPU    1: hi:  155, btch:  38 usd:   0
Jan 12 15:01:48 lb02 kernel: [88879.502227] active_anon:131242 inactive_anon:21306 isolated_anon:32
Jan 12 15:01:48 lb02 kernel: [88879.502228]  active_file:2772 inactive_file:5007 isolated_file:0
Jan 12 15:01:48 lb02 kernel: [88879.502229]  unevictable:0 dirty:0 writeback:0 unstable:0
Jan 12 15:01:48 lb02 kernel: [88879.502230]  free:101422 slab_reclaimable:16294 slab_unreclaimable:46925
Jan 12 15:01:48 lb02 kernel: [88879.502231]  mapped:2366 shmem:54 pagetables:0 bounce:0
Jan 12 15:01:48 lb02 kernel: [88879.502236] DMA free:2848kB min:76kB low:92kB high:112kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:16256kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:488kB slab_unreclaimable:1444kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jan 12 15:01:48 lb02 kernel: [88879.502241] lowmem_reserve[]: 0 696 1710 1710
Jan 12 15:01:48 lb02 kernel: [88879.502247] Normal free:4224kB min:3336kB low:4168kB high:5004kB active_anon:0kB inactive_anon:0kB active_file:108kB inactive_file:88kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:713224kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:64688kB slab_unreclaimable:186256kB kernel_stack:744kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jan 12 15:01:48 lb02 kernel: [88879.502252] lowmem_reserve[]: 0 0 8111 8111
Jan 12 15:01:48 lb02 kernel: [88879.502259] HighMem free:398616kB min:512kB low:1724kB high:2940kB active_anon:524968kB inactive_anon:85224kB active_file:10980kB inactive_file:19940kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:1038292kB mlocked:0kB dirty:0kB writeback:0kB mapped:9460kB shmem:216kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jan 12 15:01:48 lb02 kernel: [88879.502263] lowmem_reserve[]: 0 0 0 0
Jan 12 15:01:48 lb02 kernel: [88879.502266] DMA: 12*4kB 12*8kB 21*16kB 16*32kB 13*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2848kB
Jan 12 15:01:48 lb02 kernel: [88879.502274] Normal: 34*4kB 13*8kB 13*16kB 6*32kB 6*64kB 3*128kB 3*256kB 2*512kB 1*1024kB 0*2048kB 0*4096kB = 4224kB
Jan 12 15:01:48 lb02 kernel: [88879.502282] HighMem: 170*4kB 134*8kB 14*16kB 5*32kB 7*64kB 0*128kB 1*256kB 1*512kB 2*1024kB 2*2048kB 95*4096kB = 398616kB
Jan 12 15:01:48 lb02 kernel: [88879.502291] 7797 total pagecache pages
Jan 12 15:01:48 lb02 kernel: [88879.502293] 0 pages in swap cache
Jan 12 15:01:48 lb02 kernel: [88879.502294] Swap cache stats: add 0, delete 0, find 0/0
Jan 12 15:01:48 lb02 kernel: [88879.502296] Free swap  = 917496kB
Jan 12 15:01:48 lb02 kernel: [88879.502298] Total swap = 917496kB
Jan 12 15:01:48 lb02 kernel: [88879.507174] 447488 pages RAM
Jan 12 15:01:48 lb02 kernel: [88879.507176] 263682 pages HighMem
Jan 12 15:01:48 lb02 kernel: [88879.507177] 10460 pages reserved
Jan 12 15:01:48 lb02 kernel: [88879.507178] 11155 pages shared
Jan 12 15:01:48 lb02 kernel: [88879.507180] 321082 pages non-shared
Jan 12 15:01:48 lb02 kernel: [88879.891235] __ratelimit: 36466 callbacks suppressed
Jan 12 15:01:48 lb02 kernel: [88879.891238] Out of socket memory
Jan 12 15:01:48 lb02 kernel: [88879.891246] Out of socket memory
Jan 12 15:01:48 lb02 kernel: [88879.891250] Out of socket memory
Jan 12 15:01:48 lb02 kernel: [88879.891255] Out of socket memory
Jan 12 15:01:48 lb02 kernel: [88879.891260] Out of socket memory
Jan 12 15:01:48 lb02 kernel: [88879.891265] Out of socket memory
Jan 12 15:01:48 lb02 kernel: [88879.891270] Out of socket memory
Jan 12 15:01:48 lb02 kernel: [88879.891275] Out of socket memory
Jan 12 15:01:48 lb02 kernel: [88879.891279] Out of socket memory
Jan 12 15:01:48 lb02 kernel: [88879.891285] Out of socket memory

gistfile2.txt

Jan 12 16:08:54 lb02 kernel: [92905.926499] haproxy invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0
Jan 12 16:08:54 lb02 kernel: [92905.926505] haproxy cpuset=/ mems_allowed=0
Jan 12 16:08:54 lb02 kernel: [92905.926509] Pid: 27092, comm: haproxy Not tainted 2.6.32-309-ec2 #18-Ubuntu
Jan 12 16:08:54 lb02 kernel: [92905.926511] Call Trace:
Jan 12 16:08:54 lb02 kernel: [92905.926521]  [<c0199648>] oom_kill_process+0xb8/0x1e0
Jan 12 16:08:54 lb02 kernel: [92905.926581]  [<c01997c1>] __out_of_memory+0x51/0xa0
Jan 12 16:08:54 lb02 kernel: [92905.926584]  [<c0199862>] out_of_memory+0x52/0xa0
Jan 12 16:08:54 lb02 kernel: [92905.926587]  [<c019bf77>] __alloc_pages_slowpath+0x407/0x4a0
Jan 12 16:08:54 lb02 kernel: [92905.926591]  [<c019c15d>] __alloc_pages_nodemask+0x14d/0x160
Jan 12 16:08:54 lb02 kernel: [92905.926596]  [<c04d3dcb>] tcp_sendmsg+0x5db/0x8f0
Jan 12 16:08:54 lb02 kernel: [92905.926601]  [<c0492280>] sock_sendmsg+0xe0/0x110
Jan 12 16:08:54 lb02 kernel: [92905.926606]  [<c014c780>] ? autoremove_wake_function+0x0/0x40
Jan 12 16:08:54 lb02 kernel: [92905.926609]  [<c014c780>] ? autoremove_wake_function+0x0/0x40
Jan 12 16:08:54 lb02 kernel: [92905.926613]  [<c012f174>] ? load_balance_newidle+0x94/0x2f0
Jan 12 16:08:54 lb02 kernel: [92905.926616]  [<c04923f0>] sys_sendto+0xf0/0x130
Jan 12 16:08:54 lb02 kernel: [92905.926626]  [<c0138a0d>] ? __do_softirq+0x10d/0x1e0
Jan 12 16:08:54 lb02 kernel: [92905.926629]  [<c0177747>] ? handle_IRQ_event+0x97/0x180
Jan 12 16:08:54 lb02 kernel: [92905.926632]  [<c049074f>] ? sock_poll+0xf/0x20
Jan 12 16:08:54 lb02 kernel: [92905.926635]  [<c0138b65>] ? do_softirq+0x85/0x90
Jan 12 16:08:54 lb02 kernel: [92905.926638]  [<c0138c7d>] ? irq_exit+0x2d/0x40
Jan 12 16:08:54 lb02 kernel: [92905.926642]  [<c0445105>] ? evtchn_do_upcall+0x1c5/0x1f0
Jan 12 16:08:54 lb02 kernel: [92905.926645]  [<c0492466>] sys_send+0x36/0x40
Jan 12 16:08:54 lb02 kernel: [92905.926647]  [<c0493289>] sys_socketcall+0x1b9/0x280
Jan 12 16:08:54 lb02 kernel: [92905.926651]  [<c01047e0>] syscall_call+0x7/0xb
Jan 12 16:08:54 lb02 kernel: [92905.926653] Mem-Info:
Jan 12 16:08:54 lb02 kernel: [92905.926655] DMA per-cpu:
Jan 12 16:08:54 lb02 kernel: [92905.926656] CPU    0: hi:    0, btch:   1 usd:   0
Jan 12 16:08:54 lb02 kernel: [92905.926658] CPU    1: hi:    0, btch:   1 usd:   0
Jan 12 16:08:54 lb02 kernel: [92905.926660] Normal per-cpu:
Jan 12 16:08:54 lb02 kernel: [92905.926661] CPU    0: hi:  155, btch:  38 usd: 124
Jan 12 16:08:54 lb02 kernel: [92905.926663] CPU    1: hi:  155, btch:  38 usd: 134
Jan 12 16:08:54 lb02 kernel: [92905.926665] HighMem per-cpu:
Jan 12 16:08:54 lb02 kernel: [92905.926666] CPU    0: hi:  155, btch:  38 usd:   2
Jan 12 16:08:54 lb02 kernel: [92905.926668] CPU    1: hi:  155, btch:  38 usd: 122
Jan 12 16:08:54 lb02 kernel: [92905.926672] active_anon:135511 inactive_anon:17292 isolated_anon:0
Jan 12 16:08:54 lb02 kernel: [92905.926673]  active_file:3041 inactive_file:5069 isolated_file:0
Jan 12 16:08:54 lb02 kernel: [92905.926674]  unevictable:0 dirty:0 writeback:0 unstable:0
Jan 12 16:08:54 lb02 kernel: [92905.926675]  free:100589 slab_reclaimable:16343 slab_unreclaimable:46879
Jan 12 16:08:54 lb02 kernel: [92905.926676]  mapped:2416 shmem:54 pagetables:0 bounce:0
Jan 12 16:08:54 lb02 kernel: [92905.926683] DMA free:2852kB min:76kB low:92kB high:112kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:16256kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:476kB slab_unreclaimable:1416kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jan 12 16:08:54 lb02 kernel: [92905.926688] lowmem_reserve[]: 0 696 1710 1710
Jan 12 16:08:54 lb02 kernel: [92905.926696] Normal free:3320kB min:3336kB low:4168kB high:5004kB active_anon:0kB inactive_anon:0kB active_file:140kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:713224kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:64896kB slab_unreclaimable:186100kB kernel_stack:744kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:256 all_unreclaimable? yes
Jan 12 16:08:54 lb02 kernel: [92905.926700] lowmem_reserve[]: 0 0 8111 8111
Jan 12 16:08:54 lb02 kernel: [92905.926707] HighMem free:396184kB min:512kB low:1724kB high:2940kB active_anon:542044kB inactive_anon:69168kB active_file:12024kB inactive_file:20276kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1038292kB mlocked:0kB dirty:0kB writeback:0kB mapped:9660kB shmem:216kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:5760 all_unreclaimable? no
Jan 12 16:08:54 lb02 kernel: [92905.926712] lowmem_reserve[]: 0 0 0 0
Jan 12 16:08:54 lb02 kernel: [92905.926715] DMA: 437*4kB 10*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2852kB
Jan 12 16:08:54 lb02 kernel: [92905.926733] Normal: 66*4kB 34*8kB 12*16kB 5*32kB 4*64kB 1*128kB 2*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 3320kB
Jan 12 16:08:54 lb02 kernel: [92905.926741] HighMem: 1120*4kB 1833*8kB 175*16kB 1*32kB 7*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 2*2048kB 90*4096kB = 396184kB
Jan 12 16:08:54 lb02 kernel: [92905.926749] 8175 total pagecache pages
Jan 12 16:08:54 lb02 kernel: [92905.926753] 0 pages in swap cache
Jan 12 16:08:54 lb02 kernel: [92905.926769] Swap cache stats: add 0, delete 0, find 0/0
Jan 12 16:08:54 lb02 kernel: [92905.926771] Free swap  = 917496kB
Jan 12 16:08:54 lb02 kernel: [92905.926772] Total swap = 917496kB
Jan 12 16:08:54 lb02 kernel: [92905.931720] 447488 pages RAM
Jan 12 16:08:54 lb02 kernel: [92905.931723] 263682 pages HighMem
Jan 12 16:08:54 lb02 kernel: [92905.931725] 10460 pages reserved
Jan 12 16:08:54 lb02 kernel: [92905.931726] 11223 pages shared
Jan 12 16:08:54 lb02 kernel: [92905.931727] 329810 pages non-shared
Jan 12 16:08:55 lb02 kernel: [92906.410351] __ratelimit: 55289 callbacks suppressed
Jan 12 16:08:55 lb02 kernel: [92906.410354] TCP: too many of orphaned sockets
Jan 12 16:08:55 lb02 kernel: [92906.410364] TCP: too many of orphaned sockets
Jan 12 16:08:55 lb02 kernel: [92906.410373] TCP: too many of orphaned sockets
Jan 12 16:08:55 lb02 kernel: [92906.410381] TCP: too many of orphaned sockets
Jan 12 16:08:55 lb02 kernel: [92906.410389] TCP: too many of orphaned sockets
Jan 12 16:08:55 lb02 kernel: [92906.410397] TCP: too many of orphaned sockets
Jan 12 16:08:55 lb02 kernel: [92906.410406] TCP: too many of orphaned sockets
Jan 12 16:08:55 lb02 kernel: [92906.410415] TCP: too many of orphaned sockets
Jan 12 16:08:55 lb02 kernel: [92906.410422] TCP: too many of orphaned sockets
Jan 12 16:08:55 lb02 kernel: [92906.410431] TCP: too many of orphaned sockets