IPython OOM Killer

syslog.log

Mar 26 23:11:01 localhost kernel: [178164.363434] python invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Mar 26 23:11:01 localhost kernel: [178164.363441] python cpuset=/ mems_allowed=0
Mar 26 23:11:01 localhost kernel: [178164.363462] Pid: 8666, comm: python Tainted: GF            3.8.0-29-generic #42~precise1-Ubuntu
Mar 26 23:11:01 localhost kernel: [178164.363463] Call Trace:
Mar 26 23:11:01 localhost kernel: [178164.363482]  [<ffffffff816de7d6>] dump_header+0x83/0xbb
Mar 26 23:11:01 localhost kernel: [178164.363486]  [<ffffffff816de863>] oom_kill_process.part.6+0x55/0x2cf
Mar 26 23:11:01 localhost kernel: [178164.363497]  [<ffffffff8113891d>] oom_kill_process+0x4d/0x50
Mar 26 23:11:01 localhost kernel: [178164.363500]  [<ffffffff81138c55>] out_of_memory+0x145/0x1d0
Mar 26 23:11:01 localhost kernel: [178164.363502]  [<ffffffff8113e507>] __alloc_pages_nodemask+0x977/0x990
Mar 26 23:11:01 localhost kernel: [178164.363507]  [<ffffffff8117aa1a>] alloc_pages_current+0xba/0x170
Mar 26 23:11:01 localhost kernel: [178164.363518]  [<ffffffff811352a7>] __page_cache_alloc+0xb7/0xd0
Mar 26 23:11:01 localhost kernel: [178164.363520]  [<ffffffff811374bd>] filemap_fault+0x24d/0x430
Mar 26 23:11:01 localhost kernel: [178164.363525]  [<ffffffff8115b2c2>] __do_fault+0x72/0x540
Mar 26 23:11:01 localhost kernel: [178164.363529]  [<ffffffff816f254b>] ? __schedule+0x3bb/0x6b0
Mar 26 23:11:01 localhost kernel: [178164.363532]  [<ffffffff8115f326>] handle_pte_fault+0x96/0x230
Mar 26 23:11:01 localhost kernel: [178164.363535]  [<ffffffff811608f0>] handle_mm_fault+0x2a0/0x3e0
Mar 26 23:11:01 localhost kernel: [178164.363539]  [<ffffffff816f7bcf>] __do_page_fault+0x1af/0x560
Mar 26 23:11:01 localhost kernel: [178164.363547]  [<ffffffff81097d93>] ? update_curr+0x143/0x1f0
Mar 26 23:11:01 localhost kernel: [178164.363553]  [<ffffffff81044daf>] ? kvm_clock_read+0x1f/0x30
Mar 26 23:11:01 localhost kernel: [178164.363555]  [<ffffffff81044dc9>] ? kvm_clock_get_cycles+0x9/0x10
Mar 26 23:11:01 localhost kernel: [178164.363559]  [<ffffffff810ae88c>] ? ktime_get_ts+0x4c/0xe0
Mar 26 23:11:01 localhost kernel: [178164.363565]  [<ffffffff811adfc2>] ? poll_select_copy_remaining+0xf2/0x140
Mar 26 23:11:01 localhost kernel: [178164.363568]  [<ffffffff816f7f8e>] do_page_fault+0xe/0x10
Mar 26 23:11:01 localhost kernel: [178164.363570]  [<ffffffff816f7665>] do_async_page_fault+0x35/0x90
Mar 26 23:11:01 localhost kernel: [178164.363572]  [<ffffffff816f4388>] async_page_fault+0x28/0x30
Mar 26 23:11:01 localhost kernel: [178164.363574] Mem-Info:
Mar 26 23:11:01 localhost kernel: [178164.363576] Node 0 DMA per-cpu:
Mar 26 23:11:01 localhost kernel: [178164.363579] CPU    0: hi:    0, btch:   1 usd:   0
Mar 26 23:11:01 localhost kernel: [178164.363580] Node 0 DMA32 per-cpu:
Mar 26 23:11:01 localhost kernel: [178164.363582] CPU    0: hi:  186, btch:  31 usd:   1
Mar 26 23:11:01 localhost kernel: [178164.363585] active_anon:226587 inactive_anon:48 isolated_anon:0
Mar 26 23:11:01 localhost kernel: [178164.363585]  active_file:10 inactive_file:18 isolated_file:0
Mar 26 23:11:01 localhost kernel: [178164.363585]  unevictable:0 dirty:3 writeback:0 unstable:0
Mar 26 23:11:01 localhost kernel: [178164.363585]  free:12255 slab_reclaimable:2926 slab_unreclaimable:2894
Mar 26 23:11:01 localhost kernel: [178164.363585]  mapped:2 shmem:59 pagetables:2583 bounce:0
Mar 26 23:11:01 localhost kernel: [178164.363585]  free_cma:0
Mar 26 23:11:01 localhost kernel: [178164.363588] Node 0 DMA free:4648kB min:680kB low:848kB high:1020kB active_anon:11036kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictab
le:0kB isolated(anon):0kB isolated(file):0kB present:15644kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:124kB slab_unreclaimable:32kB kerne
l_stack:0kB pagetables:60kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Mar 26 23:11:01 localhost kernel: [178164.363615] lowmem_reserve[]: 0 992 992 992
Mar 26 23:11:01 localhost kernel: [178164.363618] Node 0 DMA32 free:44372kB min:44372kB low:55464kB high:66556kB active_anon:895312kB inactive_anon:192kB active_file:40kB inactive_file:7
2kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1016052kB managed:984208kB mlocked:0kB dirty:12kB writeback:0kB mapped:8kB shmem:236kB slab_reclaimable:11580kB slab_unr
eclaimable:11544kB kernel_stack:1792kB pagetables:10272kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:180 all_unreclaimable? yes
Mar 26 23:11:01 localhost kernel: [178164.363622] lowmem_reserve[]: 0 0 0 0
Mar 26 23:11:01 localhost kernel: [178164.363624] Node 0 DMA: 10*4kB (UEM) 4*8kB (UE) 24*16kB (UEM) 19*32kB (UEM) 14*64kB (UEM) 5*128kB (EM) 0*256kB 0*512kB 0*1024kB 1*2048kB (R) 0*4096k
B = 4648kB
Mar 26 23:11:01 localhost kernel: [178164.363641] Node 0 DMA32: 297*4kB (UEM) 298*8kB (UEM) 312*16kB (UEM) 204*32kB (UEM) 120*64kB (UEM) 85*128kB (UEM) 26*256kB (UEM) 0*512kB 0*1024kB 0*
2048kB 1*4096kB (R) = 44404kB
Mar 26 23:11:01 localhost kernel: [178164.363649] 79 total pagecache pages
Mar 26 23:11:01 localhost kernel: [178164.363650] 0 pages in swap cache
Mar 26 23:11:01 localhost kernel: [178164.363652] Swap cache stats: add 0, delete 0, find 0/0
Mar 26 23:11:01 localhost kernel: [178164.363653] Free swap  = 0kB
Mar 26 23:11:01 localhost kernel: [178164.363654] Total swap = 0kB
Mar 26 23:11:01 localhost kernel: [178164.366655] 262125 pages RAM
Mar 26 23:11:01 localhost kernel: [178164.366659] 7420 pages reserved
Mar 26 23:11:01 localhost kernel: [178164.366660] 261940 pages shared
Mar 26 23:11:01 localhost kernel: [178164.366661] 241952 pages non-shared
Mar 26 23:11:01 localhost kernel: [178164.366663] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
Mar 26 23:11:01 localhost kernel: [178164.366676] [  321]     0   321     4310       49      13        0             0 upstart-udev-br
Mar 26 23:11:01 localhost kernel: [178164.366679] [  325]     0   325     5380      127      16        0         -1000 udevd
Mar 26 23:11:01 localhost kernel: [178164.366682] [  374]   102   374     5989       86      16        0             0 dbus-daemon
Mar 26 23:11:01 localhost kernel: [178164.366685] [  754]     0   754     3799       48      12        0             0 upstart-socket-
Mar 26 23:11:01 localhost kernel: [178164.366688] [  829]     0   829    12510      153      29        0         -1000 sshd
Mar 26 23:11:01 localhost kernel: [178164.366690] [  908]     0   908     3948       39      13        0             0 getty
Mar 26 23:11:01 localhost kernel: [178164.366692] [  914]     0   914     3948       42      13        0             0 getty
Mar 26 23:11:01 localhost kernel: [178164.366694] [  931]     0   931     3948       42      13        0             0 getty
Mar 26 23:11:01 localhost kernel: [178164.366705] [  932]     0   932     3948       41      13        0             0 getty
Mar 26 23:11:01 localhost kernel: [178164.366708] [  936]     0   936     3948       42      13        0             0 getty
Mar 26 23:11:01 localhost kernel: [178164.366710] [  943]     0   943     1084       33       7        0             0 acpid
Mar 26 23:11:01 localhost kernel: [178164.366712] [  944]     0   944     4780       59      14        0             0 cron
Mar 26 23:11:01 localhost kernel: [178164.366714] [  945]     0   945     4229       39      13        0             0 atd
Mar 26 23:11:01 localhost kernel: [178164.366716] [  981]   103   981    46917      307      61        0             0 whoopsie
Mar 26 23:11:01 localhost kernel: [178164.366719] [  989]     0   989    19424      205      42        0             0 sshd
Mar 26 23:11:01 localhost kernel: [178164.366721] [ 1121]     0  1121     5685      571      17        0             0 bash
Mar 26 23:11:01 localhost kernel: [178164.366723] [ 1243]     0  1243     3948       41      13        0             0 getty
Mar 26 23:11:01 localhost kernel: [178164.366726] [22018]   101 22018    62370      158      23        0             0 rsyslogd
Mar 26 23:11:01 localhost kernel: [178164.366728] [24224]     0 24224     5379      128      15        0         -1000 udevd
Mar 26 23:11:01 localhost kernel: [178164.366732] [ 4742]     0  4742    10522       81      26        0             0 su
Mar 26 23:11:01 localhost kernel: [178164.366734] [ 4744]     0  4744    32740     4336      69        0             0 salt-minion
Mar 26 23:11:01 localhost kernel: [178164.366736] [28950]     0 28950    10770       91      26        0             0 sudo
Mar 26 23:11:01 localhost kernel: [178164.366738] [28951]  2000 28951    11235       84      27        0             0 su
Mar 26 23:11:01 localhost kernel: [178164.366740] [28952]  2000 28952     6942     1829      18        0             0 bash
Mar 26 23:11:01 localhost kernel: [178164.366743] [ 8645]  2000  8645     3078       47      11        0             0 bash
Mar 26 23:11:01 localhost kernel: [178164.366745] [ 8646]  2000  8646   159758    65619     219        0             0 ipython
Mar 26 23:11:01 localhost kernel: [178164.366747] [ 8653]  2000  8653   195901    10135     137        0             0 python
Mar 26 23:11:01 localhost kernel: [178164.366749] [ 8675]  2000  8675   166220    15576     150        0             0 python
Mar 26 23:11:01 localhost kernel: [178164.366783] [ 8740]  2000  8740   167248     4282      80        0             0 python
Mar 26 23:11:01 localhost kernel: [178164.366786] [12134]  2000 12134   199890    12561     145        0             0 python
Mar 26 23:11:01 localhost kernel: [178164.366788] [12135]  2000 12135   200544    13149     146        0             0 python
Mar 26 23:11:01 localhost kernel: [178164.366790] [12158]  2000 12158   198396    11043     141        0             0 python
Mar 26 23:11:01 localhost kernel: [178164.366792] [12170]  2000 12170   198871    11552     142        0             0 python
Mar 26 23:11:01 localhost kernel: [178164.366794] [12182]  2000 12182   204350    12795     154        0             0 python
Mar 26 23:11:01 localhost kernel: [178164.366796] [12194]  2000 12194   208086    20733     164        0             0 python
Mar 26 23:11:01 localhost kernel: [178164.366798] [12206]  2000 12206   196575    10725     138        0             0 python
Mar 26 23:11:01 localhost kernel: [178164.366800] [12218]  2000 12218   154587    10110     125        0             0 python
Mar 26 23:11:01 localhost kernel: [178164.366803] [12247]  2000 12247   126628     4076      75        0             0 python
Mar 26 23:11:01 localhost kernel: [178164.366805] [12278]  2000 12278   197567    10184     140        0             0 python
Mar 26 23:11:01 localhost kernel: [178164.366807] [12343]  2000 12343   126627     4076      78        0             0 python
Mar 26 23:11:01 localhost kernel: [178164.366809] Out of memory: Kill process 8646 (ipython) score 258 or sacrifice child
Mar 26 23:11:01 localhost kernel: [178164.367018] Killed process 12194 (python) total-vm:832344kB, anon-rss:82932kB, file-rss:0kB