This ain't memory leak (I think)

logistic_regression_memory_test.md

Test script

import os, time, gc, psutil
from pympler import tracker

import numpy as np
from sklearn.linear_model import LogisticRegression

tracker.memory_tracker = tracker.SummaryTracker()
def get_mem():
    return "{:.0f}MB".format(p.memory_info().rss / 1e6)

p = psutil.Process()
X = np.random.normal(size=(500000, 120))
Y = np.random.binomial(1, 0.5, size=(500000, ))

def sleep_and_print_mem(title, sleep=3):
    time.sleep(sleep)
    tracker.memory_tracker.print_diff()
    print("\n\n" + title + " : " + get_mem() + "   " + "=" * 50 + "\n\n")

sleep_and_print_mem("Initial memory")

for i in range(7):
    et = LogisticRegression(n_jobs=8).fit(X, Y)
    del et
    gc.collect()

    sleep_and_print_mem("After iteration %d" % i)

Output

                            types |   # objects |   total size
================================= | =========== | ============
            <class 'numpy.ndarray |           2 |    461.58 MB
                     <class 'list |       10490 |    986.98 KB
                      <class 'str |       10489 |    753.66 KB
                      <class 'int |        2229 |     61.02 KB
                     <class 'dict |          19 |      5.75 KB
       <class 'wrapper_descriptor |          35 |      2.73 KB
                  <class 'weakref |          19 |      1.48 KB
        <class 'method_descriptor |          12 |    864     B
                     <class 'type |           0 |    672     B
                     <class 'code |           3 |    432     B
               function (get_mem) |           1 |    136     B
   function (sleep_and_print_mem) |           1 |    136     B
            function (store_info) |           1 |    136     B
                     <class 'cell |           2 |     96     B
  <class 'psutil._pslinux.Process |           1 |     72     B


Initial memory : 593MB   ==================================================


            types |   # objects |   total size
================= | =========== | ============
     <class 'type |           0 |    960     B
     <class 'list |           5 |    408     B
      <class 'str |           5 |    367     B
  <class 'weakref |           3 |    240     B
     <class 'dict |           0 |     96     B


After iteration 0 : 1614MB   ==================================================


  types |   # objects |   total size
======= | =========== | ============


After iteration 1 : 1614MB   ==================================================


  types |   # objects |   total size
======= | =========== | ============


After iteration 2 : 1618MB   ==================================================


  types |   # objects |   total size
======= | =========== | ============


After iteration 3 : 1619MB   ==================================================


  types |   # objects |   total size
======= | =========== | ============


After iteration 4 : 1619MB   ==================================================


  types |   # objects |   total size
======= | =========== | ============


After iteration 5 : 1619MB   ==================================================


  types |   # objects |   total size
======= | =========== | ============


After iteration 6 : 1619MB   ==================================================