Python over allocation

over_allocation.py

import os
import sys

import psutil

UNIT = 2 ** 20  # MB
print("All size units are in M\n")

proc = psutil.Process(pid=os.getpid())
m0 = proc.memory_info().rss

size = int(sys.argv[1]) * (2 ** 20)

print(f"Creating an int list `xs` of size {size // UNIT}.")
obj = []
for i in range(size):
    obj.append(i)


obj_size_M = sys.getsizeof(obj) / UNIT
print(f"\tgetsizeof(xs) = {obj_size_M:.3f}")

m1 = proc.memory_info().rss
print(f"\tresident mem difference (creating `xs`) = {(m1 - m0) / UNIT:.3f}")

print()

print("Copying `xs` to `ys`")
copy = obj[:]
copy_size_M = sys.getsizeof(copy) / UNIT
print(f"\tgetsizeof(ys) = {copy_size_M:.3f}")

m2 = proc.memory_info().rss
print(f"\tresident mem difference (creating `ys` copy) = {(m2 - m1) / UNIT:.3f}")

All size units are in M

---

list-3M

Creating an int list xs of size 3.

getsizeof(xs) = 26.936
resident mem difference (creating `xs`) = 121.988

Copying xs to ys

getsizeof(ys) = 24.000
resident mem difference (creating `ys` copy) = 23.977

Unpickling int list xs of size 3 (~15M on disk)

getsizeof(xs) = 25.749
resident mem difference (creating `xs`) = 121.914

---

list-128M

Creating an int list xs of size 128.

getsizeof(xs) = 1037.682
resident mem difference (creating `xs`) = 5187.039

Copying xs to ys

getsizeof(ys) = 1024.000
resident mem difference (creating `ys` copy) = 1024.086

Unpickling int list xs of size 128 (~641M on disk)

getsizeof(xs) = 1117.807
resident mem difference (creating `xs`) = 5186.930

---

numpy-128M

Creating an int32 numpy array xs of size 128.

getsizeof(xs) = 512.000
resident mem difference (creating `xs`) = 512.109

Copying xs to ys

getsizeof(ys) = 512.000
resident mem difference (creating `ys` copy) = 512.055

Unpickling int32 numpy array xs of size 128 (~513M on disk)

resident mem difference = 511.984375