Skip to content

Instantly share code, notes, and snippets.

@vladignatyev
Last active February 2, 2017 11:26
Show Gist options
  • Save vladignatyev/ec7a26b7042efd6f710d436afbfb87de to your computer and use it in GitHub Desktop.
Save vladignatyev/ec7a26b7042efd6f710d436afbfb87de to your computer and use it in GitHub Desktop.
The example code to check the problem in a question problem, posted on StackOverflow
#!/env/bin/python
# See: http://stackoverflow.com/questions/41893967/increase-in-memory-usage-on-pandas-dataframe-creation/42000635
import sys
import pandas as pd
import gc
# @profile ## uncomment this line if you want to profile with memory-profiler Python's module
def make_list():
pd_arr = []
for i in range(0,10000):
pd_arr.append([x for x in range(0,1000)])
return pd_arr
# @profile
def to_profile():
pd_arr = make_list()
pd_df = pd.DataFrame.from_records(pd_arr, columns=[x for x in range(0,1000)])
# pd_df.info(memory_usage='deep')
print sys.getsizeof(pd_arr), sys.getsizeof(pd_arr[0])
print sys.getsizeof(pd_df), len(pd_arr)
print sys.getrefcount(pd_arr)
del pd_arr
gc.collect()
to_profile()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment