Skip to content

Instantly share code, notes, and snippets.

@cyingfan
Last active January 18, 2018 05:21
Show Gist options
  • Save cyingfan/d020d35436238c64173f3113c0c9fcb0 to your computer and use it in GitHub Desktop.
Save cyingfan/d020d35436238c64173f3113c0c9fcb0 to your computer and use it in GitHub Desktop.
data size in python
import json
from glob import iglob
import resource
dflist = []
for i in iglob('./data/FULL/data/visits*.json'):
print(i)
dflist.append(json.load(open(i, 'r')))
print(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)
import pandas
from glob import iglob
import resource
dflist = []
for i in iglob('./data/FULL/data/visits*.json'):
print(i)
dflist.append(pandas.read_json(i))
print(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)
@cyingfan
Copy link
Author

cyingfan commented Jan 18, 2018

list+dataframe uses 7410424KB
list+dict uses 4345368KB

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment