Last active
January 18, 2018 05:21
-
-
Save cyingfan/d020d35436238c64173f3113c0c9fcb0 to your computer and use it in GitHub Desktop.
data size in python
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import json | |
from glob import iglob | |
import resource | |
dflist = [] | |
for i in iglob('./data/FULL/data/visits*.json'): | |
print(i) | |
dflist.append(json.load(open(i, 'r'))) | |
print(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import pandas | |
from glob import iglob | |
import resource | |
dflist = [] | |
for i in iglob('./data/FULL/data/visits*.json'): | |
print(i) | |
dflist.append(pandas.read_json(i)) | |
print(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
list+dataframe uses 7410424KB
list+dict uses 4345368KB