Created
September 18, 2018 01:31
-
-
Save marekyggdrasil/14bbb9e2ca91f9a1fad8495b5ac59354 to your computer and use it in GitHub Desktop.
Example how to read/write large files in incremental way using cPickle. Run `gen.py` to generate a file containing list of integers then use `read.py` to read this list incrementally from the file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import cPickle as pickle | |
import io | |
lst = range(16) | |
with io.open('list.p', 'wb') as f : | |
pickler = pickle.Pickler(f) | |
for l in lst : | |
pickler.dump(l) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import cPickle as pickle | |
import io | |
with io.open('list.p', 'rb') as p : | |
unpickler = pickle.Unpickler(p) | |
while p.peek(1) : | |
print unpickler.load() |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment