Skip to content

Instantly share code, notes, and snippets.

@kain88-de

kain88-de/speed benchmark.ipynb

Last active December 17, 2015 20:33
Show Gist options
  • Save kain88-de/5487e9d9c8f7d0e4995b to your computer and use it in GitHub Desktop.
Save kain88-de/5487e9d9c8f7d0e4995b to your computer and use it in GitHub Desktop.
Download ZIP
offset fileformat speed benchmark
Display the source blob
Display the rendered blob
Raw
speed benchmark.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@orbeckst
Copy link

I forked the notebook https://gist.github.com/orbeckst/2d2e8c12288b202a6a02 and added benchmarks for

  • cPickle.HIGHEST_PROTOCOL
  • numpy.savez

Bottomline: if pickle then always use the latest protocol because it reduces file size to 1/3 and speeds up loading by a factor of 160 (in this case). The numpy native format is even faster, maybe another factor of 50.

@kain88-de
Copy link
Author

Nope numpy is slower. once you actually read the data

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment