Skip to content

Instantly share code, notes, and snippets.

@rossant
Last active July 12, 2023 09:34
Show Gist options
  • Star 25 You must be signed in to star a gist
  • Fork 5 You must be signed in to fork a gist
  • Save rossant/7b4704e8caeb8f173084 to your computer and use it in GitHub Desktop.
Save rossant/7b4704e8caeb8f173084 to your computer and use it in GitHub Desktop.
Quick HDF5 benchmark
Display the source blob
Display the rendered blob
Raw
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@rossant
Copy link
Author

rossant commented Jan 7, 2016

Benchmark updated following comment by Stuart Berg.

@andrewcollette
Copy link

@rossant, a big part of this is that the fancy-indexing code in h5py uses a naive algorithm based on repeated hyperslab selection, which is quadratic in the number of indices. It was designed/tested for small numbers of indices.

The particular example you have here (0 to 10000 in steps of 10) can be mapped to slices (although of course this is not generally true). In this case the results are:

%%timeit f = h5py.File('test.h5','r')
f['/test'][0:10000:10]

100 loops, best of 3: 12 ms per loop

This is a great argument to improve the implementation of fancy indexing in h5py, but I would hesitate to conclude "HDF5 is slow".

@rossant
Copy link
Author

rossant commented Jan 8, 2016

@andrewcollette thanks for your comment, I've updated the benchmarks and the post accordingly. And thanks for doing h5py! Despite the problems we've had with HDF5, I actually like the h5py API and how it fits so naturally with NumPy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment