-
-
Save bendichter/f28d69184ac05fa99982288daed5541d to your computer and use it in GitHub Desktop.
| ## set up env via bash: | |
| conda create -n stream_nwb -c conda-forge -y python=3.7 hdf5=1.12.0 jupyter pip | |
| conda activate stream_nwb | |
| pip install nwbwidgets | |
| pip install git+https://github.com/NeurodataWithoutBorders/pynwb | |
| pip install git+https://github.com/hdmf-dev/hdmf | |
| git clone https://github.com/satra/h5py.git | |
| cd h5py | |
| HDF5_DIR=~/Users/bendichter/opt/anaconda3/envs/stream_nwb/ pip install --force-reinstall --no-binary=h5py . | |
| ## test out streaming | |
| from pynwb import NWBHDF5IO | |
| from nwbwidgets import nwb2widget | |
| import requests | |
| def get_s3_url(url): | |
| s3_url = requests.request(url=url, method='head').url | |
| return s3_url[:s3_url.index('?')] | |
| path = "https://dandiarchive.s3.amazonaws.com/girder-assetstore/58/07/58074d0a5a4a4086afebb4d29912c419" | |
| io = NWBHDF5IO(get_s3_url(url), mode='r', load_namespaces=True, driver='ros3') | |
| nwb = io.read() | |
| nwb2widget(nwb) |
@rly @satra there are some errors when I try the above steps to stream:
I'm on Windows10,
conda create -n stream_nwb -c conda-forge -y python=3.7 hdf5=1.12.0 jupyter pip
conda activate stream_nwb
pip install nwbwidgets
pip install git+https://github.com/NeurodataWithoutBorders/pynwb.git@ros3v2
pip install git+https://github.com/hdmf-dev/hdmf
setx HDF5_DIR "C:\Program Files\HDF_Group\HDF5\1.12.0"
pip install git+https://github.com/satra/h5py.git@enh-ros3I had also got this warning after reinstalling h5py:
hdmf 2.4.0 requires h5py<3,>=2.9, but you have h5py 3.1.0 which is incompatible
Ignoring that:
Run:
from pynwb import NWBHDF5IO
path = "https://dandiarchive.s3.amazonaws.com/girder-assetstore/58/07/58074d0a5a4a4086afebb4d29912c419"
io = NWBHDF5IO(path, mode='r', load_namespaces=True, driver='ros3')Error:
Traceback (most recent call last):
File "C:\Users\Saksham\anaconda3\envs\stream_nwb\lib\site-packages\IPython\core\interactiveshell.py", line 3437, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-2-b27bc7e86954>", line 3, in <module>
io = NWBHDF5IO(path, mode='r', load_namespaces=True, driver='ros3')
File "C:\Users\Saksham\AppData\Roaming\Python\Python37\site-packages\hdmf\utils.py", line 580, in func_call
return func(args[0], **pargs)
File "C:\Users\Saksham\AppData\Roaming\Python\Python37\site-packages\pynwb\__init__.py", line 232, in __init__
super(NWBHDF5IO, self).load_namespaces(tm, path, file=file_obj, driver=driver)
File "C:\Users\Saksham\AppData\Roaming\Python\Python37\site-packages\hdmf\utils.py", line 580, in func_call
return func(args[0], **pargs)
File "C:\Users\Saksham\AppData\Roaming\Python\Python37\site-packages\hdmf\backends\hdf5\h5tools.py", line 130, in load_namespaces
with File(path, 'r', **file_kwargs) as f:
File "C:\Users\Saksham\anaconda3\envs\stream_nwb\lib\site-packages\h5py\_hl\files.py", line 403, in __init__
"h5py was built without ROS3 support, can't use ros3 driver")
ValueError: h5py was built without ROS3 support, can't use ros3 driver
just hold on a bit. conda-forge should have both hdf5 and h5py built with this in the next day or two.
That's great news! Thanks for the update @satra
Hi @satra , any updates on hdf5 release with ROS3 support?
I can reproduce @Saksham20 error on ubuntu as well
hdf5 is available on conda-forge with ros3 support. the h5py build is currently unavailable due to a failure on arm cross-compiling, but if you have a compiler, you should be able to pip install h5py .
HDF5_DIR=~/Users/bendichter/opt/anaconda3/envs/stream_nwb/ pip install --force-reinstall --no-binary=h5py
you will have to adjust the location of the HDF5 libraries that you install from conda-forge.
@satra thanks, that works indeed!
we are putting it to work on a server and the current solution is still a bit hacky I feel... do you think this h5py build from conda/pip will be available anytime soon?
that's an issue on the conda-forge side that they have not fully figured out: conda-forge/h5py-feedstock#87
i have no idea of a timeline there.
hdf5 is available on conda-forge with ros3 support. the h5py build is currently unavailable due to a failure on arm cross-compiling, but if you have a compiler, you should be able to pip install h5py .
HDF5_DIR=~/Users/bendichter/opt/anaconda3/envs/stream_nwb/ pip install --force-reinstall --no-binary=h5pyyou will have to adjust the location of the HDF5 libraries that you install from conda-forge.
@satra It still did not work for Windows even after adjusting for the conda installed hdf5 libraries. Same error
can you check the hdf settings to make sure the driver is in fact enabled? it's a file called libhdf5.settings inside the conda environment where you have installed hdf5 and it should have a line that says (Read-Only) S3 VFD: yes
can you check the hdf settings to make sure the driver is in fact enabled? it's a file called libhdf5.settings inside the conda environment where you have installed hdf5 and it should have a line that says
(Read-Only) S3 VFD: yes
It wasn't initially, but even when I changed it to yes, the h5py did not build correctly. I think the folder structure in the conda/envs/stream_nwb/is different in case of linux vs windows. So the build process is not able to look for the correct files when the HDF5_DIR is set to '*/conda/envs/stream_nwb/' in windows. I tried to follow the same steps in linux and it works fine.
you shouldn't change to yes manually. it reflects the hdf5 version that is installed. so if it says no, then your hdf5 library doesn't have the driver enabled. it means probably an incorrect install from conda-forge.
I can reproduce the problem on my windows machine, with both conda-forge installation and manual installation of hdf5.
Seems to be a problem specific to windows. I found this, it might help (search for Windows): https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.12/hdf5-1.12.0/src/hdf5-1.12.0-RELEASE.txt
we managed to make it work on a Binder deployment, if anyone is interested: https://github.com/catalystneuro/binder-nwb-stream
you may want to file an issue on conda-forge for hdf5-feedstock, if it is not being compiled on windows with ROS3 VFD support.
@rly - if you use conda on windows you can install hdf5 from conda-forge (see the updated conda lines above).