Created
October 18, 2023 18:14
-
-
Save ivirshup/9ba2b570d541ff1393990f632bc7a6ea to your computer and use it in GitHub Desktop.
Hello,
I'm trying to apply your method to NetCDF files.
!pip install git+https://github.com/ivirshup/kerchunk.git@concat-varchunks
#Create example separated files
import xarray as xr
ds = xr.tutorial.load_dataset('air_temperature')
ds.isel(lon=slice(0,5)).to_netcdf('lon0.nc')
ds.isel(lon=slice(5,10)).to_netcdf('lon1.nc')
ds.isel(lon=slice(10,13)).to_netcdf('lon2.nc')
import glob
dir_url = "/Users/todaka/"
file_pattern = "/data_xios/lon*.nc"
file_paths = glob.glob(dir_url + file_pattern)
file_paths=file_paths[0:2]
import fsspec
from kerchunk.hdf import SingleHdf5ToZarr
def translate_dask(file):
url = "file://" + file
with fsspec.open(url) as inf:
h5chunks = SingleHdf5ToZarr(inf, url, inline_threshold=100)
return h5chunks.translate()
result=[translate_dask(file) for file in file_paths]
from kerchunk.combine import MultiZarrToZarr
mzz = MultiZarrToZarr(
result,
concat_dims=["lon"],
)
a = mzz.translate()
xr.open_dataset(a,engine='kerchunk',chunks={})
result = result_indask.compute()
from kerchunk.combine import MultiZarrToZarr
mzz = MultiZarrToZarr(
result,
concat_dims=["y"],
)
a = mzz.translate()
xr.open_dataset(a,engine='kerchunk',chunks={})
I tried to apply your function
concatenate_zarr_csr_arrays(result)
But I age err :
AttributeError: 'dict' object has no attribute 'attrs'
What should I do if I want to connect not the zarr files, but NetCDF files using your approach?
Thank you very much for your help.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Well, it's all "motivated" already in https://github.com/orgs/zarr-developers/discussions/52#discussioncomment-7435293 :-)