Skip to content

Instantly share code, notes, and snippets.

@mdsumner

mdsumner/quick_vsi_icechunk.md

Last active June 17, 2026 01:28
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save mdsumner/2aa9ba53a9272c82de72b62f6f19ce1c to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save mdsumner/2aa9ba53a9272c82de72b62f6f19ce1c to your computer and use it in GitHub Desktop.
Download ZIP
Raw
quick_vsi_icechunk.md 
## list the structure elements (Icechunk) gdal vsi ls  "/vsis3/earthmover-icechunk-era5/icechunkV2" --config AWS_NO_SIGN_REQUEST YES --config AWS_REGION us-east-1
repo
chunks
manifests
overwritten
snapshots
transactions

## list the logical (virtual) elements (Zarr)

gdal vsi ls  /vsiicechunk/{/vsis3/earthmover-icechunk-era5/icechunkV2} --config AWS_NO_SIGN_REQUEST YES --config AWS_REGION us-east-1
zarr.json
500hPa
pressure
single

## delve deeper: 

gdal vsi ls  /vsiicechunk/{/vsis3/earthmover-icechunk-era5/icechunkV2}/pressure/spatial --config AWS_NO_SIGN_REQUEST YES --config AWS_REGION us-east-1
zarr.json
latitude
longitude
lsm
pressure_level
pv
q
r
status
t
u
v
valid_time
w
z

## spit out the zarr.json to virtual stdout

gdal vsi copy  /vsiicechunk/{/vsis3/earthmover-icechunk-era5/icechunkV2}/pressure/spatial/pv/zarr.json /vsistdout  --config AWS_NO_SIGN_REQUEST YES --config AWS_REGION us-east-1
{
  "shape": [
    753888,
    13,
    721,
    1440
  ],
  "data_type": "float32",
  "chunk_grid": {
    "name": "regular",
    "configuration": {
      "chunk_shape": [
        1,
        1,
        721,
        1440
      ]
    }
  },
  "chunk_key_encoding": {
    "name": "default",
    "configuration": {
      "separator": "/"
    }
  },
  "fill_value": "NaN",
  "codecs": [
    {
      "name": "numcodecs.pcodec",
      "configuration": {
        "level": 8
      }
    }
  ],
  "attributes": {
    "GRIB_paramId": 60,
    "GRIB_dataType": "an",
    "GRIB_numberOfPoints": 1038240,
    "GRIB_typeOfLevel": "isobaricInhPa",
    "GRIB_stepUnits": 1,
    "GRIB_stepType": "instant",
    "GRIB_gridType": "regular_ll",
    "GRIB_uvRelativeToGrid": 0,
    "GRIB_NV": 0,
    "GRIB_Nx": 1440,
    "GRIB_Ny": 721,
    "GRIB_cfName": "unknown",
    "GRIB_cfVarName": "pv",
    "GRIB_gridDefinitionDescription": "Latitude/Longitude Grid",
    "GRIB_iDirectionIncrementInDegrees": 0.25,
    "GRIB_iScansNegatively": 0,
    "GRIB_jDirectionIncrementInDegrees": 0.25,
    "GRIB_jPointsAreConsecutive": 0,
    "GRIB_jScansPositively": 0,
    "GRIB_latitudeOfFirstGridPointInDegrees": 90.0,
    "GRIB_latitudeOfLastGridPointInDegrees": -90.0,
    "GRIB_longitudeOfFirstGridPointInDegrees": 0.0,
    "GRIB_longitudeOfLastGridPointInDegrees": 359.75,
    "GRIB_missingValue": 3.4028234663852886e+38,
    "GRIB_name": "Potential vorticity",
    "GRIB_shortName": "pv",
    "GRIB_totalNumber": 0,
    "GRIB_units": "K m**2 kg**-1 s**-1",
    "long_name": "Potential vorticity",
    "units": "K m**2 kg**-1 s**-1",
    "standard_name": "unknown",
    "coordinates": "lsm",
    "_FillValue": "AAAAAAAA+H8="
  },
  "dimension_names": [
    "valid_time",
    "pressure_level",
    "latitude",
    "longitude"
  ],
  "zarr_format": 3,
  "node_type": "array",
  "storage_transformers": []
}0...10...20...30...40...50...60...70...80...90...100 - done.

and much more: https://github.com/rouault/gdal/blob/eaf6001c655ac2004e6c61735f6de1db83d4773e/doc/source/drivers/raster/icechunk.rst

@mdsumner

mdsumner commented Jun 12, 2026

Copy link
Copy Markdown
Author

here's a minimal example opening in gdalxarray at different levels in the tree (no datatree support yet)

import os
os.environ["AWS_NO_SIGN_REQUEST"] = "YES"
os.environ["AWS_REGION"] = "us-east-1"
from gdalxarray import GDALBackendEntrypoint
backend = GDALBackendEntrypoint()
ds = backend.open_dataset("/vsiicechunk/{/vsis3/earthmover-icechunk-era5/icechunkV2}/pressure/spatial/pv")
<xarray.Dataset> Size: 41TB
Dimensions:         (valid_time: 753888, pressure_level: 13, latitude: 721,
                     longitude: 1440)
Coordinates:
  * valid_time      (valid_time) datetime64[ns] 6MB 1940-01-01 ... 2025-12-31...
  * pressure_level  (pressure_level) float64 104B 1e+03 925.0 ... 100.0 50.0
  * latitude        (latitude) float64 6kB 90.0 89.75 89.5 ... -89.75 -90.0
  * longitude       (longitude) float64 12kB 0.0 0.25 0.5 ... 359.2 359.5 359.8
Data variables:
    pv              (valid_time, pressure_level, latitude, longitude) float32 41TB ...
Attributes: (12/33)
    GRIB_NV:                                  0
    GRIB_Nx:                                  1440
    GRIB_Ny:                                  721
    GRIB_cfName:                              unknown
    GRIB_cfVarName:                           pv
    GRIB_dataType:                            an
    ...                                       ...
    GRIB_uvRelativeToGrid:                    0
    _FillValue:                               AAAAAAAA+H8=
    coordinates:                              lsm
    long_name:                                Potential vorticity
    standard_name:                            unknown
    units:                                    K m**2 kg**-1 s**-1
ds = backend.open_dataset("/vsiicechunk/{/vsis3/earthmover-icechunk-era5/icechunkV2}/pressure/spatial")

<xarray.Dataset> Size: 326TB
Dimensions:         (latitude: 721, longitude: 1440, pressure_level: 13,
                     valid_time: 753888)
Coordinates:
  * latitude        (latitude) float64 6kB 90.0 89.75 89.5 ... -89.75 -90.0
  * longitude       (longitude) float64 12kB 0.0 0.25 0.5 ... 359.2 359.5 359.8
  * pressure_level  (pressure_level) float64 104B 1e+03 925.0 ... 100.0 50.0
  * valid_time      (valid_time) datetime64[ns] 6MB 1940-01-01 ... 2025-12-31...
Data variables:
    lsm             (latitude, longitude) float32 4MB ...
    pv              (valid_time, pressure_level, latitude, longitude) float32 41TB ...
    q               (valid_time, pressure_level, latitude, longitude) float32 41TB ...
    r               (valid_time, pressure_level, latitude, longitude) float32 41TB ...
    t               (valid_time, pressure_level, latitude, longitude) float32 41TB ...
    u               (valid_time, pressure_level, latitude, longitude) float32 41TB ...
    v               (valid_time, pressure_level, latitude, longitude) float32 41TB ...
    w               (valid_time, pressure_level, latitude, longitude) float32 41TB ...
    z               (valid_time, pressure_level, latitude, longitude) float32 41TB ...
Attributes: (12/46)
    Conventions:                CF-1.7
    GRIB_centre:                ecmf
    GRIB_centreDescription:     European Centre for Medium-Range Weather Fore...
    GRIB_subCentre:             0
    acknowledgement:            Generated using Copernicus Climate Change Ser...
    cdm_data_type:              Grid
    ...                         ...
    standard_name_vocabulary:   CF Standard Name Table v85
    summary:                    ERA5 is the fifth generation ECMWF atmospheri...
    time_coverage_end:          2025-12-31T23:00:00Z
    time_coverage_resolution:   PT1H
    time_coverage_start:        1940-01-01T00:00:00Z
    title:                      ERA5 Hourly Global Reanalysis - chunked for s...

@mdsumner

mdsumner commented Jun 17, 2026

Copy link
Copy Markdown
Author

or

import os
os.environ["AWS_NO_SIGN_REQUEST"]="YES"
os.environ["AWS_REGION"]="us-east-1" 
import xarray
dsn = "/vsis3/earthmover-icechunk-era5/icechunkV2" 

xarray.open_dataset(dsn, engine = "gdalxarray", group = "/pressure/spatial")                                                                                                                            <xarray.Dataset> Size: 326TB
Dimensions:         (latitude: 721, longitude: 1440, pressure_level: 13,
                     valid_time: 753888)
Coordinates:
  * latitude        (latitude) float64 6kB 90.0 89.75 89.5 ... -89.75 -90.0
  * longitude       (longitude) float64 12kB 0.0 0.25 0.5 ... 359.2 359.5 359.8
  * pressure_level  (pressure_level) float64 104B 1e+03 925.0 ... 100.0 50.0
  * valid_time      (valid_time) datetime64[ns] 6MB 1940-01-01 ... 2025-12-31...
Data variables:
    lsm             (latitude, longitude) float32 4MB ...
    pv              (valid_time, pressure_level, latitude, longitude) float32 41TB ...
    q               (valid_time, pressure_level, latitude, longitude) float32 41TB ...
    r               (valid_time, pressure_level, latitude, longitude) float32 41TB ...
    t               (valid_time, pressure_level, latitude, longitude) float32 41TB ...
    u               (valid_time, pressure_level, latitude, longitude) float32 41TB ...
    v               (valid_time, pressure_level, latitude, longitude) float32 41TB ...
    w               (valid_time, pressure_level, latitude, longitude) float32 41TB ...
    z               (valid_time, pressure_level, latitude, longitude) float32 41TB ...
Attributes: (12/46)
    Conventions:                CF-1.7
    GRIB_centre:                ecmf
    GRIB_centreDescription:     European Centre for Medium-Range Weather Fore...
    GRIB_subCentre:             0
    acknowledgement:            Generated using Copernicus Climate Change Ser...
    cdm_data_type:              Grid
    ...                         ...
    standard_name_vocabulary:   CF Standard Name Table v85
    summary:                    ERA5 is the fifth generation ECMWF atmospheri...
    time_coverage_end:          2025-12-31T23:00:00Z
    time_coverage_resolution:   PT1H
    time_coverage_start:        1940-01-01T00:00:00Z
    title:                      ERA5 Hourly Global Reanalysis - chunked for s...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment