Date: 2026-07-04 Status: approved design, pre-implementation
Narrow the scope of the array/group metadata classes to one job: forming a canonical, lossless representation of the JSON metadata documents. Everything the metadata classes currently do beyond that — instantiating codecs, chunk grids, chunk key encodings, and data types; casting fill values; validating codec/dtype compatibility; encoding chunk keys — moves to the metadata consumer, which builds runtime objects from the thin metadata.
The role model is zarrs: zarrs_metadata holds pure serde types where every
extension point is a MetadataV3 (name + configuration) pair, and the main
crate builds DataType / ChunkGrid / ChunkKeyEncoding / CodecChain from
that metadata via plugin registries at Array construction time, caching the
built objects on the Array struct. We follow that structure with one
deliberate divergence, described in "V2 handling" below: zarrs normalizes v2
metadata to v3 at the document level and rejects the v2 features that do not
translate; zarr-python supports all of v2, so unification happens at the
runtime-builder level and metadata documents are never rewritten.
These were settled during design discussion:
- The thin layer is the
zarr-metadatapackage's types, used directly. The model dataclasses currently prototyped inzng_metadata(ArrayMetadataModelV3,ArrayMetadataModelV2,NamedConfigModelV3) are upstreamed intozarr-metadataitself, released, and consumed by zarr-python after a floor-pin bump. - V2/v3 unification happens in runtime builders, not documents. Metadata documents stay verbatim v2 or v3 (lossless round-trip). Version-aware builders produce one uniform runtime model from either version.
- Runtime objects are cached on
AsyncArray. Resolution happens once at array construction; consumers read resolved fields from the array, never thick objects from the metadata. - Clean break. The refactor lands as one breaking change at the next major release. No deprecation shims that rebuild thick objects from thin metadata, no parallel opt-in path.
| Layer | Home | Job |
|---|---|---|
| Raw document types | zarr-metadata (exists) |
TypedDicts + structural validators for the JSON documents (ArrayMetadataV3, ArrayMetadataV2, MetadataFieldV3, ...) |
| Metadata models | zarr-metadata (upstreamed from zng_metadata) |
Frozen dataclasses: the canonical representation. Behavior limited to from_json / to_json / update / create_default / from_key_value / to_key_value |
| Runtime resolution | zarr-python | Build ZDType, chunk grid, chunk key encoding, fill-value scalar, and codec pipeline from the models via registry; cache on AsyncArray |
Release choreography (enforced by the min_deps hatch env and
ci/check_min_deps_floor.py): models land upstream in zarr-metadata, a
release is cut, zarr-python's floor pin is bumped, and only then does
zarr-python import the new symbols.
zarr.core.metadata.ArrayV3Metadata, ArrayV2Metadata, and GroupMetadata
become (aliases of) the upstreamed models.
Today's ArrayV3Metadata (src/zarr/core/metadata/v3.py) holds thick objects;
the model holds JSON-shaped values:
| Field | Today | After |
|---|---|---|
data_type |
ZDType |
MetadataFieldModelV3 (role alias; today = NamedConfigModelV3, name + configuration) |
chunk_grid |
ChunkGridMetadata object |
MetadataFieldModelV3 |
chunk_key_encoding |
ChunkKeyEncoding object |
MetadataFieldModelV3 |
codecs |
tuple[Codec, ...] |
tuple[MetadataFieldModelV3, ...] |
fill_value |
cast scalar (Any) |
JSON verbatim (NaN as the spec string "NaN", etc.) |
storage_transformers |
tuple[dict, ...] |
tuple[MetadataFieldModelV3, ...] |
shape, dimension_names, attributes, extra_fields, zarr_format, and
node_type are already thin and keep their shapes. JSON arrays are held as
tuples throughout, per the established convention.
dtype stays the raw v2 dtype spec (string or structured list), compressor
and filters stay codec-config mappings (CodecMetadataV2), order and
dimension_separator stay literals. from_key_value / to_key_value handle
the .zarray / .zattrs split.
Everything the current __init__ does beyond structural validation leaves
this layer: codec instantiation (parse_codecs), codec evolution
(evolve_from_array_spec threading against an ArraySpec), fill-value
casting (data_type.cast_scalar), and codec/dtype validation
(validate_codecs). The work-doing members leave too: encode_chunk_key,
inner_codecs, chunks, shards, dtype, get_chunk_spec, and the v2
chunk_grid property. Their functionality reappears at the array/runtime
layer.
__eq__ / __hash__ by JSON form stops being a hand-written override and
becomes the natural dataclass field comparison — the destination the current
JSON-form __eq__ was anticipating.
to_buffer_dict moves to the I/O glue (zarr.core.metadata.io), since
Buffer is a zarr-python concept the models must not know about.
GroupMetadata is nearly thin already (attributes + consolidated metadata).
A group model is added upstream alongside the array models. Consolidated
metadata holds thin child models; no thick objects are ever nested inside it.
Refinement (recorded during implementation, 2026-07-05): "consolidated
metadata holds thin child models" applies to v3, where each entry is a
complete zarr.json document. For v2, .zmetadata entries are file
fragments (.zarray / .zgroup / .zattrs as separate flat keys); merging
them into node models would lose file-level structure — e.g. which nodes had
a .zattrs entry at all — and break byte-faithful round-tripping, the
layer's first duty. So ConsolidatedMetadataModelV2 holds the flat
file-keyed map verbatim; interpreting entries into node models is consumer
work (the resolve layer, part 2).
A new module, zarr.core.resolve (name negotiable at implementation time),
with one builder per extension point, dispatching through the existing
zarr.registry:
def resolve_dtype(meta: ArrayMetadataModelV3 | ArrayMetadataModelV2) -> ZDType: ...
def resolve_chunk_grid(meta) -> ChunkGrid: ...
def resolve_chunk_key_encoding(meta) -> ChunkKeyEncoding: ...
def resolve_fill_value(meta, dtype: ZDType) -> TBaseScalar: ...
def resolve_codec_chain(meta, dtype, ...) -> tuple[Codec, ...]: ...Codec, ChunkGrid, ChunkKeyEncoding, and ZDType gain
from_metadata(cls, meta: MetadataFieldModelV3) -> Self and
to_metadata(self) -> MetadataFieldModelV3, replacing today's from_dict /
to_dict. This is zarrs' Plugin + from_metadata pattern. The registry
maps meta.name to the class; unknown names raise the existing
UnknownCodecError (and analogous errors for the other extension points).
- Parse time (models): structural validation only — the document has the
right keys with the right JSON shapes. This is
zarr-metadata'svalidate_*machinery. - Resolve time (zarr-python): semantic validation — exactly one
ArrayBytesCodec, codec/dtype compatibility (e.g. vlen-utf8 pairing), codec evolution against the array spec, fill-value castability. Resolution happens atAsyncArrayconstruction, so an invalid stored document fails when the array is opened, not when the document is parsed. - Must-understand extension fields (spec obligation, resolve time): the
v3 spec requires an implementation to fail to open a group or array
carrying metadata fields it does not recognize that are not explicitly
must_understand: false(fields are implicitly must-understand). The models exposemust_understand_fields(the obligation partition ofextra_fields); the resolve layer MUST fail whenmeta.must_understand_fields.keys() - recognized_extensionsis non-empty. Core zarr-python recognizes no extra fields, so any unwaived unknown field fails open — matching currentparse_extra_fieldsbehavior. Part 2's tests must pin this. Documents carrying such fields remain valid documents (another implementation may understand them), which is why the duty sits at resolve, not in document validation.
AsyncArray.__init__ resolves once and stores results as fields alongside
.metadata. Precedent: codec_pipeline is already built and cached exactly
this way via create_codec_pipeline (src/zarr/core/array.py). The resolved
set is: dtype (ZDType), chunk grid, chunk key encoding, fill-value scalar,
codec chain / pipeline.
Consumer rewiring is contained: only five files in src/zarr read thick
metadata fields today (core/array.py, api/asynchronous.py,
core/chunk_grids.py, testing/strategies.py, testing/stateful.py).
The filters / compressors / serializer properties, chunk-key iteration,
and create_codec_pipeline all switch from self.metadata.X to the resolved
fields.
Full v2 support is a requirement, and a faithful v2→v3 document translation
does not exist for the full v2 feature set (fill_value: null, object dtypes
and object codecs, structured dtypes, arbitrary numcodecs filters with no v3
identity). So, diverging from zarrs:
ArrayMetadataModelV2round-trips.zarray/.zattrsbyte-faithfully. Documents are never rewritten into v3 form.- The v2 builders produce the same runtime types as the v3 builders:
RegularChunkGridfromchunks;V2ChunkKeyEncodingfromdimension_separator;ZDTypefrom the v2 dtype spec (structured, datetime, object — all of it); a codec chain absorbingorder,filters,compressor, and object codecs (asV2Codecdoes today). - Downstream of resolution, nothing branches on
zarr_format.
create_array and friends keep accepting thick objects from users (codec
instances, ZDTypes, numpy dtypes). The creation path serializes them to a
thin model via to_metadata(), constructs the AsyncArray, and the array
resolves back. Validation therefore runs exactly once, at resolve time. The
heavy work currently in ArrayV3Metadata.__init__ (codec evolution against
an ArraySpec) moves here.
update_shape / update_attributes become model.update(...)
(dataclasses.replace). Resize swaps the metadata field on the array and
keeps the resolved runtime — nothing shape-dependent is cached.
metadata.codecs,.data_type,.chunk_grid,.chunk_key_encoding,.compressor,.filterschange type from thick objects to JSON-shaped values.metadata.dtype,.chunks,.shards,.inner_codecs,.encode_chunk_key,.get_chunk_specare removed from metadata; public equivalents live at the array layer (mostly already existing).- The permissive, coercing
ArrayV3Metadata.__init__(accepts dicts, codec instances, strings) becomes the strict model constructor; coercion happens increate_array's normalization, not in the metadata class. from_dict/to_dictbecomefrom_json/to_json.zarr.abc.metadata.Metadataand itsto_dictmachinery are retired.- Lands in one stroke at the next major release; internal consumers updated in the same change.
- Round-trip property tests: JSON document → model → JSON is identity,
including v2 edge cases (structured dtypes,
fill_value: null, object codecs) and v3 extension fields. These live upstream with the models; zarr-python re-exercises them through store I/O. - Resolve-layer tests: per extension point, happy path parametrized compactly; one named test per failure mode (unknown codec name, zero array-bytes codecs, two array-bytes codecs, dtype/codec mismatch, uncastable fill value).
- Regression net: the existing suite exercises every consumer path
through
Array; it is the real gate on the consumer-side rewiring.
- Removing donfig, changing the config system, or anything about
ArrayConfig.enginerouting. - New public API for the resolve layer beyond what
AsyncArrayexposes; builder functions start private. - Changing storage, indexing, or codec implementations themselves — only how they are constructed from metadata.