zarr-python thin metadata refactor — plan 1: upstream models into zarr-metadata

2026-07-04-thin-metadata-01-upstream-models.md

Thin Metadata Refactor — Part 1: Upstream Metadata Models into zarr-metadata

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add a zarr_metadata.model subpackage — frozen-dataclass models that are canonical, lossless representations of Zarr v2/v3 metadata documents — by porting the prototyped array models from ~/dev/zng/zng_metadata and adding new group/consolidated models, so a later zarr-python PR can consume them.

Architecture: zarr_metadata.model sits on top of the existing raw TypedDicts (ArrayMetadataV3, GroupMetadataV3, ConsolidatedMetadataV3, ZGroupMetadata, ...). Every v3 extension point is a ZarrMetadataV3 (name + configuration) value. Models do structural validation only; no codec/dtype/grid interpretation. One PR against d-v-b/zarr-python main.

Tech Stack: Python ≥3.11, frozen slotted kw-only dataclasses, typing_extensions (TypedDict/TypeIs/Unpack/TypeAliasType), pytest, pyright strict.

Global Constraints

• Package floor is requires-python = ">=3.11": no type X = ... statements, no bare from typing import TypeIs/Unpack — use typing_extensions.
• Reuse JSONValue from zarr_metadata._common (a TypeAliasType); do NOT port zng's type JSON = ... alias. Recursive aliases must stay TypeAliasType (pydantic TypeAdapter fixture tests break otherwise).
• JSON arrays are tuples: _arrays_to_tuples converts on ingest; models hold tuples.
• fill_value is held verbatim in JSON form (NaN as the string "NaN", etc.); models never interpret dtypes.
• No falsy conditionals: is None / explicit tests, never if not x: on possibly-falsy values.
• Docstrings use plain single backticks (no Sphinx roles, no double backticks).
• Every new test function gets a docstring stating what behavior it verifies. Compact happy paths via parametrization; one named test per failure mode.
• Type must_understand as bool, enforce False at runtime (__post_init__).
• pyright strict is the checker for this package (cd packages/zarr-metadata && uvx pyright src); root-repo prek run mypy does not cover it. Ruff extends the root config with target-version = "py311".
• Tests run as cd packages/zarr-metadata && uv run --group test pytest tests.
• Commits: conventional commits + trailer Assisted-by: ClaudeCode:claude-fable-5.
• Port editorially (capture spirit, not letter): apply the adaptation table exactly, but keep zng's structure/docstrings where they are already good.

Source Material

• Port source: /Users/d-v-b/dev/zng/zng_metadata/src/zng_metadata/_metadata.py (580 lines), /Users/d-v-b/dev/zng/zng_metadata/src/zng_metadata/__init__.py, /Users/d-v-b/dev/zng/zng_metadata/tests/test_metadata.py (1026 lines), /Users/d-v-b/dev/zng/zng_metadata/tests/_cases.py (42 lines).
• Existing raw types: packages/zarr-metadata/src/zarr_metadata/ — _common.py (JSONValue, MetadataFieldV3 area), v3/array.py (ArrayMetadataV3, ExtensionFieldV3), v3/group.py (GroupMetadataV3), v3/consolidated.py (ConsolidatedMetadataV3), v2/array.py (ZArrayMetadata, ArrayMetadataV2, CodecMetadataV2, DataTypeMetadataV2), v2/group.py (ZGroupMetadata, GroupMetadataV2), v2/consolidated.py (ConsolidatedMetadataV2), v2/attributes.py (ZAttrsMetadata).

Adaptation Table (applies to every ported file)

zng source	ported form
from zng_metadata ... imports	from zarr_metadata.model ... (absolute, matching package convention)
type JSON = str | int | ... alias + JSON exports	delete; use JSONValue from zarr_metadata._common everywhere (incl. TypeIs[JSONValue])
type ArrayMetadataStoreKeyV3 = Literal["zarr.json"] (and the other two type aliases)	plain assignment: ArrayMetadataStoreKeyV3 = Literal["zarr.json"]
from typing import ... TypeIs, Unpack, TypedDict	TypeIs, Unpack, TypedDict from typing_extensions; the rest stays typing
from zarr_metadata import (...) (zng imported the released package)	same names, now in-tree: keep absolute from zarr_metadata import ... / from zarr_metadata.v2 import ...
module name _metadata.py	split: model/_validation.py + model/_array.py (boundaries specified in Tasks 1–2)

File Structure

packages/zarr-metadata/src/zarr_metadata/model/
    __init__.py       # public re-exports of everything below
    _validation.py    # ValidationProblem, MetadataValidationError, _prefix,
                      # validate/is/parse_json, validate/is/parse_metadata_field_v3,
                      # validate/is/parse_array_metadata_v3, validate/is/parse_array_metadata_v2,
                      # validate/is/parse_group_metadata_v3, validate/is/parse_group_metadata_v2,
                      # _is_int_sequence, _validate_attributes, _arrays_to_tuples,
                      # ARRAY_METADATA_*_KEYS_* and GROUP_METADATA_*_KEYS_* constants
    _array.py         # ZarrMetadataV3, ArrayMetadataModelV3(+Partial),
                      # ArrayMetadataModelV2(+Partial), store-key constants
    _group.py         # GroupMetadataModelV3(+Partial), GroupMetadataModelV2(+Partial),
                      # ConsolidatedMetadataModelV3, ConsolidatedMetadataModelV2,
                      # group/consolidated store-key constants
packages/zarr-metadata/tests/model/
    __init__.py
    _cases.py         # ported Expect/ExpectFail helpers
    test_array.py     # ported test_metadata.py
    test_group.py     # new group/consolidated model tests
packages/zarr-metadata/changes/
    +upstream_models.feature.md   # towncrier fragment (rename to <PR#>.feature.md after PR opens if the checker requires it)

---

Task 1: Port the validation module

Files:

• Create: packages/zarr-metadata/src/zarr_metadata/model/__init__.py
• Create: packages/zarr-metadata/src/zarr_metadata/model/_validation.py
• Source: /Users/d-v-b/dev/zng/zng_metadata/src/zng_metadata/_metadata.py lines 1–274 (imports through parse_array_metadata_v2), plus _arrays_to_tuples (lines 286–292)

Interfaces:

• Produces: ValidationProblem(loc: tuple[str | int, ...], message: str); MetadataValidationError(problems: list[ValidationProblem]); validate_json(value: object) -> list[ValidationProblem]; is_json(value: object) -> TypeIs[JSONValue]; parse_json(value: object) -> JSONValue; validate_metadata_field_v3 / is_metadata_field_v3 / parse_metadata_field_v3; validate_array_metadata_v3 / is_array_metadata_v3 / parse_array_metadata_v3; validate_array_metadata_v2 / is_array_metadata_v2 / parse_array_metadata_v2; _arrays_to_tuples(obj: object) -> object; _prefix(loc_head, problems); constants ARRAY_METADATA_REQUIRED_KEYS_V3, ARRAY_METADATA_OPTIONAL_KEYS_V3, ARRAY_METADATA_STANDARD_KEYS_V3, ARRAY_METADATA_REQUIRED_KEYS_V2.

• Step 1: Create the module skeleton. model/__init__.py starts as a docstring only ("""In-memory models for Zarr metadata documents."""); exports are added in Task 6.

• Step 2: Port _validation.py. Copy the listed line ranges from the zng source verbatim, then apply the Adaptation Table. The import header becomes:

"""Structural validation for Zarr metadata documents.

Validators check JSON structure (shapes, key presence, primitive kinds),
not domain validity. Each concept gets a `validate_*` (all problems),
`is_*` (TypeIs), and `parse_*` (narrow or raise) trio.
"""

from __future__ import annotations

from collections.abc import Mapping, Sequence
from typing import Final, cast

from typing_extensions import TypeIs

from zarr_metadata import ArrayMetadataV2, ArrayMetadataV3, MetadataFieldV3
from zarr_metadata._common import JSONValue

All JSON mentions become JSONValue (annotations, TypeIs[JSONValue], docstrings). _arrays_to_tuples moves here (both _array.py and _group.py will import it).

• Step 3: Type-check. Run: cd packages/zarr-metadata && uvx pyright src — expect 0 errors (iterate on strict-mode complaints; do not loosen typeCheckingMode).

• Step 4: Commit.

git add packages/zarr-metadata/src/zarr_metadata/model/
git commit -m "feat(zarr-metadata): add model._validation — structural validators

Assisted-by: ClaudeCode:claude-fable-5"

Task 2: Port the array models

Files:

• Create: packages/zarr-metadata/src/zarr_metadata/model/_array.py
• Source: /Users/d-v-b/dev/zng/zng_metadata/src/zng_metadata/_metadata.py lines 276–580 (type store-key aliases, ZarrMetadataV3, ArrayMetadataModelV3Partial, ArrayMetadataModelV3, ArrayMetadataModelV2Partial, ArrayMetadataModelV2)

Interfaces:

• Consumes: parse_metadata_field_v3, parse_array_metadata_v3, parse_array_metadata_v2, _arrays_to_tuples, ARRAY_METADATA_STANDARD_KEYS_V3 from zarr_metadata.model._validation.

• Produces:

  • ZarrMetadataV3(name: str, configuration: dict[str, object]) with to_json() -> MetadataFieldV3, from_json(data: object) -> ZarrMetadataV3
  • ArrayMetadataModelV3 (fields: zarr_format=3, node_type="array" both init=False; shape: tuple[int, ...]; fill_value: JSONValue; data_type: ZarrMetadataV3; chunk_grid: ZarrMetadataV3; codecs: tuple[ZarrMetadataV3, ...]; chunk_key_encoding: ZarrMetadataV3; dimension_names: tuple[str | None, ...] | None; attributes: dict[str, object]; storage_transformers: tuple[ZarrMetadataV3, ...]; extra_fields: dict[str, ExtensionFieldV3]) with create_default(**overrides), update(**kwargs), to_json() -> ArrayMetadataV3, from_json(data: object), from_key_value(mapping: Mapping[str, bytes]), to_key_value(*, indent=None) -> Mapping[str, bytes]
  • ArrayMetadataModelV2 (fields: zarr_format=2 init=False; shape; dtype: DataTypeMetadataV2; chunks; fill_value: JSONValue; order: ArrayOrderV2; compressor: CodecMetadataV2 | None; filters: tuple[CodecMetadataV2, ...] | None; dimension_separator: ArrayDimensionSeparatorV2 = "/"; attributes) with the same method set; from_key_value/to_key_value handle the .zarray / .zattrs split
  • ArrayMetadataModelV3Partial / ArrayMetadataModelV2Partial TypedDicts
  • Constants ARRAY_METADATA_STORE_KEY_V3 = "zarr.json", ARRAY_METADATA_STORE_KEY_V2 = ".zarray", ATTRIBUTES_STORE_KEY_V2 = ".zattrs"

• Step 1: Port _array.py. Copy the line ranges, apply the Adaptation Table. Import header:

"""In-memory models for Zarr array metadata documents."""

from __future__ import annotations

import dataclasses
import json
from collections.abc import Mapping
from dataclasses import dataclass, field
from typing import Final, Literal, cast

from typing_extensions import TypedDict, Unpack

from zarr_metadata import (
    ArrayDimensionSeparatorV2,
    ArrayMetadataV2,
    ArrayMetadataV3,
    ArrayOrderV2,
    ExtensionFieldV3,
    MetadataFieldV3,
)
from zarr_metadata._common import JSONValue
from zarr_metadata.model._validation import (
    ARRAY_METADATA_STANDARD_KEYS_V3,
    _arrays_to_tuples,
    parse_array_metadata_v2,
    parse_array_metadata_v3,
    parse_metadata_field_v3,
)
from zarr_metadata.v2 import CodecMetadataV2, DataTypeMetadataV2

Store-key aliases become plain assignments per the Adaptation Table. Preserve zng's docstrings, __post_init__ overlap check, and the # type: ignore[...] comments (convert mypy codes to plain # type: ignore[...] forms pyright accepts, or # pyright: ignore[...] where strict mode names a different rule).

• Step 2: Type-check. Run: cd packages/zarr-metadata && uvx pyright src — expect 0 errors.

• Step 3: Smoke round-trip. Run:

cd packages/zarr-metadata && uv run python -c "
from zarr_metadata.model._array import ArrayMetadataModelV3
m = ArrayMetadataModelV3.create_default(shape=(4,), attributes={'a': 1})
assert ArrayMetadataModelV3.from_json(m.to_json()) == m
print('roundtrip ok')"

Expected: roundtrip ok

• Step 4: Commit.

git add packages/zarr-metadata/src/zarr_metadata/model/_array.py
git commit -m "feat(zarr-metadata): add model._array — array metadata models

Assisted-by: ClaudeCode:claude-fable-5"

Task 3: Port the array-model test suite

Files:

• Create: packages/zarr-metadata/tests/model/__init__.py (empty)
• Create: packages/zarr-metadata/tests/model/_cases.py — port of /Users/d-v-b/dev/zng/zng_metadata/tests/_cases.py
• Create: packages/zarr-metadata/tests/model/test_array.py — port of /Users/d-v-b/dev/zng/zng_metadata/tests/test_metadata.py

Interfaces:

• Consumes: everything from Task 1–2 via from zarr_metadata.model import ... (public names) and from zarr_metadata.model._validation import _arrays_to_tuples, _prefix (private helpers, imported from their module directly). Case helpers via from tests.model._cases import Expect, ExpectFail (the tests/ directory is already a package with __init__.py).

• Step 1: Port _cases.py verbatim (only change: none expected — it imports re, dataclasses, pytest only). Keep the ExpectFail.raises() regex-msg behavior exactly (msg is an optional regex; never pass msg="").

• Step 2: Port test_array.py. Apply the Adaptation Table to imports: from zng_metadata import (...) → from zarr_metadata.model import (...); JSON → JSONValue; from zng_metadata._metadata import _arrays_to_tuples, _prefix → from zarr_metadata.model._validation import _arrays_to_tuples, _prefix; from _cases import ... → from tests.model._cases import .... Note the ported public-name imports require Task 6's exports — until then import from the private modules (zarr_metadata.model._validation, zarr_metadata.model._array); flip to zarr_metadata.model in Task 6.

• Step 3: Run the ported suite. Run: cd packages/zarr-metadata && uv run --group test pytest tests/model -q Expected: all pass (zng suite is green upstream; failures indicate a porting error — fix the port, not the test, unless the failure is a genuine 3.11/typing_extensions incompatibility).

• Step 4: Run the whole package suite. Run: cd packages/zarr-metadata && uv run --group test pytest tests -q Expected: all pass (existing test_public_api.py untouched so far).

• Step 5: Commit.

git add packages/zarr-metadata/tests/model/
git commit -m "test(zarr-metadata): port array model test suite

Assisted-by: ClaudeCode:claude-fable-5"

Task 4: Group metadata models (v3 + v2), TDD

Files:

• Create: packages/zarr-metadata/src/zarr_metadata/model/_group.py
• Modify: packages/zarr-metadata/src/zarr_metadata/model/_validation.py (add group validators + key constants)
• Test: packages/zarr-metadata/tests/model/test_group.py

Interfaces:

• Consumes: ValidationProblem, _prefix, _validate_attributes, _arrays_to_tuples, MetadataValidationError from ._validation; ArrayMetadataModelV3, ArrayMetadataModelV2 from ._array (for Task 5's consolidated models, defined in the same file).
• Produces:
  • validate_group_metadata_v3 / is_group_metadata_v3 / parse_group_metadata_v3 and validate_group_metadata_v2 / is_group_metadata_v2 / parse_group_metadata_v2 in _validation.py, plus GROUP_METADATA_REQUIRED_KEYS_V3, GROUP_METADATA_OPTIONAL_KEYS_V3, GROUP_METADATA_STANDARD_KEYS_V3 (built from GroupMetadataV3.__required_keys__ / __optional_keys__ like the array constants)
  • GroupMetadataModelV3 (fields below) + GroupMetadataModelV3Partial
  • GroupMetadataModelV2 + GroupMetadataModelV2Partial
  • Constants GROUP_METADATA_STORE_KEY_V3 = "zarr.json", GROUP_METADATA_STORE_KEY_V2 = ".zgroup"

Behavioral contract (mirror the array models' conventions exactly — same method names, same create_default/update idiom, same overlap __post_init__):

@dataclass(frozen=True, slots=True, kw_only=True)
class GroupMetadataModelV3:
    zarr_format: Literal[3] = field(default=3, init=False)
    node_type: Literal["group"] = field(default="group", init=False)
    attributes: dict[str, JSONValue]
    consolidated_metadata: ConsolidatedMetadataModelV3 | None
    extra_fields: dict[str, ExtensionFieldV3]

• from_json: parse_group_metadata_v3(_arrays_to_tuples(data)); the consolidated_metadata key (a zarr-python convention stored as an extension field, already typed by zarr_metadata.v3.consolidated.ConsolidatedMetadataV3) is pulled out into the typed field; remaining non-standard keys land in extra_fields. __post_init__ rejects extra_fields overlapping standard keys or "consolidated_metadata".
• to_json: emits zarr_format, node_type; attributes only when non-empty; consolidated_metadata only when not None; then extra fields — matching the array model's omit-empty convention.
• from_key_value/to_key_value: single zarr.json key.
• create_default() returns GroupMetadataModelV3(attributes={}, consolidated_metadata=None, extra_fields={}) then .update(**overrides).

@dataclass(frozen=True, slots=True, kw_only=True)
class GroupMetadataModelV2:
    zarr_format: Literal[2] = field(default=2, init=False)
    attributes: dict[str, JSONValue]

• from_json takes the merged form (GroupMetadataV2 TypedDict: zarr_format + optional attributes); from_key_value merges .zgroup + optional .zattrs; to_key_value splits back (always emitting both keys, matching ArrayMetadataModelV2.to_key_value); to_json emits zarr_format and attributes only when non-empty.

validate_group_metadata_v3(value): mapping check; missing required keys (zarr_format, node_type) reported per-key; attributes via _validate_attributes; if consolidated_metadata present it must be a Mapping (problem at ("consolidated_metadata",) otherwise; deep validation of its entries belongs to the consolidated model in Task 5). Like the array validators, literal values are not checked — presence and shape only.

validate_group_metadata_v2(value): mapping check; missing zarr_format reported; attributes via _validate_attributes.

• Step 1: Write the failing tests in tests/model/test_group.py. Docstring on every test. Cover, at minimum:

"""Tests for the group metadata models in `zarr_metadata.model`."""

import json

import pytest

from tests.model._cases import Expect, ExpectFail  # same helpers as test_array
from zarr_metadata.model._group import (
    GroupMetadataModelV2,
    GroupMetadataModelV3,
)
from zarr_metadata.model._validation import (
    MetadataValidationError,
    parse_group_metadata_v2,
    parse_group_metadata_v3,
)


def test_group_v3_roundtrip() -> None:
    """A v3 group document round-trips through the model unchanged."""
    doc = {"zarr_format": 3, "node_type": "group", "attributes": {"a": (1, 2)}}
    model = GroupMetadataModelV3.from_json(doc)
    assert model.to_json() == doc


def test_group_v3_omits_empty_attributes() -> None:
    """to_json omits the attributes key when attributes is empty."""
    model = GroupMetadataModelV3.create_default()
    assert "attributes" not in model.to_json()


def test_group_v3_extra_fields_roundtrip() -> None:
    """Unknown top-level keys land in extra_fields and reappear in to_json."""
    doc = {
        "zarr_format": 3,
        "node_type": "group",
        "my_extension": {"name": "thing", "must_understand": False},
    }
    model = GroupMetadataModelV3.from_json(doc)
    assert model.extra_fields == {"my_extension": {"name": "thing", "must_understand": False}}
    assert model.to_json() == doc


def test_group_v3_extra_fields_overlap_rejected() -> None:
    """Constructing a v3 group model with extra_fields shadowing a standard key raises."""
    with pytest.raises(ValueError, match="Extra fields"):
        GroupMetadataModelV3(
            attributes={},
            consolidated_metadata=None,
            extra_fields={"node_type": {"name": "x", "must_understand": False}},
        )


def test_group_v3_missing_required_key() -> None:
    """parse_group_metadata_v3 reports each missing required key."""
    with pytest.raises(MetadataValidationError, match="node_type"):
        parse_group_metadata_v3({"zarr_format": 3})


def test_group_v3_key_value_roundtrip() -> None:
    """from_key_value(to_key_value()) is the identity for v3 groups."""
    model = GroupMetadataModelV3.create_default(attributes={"a": 1})
    assert GroupMetadataModelV3.from_key_value(model.to_key_value()) == model


def test_group_v2_key_value_split() -> None:
    """v2 to_key_value writes .zgroup and .zattrs; from_key_value merges them."""
    model = GroupMetadataModelV2.create_default(attributes={"a": 1})
    kv = model.to_key_value()
    assert set(kv) == {".zgroup", ".zattrs"}
    assert json.loads(kv[".zgroup"]) == {"zarr_format": 2}
    assert GroupMetadataModelV2.from_key_value(kv) == model


def test_group_v2_from_key_value_without_zattrs() -> None:
    """A v2 group with no .zattrs file parses with empty attributes."""
    model = GroupMetadataModelV2.from_key_value({".zgroup": b'{"zarr_format": 2}'})
    assert model.attributes == {}


def test_group_v2_not_a_mapping() -> None:
    """parse_group_metadata_v2 rejects a non-mapping document."""
    with pytest.raises(MetadataValidationError, match="expected a mapping"):
        parse_group_metadata_v2([1, 2, 3])

(Adjust Expect/ExpectFail usage to match the ported test_array.py idiom where parametrized; the inline forms above are acceptable.)

• Step 2: Run to verify failure. cd packages/zarr-metadata && uv run --group test pytest tests/model/test_group.py -q — expect ImportError (_group does not exist).

• Step 3: Implement _group.py (models per the contract above; GroupMetadataModelV3.consolidated_metadata typed as ConsolidatedMetadataModelV3 | None with the class added in Task 5 — for this task, define a forward-declared placeholder is NOT allowed; instead implement Tasks 4+5's _group.py module in one pass if easier, keeping test commits separate) and the group validators + constants in _validation.py.

• Step 4: Run to verify pass. cd packages/zarr-metadata && uv run --group test pytest tests/model -q — expect all pass.

• Step 5: Type-check. cd packages/zarr-metadata && uvx pyright src — expect 0 errors.

• Step 6: Commit.

git add packages/zarr-metadata/src/zarr_metadata/model/ packages/zarr-metadata/tests/model/test_group.py
git commit -m "feat(zarr-metadata): add group metadata models

Assisted-by: ClaudeCode:claude-fable-5"

Task 5: Consolidated metadata models, TDD

Files:

• Modify: packages/zarr-metadata/src/zarr_metadata/model/_group.py
• Test: packages/zarr-metadata/tests/model/test_group.py (append)

Interfaces:

• Produces:
  • ConsolidatedMetadataModelV3 — kind: Literal["inline"] (init=False), must_understand: bool = False (typed permissive, __post_init__ enforces is False), metadata: dict[str, ArrayMetadataModelV3 | GroupMetadataModelV3]; from_json discriminates entries on their node_type key; to_json emits {kind, must_understand, metadata} with each entry's to_json().
  • ConsolidatedMetadataModelV2 — zarr_consolidated_format: Literal[1] (init=False), metadata: dict[str, JSONValue] holding the flat file-keyed map ("path/.zarray" → raw doc) verbatim; from_json validates only the envelope (both keys present, metadata a string-keyed mapping) and tuple-izes; to_json is the inverse. Store-key constant CONSOLIDATED_METADATA_STORE_KEY_V2 = ".zmetadata".

Design note (records a spec refinement): the design spec says consolidated metadata holds thin child models. That holds for v3, where each entry is a complete zarr.json document. For v2, .zmetadata entries are file fragments (.zarray / .zgroup / .zattrs as separate keys); merging them into node models would lose file-level structure (e.g. which nodes had a .zattrs entry at all) and break byte-faithful round-tripping — the layer's first duty. So the v2 consolidated model stays flat and verbatim; interpreting entries into node models is consumer work (zarr-python's resolve layer, part 2). Record this refinement in the design spec when this task completes.

• Step 1: Write the failing tests (append to test_group.py):

from zarr_metadata.model._group import (
    ConsolidatedMetadataModelV2,
    ConsolidatedMetadataModelV3,
)
from zarr_metadata.model._array import ArrayMetadataModelV3


def test_consolidated_v3_roundtrip() -> None:
    """A v3 group with inline consolidated metadata round-trips, with child
    entries parsed into array/group models."""
    child = ArrayMetadataModelV3.create_default(shape=(2,)).to_json()
    doc = {
        "zarr_format": 3,
        "node_type": "group",
        "consolidated_metadata": {
            "kind": "inline",
            "must_understand": False,
            "metadata": {"a": child, "g": {"zarr_format": 3, "node_type": "group"}},
        },
    }
    model = GroupMetadataModelV3.from_json(doc)
    assert isinstance(model.consolidated_metadata, ConsolidatedMetadataModelV3)
    assert isinstance(model.consolidated_metadata.metadata["a"], ArrayMetadataModelV3)
    assert isinstance(model.consolidated_metadata.metadata["g"], GroupMetadataModelV3)
    assert model.to_json() == doc


def test_consolidated_v3_must_understand_true_rejected() -> None:
    """ConsolidatedMetadataModelV3 enforces must_understand=False at runtime."""
    with pytest.raises(ValueError, match="must_understand"):
        ConsolidatedMetadataModelV3(must_understand=True, metadata={})


def test_consolidated_v2_verbatim_roundtrip() -> None:
    """The v2 .zmetadata model holds the flat file-keyed map verbatim,
    including nodes that have no .zattrs entry."""
    doc = {
        "zarr_consolidated_format": 1,
        "metadata": {
            ".zgroup": {"zarr_format": 2},
            "a/.zarray": {
                "zarr_format": 2, "shape": (2,), "chunks": (2,), "dtype": "|u1",
                "fill_value": 0, "order": "C", "compressor": None, "filters": None,
            },
        },
    }
    model = ConsolidatedMetadataModelV2.from_json(doc)
    assert model.to_json() == doc


def test_consolidated_v2_envelope_validation() -> None:
    """from_json rejects a .zmetadata document missing the metadata key."""
    with pytest.raises(MetadataValidationError, match="metadata"):
        ConsolidatedMetadataModelV2.from_json({"zarr_consolidated_format": 1})

• Step 2: Run to verify failure, Step 3: Implement, Step 4: Run to verify pass (pytest tests/model -q), Step 5: pyright clean, Step 6: Update the design spec (docs/superpowers/specs/2026-07-04-thin-metadata-refactor-design.md "Groups and consolidated metadata" section — add the v2-verbatim refinement; re-push the gist with gh gist edit).

• Step 7: Commit.

git add packages/zarr-metadata/src/zarr_metadata/model/_group.py packages/zarr-metadata/tests/model/test_group.py
git commit -m "feat(zarr-metadata): add consolidated metadata models

Assisted-by: ClaudeCode:claude-fable-5"

Task 6: Public exports

Files:

• Modify: packages/zarr-metadata/src/zarr_metadata/model/__init__.py (re-export every public name from _validation, _array, _group with __all__, mirroring zng's __init__.py plus the new group/consolidated/validator names)
• Modify: packages/zarr-metadata/src/zarr_metadata/__init__.py (top-level front door)
• Modify: packages/zarr-metadata/tests/test_public_api.py (EXPECTED list)
• Modify: packages/zarr-metadata/tests/model/test_array.py (flip private-module imports to zarr_metadata.model public names, per Task 3 Step 2 note)

Interfaces:

• Produces: zarr_metadata.model exporting all model/validator names; top-level zarr_metadata additionally exporting the class names only: ArrayMetadataModelV2, ArrayMetadataModelV2Partial, ArrayMetadataModelV3, ArrayMetadataModelV3Partial, GroupMetadataModelV2, GroupMetadataModelV2Partial, GroupMetadataModelV3, GroupMetadataModelV3Partial, ConsolidatedMetadataModelV2, ConsolidatedMetadataModelV3, ZarrMetadataV3, ValidationProblem, MetadataValidationError. Validator functions and key/store constants stay under zarr_metadata.model (front door promotes types; the function trio family is large and submodule-scoped).

• Step 1: Write model/__init__.py re-exports + __all__ (RUF022 ordering is enforced by ruff — run prek run ruff --files ... or let the pre-commit fix it).

• Step 2: Add the 13 class names to the top-level __init__.py import-and-__all__, and to EXPECTED in tests/test_public_api.py (grouping: TitleCase group — the test asserts SCREAMING/TitleCase/dunder grouping only).

• Step 3: Flip test_array.py imports to public from zarr_metadata.model import (...) (keep _arrays_to_tuples/_prefix private imports as-is).

• Step 4: Run everything. cd packages/zarr-metadata && uv run --group test pytest tests -q — all pass; uvx pyright src — 0 errors.

• Step 5: Commit.

git add packages/zarr-metadata/src/zarr_metadata/__init__.py packages/zarr-metadata/src/zarr_metadata/model/__init__.py packages/zarr-metadata/tests/
git commit -m "feat(zarr-metadata): export model layer from package front door

Assisted-by: ClaudeCode:claude-fable-5"

Task 7: Changelog, lint, PR

Files:

• Create: packages/zarr-metadata/changes/+upstream_models.feature.md

• Step 1: Check the fragment-name rules: uv run --no-sync python ci/check_changelog_entries.py packages/zarr-metadata/changes (run from repo root). If numeric names are required, defer naming to after the PR exists and use the PR number.

• Step 2: Fragment content:

Added `zarr_metadata.model`: frozen-dataclass models (`ArrayMetadataModelV2`,
`ArrayMetadataModelV3`, `GroupMetadataModelV2`, `GroupMetadataModelV3`,
`ConsolidatedMetadataModelV2`, `ConsolidatedMetadataModelV3`, `ZarrMetadataV3`)
that are canonical, lossless representations of Zarr metadata documents, plus
structural validators (`validate_*` / `is_*` / `parse_*`). Every v3 extension
point (data type, chunk grid, chunk key encoding, codecs, storage transformers)
is held as a name + configuration pair; nothing is interpreted.

• Step 3: Lint. prek run --all-files from repo root (ruff formatting/RUF022 for the new files; mypy does not cover this package — pyright already ran).
• Step 4: Create branch off origin/main, cherry-pick/rebase the task commits onto it (work happens in this worktree's branch; final branch name zarr-metadata-model-layer), push to origin (d-v-b/zarr-python).
• Step 5: Open the PR against d-v-b/zarr-python main:
  • Title: feat(zarr-metadata): add metadata model layer
  • Body starts with :robot: _AI text below_ :robot:, links the design gist, summarizes the layer, and flags the two recorded follow-ups: (a) validators check key presence/shape but not literal values (zarr_format, node_type) — inherited from the zng prototype, flagged for reviewer judgment; (b) ArrayMetadataModelV2.to_key_value always emits .zattrs even when attributes are empty — fidelity question to settle before the zarr-python consumption PR.
  • End body with the standard Claude Code attribution line.
• Step 6: Rename the changelog fragment to <PR#>.feature.md if Step 1 found numeric names are required; amend/push.

---

Sequencing After This Plan

1. This PR merges → cut zarr_metadata-v0.4.0 (release is the user's call; the zarr-metadata-release.yml workflow publishes from the tag).
2. Part 2 plan (separate document, written after this lands): zarr-python gains zarr-metadata>=0.4.0, the from_metadata/to_metadata hooks, zarr.core.resolve, the metadata-class swap, consumer rewiring, and the old-machinery deletion — the clean break described in the design spec.

Self-Review Notes

• Spec coverage: this plan implements the spec's "Metadata models" layer + "Groups and consolidated metadata" (with the v2-verbatim refinement recorded in Task 5); the runtime-resolution/AsyncArray/v2-builder sections are explicitly Part 2.
• Type consistency: JSONValue (not JSON) throughout; ZarrMetadataV3 name+configuration everywhere; group model field set matches between Task 4 contract and Task 5 tests.
• No placeholders: ported code is specified by exact source path + line range + adaptation table; new code is given in full or by exact behavioral contract with complete test code.