feat!:pydantic zarr scoped validation - #330

…bers override

The pyz ArraySpec type parameter was phantom (both wrappers subclassed
ArraySpec[Any]), so every declared attributes model on the Sentinel
array classes was decorative: Sentinel2DataArray accepted arbitrary
attributes and held plain dicts. The parameter is now passed through,
so attributes validate against the declared model and read back typed.
The GroupSpec wrappers stop binding the members mapping type into the
parent's member-item slot (the old type-var hack and both its ignores):
with pydantic-zarr's covariant parameters, parameterizing the parent at
its widest member type and narrowing the members field is a legal
override. s3_olci spells unconstrained attributes Mapping[str, object]
(object no longer satisfies the enforced bound), and the GeoZarr v3
group classes use AnyGroupSpec, now an ordinary wide type, so member
reads stay checked. pydantic-zarr is bumped to main with covariant
spec parameters (zarr-developers/pydantic-zarr#191).

Assisted-by: ClaudeCode:claude-fable-5
Co-Authored-By: Claude Fable 5 <[email protected]>

load_sentinel2 reads a Zarr store into the validated Sentinel2Root
input model — keeping the model as the carrier instead of validating
and discarding it — and plan_geozarr is a pure function from that model
to the GeoZarr hierarchy to write, as a Zarr V3 GroupSpec tree: the
metadata plane of the conversion, independent of chunk copying. The
reflectance pyramid (native levels with the product level's finer bands
injected, coarser levels derived, sharded uint16 bands, float64
coordinates, CF grid mappings from the product's own proj:wkt2 via a
CF round-trip, spatial/proj/multiscales convention attributes) is
derived entirely from the input model's metadata, and reproduces the
converter's stored snapshot byte for byte for the product whose tile
the snapshot harness models; for the other products the snapshots carry
the harness's synthetic tile georeferencing, so the tests mask those
values and pin the plan's georeferencing to the product's own band
attributes instead. conditions and quality stay on the DataTree
converter for now.

Assisted-by: ClaudeCode:claude-fable-5
Co-Authored-By: Claude Fable 5 <[email protected]>

…quality

plan_geozarr now plans the whole hierarchy. The aux multiscale groups
(l1c/l2a classification, cloud probabilities) get the same pyramid
treatment as reflectance via AUX_MULTISCALE_LEVELS — native levels
carried, coarser levels derived with attributes verbatim, nearest
resampling when the arrays carry a flag table — and every other
conditions/quality dataset is carried: georeferenced datasets (detector
footprints, quality masks, atmosphere) gain spatial attributes and a
spatial_ref grid mapping derived from their own proj attributes,
geometry and meteorology are translated as-is (string arrays to
fixed_length_utf32, coordinates uncompressed, data arrays sharded), and
the quicklook groups and scalar arrays are dropped, as the converter
drops them. The planner walks the input model's dump, since the input
model types these subtrees loosely. Runtime assert statements are
replaced with raises or restructured away.

The whole planned tree is byte-for-byte equal to the stored converter
snapshot for the product whose tile the snapshot harness models; for
the others the tests mask the harness's synthetic-tile georeferencing
and the generated aux levels' chunk shapes, which the converter derives
from incidental dask chunking (inner chunks of 915, 512 and 1830 for
the same situation across the snapshots) where the plan uses its
uniform rule.

Assisted-by: ClaudeCode:claude-fable-5
Co-Authored-By: Claude Fable 5 <[email protected]>

…re encoding

Coarsening divides whatever dask chunks the source dataset happens to
have, so the stored encoding of a generated level depended on the
incidental chunking of the object it was coarsened from: the snapshots
held inner chunks of 915, 512 and 1830 for like levels. generic
multiscales now rechunks the coarsened dataset to the spatial chunk
target (min(spatial_chunk, dim) per dimension) before deriving the
encoding, matching the native levels' policy and the spec-level plan's
uniform rule. The snapshots are regenerated (the only other change is
the omission of null dimension_names, which current pydantic-zarr no
longer serializes), and the plan tests drop their generated-level
chunk masking: the whole tree is now pinned.

Assisted-by: ClaudeCode:claude-fable-5
Co-Authored-By: Claude Fable 5 <[email protected]>

Picks up the unreleased extra-fields support (consolidated root documents
now validate, carried in extra_fields and refused by to_zarr), the typed
to_metadata_json / to_stored_metadata_json exports, the always-complete
GroupSpec hierarchy, and whole-document validation for Zarr V2 arrays.
Whole-repo pyright and the full non-network suite are green against it.

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <[email protected]>

…data and zarr-cm

Every zarr metadata field the Sentinel-2 plan writes now carries its
published type: zarr-metadata's *Object TypedDicts for the extension
points (regular chunk grid, default chunk key encoding, and the bytes,
blosc, zstd, crc32c and sharding_indexed codecs), zarr-cm's convention
types for the zarr-conventions attributes (the CMO declarations replace
the hand-written registry dicts, and layout entries are LayoutObject
with the spatial convention's keys as typed extra items), and dedicated
TypedDicts for the fixed-shape documents the plan builds itself:
GridMappingAttrs (narrowed at runtime from pyproj's CF output via
zarr_metadata.typed_json.check), SpatialGroupAttrs,
MultiscaleGroupAttrs, and FixedLengthUtf32Object in the GeoZarr scope.
Function-scoped imports move to the top of their modules. No runtime
change: the stored snapshots are byte-identical.

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <[email protected]>

Adds data_api/s2_write.py: write_geozarr realizes the plan with
zarr-python and numpy alone. The planned metadata documents are written
first (pydantic-zarr to_zarr, so every stored document is exactly the
plan), then the chunk data is filled in: native arrays copied from the
input store, injected bands block-resampled from the finest input level
that holds them, generated pyramid levels chained from the level before
them with the converter's arithmetic (nodata-masked rounded block mean,
window-centre subsampling for classifications, window means for the
coordinates), and grid-mapping arrays set to zero. Product-level
publication (root attributes, consolidation, STAC) stays the converter
pipeline's separate final step.

The parity test runs write_geozarr and create_multiscale_from_datatree
on the same seeded input (each product's own proj:transform drives the
coordinates; small arrays and the scene classifications carry patterned
values) and compares the stores file by file: identical file sets,
identical metadata documents, and byte-identical chunks, for all three
fixture products.

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <[email protected]>

Converts the pyz GroupSpec/ArraySpec wrappers from Generic[...] bases
with module-level TypeVars to PEP 695 type parameters (ruff UP046), and
drops an unused import this branch had left in geozarr/v3.py.

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <[email protected]>

…and drop the scratch copy

The converter no longer materializes in-memory or Zarr V3-backed trees to
a scratch Zarr V2 store. New `pyz.datatree` is the DataTree analogue of
`GroupSpec.from_zarr`: `group_spec_from_datatree` extracts a tree's
metadata plane as the Zarr V2 documents xarray's own zarr writer would
store (every variable through `encode_zarr_variable`, CF packing applied,
storage-layout encoding keys dropped), so `Sentinel2Root` validates a
tree's named members and typed attributes exactly as it validates a store.
`load_sentinel2` accepts a DataTree, and `write_geozarr` reads the input
through a `SourceReader` protocol with a zarr-store and a DataTree
implementation, the latter reading values in the same encoded form.

A parity test pins the equivalence: on the seeded Sentinel-2 fixtures,
the tree source writes the same store as the zarr-store source, chunk for
chunk, whether the tree holds the raw packed values or the decoded ones
(which the reader re-encodes as xarray would have on a store write).

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <[email protected]>

Correctness and robustness:
- `_window_mean` keeps an all-NaN float window NaN instead of writing 0.0,
  matching the removed converter's skipna mean and `_coarsen`'s docstring.
- `_plan_subtree` carries a mixed group's own arrays alongside its planned
  subgroups instead of dropping them silently.
- `plan_reflectance` raises its deliberate ValueErrors (no band arrays;
  missing proj:transform/proj:epsg/proj:wkt2 with no product fallback)
  instead of bare StopIteration/KeyError.

Consistency and single sources of truth:
- Root consolidation consolidates every multiscale root the plan maps to
  None, dropping the converter's dependency on the duplicate
  `_AUX_MULTISCALES` table; the dead `REFLECTANCE_MULTISCALE` is deleted.
- The CPM writer forwards spatial_chunk and compression_level to the S2
  policy check, so an explicit deviation is refused loudly rather than
  silently ignored (enable_sharding stays unforwarded: sharding is fixed on
  and its CPM default of False is indistinguishable from an explicit False).
- The pydantic-zarr git source pins the validated commit (9f2e847) instead
  of floating on main.

Snapshots and docs:
- The plan snapshots are regenerated from the plan, replacing the snapshot
  harness's synthetic-tile georeferencing with each product's own; the
  masking machinery and `_EXACT_PRODUCT` special case are retired, so every
  product's snapshot comparison is now exact.
- The docs drop the removed convert-s2-optimized layout flags, scope
  --scale-offset-codec to the generic rechunker, document the fixed S2
  write policy, and replace the deleted create_multiscale_from_datatree
  reference with the plan/write API.

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <[email protected]>

…to feat/pydantic-zarr-scoped-validation

Resolves the conflicts between this branch's plan-driven S2 pipeline and
upstream's root-consolidation harmonization (EOPF-Explorer#328) and spatial-chunking fix
(EOPF-Explorer#331):

- convert_s2_optimized now feeds the planned dataset index into the
  harmonized utils.updated_root_consolidation, passing the plan's None-mapped
  multiscale roots as additional_consolidate_paths; the converter-local
  simple_root_consolidation, add_multiscale_pyramids_to_stac_metadata and
  write_store_root_bbox are gone, and add_s2_stac_assets (from EOPF-Explorer#328) is
  adapted to take the index and the planned reflectance shape.
- utils.add_multiscale_pyramids_to_stac_metadata accepts a dataset's variable
  names in place of the dataset, which is what the plan-driven converter has.
- convert_s2 and the DataTree-writer code EOPF-Explorer#328/EOPF-Explorer#331 touched stay deleted
  (no remaining callers); max_retries leaves convert_s2_optimized as upstream
  removed it from the call sites.
- tests follow: the consolidation tests run against updated_root_consolidation
  with explicit None roots, upstream's revived test_s2_multiscale_geo_metadata
  (now against conversion.utils) is kept, and the DataTree-writer tests stay
  deleted.
- uv.lock re-locked: upstream's bumps plus this branch's pinned pydantic-zarr
  commit; the regenerated plan snapshots are byte-identical to the merged
  files, so both pipelines had converged on the same output.

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <[email protected]>

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

- Require pydantic-zarr>=0.11.0, sourced from the v0.11.0 tag until the
  release reaches PyPI, instead of a commit on main.
- s2_plan converts a V2 dtype to its Zarr V3 form with
  `parse_dtype_v3(..., scope=GEOZARR_SCOPE)` instead of its own mapping.
  0.11 reads numpy's fixed-length unicode and bytes types as zarr-python
  writes them. The local mapping turned a datetime dtype into an
  unclaimed name such as "datetime64[ns]"; `parse_dtype_v3` gives the
  `numpy.datetime64` form.
- 0.11 accepts zarr-metadata's typed codec objects on `create_default`,
  so the builders that pass them type-check.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

A carried V2 `<M8[ns]` array is planned as `numpy.datetime64` with its
unit, the form zarr-python writes and the GeoZarr scope claims. The local
mapping that `parse_dtype_v3` replaced gave the unclaimed name
"datetime64[ns]".

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>