Skip to content

Consolidate the EOPF Zarr product schema through pydantic on Zarr V3 #258

Open

Description

@emmanuelmathot

Problem

Every input product model is built on the Zarr V2 pydantic model, while CPM 3.0.0 now emits Zarr V3. So validation silently stops working on current products, and the code works around it by hand.

data_api/s2.py        -> from eopf_geozarr.pyz.v2 import ArraySpec, GroupSpec
data_api/s1.py        -> from eopf_geozarr.pyz.v2 import ArraySpec, GroupSpec
data_api/s3_olci.py   -> from eopf_geozarr.pyz.v2 import ArraySpec, GroupSpec

pyz/v3.py already exists and provides GroupSpec / ArraySpec over pydantic_zarr.v3. Nothing in src/ imports it. The scaffolding is there; the models were never ported.

Symptoms accumulating in the meantime

  • Cpm geozarr eodc changes #247 adds _is_sentinel2_dataset_v3() — a hand-rolled structural check (set(group.keys()) == {"measurements","quality","conditions"}, then look for measurements/reflectance). Its own docstring says why: "Sentinel2Root ... is defined against the Zarr V2 pydantic model only; there is no V3 counterpart yet." Reasonable as a stopgap, but it is a schema expressed as imperative code.
  • CRS extraction from other_metadata (s2_converter.py:41-60) reads horizontal_CRS_code / horizontal_crs_code, coerces "EPSG:32632" / "32632" / 32632, and swallows failures in a broad except Exception. Two spellings and three representations of one field, parsed by hand — exactly what a model should own.
  • Earlier in the CPM writer work, is_sentinel2_dataset() and is_sentinel3_olci_dataset() both had to be defensively guarded because they validate against V2 models and fail on V3 stores.

Each workaround is individually defensible. Together they mean the product schema now lives in several places, none of them the model.

Proposal

Consolidate the schema of the EOPF Zarr product through pydantic, on pyz.v3:

  1. Port Sentinel2Root, Sentinel1Root and Sentinel3OlciRoot to pyz.v3, keeping the V2 models only as long as V2 inputs must still be read.
  2. Model the CPM root metadata we actually depend on — at minimum the other_metadata CRS field, with its spelling and type variants handled once, in the model.
  3. Replace the hand-rolled structural detectors (_is_sentinel2_dataset_v3, and the guards around the existing is_*_dataset helpers) with model validation, so "is this an S2 product?" has one answer.
  4. Decide what the models own versus what stays imperative, and write that down — the point is one source of truth for the product schema, not maximal coverage.

Notes

Activity

  1. shollendEODC commented on Sep 3, 2026

    @shollendEODC
    Collaborator

    this opens the general discussion of porting everything to the v3 pydantic model -> I think we need to consider if/how/when pydantic models are created for all the input data modes. Its likely a lot of work and as far as I could backtrace it they are only used for the tests. Maybe an end-to-end conversion test with selected files (fetched and converted) for each product can be used instead?

  2. d-v-b commented on Oct 27, 2026

    @d-v-b
    Contributor

    an update on this:

    • I agree that we need to update the pydantic models that describe the stored state of the input data. I'm happy to take this on
    • We also need to use the models of the output product for more than validation. they are designed to generate a spec-compliant zarr layout that can then be filled with chunks. this requires taking xarray out of the loop for creating arrays / groups and writing attributes, which IMO is the correct direction anyway.

    some of this depends on upstreaming local changes here to pydantic-zarr, and also bringing in the latest release of zarr-metadata that consolidates type checking logic for zarr metadata in one package. I'm pushing hard on this and I aim to have something ready for review this week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

data-apiPydantic-zarr models of the products (data_api)enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions