Repository navigation
Open
Labels
data-apiPydantic-zarr models of the products (data_api)Pydantic-zarr models of the products (data_api)enhancementNew feature or requestNew feature or request
Description
Problem
Every input product model is built on the Zarr V2 pydantic model, while CPM 3.0.0 now emits Zarr V3. So validation silently stops working on current products, and the code works around it by hand.
data_api/s2.py -> from eopf_geozarr.pyz.v2 import ArraySpec, GroupSpec
data_api/s1.py -> from eopf_geozarr.pyz.v2 import ArraySpec, GroupSpec
data_api/s3_olci.py -> from eopf_geozarr.pyz.v2 import ArraySpec, GroupSpec
pyz/v3.py already exists and provides GroupSpec / ArraySpec over pydantic_zarr.v3. Nothing in src/ imports it. The scaffolding is there; the models were never ported.
Symptoms accumulating in the meantime
- Cpm geozarr eodc changes #247 adds
_is_sentinel2_dataset_v3()— a hand-rolled structural check (set(group.keys()) == {"measurements","quality","conditions"}, then look formeasurements/reflectance). Its own docstring says why: "Sentinel2Root... is defined against the Zarr V2 pydantic model only; there is no V3 counterpart yet." Reasonable as a stopgap, but it is a schema expressed as imperative code. - CRS extraction from
other_metadata(s2_converter.py:41-60) readshorizontal_CRS_code/horizontal_crs_code, coerces"EPSG:32632"/"32632"/32632, and swallows failures in a broadexcept Exception. Two spellings and three representations of one field, parsed by hand — exactly what a model should own. - Earlier in the CPM writer work,
is_sentinel2_dataset()andis_sentinel3_olci_dataset()both had to be defensively guarded because they validate against V2 models and fail on V3 stores.
Each workaround is individually defensible. Together they mean the product schema now lives in several places, none of them the model.
Proposal
Consolidate the schema of the EOPF Zarr product through pydantic, on pyz.v3:
- Port
Sentinel2Root,Sentinel1RootandSentinel3OlciRoottopyz.v3, keeping the V2 models only as long as V2 inputs must still be read. - Model the CPM root metadata we actually depend on — at minimum the
other_metadataCRS field, with its spelling and type variants handled once, in the model. - Replace the hand-rolled structural detectors (
_is_sentinel2_dataset_v3, and the guards around the existingis_*_datasethelpers) with model validation, so "is this an S2 product?" has one answer. - Decide what the models own versus what stays imperative, and write that down — the point is one source of truth for the product schema, not maximal coverage.
Notes
- Related, but separate: Specify which CPM root metadata is carried through to GeoZarr output (stac_discovery is stale by construction) #257 covers which CPM root metadata we carry through to the output. This issue is about typing what we read.
- @d-v-b — flagging you as the author of the pydantic model layer; happy to discuss scope before anyone starts.
Metadata
Metadata
Assignees
Labels
data-apiPydantic-zarr models of the products (data_api)Pydantic-zarr models of the products (data_api)enhancementNew feature or requestNew feature or request
Activity
shollendEODC commentedon Sep 3, 2026
this opens the general discussion of porting everything to the v3 pydantic model -> I think we need to consider if/how/when pydantic models are created for all the input data modes. Its likely a lot of work and as far as I could backtrace it they are only used for the tests. Maybe an end-to-end conversion test with selected files (fetched and converted) for each product can be used instead?
d-v-b commentedon Oct 27, 2026
an update on this:
some of this depends on upstreaming local changes here to
pydantic-zarr, and also bringing in the latest release ofzarr-metadatathat consolidates type checking logic for zarr metadata in one package. I'm pushing hard on this and I aim to have something ready for review this week.