Skip to content

refactor!: Adding refactoring and harmonization of root consolidation (supersedes #314) - #328

Merged
claxn merged 68 commits into
mainfrom
claxn/feat/harmonize-root-consolidation
Oct 29, 2026
Merged

claxn merged 68 commits into
mainfrom
claxn/feat/harmonize-root-consolidation

Conversation

@claxn

@claxn claxn commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

This PR supersedes #314 and addresses the latest review comments. After all open questions are solved in #314, it will be closed and the next review round will be done in this PR.

d-v-b and others added 30 commits August 11, 2026 13:49
Adds eopf_geozarr.cpm, an integration package for eopf-cpm (the eopf
package, available via the new 'cpm' extra on Python >= 3.13):

- GeoZarrWriter, registered in CPM's EOWriterRegistry as engine
  "geozarr" (by-name only: cpm_zarr already claims .zarr targets in
  by-target discovery). Routes products to the Sentinel-2 optimized
  pipeline or the generic pipeline based on the declared CPM
  product:type, with a structural fallback. Zarr v3 only; rejects
  unsupported modes, formats, and options loudly.
- an 'eopf convert-geozarr' subcommand exposed through CPM's eopf.cli
  entry-point group, delegating to eopf.store.convert with the engine
  preset, enabling one-pass SAFE -> GeoZarr conversion.
- routing helpers importable without eopf installed.

convert_s2/convert_s2_optimized now skip backing-store validation for
DataTrees with no zarr backend (e.g. built in memory by the CPM SAFE
reader) instead of crashing in get_zarr_group.

Verified against eopf 3.0.0 on Python 3.13: engine registration,
convert-geozarr discovery in the eopf CLI, and an end-to-end
write_datatree(engine="geozarr") producing the flat r10m..r720m
pyramid from an S2 example store.

Assisted-by: ClaudeCode:claude-fable-5
Fixes the confirmed findings from a multi-agent adversarial review:

- write() now runs every argument-derivable check (target type, remote
  scheme, w- existence, pipeline routing, required 'groups') BEFORE the
  destructive mode="w" removal of an existing store. Previously a pure
  options mistake (e.g. missing groups) deleted the user's previous
  product and then failed with nothing written.
- accept compute=True (rejecting only compute=False): CPM's staged-output
  path (staging.py) injects compute=True when a remote Dask client is
  active, so the strict kwargs rejection broke the exact --stage-output
  flow the writer's own error message recommends.
- validate_write_options now applies the target checks (type, remote
  scheme), so CPM's triggering preflight rejects unsupported targets at
  parse time instead of at write time.
- routing: only S02MSIL1C/S02MSIL2A products go to the S2-optimized
  pipeline; other MSI types (S02MSIL0_, S02MSIRAW) lack the reflectance
  pyramid structure and now route generic.
- eopf_geozarr.cpm raises a clear ImportError explaining the Python
  >= 3.13 floor when eopf is missing (pip install 'eopf-geozarr[cpm]'
  silently installs nothing on 3.12).
- _validate_s2_input: cover both branches with tests; also skip
  validation for zarr-v3-backed trees (the S2 input model reads Zarr v2
  only and raised TypeError on CPM 3.x products).

Verified on Python 3.13 with eopf 3.0.0: 42 tests pass, including new
regression tests for the destroy-then-fail ordering and compute
handling.

Assisted-by: ClaudeCode:claude-fable-5
- test-cpm job: installs the cpm extra (eopf pinned via uv.lock) on
  Python 3.13 and runs the CPM writer/routing tests on every push/PR,
  so they no longer silently skip in CI.
- test-cpm-latest job: weekly scheduled run against the newest published
  eopf release to catch upstream contract drift (writer registry,
  staging, CLI entry-point loading) independently of PR failures.
- new test: invoke the real `eopf convert-geozarr` console script via
  subprocess on a synthetic S2 store, covering entry-point discovery,
  click option wiring, and convert()'s reader dispatch through to the
  geozarr engine.

Assisted-by: ClaudeCode:claude-fable-5
PR #212 (Sentinel-3 OLCI GeoZarr export) merged to main after #238 was
branched, leaving select_pipeline unable to reach it: OLCI products fell
into the generic pipeline and failed on the missing 'groups' option.

Adds S3_OLCI_PRODUCT_TYPE_PREFIXES and looks_like_sentinel3_olci to
routing.py, structural fallback included -- oa01_radiance is a data
variable of the measurements node, not a child node, unlike the S2
r{N}m pyramid groups. Wires convert_olci_optimized into the writer with
a new s3_olci_optimized flag and an output_grid option, kept additive
to the existing write() signature (s2_optimized is untouched) since
that's the EOWriter entry point other CPM code calls into.
* ci: run CI on the cpm-geozarr branch

The workflow only triggered on main, so no work on the cpm-geozarr
feature branch ever got checks -- including #247, which is 20 commits
and +29105/-3763 across 23 files. The Samples Service runs this branch
in its September GeoZarr production track, so changes reached a
production pipeline unchecked.

Closes #253

* style: wrap the resolved_spatial_chunk conditional to satisfy ruff format

Introduced by the OLCI routing commit, which renamed the local
`pipeline` variable to `selected_pipeline` and pushed the line past the
limit. It went unnoticed because the branch had no CI -- which is what
this PR fixes.
* init project setup

* adding and harmonising the chunking/sharding functionality across measuremetn and non-measurement groups - allows the contuniation of chunking behaviour from input eopf datatree instead if overwriting by force - scale_offset logic not touched

* addign nan masking for coarsening operation for multiscale calualtation (only for s2_optimising, later extend!) response to issue #246

* addign rechunking after readin to circumvent chunking at read-in (which is hardwired to eopf/store/mappers)

* adding L1C injected bands for mutliscale calculation -> with rechukning of added bands to spatial_chunk size

* readding prior launch script

* implementing changes recommended by claxn and d-v-b, mainly docs, cleanup, simplification

* adressing change requests, improving chunk workflow

* simplifing encoding retrieval for (all and) 1d arrays

* hotfix: adding stac metadata to root level was previously not inlcuded -> copying root metadata to new tree root now

* adding docstrings

* working on the tests, frustrated with test layout

* adding zarr v3 test samples

* change of the data_api tests

* removing test structure which is not adhering to s2 specs, also 'notebook' scripts referenced in tests didnt seem to exist

* changing the s2 zarr v3 structure and all associated tests. Also removing all emojis from output messages

* adapting to new zarr layout, changed function calls, and multiscale calculation changes

* removing emojis

* adding checks to add metadata to arrays if keep_scale_offset=False, modifying checks for s2 structure to align with tests, addig docs

* fixing enum to strenum to allow detection of S2type in simple == comparison

* applying changes to validate pre-commit pyright issues

* addressign requested chagnes for stAc metadata addition adn fill_value fallbacks, removing 'experimental_scale_offset_codec'functionality for encoding2

* small issue of not rechuking in tests

* removing addition of multiscales to stac asset metadata, other small commetns

* adding _FillValue to encoding to allow xarray decode_cf to correctly read zarr store in abscence of eopf-xarray reader, removed large commented section

---------

Co-authored-by: Emmanuel Mathot <[email protected]>
…ng pr247 (#267)

* adding single /measurement/refectance group to stac as requested during pr247

* adressing comments for crs reuse and proj_attrs_for_crs use

* adding test to incorporate new functionalities and expanding comments
Brings in S1 GRD RTC ingestion (#216), release 0.11.0 (#235) and dependency
bumps (#268, #269). Conflicts: tests/conftest.py keeps the cpm-geozarr v3 S2
fixtures and adds main's S1 RTC fixtures; uv.lock re-locked from main's lock.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Brings in the spatial:bbox pixel-edge fix (#275, fixes #266).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ition of generalised functionality to utils which are not used in other functions
…ozarr) but allows a path to which is needed for rechunking
… due to utils.create_uniform_encoding, Requesting input to check if tests or encoding needs to be adapted (in sight of new scale/offset codec)
…float32 per default if keep_scale_offset=False;
…ate_write_options', removing assert and introducing new pass-through error
…esult_datatree, rechunk_dataset_for_encoding, get_chunking_for_encoding), keeping functions from s2 processing (simple_root_consolidation, stream_write_dataset->stream_write_s2dataset)
…_store_root_geo_metadata() which uses shapely for bounds calculation from geojson geometry
@claxn claxn self-assigned this Oct 28, 2026
@claxn claxn added the enhancement New feature or request label Oct 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Security Audit Report

View workflow run

Bandit — Static Security Analysis (Security tab)

12 issue(s) found: 1 medium, 11 low

✅ No issues at or above HIGH severity.

1 medium, 11 low issue(s) below threshold not shown in table.

pip-audit — Dependency Vulnerabilities (Security tab)

Dependencies audited: 121, skipped: 0.

Package Version ID Fix Versions Description
msgpack 1.2.2 CVE-2026-114613 1.2.3 ### Impact The msgpack-python C extension (msgpack._cmsgpack) leaks a decoded map value when deserialization rejects

1 vulnerability/vulnerabilities found (1 fixable) across 1 package(s).


Result: ❌ Blocking issues found — see details above.

@claxn
claxn marked this pull request as ready for review October 28, 2026 10:32

@lhoupert lhoupert left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @claxn, routing S2 through the same updated_root_consolidation makes this much easier to follow!

A few points that came during my review

compression_level for coarsened levels

Claude found the issue below:

compression_level for coarsened levels now reaches the native levels, but neither generic_multiscales(...) call in create_multiscale_from_datatree passes it (s2_multiscale.py L710, L726). So every coarsened level (reflectance and both classification pyramids from r120m, quality/probability from r60m) stays at 3 when a caller asks for another level.

Could it be passed there too? test_convert_s2_optimized_convenience_function passes compression_level=2 but doesn't assert that it reaches the encoding.

Test for the generic /measurements consolidation?

The new None parents in generic_converter.py are what give /measurements of S3 SLSTR its own consolidated metadata, but no test checks it (test_generic_rechunker.py only asserts the root). A tree with only measurements/inadir and an assert on measurements/zarr.json would cover it.

Breaking change and docs update

convert_s2_optimized no longer accepts max_retries (still documented in docs/api-reference.md), and a few functions are removed. Could the docs be updated, and the title made refactor!: (or a BREAKING CHANGE: footer added) so it shows in the release notes?

Thank you!

@claxn claxn changed the title refactor: Adding refactoring and harmonization of root consolidation (supersedes #314) refactor!: Adding refactoring and harmonization of root consolidation (supersedes #314) Oct 28, 2026
Comment thread src/eopf_geozarr/conversion/utils.py Resolved
Comment thread src/eopf_geozarr/conversion/utils.py Resolved
Comment thread src/eopf_geozarr/conversion/utils.py Resolved
Comment thread src/eopf_geozarr/conversion/utils.py Resolved
@claxn

claxn commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator Author

@lhoupert @d-v-b thx for the quick review!

@lhoupert I addressed all your remarks and uploaded the latest S2 data to s3://continuous-integration/eopf-geozarr/branches/feat-harmonize-root-consolidation.

@d-v-b I addressed all your comments - please let me know if you are not done yet with the review and have some further comments

@claxn
claxn merged commit 4b25785 into main Oct 29, 2026
8 checks passed
@github-actions github-actions Bot mentioned this pull request Oct 29, 2026
d-v-b added a commit to d-v-b/data-model that referenced this pull request Oct 29, 2026
…to feat/pydantic-zarr-scoped-validation

Resolves the conflicts between this branch's plan-driven S2 pipeline and
upstream's root-consolidation harmonization (EOPF-Explorer#328) and spatial-chunking fix
(EOPF-Explorer#331):

- convert_s2_optimized now feeds the planned dataset index into the
  harmonized utils.updated_root_consolidation, passing the plan's None-mapped
  multiscale roots as additional_consolidate_paths; the converter-local
  simple_root_consolidation, add_multiscale_pyramids_to_stac_metadata and
  write_store_root_bbox are gone, and add_s2_stac_assets (from EOPF-Explorer#328) is
  adapted to take the index and the planned reflectance shape.
- utils.add_multiscale_pyramids_to_stac_metadata accepts a dataset's variable
  names in place of the dataset, which is what the plan-driven converter has.
- convert_s2 and the DataTree-writer code EOPF-Explorer#328/EOPF-Explorer#331 touched stay deleted
  (no remaining callers); max_retries leaves convert_s2_optimized as upstream
  removed it from the call sites.
- tests follow: the consolidation tests run against updated_root_consolidation
  with explicit None roots, upstream's revived test_s2_multiscale_geo_metadata
  (now against conversion.utils) is kept, and the DataTree-writer tests stay
  deleted.
- uv.lock re-locked: upstream's bumps plus this branch's pinned pydantic-zarr
  commit; the regenerated plan snapshots are byte-identical to the merged
  files, so both pipelines had converged on the same output.

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants