Repository navigation
refactor!: Adding refactoring and harmonization of root consolidation (supersedes #314) - #328
Conversation
Adds eopf_geozarr.cpm, an integration package for eopf-cpm (the eopf package, available via the new 'cpm' extra on Python >= 3.13): - GeoZarrWriter, registered in CPM's EOWriterRegistry as engine "geozarr" (by-name only: cpm_zarr already claims .zarr targets in by-target discovery). Routes products to the Sentinel-2 optimized pipeline or the generic pipeline based on the declared CPM product:type, with a structural fallback. Zarr v3 only; rejects unsupported modes, formats, and options loudly. - an 'eopf convert-geozarr' subcommand exposed through CPM's eopf.cli entry-point group, delegating to eopf.store.convert with the engine preset, enabling one-pass SAFE -> GeoZarr conversion. - routing helpers importable without eopf installed. convert_s2/convert_s2_optimized now skip backing-store validation for DataTrees with no zarr backend (e.g. built in memory by the CPM SAFE reader) instead of crashing in get_zarr_group. Verified against eopf 3.0.0 on Python 3.13: engine registration, convert-geozarr discovery in the eopf CLI, and an end-to-end write_datatree(engine="geozarr") producing the flat r10m..r720m pyramid from an S2 example store. Assisted-by: ClaudeCode:claude-fable-5
Fixes the confirmed findings from a multi-agent adversarial review: - write() now runs every argument-derivable check (target type, remote scheme, w- existence, pipeline routing, required 'groups') BEFORE the destructive mode="w" removal of an existing store. Previously a pure options mistake (e.g. missing groups) deleted the user's previous product and then failed with nothing written. - accept compute=True (rejecting only compute=False): CPM's staged-output path (staging.py) injects compute=True when a remote Dask client is active, so the strict kwargs rejection broke the exact --stage-output flow the writer's own error message recommends. - validate_write_options now applies the target checks (type, remote scheme), so CPM's triggering preflight rejects unsupported targets at parse time instead of at write time. - routing: only S02MSIL1C/S02MSIL2A products go to the S2-optimized pipeline; other MSI types (S02MSIL0_, S02MSIRAW) lack the reflectance pyramid structure and now route generic. - eopf_geozarr.cpm raises a clear ImportError explaining the Python >= 3.13 floor when eopf is missing (pip install 'eopf-geozarr[cpm]' silently installs nothing on 3.12). - _validate_s2_input: cover both branches with tests; also skip validation for zarr-v3-backed trees (the S2 input model reads Zarr v2 only and raised TypeError on CPM 3.x products). Verified on Python 3.13 with eopf 3.0.0: 42 tests pass, including new regression tests for the destroy-then-fail ordering and compute handling. Assisted-by: ClaudeCode:claude-fable-5
- test-cpm job: installs the cpm extra (eopf pinned via uv.lock) on Python 3.13 and runs the CPM writer/routing tests on every push/PR, so they no longer silently skip in CI. - test-cpm-latest job: weekly scheduled run against the newest published eopf release to catch upstream contract drift (writer registry, staging, CLI entry-point loading) independently of PR failures. - new test: invoke the real `eopf convert-geozarr` console script via subprocess on a synthetic S2 store, covering entry-point discovery, click option wiring, and convert()'s reader dispatch through to the geozarr engine. Assisted-by: ClaudeCode:claude-fable-5
PR #212 (Sentinel-3 OLCI GeoZarr export) merged to main after #238 was branched, leaving select_pipeline unable to reach it: OLCI products fell into the generic pipeline and failed on the missing 'groups' option. Adds S3_OLCI_PRODUCT_TYPE_PREFIXES and looks_like_sentinel3_olci to routing.py, structural fallback included -- oa01_radiance is a data variable of the measurements node, not a child node, unlike the S2 r{N}m pyramid groups. Wires convert_olci_optimized into the writer with a new s3_olci_optimized flag and an output_grid option, kept additive to the existing write() signature (s2_optimized is untouched) since that's the EOWriter entry point other CPM code calls into.
* ci: run CI on the cpm-geozarr branch The workflow only triggered on main, so no work on the cpm-geozarr feature branch ever got checks -- including #247, which is 20 commits and +29105/-3763 across 23 files. The Samples Service runs this branch in its September GeoZarr production track, so changes reached a production pipeline unchecked. Closes #253 * style: wrap the resolved_spatial_chunk conditional to satisfy ruff format Introduced by the OLCI routing commit, which renamed the local `pipeline` variable to `selected_pipeline` and pushed the line past the limit. It went unnoticed because the branch had no CI -- which is what this PR fixes.
* init project setup * adding and harmonising the chunking/sharding functionality across measuremetn and non-measurement groups - allows the contuniation of chunking behaviour from input eopf datatree instead if overwriting by force - scale_offset logic not touched * addign nan masking for coarsening operation for multiscale calualtation (only for s2_optimising, later extend!) response to issue #246 * addign rechunking after readin to circumvent chunking at read-in (which is hardwired to eopf/store/mappers) * adding L1C injected bands for mutliscale calculation -> with rechukning of added bands to spatial_chunk size * readding prior launch script * implementing changes recommended by claxn and d-v-b, mainly docs, cleanup, simplification * adressing change requests, improving chunk workflow * simplifing encoding retrieval for (all and) 1d arrays * hotfix: adding stac metadata to root level was previously not inlcuded -> copying root metadata to new tree root now * adding docstrings * working on the tests, frustrated with test layout * adding zarr v3 test samples * change of the data_api tests * removing test structure which is not adhering to s2 specs, also 'notebook' scripts referenced in tests didnt seem to exist * changing the s2 zarr v3 structure and all associated tests. Also removing all emojis from output messages * adapting to new zarr layout, changed function calls, and multiscale calculation changes * removing emojis * adding checks to add metadata to arrays if keep_scale_offset=False, modifying checks for s2 structure to align with tests, addig docs * fixing enum to strenum to allow detection of S2type in simple == comparison * applying changes to validate pre-commit pyright issues * addressign requested chagnes for stAc metadata addition adn fill_value fallbacks, removing 'experimental_scale_offset_codec'functionality for encoding2 * small issue of not rechuking in tests * removing addition of multiscales to stac asset metadata, other small commetns * adding _FillValue to encoding to allow xarray decode_cf to correctly read zarr store in abscence of eopf-xarray reader, removed large commented section --------- Co-authored-by: Emmanuel Mathot <[email protected]>
…ng pr247 (#267) * adding single /measurement/refectance group to stac as requested during pr247 * adressing comments for crs reuse and proj_attrs_for_crs use * adding test to incorporate new functionalities and expanding comments
Brings in S1 GRD RTC ingestion (#216), release 0.11.0 (#235) and dependency bumps (#268, #269). Conflicts: tests/conftest.py keeps the cpm-geozarr v3 S2 fixtures and adds main's S1 RTC fixtures; uv.lock re-locked from main's lock. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Brings in the spatial:bbox pixel-edge fix (#275, fixes #266). Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ition of generalised functionality to utils which are not used in other functions
…p traversal across metadata-ingestion mdoes
…ozarr) but allows a path to which is needed for rechunking
…-main-feat-generic-conversion-s1slc
…justed error messge from writer
… due to utils.create_uniform_encoding, Requesting input to check if tests or encoding needs to be adapted (in sight of new scale/offset codec)
…float32 per default if keep_scale_offset=False;
…s an atomic product due to SLC size
…ate_write_options', removing assert and introducing new pass-through error
…esult_datatree, rechunk_dataset_for_encoding, get_chunking_for_encoding), keeping functions from s2 processing (simple_root_consolidation, stream_write_dataset->stream_write_s2dataset)
…eeds to be discussed
…_store_root_geo_metadata() which uses shapely for bounds calculation from geojson geometry
Security Audit ReportBandit — Static Security Analysis (Security tab)12 issue(s) found: 1 medium, 11 low ✅ No issues at or above HIGH severity. 1 medium, 11 low issue(s) below threshold not shown in table. pip-audit — Dependency Vulnerabilities (Security tab)Dependencies audited: 121, skipped: 0.
1 vulnerability/vulnerabilities found (1 fixable) across 1 package(s). Result: ❌ Blocking issues found — see details above. |
lhoupert
left a comment
There was a problem hiding this comment.
Thanks @claxn, routing S2 through the same updated_root_consolidation makes this much easier to follow!
A few points that came during my review
compression_level for coarsened levels
Claude found the issue below:
compression_level for coarsened levels now reaches the native levels, but neither generic_multiscales(...) call in create_multiscale_from_datatree passes it (s2_multiscale.py L710, L726). So every coarsened level (reflectance and both classification pyramids from r120m, quality/probability from r60m) stays at 3 when a caller asks for another level.
Could it be passed there too? test_convert_s2_optimized_convenience_function passes compression_level=2 but doesn't assert that it reaches the encoding.
Test for the generic /measurements consolidation?
The new None parents in generic_converter.py are what give /measurements of S3 SLSTR its own consolidated metadata, but no test checks it (test_generic_rechunker.py only asserts the root). A tree with only measurements/inadir and an assert on measurements/zarr.json would cover it.
Breaking change and docs update
convert_s2_optimized no longer accepts max_retries (still documented in docs/api-reference.md), and a few functions are removed. Could the docs be updated, and the title made refactor!: (or a BREAKING CHANGE: footer added) so it shows in the release notes?
Thank you!
…ehaviour with deterministic return types
|
@lhoupert @d-v-b thx for the quick review! @lhoupert I addressed all your remarks and uploaded the latest S2 data to @d-v-b I addressed all your comments - please let me know if you are not done yet with the review and have some further comments |
…to feat/pydantic-zarr-scoped-validation Resolves the conflicts between this branch's plan-driven S2 pipeline and upstream's root-consolidation harmonization (EOPF-Explorer#328) and spatial-chunking fix (EOPF-Explorer#331): - convert_s2_optimized now feeds the planned dataset index into the harmonized utils.updated_root_consolidation, passing the plan's None-mapped multiscale roots as additional_consolidate_paths; the converter-local simple_root_consolidation, add_multiscale_pyramids_to_stac_metadata and write_store_root_bbox are gone, and add_s2_stac_assets (from EOPF-Explorer#328) is adapted to take the index and the planned reflectance shape. - utils.add_multiscale_pyramids_to_stac_metadata accepts a dataset's variable names in place of the dataset, which is what the plan-driven converter has. - convert_s2 and the DataTree-writer code EOPF-Explorer#328/EOPF-Explorer#331 touched stay deleted (no remaining callers); max_retries leaves convert_s2_optimized as upstream removed it from the call sites. - tests follow: the consolidation tests run against updated_root_consolidation with explicit None roots, upstream's revived test_s2_multiscale_geo_metadata (now against conversion.utils) is kept, and the DataTree-writer tests stay deleted. - uv.lock re-locked: upstream's bumps plus this branch's pinned pydantic-zarr commit; the regenerated plan snapshots are byte-identical to the merged files, so both pipelines had converged on the same output. Assisted-by: ClaudeCode:claude-fable-5 Co-Authored-By: Claude Fable 5 <[email protected]>
This PR supersedes #314 and addresses the latest review comments. After all open questions are solved in #314, it will be closed and the next review round will be done in this PR.