feat(s1): Sentinel-1 GRD RTC GeoZarr ingestion, data model and STAC builders by lhoupert · Pull Request #216 · EOPF-Explorer/data-model

lhoupert · 2026-07-14T11:08:20Z

Part of #139

Summary

Integration of the s1-tiling branch: Sentinel-1 GRD γ0 RTC support, developed and reviewed incrementally on this branch.

Data model — data_api/s1_rtc.py: Pydantic models for S1 RTC GeoZarr V3 stores, following the existing S2 pattern.
Ingestion — conversion/s1_ingest.py + new CLI commands: S1Tiling GeoTIFF → GeoZarr V3 pipeline, including multi-frame acquisition discovery (fix(s1-ingest): discover multi-frame acquisitions with a masked timestamp (#183) #184).
STAC — stac/s1_rtc.py: per-acquisition STAC item builders with assets compatible with TiTiler rendering, merged into this branch via feat(s1-rtc): STAC item builders + ingest fixes for TiTiler rendering #180 (which supersedes the earlier feat(stac): add build_s1_rtc_stac_item and generate-stac-s1 CLI #173, reverted in Revert "feat(stac): add build_s1_rtc_stac_item and generate-stac-s1 CLI" #179).
Supporting material: design/implementation plans (.github/prompts/), prototype and inspection scripts (analysis/), and a real-data STAC example fixture.

The branch is already synced with main (including #199: zarr-conventions 0.4.1 + pyright migration), so the diff is almost purely additive (~11k insertions, 9 deletions).

Breaking changes for data-pipeline — coordinate with the pin bump

discover_s1tiling_acquisitions() now defaults to skip_incomplete=True. Bundles missing a key the ingester actually reads (vv, vh, vv_mask) are dropped instead of being warned about and returned anyway — they used to reach ingest_s1tiling_acquisition and die on KeyError: 'vh', after the warning that should have prevented it. Pass skip_incomplete=False for the old list; every dict now carries complete: bool. Dicts also gain orbit_direction (the long ascending/descending form) next to the short-form orbit_dir, and passing the short form now raises before any GeoTIFF is opened rather than silently creating a <store>/ASC/ group that fails hours later at STAC time. Consumer: data-pipeline scripts/ingest_v1_s1_rtc.py.
renders moved from properties to the item root, where render extension v1.0.0 requires it — emitting it under properties made every item fail the extension it declared. A properties.renders mirror is written from the same call and is permanent, not a migration shim: the STAC API models its write body as Item-Input, which lists ten root fields (no renders) and sets no additionalProperties, so only the properties copy is expected to survive a POST/PUT. Both emission sites carry the reasoning. Consumers should read the root first and fall back to properties — data-pipeline's three readers (scripts/register_per_acquisition.py:132, which subscripted it and would KeyError; scripts/register_v1_s1_rtc.py:57 and scripts/register_v1.py:191, which used .get and would silently lose the render config) are updated in fix(s1-rtc): prepare the registration and validation paths for the data-model v0.11.0 pin data-pipeline#413.

For reviewers

The substantive code was already reviewed piecewise in PRs targeting this branch — #180 (STAC builders + ingest fixes, merged today), #184, and #173/#179 — so this PR is the final integration into main. The easiest path is to review by module (data model / ingestion / STAC) rather than the full diff; analysis/ and .github/prompts/ are documentation of the design phases, not runtime code.

Tests live in tests/test_s1_rtc_ingest.py, tests/test_s1_stac.py, tests/test_s1_stac_per_acquisition.py, and tests/test_data_api/test_s1_rtc.py. STAC output was validated live against TiTiler rendering during #180.

Author attestation

I am a human, these are my changes, and I have reviewed and understood
every change and can explain why each is correct.

Development was AI-assisted (Claude Code); all changes were human-reviewed through the incremental PRs into this branch (#173/#179, #184, #180) and verified against real S1Tiling output data.

…a GeoTIFF→GeoZarr V3 prototype - Implementation plan with Zarr V3 store structure, conventions, and phased roadmap - S1Tiling Docker instructions with EODAG 4.0.0 patch (5 breaking changes fixed) - GeoTIFF inspector script for S1Tiling output validation - Synthetic prototype validating sharding, resize/append, overview generation - Real-data conversion: 3 acquisitions (31TCH) → 1.8 GB Zarr store with 6 overview levels - Key findings: chunk divisibility constraint, datetime format, coverage variability

- Update §2 array metadata example to actual zarr-python 3.1.1 on-disk format (sharding codec in codecs array, chunk_grid holds shard shape, inner chunks 366) - Update conditions group: per-orbit naming (gamma_area_{orbit}), mark LIA and incidence_angle as aspirational (not confirmed as S1Tiling outputs) - Add Operational Findings subsection: EODAG 4.0.0 patch, DEM tile extent, GAMMA_AREA filename format - Fix docker instructions: CALIBRATION tag is GammaNaughtRTC (not gamma_naught_rtc) - Mark Phase 0 checklist fully complete - Update §9 API corrections to reflect plan is now corrected

- Add src/eopf_geozarr/data_api/s1_rtc.py — Zarr V3 Pydantic models for S1 GRD γ0T RTC GeoZarr stores, using pyz.v3 GroupSpec/ArraySpec with TypedDict members (same pattern as s2.py uses pyz.v2) - Models: S1RtcRoot, S1RtcOrbitGroup, S1RtcNativeResolutionDataset, S1RtcOverviewResolutionDataset, S1RtcConditionsGroup - Validation: convention UUIDs, spatial:dimensions, multiscales layout, required data arrays (vv/vh/border_mask), gamma_area presence - Add tests/_test_data/s1_rtc_examples/s1-grd-rtc-31TCH.json — realistic fixture with 3 timesteps, 6 overview levels, 3 gamma_area conditions - Add tests/test_data_api/test_s1_rtc.py — 11 tests: round-trip, structure validation, negative cases (missing orbit, r10m, UUIDs, etc.) - Add conftest fixture s1_rtc_json_example parametrized over all fixtures

…n S1 RTC module

…ssue #139

… test cases

…es in S1 GRD RTC prototype

…pipeline

…dinate arrays and enforce lowercase dimension names

Productionise the GeoTIFF → Zarr V3 ingestion pipeline for S1Tiling gamma-naught RTC outputs. New module: src/eopf_geozarr/conversion/s1_ingest.py (~500 lines) - extract_geotiff_metadata(): rasterio-based metadata extraction with tag validation and S1Tiling datetime normalisation - create_s1_store(): Zarr V3 store creation with full GeoZarr conventions (multiscales, proj:, spatial:), sharded arrays, 1D spatial coordinate arrays at every resolution level - ingest_s1tiling_acquisition(): main API — create-or-open store, read GeoTIFFs, generate overviews (average/nearest), write all levels, append coordinate variables. CRS/shape consistency validation on append. - consolidate_s1_store(): post-batch metadata consolidation - discover_s1tiling_acquisitions(): file discovery with grouping Tests: tests/test_s1_rtc_ingest.py (27 tests) - Metadata extraction, store creation, ingestion (create + append), data integrity, xarray roundtrip, CRS/shape mismatch rejection, consolidation, file discovery Uses zarr_cm CMO dicts for convention metadata (not hardcoded UUIDs). Reuses calculate_aligned_chunk_size() from utils.py. Exports wired in conversion/__init__.py. Refs: #139

- Replace dict[str, Any] multiscales field with zcm.Multiscales import - Remove inline MultiscalesTransform/ScaleLevel/Multiscales classes - Update spatial:dimensions validator to lowercase ['y', 'x'] (Python 3.12) - Remove typing_extensions import (native in Python 3.12) - Update test assertions for Pydantic model attribute access - Auto-format fixes from ruff (imports, fixture parens, line length)

- Add [tool.ruff.lint.flake8-type-checking] runtime-evaluated-base-classes for pydantic.BaseModel so Pydantic field type imports aren't flagged - Remove 4 stale noqa comments auto-fixed by ruff

…in CLI

@vincentsarago

* feat(stac): add build_s1_rtc_stac_item and generate-stac-s1 CLI command Adds eopf_geozarr.stac.s1_rtc.build_s1_rtc_stac_item, which reads a consolidated S1 GRD RTC Zarr V3 store and returns a pystac.Item with SAR/SAT/projection extensions, WGS84 bbox derived via pyproj, and vv/vh asset sub-paths. Ascending orbit is preferred when both are present. Also adds the generate-stac-s1 CLI subcommand that prints the item as JSON. 8 unit tests cover all acceptance criteria from the plan. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix: support s3:// URIs in S1Tiling discovery and ingest (#175) Four functions in s1_ingest.py silently failed for s3:// paths: - discover_s1tiling_acquisitions / discover_s1tiling_conditions used pathlib.Path.glob(), which normalises s3:// to s3:/ and returns 0 matches - ingest_s1tiling_acquisition / ingest_s1tiling_conditions coerced paths via Path(), corrupting s3:// URIs, and called .exists() which always returns False for S3 (existence checks now use path_exists() from fs_utils) Adds three private helpers: - _list_tifs(): uses s3fs.S3FileSystem.glob for s3:// prefixes - _coerce_input_path(): preserves str for s3://, returns Path otherwise - _rasterio_env(): rasterio.Env(AWSSession) context for s3:// paths; strips https:// from AWS_ENDPOINT_URL since GDAL expects hostname only Tested: rasterio.open("s3://...") confirmed working on OVH S3 with AWS_S3_ENDPOINT or AWS_ENDPOINT_URL set; hostname-only required by GDAL. Refs #139 Co-authored-by: Claude Sonnet 4.6 <[email protected]> * fix(stac): point vv/vh asset hrefs to orbit group root; add tile_matrix_set Per Emmanuel's review on #173: - vv/vh asset hrefs now point to {store}/{orbit} (the zarr group that carries multiscales), matching how S2 reflectance assets point to measurements/reflectance. TiTiler reads tile_matrix_set from that group's multiscales attributes. - create_s1_store now writes tile_matrix_set into the orbit group's multiscales so TiTiler can discover the tiling scheme without error. - Updated test_asset_hrefs to assert the new orbit-group href. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(ingest): add proj:code to resolution group attrs for TiTiler compatibility TiTiler's _validate_zarr checks ds.rio.crs on the native resolution group (r10m). Without proj:code in the resolution group's attrs, rioxarray returns None and the group is excluded → groups=[] → bounds unpack error. S2 zarr already has proj:code at the resolution group level; mirror that for S1. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(s1): write CF grid_mapping, drop tile_matrix_set from RTC store (#176) The deployed titiler-eopf reader resolves a group's CRS via rioxarray, which reads a CF spatial_ref/crs_wkt coordinate -- not the GeoZarr proj:code attr. S1 GRD RTC stores carried only proj:code, so rio.crs was None and every multiscale group was rejected (HTTP 500 on render/info). The S2 converter already writes a spatial_ref grid-mapping; this brings S1 in line. Also stop writing tile_matrix_set into the orbit-group multiscales: the data-model owner confirmed it is not part of the S1 GRD RTC data model, and the data_api reader already treats it as optional (MISSING default). - add _add_grid_mapping() using pyproj CRS.to_cf() (the same source rioxarray uses, so no hard-coded projection) and call it for every resolution level (both store-creation paths) and the conditions group - remove _create_tile_matrix_set() and its multiscales entry - tests: assert no tile_matrix_set, spatial_ref present + rio.crs resolves to the native EPSG, conditions group carries grid_mapping This fixes at the source what data-pipeline currently works around in ingest_v1_s1_rtc.py (_patch_cf_grid_mapping / _patch_tile_matrix_limits). Co-authored-by: Claude Opus 4.8 <[email protected]> * Merge #172 (deprecate v0, fix fill values, sanitize attrs) into s1-rtc (#178) * chore: release 0.9.0 * feat: implement scale-offset and data type casting via codecs * fix: fix dependency declaration * chore: use latest version of cast value * chore: make cast-value a project dependency * test: expand test coverage * fix: upgrade pytest to 9.0.3 (CVE-2025-71176) Fixes insecure /tmp directory handling on UNIX systems. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * - Use zarr-python's presumptive implementation of scale-offset - use scalar map for handling NaN - ensure that downsampled arrays use scalar map + cast value - improve tests across parametrization of relelvant functions * feat: add store-root spatial:bbox and tighten minispec requirements (#164) * feat: add store-root spatial:bbox and tighten minispec requirements Introduces a GeoZarr "Store Root" layer in the minispec so clients can read a summary footprint without walking into child groups, and tightens the multiscale profile so `spatial:bbox` at the root and `spatial:transform` + `spatial:shape` on every layout entry are mandatory. Adds a new `geozarr.store` pydantic module (`GeoZarrStoreAttrs`, `GeoZarrMultiscaleGroupAttrs`, `GeoZarr`) enforcing the tightened profile, and updates the S2 converter to union child-group bboxes into an EPSG:4326 footprint written at the store root. Closes #156. Addresses the clear parts of #163; the array-level and non-multiscale-group parts of that issue need further clarification. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat: update GeoZarr store model and optimize S2 converter for improved bbox handling * Addresses review feedback on #164: - Adds a "GeoZarr Hierarchy & Identification" subsection to the Store Root section of the minispec, codifying d-v-b's proposed rules: single root, root prefix ends with `.zarr`, suffix occurs at most once in the hierarchy, and explicit terminal-path conditions. - Promotes the store-root CRS to mandatory: at least one of `proj:code`, `proj:wkt2`, `proj:projjson` MUST be set; there is no implicit EPSG:4326 default. Per @vincentsarago's review. - Promotes `zarr_conventions` declaration at the store root from RECOMMENDED to required. - Converter now writes `proj:code: "EPSG:4326"` at the root alongside `spatial:bbox`. - Pydantic `GeoZarrStoreAttrs` enforces the new CRS-required rule. Cross-links the new hierarchy + spatial:extent follow-ups upstream: zarr-developers/geozarr-spec#132 (hierarchy & root identification, also addresses #124 URL parsing) and #133 (STAC-style spatial:extent). --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> * chore: uv.lock * refactor: use zarr-python 3.2.0 * chore: lockfile * fix: pin fill value based on data type instead of relying on xarray * test: update tests to consistently use nan fill value for floats * chore: bump urllib3 to 2.7.0 * chore: skip quicklook groups (#165) * chore: group dependabot updates for actions and pip (#160) * chore: group dependabot updates for actions and pip - Group all GitHub Actions bumps (minor, patch, major) into one PR - Group minor and patch pip bumps into one PR; major bumps remain ungrouped for individual review Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * chore: change dependabot schedule day to wednesday * ci: add comment_on option to security auditing action * ci: switch dependabot Python ecosystem from pip to uv uv ecosystem reads uv.lock directly, enabling Dependabot to raise PRs for lockfile-pinned versions including security patches. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> --------- Co-authored-by: Claude Sonnet 4.6 <[email protected]> * fix: derive coarse spatial transforms from coordinates (#168) * fix: derive coarse spatial transforms from coordinates * refactor: improve function definitions for clarity and consistency * chore: release 0.10.0 (#162) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix(converter): align convert layout to S2, strip _eopf_attrs, fix _FillValue (#171) - Remove TMS layout and /0 /1 /2 numeric overview groups from the general 'convert' CLI; produce S2-style sibling r{2**level} overviews instead. 'convert-s2-optimized' continues to emit the same layout. - Drop the '--tile-width' CLI option. - Sanitize array attrs in all converters: strip stale '_eopf_attrs' and source-only '_FillValue' / 'dtype' / 'valid_min' / 'valid_max' on decoded floats; rewrite 'units: digital_counts' -> 'units: 1'. The sanitizer now returns a new dict and callers reassign 'attrs', so stale keys are actually removed (a previous '.update()' pattern left them in place). - Sanitize coord attrs too (datetime coords in /conditions/meteorology were leaking '_eopf_decode_datetime64' via '_eopf_attrs'). - Set '_FillValue' properly for float measurements so xarray's CF encoder produces the base64-NaN representation needed for 'use_zarr_fill_value_as_mask=True' round-trip (xarray#11345). Fixes #171. - Remove dead 'utils.encode_cf_fill_value' (no callers). - Tests: open each r{N} multiscale level separately; migrate 'test_multiscales_round_trip' from deprecated tms.Multiscales to zcm.Multiscales (with tuplify_json to handle extra='allow' fields). - Regenerate snapshot fixtures; verified 0 '_eopf_attrs', 'tile_matrix*', or 'digital_counts' markers across S2A/S2B/S2C. * test: add guardrails for converter output attrs (#171) Walks the snapshotted GroupSpec JSON fixtures and asserts: - no '_eopf_attrs' anywhere; - no 'tile_matrix*' markers (TMS removal); - no 'units: digital_counts' on float arrays (decoded scale/offset); - every float array under '/measurements/' has '_FillValue' set (required for CF NaN-mask round-trip, xarray#11345). * docs: deprecate v0 references, document r{N} overview layout (#171) - Drop 'tile_width' parameter from API examples (option removed from CLI/API). - Replace '/measurements/r10m/{0,1,2}' nested-pyramid examples with the current flat 'r{2**level}' sibling-group layout. - Remove the V0 vs V1 split in converter.md / examples.md / quickstart.md; the general 'convert' command now produces the same flat layout as 'convert-s2-optimized'. The S2-optimized section is reframed as a feature description rather than a 'V1 vs V0' comparison. - Update architecture.md multiscales snippet to the new model ('layout' / 'asset' / 'derived_from' / 'transform'). - Update faq.md inspection snippet to walk the new layout. * refactor: clean up code formatting and enhance attribute handling in geozarr conversion * fix(cli-e2e): drop deprecated --tile-width flag from tests and example * refactor: remove deprecated v0 layout, sanitize attributes, and fix _FillValue handling * test: add _FillValue masking roundtrip test (#172) Addresses @vincentsarago's PR review: assert float arrays written with the converter's _FillValue convention round-trip through xarray's use_zarr_fill_value_as_mask=True so NaN cells come back masked. * test: drive _FillValue masking test through create_geozarr_dataset (#172) Replace the standalone xarray write with a real converter run: build a small float DataTree, invoke create_geozarr_dataset, then reopen the output with use_zarr_fill_value_as_mask=True and assert masking on the nodata patch. Exercises the converter's _FillValue + encoding path end-to-end, per @vincentsarago's review. * fix: green CI for #172 — S2 convert flat layout + idna CVE (#177) * fix(security): bump idna 3.13 -> 3.17 to clear CVE-2026-45409 pip-audit in the Security Audit workflow blocks on idna 3.13 (CVE-2026-45409, incomplete fix of CVE-2024-3651; fixed in 3.15). Bump the transitive pin via uv lock. pip-audit now reports no known vulnerabilities. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(cli): route Sentinel-2 convert through the optimized flat layout Generic `convert` emitted GeoZarr-0.4 multiscales with native data at the group root and overviews as nested r{2**N} sub-groups. That tree cannot be opened by xr.open_datatree (an overview child's x/y conflict with the parent's inherited x/y), so the `info` and `validate` CLI commands crashed on the converter's own Sentinel-2 output. Detect Sentinel-2 inputs and delegate to the existing, tested convert_s2_optimized path, which emits flat sibling r{N}m levels (the layout PR #172 documents). Non-S2 inputs keep the generic create_geozarr_dataset path. Detection is best-effort and never aborts conversion. Regenerate the geozarr_examples structure snapshots to the flat layout. Co-Authored-By: Claude Opus 4.8 <[email protected]> * test: make CLI e2e assertions debuggable and fix multiscale round-trip - The CLI logs errors to stdout via structlog; assert on stdout+stderr in test_cli_e2e so a non-zero exit shows the actual error instead of an empty stderr. - test_multiscale_attrs_round_trip tuplified only one side of its equality, so flat-layout level groups carrying list-valued attrs (e.g. spatial:bbox) broke a check that is only about JSON list/tuple normalisation. Tuplify both sides; the model still round-trips losslessly. Co-Authored-By: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Davis Vann Bennett <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: Emmanuel Mathot <[email protected]> --------- Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Davis Vann Bennett <[email protected]> Co-authored-by: Emmanuel Mathot <[email protected]>

…LI (#173)" (#179) This reverts commit f749368.

…tamp (#183) (#184) discover_s1tiling_acquisitions returned 0 acquisitions for multi-frame S1Tiling products: their GeoTIFF filenames mask the acquisition time as '…txxxxxx', which the filename regex (\d{8}t\d{6}) rejected, so parse_s1tiling_filename returned None and every file was skipped — silently, with no error. - Relax the acq_stamp pattern to accept a masked time (\d{6}|x{6}), so masked files are no longer dropped. - In discover, when the stamp is masked, resolve the real YYYYMMDDtHHMMSS stamp from the GeoTIFF ACQUISITION_DATETIME tag (via extract_geotiff_metadata) and use it for grouping + downstream STAC datetime. This is the durable upstream fix for the data-pipeline #237 workaround (which renamed masked products before discovery); that workaround can now be retired. Closes #183. Co-authored-by: Claude Opus 4.8 <[email protected]>

@vincentsarago

* chore: release 0.9.0 * feat: implement scale-offset and data type casting via codecs * fix: fix dependency declaration * chore: use latest version of cast value * chore: make cast-value a project dependency * test: expand test coverage * fix: upgrade pytest to 9.0.3 (CVE-2025-71176) Fixes insecure /tmp directory handling on UNIX systems. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * - Use zarr-python's presumptive implementation of scale-offset - use scalar map for handling NaN - ensure that downsampled arrays use scalar map + cast value - improve tests across parametrization of relelvant functions * feat: add store-root spatial:bbox and tighten minispec requirements (#164) * feat: add store-root spatial:bbox and tighten minispec requirements Introduces a GeoZarr "Store Root" layer in the minispec so clients can read a summary footprint without walking into child groups, and tightens the multiscale profile so `spatial:bbox` at the root and `spatial:transform` + `spatial:shape` on every layout entry are mandatory. Adds a new `geozarr.store` pydantic module (`GeoZarrStoreAttrs`, `GeoZarrMultiscaleGroupAttrs`, `GeoZarr`) enforcing the tightened profile, and updates the S2 converter to union child-group bboxes into an EPSG:4326 footprint written at the store root. Closes #156. Addresses the clear parts of #163; the array-level and non-multiscale-group parts of that issue need further clarification. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat: update GeoZarr store model and optimize S2 converter for improved bbox handling * Addresses review feedback on #164: - Adds a "GeoZarr Hierarchy & Identification" subsection to the Store Root section of the minispec, codifying d-v-b's proposed rules: single root, root prefix ends with `.zarr`, suffix occurs at most once in the hierarchy, and explicit terminal-path conditions. - Promotes the store-root CRS to mandatory: at least one of `proj:code`, `proj:wkt2`, `proj:projjson` MUST be set; there is no implicit EPSG:4326 default. Per @vincentsarago's review. - Promotes `zarr_conventions` declaration at the store root from RECOMMENDED to required. - Converter now writes `proj:code: "EPSG:4326"` at the root alongside `spatial:bbox`. - Pydantic `GeoZarrStoreAttrs` enforces the new CRS-required rule. Cross-links the new hierarchy + spatial:extent follow-ups upstream: zarr-developers/geozarr-spec#132 (hierarchy & root identification, also addresses #124 URL parsing) and #133 (STAC-style spatial:extent). --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> * chore: uv.lock * refactor: use zarr-python 3.2.0 * chore: lockfile * fix: pin fill value based on data type instead of relying on xarray * test: update tests to consistently use nan fill value for floats * chore: bump urllib3 to 2.7.0 * chore: skip quicklook groups (#165) * chore: group dependabot updates for actions and pip (#160) * chore: group dependabot updates for actions and pip - Group all GitHub Actions bumps (minor, patch, major) into one PR - Group minor and patch pip bumps into one PR; major bumps remain ungrouped for individual review Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * chore: change dependabot schedule day to wednesday * ci: add comment_on option to security auditing action * ci: switch dependabot Python ecosystem from pip to uv uv ecosystem reads uv.lock directly, enabling Dependabot to raise PRs for lockfile-pinned versions including security patches. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> --------- Co-authored-by: Claude Sonnet 4.6 <[email protected]> * fix: derive coarse spatial transforms from coordinates (#168) * fix: derive coarse spatial transforms from coordinates * refactor: improve function definitions for clarity and consistency * chore: release 0.10.0 (#162) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * enh: deprecate v0, fix fill values and sanitize attributes (#172) * fix(converter): align convert layout to S2, strip _eopf_attrs, fix _FillValue (#171) - Remove TMS layout and /0 /1 /2 numeric overview groups from the general 'convert' CLI; produce S2-style sibling r{2**level} overviews instead. 'convert-s2-optimized' continues to emit the same layout. - Drop the '--tile-width' CLI option. - Sanitize array attrs in all converters: strip stale '_eopf_attrs' and source-only '_FillValue' / 'dtype' / 'valid_min' / 'valid_max' on decoded floats; rewrite 'units: digital_counts' -> 'units: 1'. The sanitizer now returns a new dict and callers reassign 'attrs', so stale keys are actually removed (a previous '.update()' pattern left them in place). - Sanitize coord attrs too (datetime coords in /conditions/meteorology were leaking '_eopf_decode_datetime64' via '_eopf_attrs'). - Set '_FillValue' properly for float measurements so xarray's CF encoder produces the base64-NaN representation needed for 'use_zarr_fill_value_as_mask=True' round-trip (xarray#11345). Fixes #171. - Remove dead 'utils.encode_cf_fill_value' (no callers). - Tests: open each r{N} multiscale level separately; migrate 'test_multiscales_round_trip' from deprecated tms.Multiscales to zcm.Multiscales (with tuplify_json to handle extra='allow' fields). - Regenerate snapshot fixtures; verified 0 '_eopf_attrs', 'tile_matrix*', or 'digital_counts' markers across S2A/S2B/S2C. * test: add guardrails for converter output attrs (#171) Walks the snapshotted GroupSpec JSON fixtures and asserts: - no '_eopf_attrs' anywhere; - no 'tile_matrix*' markers (TMS removal); - no 'units: digital_counts' on float arrays (decoded scale/offset); - every float array under '/measurements/' has '_FillValue' set (required for CF NaN-mask round-trip, xarray#11345). * docs: deprecate v0 references, document r{N} overview layout (#171) - Drop 'tile_width' parameter from API examples (option removed from CLI/API). - Replace '/measurements/r10m/{0,1,2}' nested-pyramid examples with the current flat 'r{2**level}' sibling-group layout. - Remove the V0 vs V1 split in converter.md / examples.md / quickstart.md; the general 'convert' command now produces the same flat layout as 'convert-s2-optimized'. The S2-optimized section is reframed as a feature description rather than a 'V1 vs V0' comparison. - Update architecture.md multiscales snippet to the new model ('layout' / 'asset' / 'derived_from' / 'transform'). - Update faq.md inspection snippet to walk the new layout. * refactor: clean up code formatting and enhance attribute handling in geozarr conversion * fix(cli-e2e): drop deprecated --tile-width flag from tests and example * refactor: remove deprecated v0 layout, sanitize attributes, and fix _FillValue handling * test: add _FillValue masking roundtrip test (#172) Addresses @vincentsarago's PR review: assert float arrays written with the converter's _FillValue convention round-trip through xarray's use_zarr_fill_value_as_mask=True so NaN cells come back masked. * test: drive _FillValue masking test through create_geozarr_dataset (#172) Replace the standalone xarray write with a real converter run: build a small float DataTree, invoke create_geozarr_dataset, then reopen the output with use_zarr_fill_value_as_mask=True and assert masking on the nodata patch. Exercises the converter's _FillValue + encoding path end-to-end, per @vincentsarago's review. * fix: green CI for #172 — S2 convert flat layout + idna CVE (#177) * fix(security): bump idna 3.13 -> 3.17 to clear CVE-2026-45409 pip-audit in the Security Audit workflow blocks on idna 3.13 (CVE-2026-45409, incomplete fix of CVE-2024-3651; fixed in 3.15). Bump the transitive pin via uv lock. pip-audit now reports no known vulnerabilities. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(cli): route Sentinel-2 convert through the optimized flat layout Generic `convert` emitted GeoZarr-0.4 multiscales with native data at the group root and overviews as nested r{2**N} sub-groups. That tree cannot be opened by xr.open_datatree (an overview child's x/y conflict with the parent's inherited x/y), so the `info` and `validate` CLI commands crashed on the converter's own Sentinel-2 output. Detect Sentinel-2 inputs and delegate to the existing, tested convert_s2_optimized path, which emits flat sibling r{N}m levels (the layout PR #172 documents). Non-S2 inputs keep the generic create_geozarr_dataset path. Detection is best-effort and never aborts conversion. Regenerate the geozarr_examples structure snapshots to the flat layout. Co-Authored-By: Claude Opus 4.8 <[email protected]> * test: make CLI e2e assertions debuggable and fix multiscale round-trip - The CLI logs errors to stdout via structlog; assert on stdout+stderr in test_cli_e2e so a non-zero exit shows the actual error instead of an empty stderr. - test_multiscale_attrs_round_trip tuplified only one side of its equality, so flat-layout level groups carrying list-valued attrs (e.g. spatial:bbox) broke a check that is only about JSON list/tuple normalisation. Tuplify both sides; the model still round-trips losslessly. Co-Authored-By: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: Loïc Houpert <[email protected]> Co-authored-by: Claude Opus 4.8 <[email protected]> * ci(deps): bump the actions group across 1 directory with 8 updates (#167) Bumps the actions group with 8 updates in the / directory: | Package | From | To | | --- | --- | --- | | [actions/checkout](https://github.com/actions/checkout) | `5.0.1` | `6.0.2` | | [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) | `7.6.0` | `8.1.0` | | [actions/cache](https://github.com/actions/cache) | `5.0.4` | `5.0.5` | | [codecov/codecov-action](https://github.com/codecov/codecov-action) | `5.5.3` | `6.0.0` | | [googleapis/release-please-action](https://github.com/googleapis/release-please-action) | `4.4.0` | `5.0.0` | | [actions/upload-artifact](https://github.com/actions/upload-artifact) | `6.0.0` | `7.0.1` | | [actions/download-artifact](https://github.com/actions/download-artifact) | `7.0.0` | `8.0.1` | | [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) | `1.13.0` | `1.14.0` | Updates `actions/checkout` from 5.0.1 to 6.0.2 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@93cb6ef...de0fac2) Updates `astral-sh/setup-uv` from 7.6.0 to 8.1.0 - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](astral-sh/setup-uv@37802ad...0880764) Updates `actions/cache` from 5.0.4 to 5.0.5 - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@6682284...27d5ce7) Updates `codecov/codecov-action` from 5.5.3 to 6.0.0 - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@1af5884...57e3a13) Updates `googleapis/release-please-action` from 4.4.0 to 5.0.0 - [Release notes](https://github.com/googleapis/release-please-action/releases) - [Changelog](https://github.com/googleapis/release-please-action/blob/main/CHANGELOG.md) - [Commits](googleapis/release-please-action@16a9c90...45996ed) Updates `actions/upload-artifact` from 6.0.0 to 7.0.1 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@b7c566a...043fb46) Updates `actions/download-artifact` from 7.0.0 to 8.0.1 - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](actions/download-artifact@37930b1...3e5f45b) Updates `pypa/gh-action-pypi-publish` from 1.13.0 to 1.14.0 - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](pypa/gh-action-pypi-publish@ed0c539...cef2210) --- updated-dependencies: - dependency-name: actions/cache dependency-version: 5.0.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: actions/checkout dependency-version: 6.0.2 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/download-artifact dependency-version: 8.0.1 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/upload-artifact dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: astral-sh/setup-uv dependency-version: 8.1.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: codecov/codecov-action dependency-version: 6.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: googleapis/release-please-action dependency-version: 5.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: pypa/gh-action-pypi-publish dependency-version: 1.14.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * deps(deps-dev): bump mypy from 1.20.2 to 2.1.0 (#170) Bumps [mypy](https://github.com/python/mypy) from 1.20.2 to 2.1.0. - [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md) - [Commits](python/mypy@v1.20.2...v2.1.0) --- updated-dependencies: - dependency-name: mypy dependency-version: 2.1.0 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * deps(deps): bump the uv-minor-patch group across 1 directory with 4 updates (#174) Bumps the uv-minor-patch group with 4 updates in the / directory: [pydantic](https://github.com/pydantic/pydantic), [numpy](https://github.com/numpy/numpy), [cf-xarray](https://github.com/xarray-contrib/cf-xarray) and [pymdown-extensions](https://github.com/facelessuser/pymdown-extensions). Updates `pydantic` from 2.13.3 to 2.13.4 - [Release notes](https://github.com/pydantic/pydantic/releases) - [Changelog](https://github.com/pydantic/pydantic/blob/main/HISTORY.md) - [Commits](pydantic/pydantic@v2.13.3...v2.13.4) Updates `numpy` from 2.4.4 to 2.4.6 - [Release notes](https://github.com/numpy/numpy/releases) - [Changelog](https://github.com/numpy/numpy/blob/main/doc/RELEASE_WALKTHROUGH.rst) - [Commits](numpy/numpy@v2.4.4...v2.4.6) Updates `cf-xarray` from 0.11.0 to 0.11.1 - [Release notes](https://github.com/xarray-contrib/cf-xarray/releases) - [Commits](xarray-contrib/cf-xarray@v0.11.0...v0.11.1) Updates `pymdown-extensions` from 10.21.2 to 10.21.3 - [Release notes](https://github.com/facelessuser/pymdown-extensions/releases) - [Commits](facelessuser/pymdown-extensions@10.21.2...10.21.3) --- updated-dependencies: - dependency-name: pydantic dependency-version: 2.13.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: uv-minor-patch - dependency-name: numpy dependency-version: 2.4.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: uv-minor-patch - dependency-name: cf-xarray dependency-version: 0.11.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: uv-minor-patch - dependency-name: pymdown-extensions dependency-version: 10.21.3 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: uv-minor-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(deps): bump aiohttp to >=3.14.0 to resolve CVE-2026-34993 and CVE-2026-47265 (#182) The Security Audit job's pip-audit step flagged aiohttp 3.13.5 for CVE-2026-34993 and CVE-2026-47265 (both fixed in 3.14.0). Raise the floor and refresh the lock to 3.14.1. Co-authored-by: Claude Opus 4.8 <[email protected]> * chore: release 0.10.1 (#181) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * ci(deps): bump the actions group with 3 updates (#185) Bumps the actions group with 3 updates: [actions/checkout](https://github.com/actions/checkout), [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) and [codecov/codecov-action](https://github.com/codecov/codecov-action). Updates `actions/checkout` from 6.0.2 to 6.0.3 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@de0fac2...df4cb1c) Updates `astral-sh/setup-uv` from 8.1.0 to 8.2.0 - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](astral-sh/setup-uv@0880764...fac544c) Updates `codecov/codecov-action` from 6.0.0 to 6.0.1 - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@57e3a13...e79a696) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 6.0.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: astral-sh/setup-uv dependency-version: 8.2.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: codecov/codecov-action dependency-version: 6.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Davis Vann Bennett <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: Emmanuel Mathot <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…right) Brings main's #199 changes into s1-tiling without rewriting history (no force-push). The merge is resolved to exactly match the verified s1-tiling-rebased tree: main's #199 content + the S1 commits, with the pyproject mypy->pyright + ruff-TC conflict resolutions already tested there. Co-Authored-By: Claude Fable 5 <[email protected]>

@vincentsarago

…#180) * feat(stac): add build_s1_rtc_stac_item and generate-stac-s1 CLI command Adds eopf_geozarr.stac.s1_rtc.build_s1_rtc_stac_item, which reads a consolidated S1 GRD RTC Zarr V3 store and returns a pystac.Item with SAR/SAT/projection extensions, WGS84 bbox derived via pyproj, and vv/vh asset sub-paths. Ascending orbit is preferred when both are present. Also adds the generate-stac-s1 CLI subcommand that prints the item as JSON. 8 unit tests cover all acceptance criteria from the plan. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix: support s3:// URIs in S1Tiling discovery and ingest (#175) Four functions in s1_ingest.py silently failed for s3:// paths: - discover_s1tiling_acquisitions / discover_s1tiling_conditions used pathlib.Path.glob(), which normalises s3:// to s3:/ and returns 0 matches - ingest_s1tiling_acquisition / ingest_s1tiling_conditions coerced paths via Path(), corrupting s3:// URIs, and called .exists() which always returns False for S3 (existence checks now use path_exists() from fs_utils) Adds three private helpers: - _list_tifs(): uses s3fs.S3FileSystem.glob for s3:// prefixes - _coerce_input_path(): preserves str for s3://, returns Path otherwise - _rasterio_env(): rasterio.Env(AWSSession) context for s3:// paths; strips https:// from AWS_ENDPOINT_URL since GDAL expects hostname only Tested: rasterio.open("s3://...") confirmed working on OVH S3 with AWS_S3_ENDPOINT or AWS_ENDPOINT_URL set; hostname-only required by GDAL. Refs #139 Co-authored-by: Claude Sonnet 4.6 <[email protected]> * fix(stac): point vv/vh asset hrefs to orbit group root; add tile_matrix_set Per Emmanuel's review on #173: - vv/vh asset hrefs now point to {store}/{orbit} (the zarr group that carries multiscales), matching how S2 reflectance assets point to measurements/reflectance. TiTiler reads tile_matrix_set from that group's multiscales attributes. - create_s1_store now writes tile_matrix_set into the orbit group's multiscales so TiTiler can discover the tiling scheme without error. - Updated test_asset_hrefs to assert the new orbit-group href. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(ingest): add proj:code to resolution group attrs for TiTiler compatibility TiTiler's _validate_zarr checks ds.rio.crs on the native resolution group (r10m). Without proj:code in the resolution group's attrs, rioxarray returns None and the group is excluded → groups=[] → bounds unpack error. S2 zarr already has proj:code at the resolution group level; mirror that for S1. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(s1): write CF grid_mapping, drop tile_matrix_set from RTC store (#176) The deployed titiler-eopf reader resolves a group's CRS via rioxarray, which reads a CF spatial_ref/crs_wkt coordinate -- not the GeoZarr proj:code attr. S1 GRD RTC stores carried only proj:code, so rio.crs was None and every multiscale group was rejected (HTTP 500 on render/info). The S2 converter already writes a spatial_ref grid-mapping; this brings S1 in line. Also stop writing tile_matrix_set into the orbit-group multiscales: the data-model owner confirmed it is not part of the S1 GRD RTC data model, and the data_api reader already treats it as optional (MISSING default). - add _add_grid_mapping() using pyproj CRS.to_cf() (the same source rioxarray uses, so no hard-coded projection) and call it for every resolution level (both store-creation paths) and the conditions group - remove _create_tile_matrix_set() and its multiscales entry - tests: assert no tile_matrix_set, spatial_ref present + rio.crs resolves to the native EPSG, conditions group carries grid_mapping This fixes at the source what data-pipeline currently works around in ingest_v1_s1_rtc.py (_patch_cf_grid_mapping / _patch_tile_matrix_limits). Co-authored-by: Claude Opus 4.8 <[email protected]> * Merge #172 (deprecate v0, fix fill values, sanitize attrs) into s1-rtc (#178) * chore: release 0.9.0 * feat: implement scale-offset and data type casting via codecs * fix: fix dependency declaration * chore: use latest version of cast value * chore: make cast-value a project dependency * test: expand test coverage * fix: upgrade pytest to 9.0.3 (CVE-2025-71176) Fixes insecure /tmp directory handling on UNIX systems. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * - Use zarr-python's presumptive implementation of scale-offset - use scalar map for handling NaN - ensure that downsampled arrays use scalar map + cast value - improve tests across parametrization of relelvant functions * feat: add store-root spatial:bbox and tighten minispec requirements (#164) * feat: add store-root spatial:bbox and tighten minispec requirements Introduces a GeoZarr "Store Root" layer in the minispec so clients can read a summary footprint without walking into child groups, and tightens the multiscale profile so `spatial:bbox` at the root and `spatial:transform` + `spatial:shape` on every layout entry are mandatory. Adds a new `geozarr.store` pydantic module (`GeoZarrStoreAttrs`, `GeoZarrMultiscaleGroupAttrs`, `GeoZarr`) enforcing the tightened profile, and updates the S2 converter to union child-group bboxes into an EPSG:4326 footprint written at the store root. Closes #156. Addresses the clear parts of #163; the array-level and non-multiscale-group parts of that issue need further clarification. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat: update GeoZarr store model and optimize S2 converter for improved bbox handling * Addresses review feedback on #164: - Adds a "GeoZarr Hierarchy & Identification" subsection to the Store Root section of the minispec, codifying d-v-b's proposed rules: single root, root prefix ends with `.zarr`, suffix occurs at most once in the hierarchy, and explicit terminal-path conditions. - Promotes the store-root CRS to mandatory: at least one of `proj:code`, `proj:wkt2`, `proj:projjson` MUST be set; there is no implicit EPSG:4326 default. Per @vincentsarago's review. - Promotes `zarr_conventions` declaration at the store root from RECOMMENDED to required. - Converter now writes `proj:code: "EPSG:4326"` at the root alongside `spatial:bbox`. - Pydantic `GeoZarrStoreAttrs` enforces the new CRS-required rule. Cross-links the new hierarchy + spatial:extent follow-ups upstream: zarr-developers/geozarr-spec#132 (hierarchy & root identification, also addresses #124 URL parsing) and #133 (STAC-style spatial:extent). --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> * chore: uv.lock * refactor: use zarr-python 3.2.0 * chore: lockfile * fix: pin fill value based on data type instead of relying on xarray * test: update tests to consistently use nan fill value for floats * chore: bump urllib3 to 2.7.0 * chore: skip quicklook groups (#165) * chore: group dependabot updates for actions and pip (#160) * chore: group dependabot updates for actions and pip - Group all GitHub Actions bumps (minor, patch, major) into one PR - Group minor and patch pip bumps into one PR; major bumps remain ungrouped for individual review Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * chore: change dependabot schedule day to wednesday * ci: add comment_on option to security auditing action * ci: switch dependabot Python ecosystem from pip to uv uv ecosystem reads uv.lock directly, enabling Dependabot to raise PRs for lockfile-pinned versions including security patches. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> --------- Co-authored-by: Claude Sonnet 4.6 <[email protected]> * fix: derive coarse spatial transforms from coordinates (#168) * fix: derive coarse spatial transforms from coordinates * refactor: improve function definitions for clarity and consistency * chore: release 0.10.0 (#162) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix(converter): align convert layout to S2, strip _eopf_attrs, fix _FillValue (#171) - Remove TMS layout and /0 /1 /2 numeric overview groups from the general 'convert' CLI; produce S2-style sibling r{2**level} overviews instead. 'convert-s2-optimized' continues to emit the same layout. - Drop the '--tile-width' CLI option. - Sanitize array attrs in all converters: strip stale '_eopf_attrs' and source-only '_FillValue' / 'dtype' / 'valid_min' / 'valid_max' on decoded floats; rewrite 'units: digital_counts' -> 'units: 1'. The sanitizer now returns a new dict and callers reassign 'attrs', so stale keys are actually removed (a previous '.update()' pattern left them in place). - Sanitize coord attrs too (datetime coords in /conditions/meteorology were leaking '_eopf_decode_datetime64' via '_eopf_attrs'). - Set '_FillValue' properly for float measurements so xarray's CF encoder produces the base64-NaN representation needed for 'use_zarr_fill_value_as_mask=True' round-trip (xarray#11345). Fixes #171. - Remove dead 'utils.encode_cf_fill_value' (no callers). - Tests: open each r{N} multiscale level separately; migrate 'test_multiscales_round_trip' from deprecated tms.Multiscales to zcm.Multiscales (with tuplify_json to handle extra='allow' fields). - Regenerate snapshot fixtures; verified 0 '_eopf_attrs', 'tile_matrix*', or 'digital_counts' markers across S2A/S2B/S2C. * test: add guardrails for converter output attrs (#171) Walks the snapshotted GroupSpec JSON fixtures and asserts: - no '_eopf_attrs' anywhere; - no 'tile_matrix*' markers (TMS removal); - no 'units: digital_counts' on float arrays (decoded scale/offset); - every float array under '/measurements/' has '_FillValue' set (required for CF NaN-mask round-trip, xarray#11345). * docs: deprecate v0 references, document r{N} overview layout (#171) - Drop 'tile_width' parameter from API examples (option removed from CLI/API). - Replace '/measurements/r10m/{0,1,2}' nested-pyramid examples with the current flat 'r{2**level}' sibling-group layout. - Remove the V0 vs V1 split in converter.md / examples.md / quickstart.md; the general 'convert' command now produces the same flat layout as 'convert-s2-optimized'. The S2-optimized section is reframed as a feature description rather than a 'V1 vs V0' comparison. - Update architecture.md multiscales snippet to the new model ('layout' / 'asset' / 'derived_from' / 'transform'). - Update faq.md inspection snippet to walk the new layout. * refactor: clean up code formatting and enhance attribute handling in geozarr conversion * fix(cli-e2e): drop deprecated --tile-width flag from tests and example * refactor: remove deprecated v0 layout, sanitize attributes, and fix _FillValue handling * test: add _FillValue masking roundtrip test (#172) Addresses @vincentsarago's PR review: assert float arrays written with the converter's _FillValue convention round-trip through xarray's use_zarr_fill_value_as_mask=True so NaN cells come back masked. * test: drive _FillValue masking test through create_geozarr_dataset (#172) Replace the standalone xarray write with a real converter run: build a small float DataTree, invoke create_geozarr_dataset, then reopen the output with use_zarr_fill_value_as_mask=True and assert masking on the nodata patch. Exercises the converter's _FillValue + encoding path end-to-end, per @vincentsarago's review. * fix: green CI for #172 — S2 convert flat layout + idna CVE (#177) * fix(security): bump idna 3.13 -> 3.17 to clear CVE-2026-45409 pip-audit in the Security Audit workflow blocks on idna 3.13 (CVE-2026-45409, incomplete fix of CVE-2024-3651; fixed in 3.15). Bump the transitive pin via uv lock. pip-audit now reports no known vulnerabilities. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(cli): route Sentinel-2 convert through the optimized flat layout Generic `convert` emitted GeoZarr-0.4 multiscales with native data at the group root and overviews as nested r{2**N} sub-groups. That tree cannot be opened by xr.open_datatree (an overview child's x/y conflict with the parent's inherited x/y), so the `info` and `validate` CLI commands crashed on the converter's own Sentinel-2 output. Detect Sentinel-2 inputs and delegate to the existing, tested convert_s2_optimized path, which emits flat sibling r{N}m levels (the layout PR #172 documents). Non-S2 inputs keep the generic create_geozarr_dataset path. Detection is best-effort and never aborts conversion. Regenerate the geozarr_examples structure snapshots to the flat layout. Co-Authored-By: Claude Opus 4.8 <[email protected]> * test: make CLI e2e assertions debuggable and fix multiscale round-trip - The CLI logs errors to stdout via structlog; assert on stdout+stderr in test_cli_e2e so a non-zero exit shows the actual error instead of an empty stderr. - test_multiscale_attrs_round_trip tuplified only one side of its equality, so flat-layout level groups carrying list-valued attrs (e.g. spatial:bbox) broke a check that is only about JSON list/tuple normalisation. Tuplify both sides; the model still round-trips losslessly. Co-Authored-By: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Davis Vann Bennett <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: Emmanuel Mathot <[email protected]> * feat(s1-stac): add render extension with dual-pol RGB composite preview The previous S1 RTC preview rendered a single VH band as grayscale with an incorrect rescale (0,219), unsuitable for linear gamma0 RTC values. Declare a render-extension `renders.rgb` config producing the standard dual-pol false- colour composite (R=VV, G=VH, B=VV/VH ratio) with rescale 0–0.1 and tilesize 256, referencing the preferred orbit group. Downstream titiler-based services consume this to generate previews/tiles/tilejson. Co-Authored-By: Claude Opus 4.8 <[email protected]> * refactor(s1-stac): emit a single rescale pair in the RGB render A single [min, max] pair applies to all bands in titiler, so emitting three identical pairs was redundant (and forced collapse logic in the consumer). Same rendered output. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): TEMPORARY name store as s1-rtc-{tile} to match titiler render path (#246) titiler-eopf reconstructs the store path as s3://{bucket}/tests-output/{collection}/{item_id}.zarr and ignores the STAC asset href, so the GeoZarr cube must be named after the item id (s1-rtc-{tile}) for new tiles to preview. Parse the tile from the s1-rtc- prefix (was s1-grd-rtc-); the item id (s1-rtc-{tile}) is unchanged, so filename == item id by construction. Pairs with the data-pipeline direct-write change (store written at the tests-output path) which replaces the #250 auto-copy. Dev-phase, no consumers — disposable test data. Revert to s1-grd-rtc- when titiler-eopf#108 (resolve store from href) lands. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): build STAC item from a non-consolidated cube store build_s1_rtc_stac_item opened the cube with zarr.open_consolidated, which raises `ValueError: Consolidated metadata ... not found` when the store lacks root consolidated metadata. A per-tile cube grown by appending a time-slice to an existing same-orbit group ends up in exactly that state — re-consolidating an append on the S3 store is unreliable — so live STAC registration failed on the 2nd+ same-orbit acquisition for a tile (it broke the Pyrenees S1 RTC soak). The builder must not require consolidated metadata: titiler reads these stores fine without it. Fall back to a direct zarr.open_group(mode="r") read when the consolidated metadata is absent; re-raise any other ValueError. Adds a regression test: build from a 2-slice same-orbit store created with consolidate=False (fails before the fix in open_consolidated, passes after). Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): CF-encode `time` at every level for datetime-based rendering (#192) (#193) Per-acquisition previews render via titiler `sel=time={INTEGER index}` — a positional index that breaks once a cube's time axis goes non-monotonic (a cross-run append of an earlier-dated scene; proved on 31TEH: `[06-08, 06-07]` → the 06-07 item has no preview). Keeping positional indices correct would need cube reorder + full re-registration on every append (not scalable). The deployed titiler (v0.5.0) already does label-based `da.sel(..., method=...)` via `open_datatree(decode_times=True)`; the only gap is the cube's `time` array, which is a bare int64 with no CF datetime metadata, so it can't be a datetime index. Encoding it lets previews select by exact datetime, which works even on a non-monotonic axis — no reorder, register only the new item. titiler picks a multiscale level by zoom (previews use a coarse level), so `time` must resolve there. Putting it only at the orbit-group level and inheriting fails to open while r10m stays bare int64 (AlignmentError: int64 vs datetime64 on the shared dim). So encode `time` consistently at EVERY level: - new `_create_time_coordinate_array` + `TIME_CF_ATTRS` (units/calendar/ standard_name); created at every level in create + append-group paths. - the append write resizes/writes `time` at every level (was r10m only). - absolute_orbit/relative_orbit/platform stay at r10m (not selected on). - stored dtype stays int64 ns, so register_per_acquisition's raw read is unaffected. Adds TestTimeCFDatetime incl. exact datetime `.sel` on a non-monotonic axis at the native and a coarse level. 55 passed. Refs #192 Co-authored-by: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): enrich STAC items & consolidate construction into the library (#195) (#196) * fix(s1-rtc): enrich cube STAC item + fix vv/vh asset model (#195) The S1 RTC cube STAC item carried duplicate vv/vh assets with byte-identical hrefs (bug #1), exposed no asset for the descending orbit group of a dual-orbit cube (bug #2), and was missing most of the metadata the S2 L2A reference carries. Rework `build_s1_rtc_stac_item`: - Asset model: replace the two indistinguishable vv/vh assets with one `gamma0-rtc-backscatter-{asc,desc}` asset per present orbit group, exposing VV/VH as named STAC 1.1 `bands` (no more duplicate hrefs), plus a `border-mask-{orbit}` asset for the valid-data mask. Each γ⁰ asset carries data_type/nodata/unit/gsd. - Identity: add `constellation`, `instruments: [c-sar]`, `gsd` (platform stays per-acquisition — a cube can mix S1A/S1C). - Projection: add proj:bbox, and proj:shape/proj:transform (best-effort from the r10m group attrs). - Datacube extension: cube:dimensions (time as a bounded extent — not a values list, the cube grows by appending — plus x/y) + cube:variables. - Descriptive: title/description; created (earliest acquisition, stable across rebuilds) / updated (build time) via the timestamps extension. - Fold the corrected render rescale (0.0,0.2) into `_rgb_render`. Out of scope (tracked follow-ups): statistics (#157), processing:software / DEM lineage, and per-product CDSE fields are not in the store; per-acquisition item construction is moved in a follow-up commit. Co-Authored-By: Claude Opus 4.8 <[email protected]> * feat(s1-rtc): move per-acquisition + coverage STAC construction into the library Consolidate the *pure* S1 RTC STAC construction in eopf_geozarr.stac.s1_rtc so item shape has a single owner (the library); registration (STAC-API upsert, S3 alternates, gateway rewrite, TiTiler links, CDSE discovery) stays in data-pipeline. Adds, ported from the data-pipeline scripts (deployment URLs stripped): - acquisition_id(tile_id, when) - build_s1_rtc_per_acquisition_items(store, *, orbit, collection_id) -> list[pystac.Item]: one single-datetime item per cube time slice of an orbit, reoriented to that orbit (sat:orbit_state + render expression + only that orbit's γ⁰/mask assets), per-slice platform, datacube structure dropped (a single acquisition is not a cube). Deployment-agnostic: carries the render config + datetime; the registration layer derives the cube-endpoint TiTiler links (sel=time={datetime}) from it. - Slice / pick_slice / slice_coverages: coverage-based preview-slice selection (reads border_mask at r720m). Ported the construction unit tests (pick_slice, slice_coverages, and the per-acquisition construction subset); the link/thumbnail/alternate-asset and rescale-override tests stay in data-pipeline as registration concerns. Follow-up (separate data-pipeline PR): bump the data-model pin, delete the moved functions from register_per_acquisition.py / register_v1_s1_rtc.py, and rewire them to consume this library output. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): per-acquisition items carry their own orbit footprint Review follow-up. A per-acquisition item cloned the cube base, so on a dual-orbit cube every item inherited the WGS84 union bbox/geometry of both orbits and the preferred (ascending) orbit's proj:bbox — advertising a footprint wider than, and a data bbox from the wrong orbit relative to, the acquisition's own orbit. Recompute bbox/geometry/proj:bbox from the run orbit's spatial:bbox (factored the bbox->polygon construction into `_bbox_to_geometry`, shared with the cube builder), and give per-acquisition items a single-acquisition description rather than the cube's "datacube" wording. Tests: per-acq footprint = run orbit (not the union); `created` = earliest acquisition (idempotent across rebuilds); ValueError on an invalid or absent orbit. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): omit orbit_state on dual-orbit cubes; expose datacube dimension sizes Two review observations from inspecting the live items: - sat:orbit_state is single-valued, so on a dual-orbit cube (both ascending and descending groups present) a single value mislabels half the slices. Set it (and declare the SAT extension) only for a single-orbit cube; per-acquisition items, which are single-orbit, still carry the real per-orbit value. Ensure per-acq items declare the SAT extension even when cloned from a dual-orbit base that omitted it. - cube:dimensions previously carried only `extent`, so the number of elements per dimension wasn't visible. The time axis is irregularly sampled, so list its discrete `values` (count = number of acquisitions); the regular x/y axes get a `step` (size derivable from extent; exact pixel count is also in proj:shape). Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): datacube time axis carries extent only (drop values enumeration) Enumerating every acquisition as cube:dimensions.time.values does not scale as the cube is appended over the mission. No STAC extension provides a scalable temporal element-count field (datacube temporal_dimension offers only values/step/extent, and step is null for S1's irregular sampling; projection proj:shape is spatial-only; raster has no dimension shape). So the time axis carries a bounded `extent` only. x/y element counts remain available via proj:shape (exact) and the datacube x/y extent+step. The number of acquisitions is obtained from the per-acquisition collection (STAC search count), which stays correct as the cube grows. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): correct datacube variable_type, platform value, created, time values Review of the live items surfaced several metadata issues: - cube:variables used `type` (a non-standard, silently-ignored key) instead of the datacube extension's `variable_type`; border_mask is now `auxiliary`, vv/vh `data`. - per-acquisition `platform` carried the store's short code (e.g. `s1a`); normalize to the STAC convention `sentinel-1a` (mirrors the Sentinel-2 reference). - `created` (timestamps ext) was set to an acquisition time, which misuses the field (it means the item's creation instant); omit it and keep `updated` = build time. - the cube's irregular time axis now lists its discrete `values` (the acquisition instants) so the number of time steps is visible; the regular x/y axes stay extent + step (their pixel count is in proj:shape). Co-Authored-By: Claude Opus 4.8 <[email protected]> * refactor(s1-rtc): robust EPSG parse for datacube reference_system + clarify per-acq proj inheritance Review follow-ups (no behaviour change for the EPSG:NNNN stores in use): - derive the datacube reference_system via pyproj.CRS(...).to_epsg() instead of splitting the proj:code string, so non-"EPSG:NNNN" CRS forms don't break it. - comment why per-acquisition items recompute proj:bbox from their orbit but inherit proj:code/shape/transform (the MGRS grid is shared across orbits). Co-Authored-By: Claude Opus 4.8 <[email protected]> * docs(s1-rtc): note the dual-orbit time axis spans both orbits in the datacube A dual-orbit cube merges two per-orbit sub-cubes (disjoint time axes) onto a shared grid. Modelling orbit as a dimension would imply a sparse, mostly-empty orbit×time grid; instead orbit stays an attribute of each acquisition, conveyed via the per-orbit assets. Add a `description` on the time dimension so the merged axis isn't misread. Single-orbit cubes get no note. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): distinct per-acquisition titles; clearer asset titles - per-acquisition items inherited the cube's "… — tile {id}" title, so every scene in the acquisitions collection was titled identically. Give each a title with its datetime + orbit so siblings are distinguishable. - simplify the border-mask asset title ("Valid-data mask") and make the zarr-store title say "Sentinel-1" for naming consistency. Co-Authored-By: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: Claude Opus 4.8 <[email protected]> * perf(s1-rtc): shard the conditions arrays (gamma_area/LIA) like the vv/vh pyramid (#197) * perf(s1-rtc): shard the conditions arrays like the vv/vh pyramid A real S1 RTC cube is 3807 objects / 3.5 GB, of which 3604 (94.7%) are the conditions/gamma_area_<relorbit> arrays: [10980,10980] float32, inner chunk 366², NO sharding_indexed -> ~900 tiny chunk objects each. They are time-invariant yet dominate the object count, which dominated the ingest's S3 upload wall-time (a live pod sat ~34 min in "Uploading store" at 9 millicores). The vv/vh/border_mask display pyramid is already sharded (one shard per time slice over the full spatial extent, inner 366²). Apply that same existing layout to the condition arrays: add shards=(h, w) to the one create_array in ingest_s1tiling_conditions. All condition arrays (gamma_area, lia, incidence_angle) share that write path and the same 2D full-resolution shape, so all collapse from ~900 chunk objects to 1 shard object (cube ~3807 -> ~210). calculate_aligned_chunk_size returns a divisor of the dimension, so (h, w) is a clean multiple of the inner chunk (Zarr v3 shard-divisibility). conditions arrays are NOT in TiTiler's render path (vv/vh/border_mask), so this does not touch the web-render layout; it only makes a client read a condition array in one ranged GET instead of ~900. Values are byte-identical. Tests: +2 (sharding codec present; 9 inner chunks -> 1 on-disk shard object + byte-identical roundtrip). 57 passed. Spec: claude-docs/specs/s1_gamma_area_sharding.md. Cross-repo Task T5 of data-pipeline/claude-docs/plans/s1_ingest_upload_perf.md. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_011LsWkVvRfkRzjqAMrzfmRP * docs(s1-rtc): record real-S3 sharding benchmark in the T5 spec Validated on the live OVH bucket (laptop->DE): object collapse 100->1 (prod ~900->1), PUT 1.7x faster even with batched concurrency on, divisibility valid at the production 10980² (aligned 366 divides 10980), reads byte-identical. Honest caveat recorded: a full-array sequential read is NOT faster sharded (same bytes, one un-parallelizable object) — the win is object-count (upload + listing) and windowed/partial cloud reads, not full-read throughput. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_011LsWkVvRfkRzjqAMrzfmRP * refactor(s1-rtc): trim the conditions-sharding comment to match vv/vh style Comment-only. The surrounding vv/vh sharding has no inline explainer; the long cloud-access rationale now lives in claude-docs/specs/s1_gamma_area_sharding.md. Keep only the non-obvious bits: why one shard, and the Zarr v3 shard-divisibility invariant. No behavior change (20 condition/shard tests green). Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_011LsWkVvRfkRzjqAMrzfmRP * chore(s1-rtc): drop the claude-docs spec from this PR The data-model repo has no claude-docs/specs convention; the spec was noise for this PR's reviewers. The problem statement, real-S3 benchmark and migration note live in the data-pipeline plan + tracking issue EOPF-Explorer/data-pipeline#288 and PR #197's description. Also drop the now-dangling spec path from the code comment (the rationale stays inline). No behavior change (20 condition/shard tests green). Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_011LsWkVvRfkRzjqAMrzfmRP --------- Co-authored-by: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): heal a multiscale level missing `time` on append (robust writer) (#200) * perf(s1-rtc): shard the conditions arrays like the vv/vh pyramid A real S1 RTC cube is 3807 objects / 3.5 GB, of which 3604 (94.7%) are the conditions/gamma_area_<relorbit> arrays: [10980,10980] float32, inner chunk 366², NO sharding_indexed -> ~900 tiny chunk objects each. They are time-invariant yet dominate the object count, which dominated the ingest's S3 upload wall-time (a live pod sat ~34 min in "Uploading store" at 9 millicores). The vv/vh/border_mask display pyramid is already sharded (one shard per time slice over the full spatial extent, inner 366²). Apply that same existing layout to the condition arrays: add shards=(h, w) to the one create_array in ingest_s1tiling_conditions. All condition arrays (gamma_area, lia, incidence_angle) share that write path and the same 2D full-resolution shape, so all collapse from ~900 chunk objects to 1 shard object (cube ~3807 -> ~210). calculate_aligned_chunk_size returns a divisor of the dimension, so (h, w) is a clean multiple of the inner chunk (Zarr v3 shard-divisibility). conditions arrays are NOT in TiTiler's render path (vv/vh/border_mask), so this does not touch the web-render layout; it only makes a client read a condition array in one ranged GET instead of ~900. Values are byte-identical. Tests: +2 (sharding codec present; 9 inner chunks -> 1 on-disk shard object + byte-identical roundtrip). 57 passed. Spec: claude-docs/specs/s1_gamma_area_sharding.md. Cross-repo Task T5 of data-pipeline/claude-docs/plans/s1_ingest_upload_perf.md. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_011LsWkVvRfkRzjqAMrzfmRP * docs(s1-rtc): record real-S3 sharding benchmark in the T5 spec Validated on the live OVH bucket (laptop->DE): object collapse 100->1 (prod ~900->1), PUT 1.7x faster even with batched concurrency on, divisibility valid at the production 10980² (aligned 366 divides 10980), reads byte-identical. Honest caveat recorded: a full-array sequential read is NOT faster sharded (same bytes, one un-parallelizable object) — the win is object-count (upload + listing) and windowed/partial cloud reads, not full-read throughput. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_011LsWkVvRfkRzjqAMrzfmRP * refactor(s1-rtc): trim the conditions-sharding comment to match vv/vh style Comment-only. The surrounding vv/vh sharding has no inline explainer; the long cloud-access rationale now lives in claude-docs/specs/s1_gamma_area_sharding.md. Keep only the non-obvious bits: why one shard, and the Zarr v3 shard-divisibility invariant. No behavior change (20 condition/shard tests green). Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_011LsWkVvRfkRzjqAMrzfmRP * chore(s1-rtc): drop the claude-docs spec from this PR The data-model repo has no claude-docs/specs convention; the spec was noise for this PR's reviewers. The problem statement, real-S3 benchmark and migration note live in the data-pipeline plan + tracking issue EOPF-Explorer/data-pipeline#288 and PR #197's description. Also drop the now-dangling spec path from the code comment (the rationale stays inline). No behavior change (20 condition/shard tests green). Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_011LsWkVvRfkRzjqAMrzfmRP * fix(s1-rtc): heal a multiscale level missing `time` on append (robust writer) `ingest_s1tiling_acquisition` resized `level["time"]` on every multiscale level, assuming a fresh build created `time` at each level (#192). A cube built before #192 -- or left half-built by an interrupted append -- can carry `r10m/time` yet lack it at a coarser level, so the resize raised `KeyError: 'time'` and, because the append consistency check validated only CRS + shape, the ingest was non-convergent (observed on 30TWM). Before the per-level write loop, recreate any missing-level `time` from `r10m/time` (backfilling the existing slices so prior timestamps are preserved), or raise a clear error when the cube is inconsistent in a way a backfill cannot fix (a level's length disagrees with `r10m/time`, or `r10m` has slices but no `time`). This is the durable upstream counterpart to the data-pipeline guard (data-pipeline #294), making that orchestration-side mitigation belt-and-suspenders. Tests: 4 new cases (heal, no-op when healthy, raise on half-built, raise on missing r10m/time); full s1_ingest suite 61 passed. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01V3qS75byrUuCSHFqcWi26B --------- Co-authored-by: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): write CF `_FillValue` on float arrays (#172 parity) (#201) * fix(s1-rtc): write CF `_FillValue` on float arrays (data-model #172 parity) The S1 RTC ingest writer (`s1_ingest.py`) was never touched by #172, which added the CF `_FillValue` attribute to the S2 and S1-GRD paths so xarray can mask NaN nodata via `use_zarr_fill_value_as_mask=True` despite xarray #11345 — the zarr-level `fill_value` field alone is not surfaced through xarray's encoding. Consequently S1 RTC cubes carried only `grid_mapping` on their float bands, while S2 cubes carry `_FillValue` (`AAAAAAAA+H8=`) + standard_name + units, leaving S1 nodata unmaskable for generic xarray/CF consumers. Set the CF `_FillValue` (FillValueCoder-encoded, matching S2) plus standard_name and units on the float backscatter bands (vv/vh) at every multiscale level, and `_FillValue` on the float condition arrays (gamma_area, lia). The two near-identical vv/vh creation loops — new store vs. new orbit on an existing store — are unified into a shared `_create_band_arrays` helper so they can't drift again (the inline path previously lacked the metadata). Not ported (S2-specific or N/A to the RTC writer): the scale/offset + CastValue codec (S1 vv/vh are float32, unpacked), v0 deprecation (RTC is v1-only), and `sanitize_array_attrs` (s1_ingest writes controlled attrs zarr-direct and never copies source EOPF attrs). Tests: create-store unit, end-to-end ingest across both orbit-creation paths, and float-conditions coverage. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_011sBdb17MWSAn7VEPSnqKJU * refactor(s1-rtc): address review — round-trip test, orbit-builder dedup, typed attrs - Add an end-to-end NaN-masking round-trip test (open with `use_zarr_fill_value_as_mask=True`, `to_masked_array()`), mirroring the S2 guarantee in `test_array_attrs.py`. Proves `_FillValue` actually masks nodata, not just that the attribute is present. - Unify the two orbit-creation paths — new store (`create_s1_store`) and new orbit added to an existing store (`ingest_s1tiling_acquisition`) — into a shared `_build_orbit_group` helper. Removes ~60 lines of duplication and fixes the inline path's missing `proj:code` on level groups (guarded by a new test). Net: `s1_ingest.py` shrinks while gaining the fix. - Model the backscatter band CF attrs as a `S1BackscatterAttrsJSON` TypedDict (PR review request), alongside the existing `Standard*CoordAttrsJSON` types. - Document the internal `xarray.backends.zarr.FillValueCoder` import. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_011sBdb17MWSAn7VEPSnqKJU --------- Co-authored-by: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): store nodata as NaN, not 0, so titiler masks it transparent (#202) S1 RTC previews render the out-of-swath region as opaque black while the S2 reference renders it transparent. The cause is the *data*, not the metadata: #201 already gave vv/vh the same dtype/`fill_value`=NaN/ `_FillValue`/`grid_mapping` encoding as S2, but those are inert because no NaN is ever written — s1tiling stores `0.0` out of swath, and titiler treats `0` as valid data and paints it black. Mask nodata to NaN at the writer: - vv/vh: `np.where(border_mask == 0, NaN, ...)` — border_mask is the authoritative valid-data mask (0 = no-data). `_downsample_2d` already uses `np.nanmean` for floats, so NaN propagates to every overview level. - float conditions (gamma_area/lia): masked read off the GeoTIFF's declared nodata (border_mask is N/A for static conditions); a no-op when no nodata is declared. Only newly re-ingested cubes get NaN; existing cubes are remediated separately. Tests: NaN ⟺ border_mask==0 at native + overview levels, masking round-trips via `use_zarr_fill_value_as_mask`, conditions declared-nodata → NaN. Co-authored-by: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): consolidate every orbit group, not just the last-ingested one (#203) S1 RTC cubes ended up with only one orbit consolidated on disk (staging: 30TWM asc✓/desc✗, 32TNN/30UVU asc✗/desc✓). Root cause: the pipeline ingests acquisitions one orbit at a time in separate pods, each of which strips all consolidated metadata (so `time` can resize), ingests its orbit, then calls `consolidate_s1_store(store, orbit_direction)` — which only re-consolidated the single orbit passed. So whichever orbit was ingested last is the only one left with on-disk consolidated metadata. Consolidate every orbit group present (iterate `root.groups()`), then the root. The minimal append-fetch already pulls all `zarr.json`, so both orbits' metadata is local — same reason the root consolidation already works. Signature unchanged (`orbit_direction` kept for logging/callers). Impact is cosmetic: the root consolidation is complete for both orbits and readers opening at the root get a synthesized child view (titiler renders the unconsolidated orbit fine). The fix matters for clients opening a single orbit group standalone (the STAC `<cube>.zarr/<orbit>` hrefs) with `consolidated=True`, which otherwise fall back to a listing. Test asserts each orbit group is consolidated *standalone* — checking via `root[orbit]` is a false-green because a consolidated root synthesizes the child's view. Existing cubes self-heal on their next re-ingest. Co-authored-by: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): per-band rescale for the RGB composite render (#204) The render expression emits 3 bands (vv; vh; vv/vh) but supplied a single rescale pair [0,0.2] applied to all three. The vv/vh ratio (natural range ~1-15) saturated under that stretch -> flat blue/purple wash, and low cross-pol water dropped to transparent, which made (correctly geolocated) ascending swaths look mislocated. Give each band its own pair: vv [0,0.4], vh [0,0.1], ratio [1,15]. Cosmetic preview-only change; pixel placement is unchanged. Verified live against the raster API and the S1 STAC tests. Co-authored-by: Claude Opus 4.8 <[email protected]> * feat(s1-rtc): add grid:code (MGRS tile) + grid extension to STAC items (#205) Both the cube item and (by inheritance) its per-acquisition items now carry the STAC grid extension with grid:code = "MGRS-{tile}". This makes the tile a first-class queryable property: the acquisitions collection becomes natively tile-filterable in STAC Browser, and cube↔acquisition cross-links can filter on grid:code instead of an id-prefix LIKE. Cube-builder only — per-acquisition items inherit it via the existing build_s1_rtc_per_acquisition_items copy ({**base_dict} carries stac_extensions; the per-acq property denylist excludes grid:code). Claude-Session: https://claude.ai/code/session_019z3eVtkSNHhN9vHd8QqGcf Co-authored-by: Claude Opus 4.8 <[email protected]> * refactor(s1): make S1 RTC code + tests pyright-clean (#199 parity) (#215) PR #199 migrated the project from mypy to pyright, but the S1 RTC code was written under mypy with zarr type-checking disabled (ignore_missing_imports). Under pyright, zarr 3.x is fully typed, so every `group[name]` is a `zarr.Array | zarr.Group` union and the S1 code + tests surfaced ~1390 type errors (index/argument/attribute-access on the union). Resolve all of them with runtime narrowing, matching #199's idioms (no new `Any`, no blanket ignores): - s1_ingest.py: add `_child_group`/`_child_array` helpers (isinstance-or-raise, mirroring geozarr.py / s2_multiscale.py) used at every store member access; narrow `arr.metadata` to ArrayV3Metadata for `dimension_names`; guard the append-validation attrs chains with isinstance; give `_rasterio_env` a real return type instead of `# type: ignore[return]`. - s1_rtc.py: narrow the NotRequired TypedDict member accessors with `.get()` + explicit `raise KeyError` (behavior-identical); drop mypy-era `# type: ignore` comments pyright flags as unnecessary. - tests: add isinstance-narrowing `_group`/`_array` helpers and `np.asarray` around zarr reads; no assertion weakened. - pyz/v3.py: add `# type: ignore[type-var]` on the GroupSpec base (parity with pyz/v2.py:43). The S1 models are the first to subclass pyz.v3.GroupSpec under a type checker, so this intentional variance now needs the same suppression v2 already carries. Full-project pyright: 0 errors. No runtime behavior or on-disk output changes; full test suite green. Co-authored-by: Claude Fable 5 <[email protected]> * refactor(s1): type CRS codes, bboxes and time-coord attrs per review (#180) Address the three open review threads from d-v-b on #180 (all typing): - "typeddictify this" (TIME_CF_ATTRS): add S1TimeCoordAttrsJSON next to its sibling S1BackscatterAttrsJSON and annotate the constant Final[...]. The call-site spread needs the same cast("dict", ...) idiom the backscatter attrs already use (zarr's attrs.update JSON union vs TypedDict). - NewType for CRS strings / BoundingBox2D: add CRSCode and BoundingBox2D NewTypes with validating make_crs_code / make_bounding_box constructors (accept object, validate at runtime) in eopf_geozarr.types, replacing the unchecked cast() reads of store attrs — corrupt attrs now fail loudly. Threaded through stac/s1_rtc.py (incl. a _OrbitInfo NamedTuple replacing the untyped per-orbit dict and its downstream cast cluster), conversion/s1_ingest.py (S1TilingMetadata.crs/bounds, _add_grid_mapping, conditions path), and the test constants the review anchored on. CRS validation is intentionally light (non-empty str); pyproj raises CRSError downstream on unparseable values. Bbox tuples convert via list(...) at pystac/JSON boundaries. - tests/test_types.py: unit tests for both constructors. Also fix four pre-existing ruff findings in s1_ingest.py that #215 (merged via GitHub, bypassing local hooks) left behind: RUF002, TC003, SIM108, RUF059 — required for pre-commit to pass on this file. No runtime behavior or on-disk output changes. Full-project pyright: 0 errors; full non-network suite: 531 passed. Co-Authored-By: Claude Fable 5 <[email protected]> --------- Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Davis Vann Bennett <[email protected]> Co-authored-by: Emmanuel Mathot <[email protected]>

Fixes the failing `lint` CI job (ruff-check + ruff-format) on the s1-tiling branch. All changes are lint-only; no behavior changes intended. analysis/ scripts: - annotate public function return types (ANN201) - prefix unused unpacked shape vars with underscore (RUF059) - use list.extend/comprehensions (PERF401) - collapse nested/redundant ifs and .keys() iteration (SIM102/108/118) - replace ambiguous unicode gamma/multiplication signs with ASCII (RUF001/RUF002) - mark the eodag4 patch script executable to match its shebang (EXE001) src/eopf_geozarr/cli.py: - use log.exception in S1 ingest error handlers (TRY400) src/eopf_geozarr/data_api/s1_rtc.py, tests/test_s1_rtc_ingest.py: - ruff-format reflow; drop unused rng binding (F841) Co-Authored-By: Claude Opus 4.8 <[email protected]>

The `open_source` re-export was appended after `s1_ingest`, tripping ruff's I001 (unsorted-imports) in the `lint` CI job. Move it into alphabetical position so `prek run --all-files` passes without the ruff-check hook rewriting the file. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_0183hi73TJBpNdroCtni1b1z

github-actions · 2026-09-09T11:45:22Z

Security Audit Report

View workflow run

Bandit — Static Security Analysis (Security tab)

12 issue(s) found: 12 low

✅ No issues at or above HIGH severity.

12 low issue(s) below threshold not shown in table.

pip-audit — Dependency Vulnerabilities (Security tab)

Package	Version	ID	Fix Versions	Description
tornado	6.5.7	GHSA-wwv5-g3v4-889x	6.5.8	## Summary The CVE-2026-35536 fix added a validation loop that rejects `[\x00-\x20\x3b\x7f]`, but only for the hardcoded
tornado	6.5.7	GHSA-8423-8fgw-73vq	6.5.8	## Description ### Summary `parse_multipart_form_data` (httputil.py:34) calls `data.split(b"--"+boundary+b"\r\n")` **b
tornado	6.5.7	CVE-2026-82397	6.5.8	## Summary Tornado parses `application/x-www-form-urlencoded` bodies with `urllib.parse.parse_qs` and does not pass `ma

3 vulnerability/vulnerabilities found (3 fixable) across 1 package(s).

Result: ❌ Blocking issues found — see details above.

… import `get_cf_standard_names` is called at module scope and was guarded by `except URLError`. A truncated HTTP response raises `http.client.IncompleteRead`, which is an `HTTPException` and not a `URLError`, so a partial download propagated out of the import and broke `import eopf_geozarr` for every mission, not just the CF validation it gates. Broaden the guard to `(URLError, HTTPException, TimeoutError)` and warn when validation is disabled: with `DO_CF_NAME_VALIDATION` false, `check_standard_name` accepts anything, so a silent degradation is indistinguishable from a passing check. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011TEYQYdyBLTXuEQJwgMZEC

…d path `zarr.open_consolidated` is an alias for `open_group`, whose signature defaults to `mode="a"`. `_open_root` therefore *created* a store at any path that did not resolve: `generate-stac-s1 --store s3://bucket/typo.zarr` wrote into the production bucket before failing. Open with `mode="r"` and `use_consolidated=None` — zarr's documented use-if-present-else-list mode. That also removes the need to tell a missing consolidated block from a missing store by exception type, which was not possible: `GroupNotFoundError` subclasses `ValueError` on zarr 3.2.0, so the previous `except ValueError` was no narrower than the substring match it guarded and additionally swallowed missing-store errors. The unconsolidated path is permanent, not transitional — appends now strip the block — so both forms are covered by tests. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011TEYQYdyBLTXuEQJwgMZEC

…store URIs Five defects in the ingest path, each of which lost or corrupted data with exit code 0. F1 — appends after `consolidate-s1` overwrote the same time slice. `consolidate_s1_store` writes a block on *every* orbit group, and `use_consolidated` applies only to the group actually opened, so array shapes read through the root stayed stale: every post-consolidation append read the pre-consolidation length and targeted the same index. Add `_open_store_for_write` (root; fixes group *membership*, so an orbit created since the last consolidation is visible) and `_open_orbit_for_write` (orbit; fixes *shapes*). Both are needed — the root flag alone leaves shapes stale. Applied at all three sites, including `consolidate_s1_store`'s own orbit enumeration, which otherwise silently skipped the newest orbit. Strip the stale block after a successful append; the write path never depends on that having run. F1.4 — make the append convergent. A transport or OOM failure inside the per-level write loop leaves r10m at N+1 and a coarser level at N; the next append then wrote NaN fill at the gap with `time == 0` (1970) and exited 0. Assert every level's arrays agree with `r10m/time` before writing anything. This has to cover the levels that *do* carry `time`, which is precisely the set the heal loop's `continue` skips, so it cannot live inside that loop. The old narrower raise is subsumed and removed; the behaviour contract is unchanged. F2 — a GeoTIFF from a different MGRS tile appended silently. The check compared only `proj:code` and native `spatial:shape`, so two adjacent same-zone tiles differing only in origin both passed and slice 1's pixels were served 100 km from where the item said they were. Compare all six terms of `spatial:transform` too, via `_assert_grid_matches`, on *both* the append and the new-orbit branch — the latter built the group straight from the incoming metadata with no comparison at all, poisoning a whole orbit group rather than one slice. F4 — `s3://` store URIs were written to local disk. `Path("s3://bucket/x.zarr")` collapses to `PosixPath("s3:/bucket/x.zarr")`, whose `.exists()` is always False, so the create branch wrote a LocalStore under `./s3:/bucket/...`, logged success and exited 0 with nothing in S3. Keep the URI a string, thread `storage_options` through every open, and probe `<store>/zarr.json` rather than the prefix, which on object storage exists only implicitly. F8.3 — `create_s1_store` now writes a complete root. `spatial:bbox` has no default in the root model, so a store that was never consolidated failed root validation outright. The native CRS and bounds are already in hand at creation, and the minispec does not require EPSG:4326 here. F9 — an integer-dtype condition GeoTIFF aborted the ingest: `.filled(np.nan)` ran before the `.astype(np.float32)` that makes NaN a legal fill. Also stamp `eopf:writer_schema` on new stores. Generation 2 is layout-compatible with earlier stores, so an absent or lower stamp only warns — refusing would brick appends to the existing archive for no corresponding hazard. The release that lands the pending chunk and coordinate changes must bump the constant and turn the warning into a hard refusal; without it one cube can carry an ascending group with cell-centre coordinates and a descending group with edge coordinates under a single STAC item. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011TEYQYdyBLTXuEQJwgMZEC

An exact float comparison against the value round-tripped through the store's JSON risks rejecting every legitimate append across the existing archive to catch a difference that cannot occur in practice. The failure mode this guard exists for is an adjacent MGRS tile, whose origin is ~100 km away — five orders of magnitude outside any tolerance — so nothing is lost by allowing for float drift between rasterio's computed transform and the stored one. This supersedes the origin-only guard on `fix/s1-ingest-tile-origin-guard` (b1fa0b5), which compared only the two translation terms and only on the append branch; that commit is not cherry-picked. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011TEYQYdyBLTXuEQJwgMZEC

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011TEYQYdyBLTXuEQJwgMZEC

`security` blocks the PR on GHSA-wwv5-g3v4-889x, GHSA-8423-8fgw-73vq and CVE-2026-82397, all fixed in tornado 6.5.8. tornado is transitive (via distributed); the existing `[tool.uv] constraint-dependencies` block is the repo's established mechanism for forcing a transitive security bump, so this raises the floor there rather than adding a direct dependency. Bandit reported nothing at or above its HIGH threshold. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011TEYQYdyBLTXuEQJwgMZEC

`validate_store` reports 86 issues per live cube. The causes are structural, not incidental, and the store the writer produces is rejected by the project's own validator. A subgroup can never inherit `zarr_conventions`: `validation._walk` passes inherited convention UUIDs down only to a group's direct child *arrays*, handing any child group an empty frozenset. Every level group was therefore writing `spatial:*` and `proj:*` keys while declaring neither convention, and none carried the required `spatial:dimensions`. Both are now built with `utils.build_convention_attrs`, the same helper the S2 and OLCI paths use, which also closes a duplication item — it emits exactly the keys that were hand-written, plus the two that were missing. It needs a real CRS object, so call sites wrap `metadata.crs`. The conditions group had the same undeclared-convention problem and, worse, no 1-D `x`/`y` coordinate arrays at all: it opened with "dimensions without coordinates" and rioxarray inferred an identity transform, so the georeferencing was absent rather than merely undeclared. `_create_spatial_coordinate_arrays` is reused verbatim, guarded so the second orbit ingested into the shared group does not attempt to re-create them. That guard makes the group's grid load-bearing. Its attributes are written once, by whichever file created it, and every orbit now shares one `x`/`y` pair — so a condition raster on a different grid would be stored under coordinates that do not describe it, and the group stops opening at all once the shapes diverge ("conflicting sizes for dimension 'y'"). The conditions path validated CRS but never the grid; it now rejects a mismatched shape or transform, using the same sub-pixel tolerance as the acquisition path. Root and orbit-group hrefs now validate COMPLIANT with zero issues, which is the WS2a acceptance target — STAC data assets point at the orbit groups, so each must stand alone. Also corrects a comment claiming `consolidate_s1_store` calls `write_store_root_geo_metadata`. It does not; that helper is never called on the S1 path. Adopting it would rewrite the root to EPSG:4326 with a reprojected lon/lat bbox, changing the root's CRS semantics for every reader, so it stays a deliberate behaviour decision rather than part of a conformance fix — the native-CRS root already validates. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011TEYQYdyBLTXuEQJwgMZEC

Two decisions, both taken by the maintainer. D1 (F3) — raise on an out-of-order append. `discover_s1tiling_acquisitions` sorted its group key verbatim, and the key is (platform, tile, orbit_dir, rel_orbit, acq_stamp), so `platform` dominated the timestamp: a mixed S1A/S1C archive came back as every S1A date followed by every S1C date. Ingesting in that order writes a non-monotonic time axis, and `.sel(time=slice(...))` raises on one — the cube is silently unqueryable by time from then on. Discovery now orders by acquisition time first. Sorting discovery is not sufficient on its own, so the invariant is also enforced where the write happens: the real caller iterates the discovered list, but other callers never go through discovery at all. An append whose timestamp precedes the cube's last slice now raises, as does one that duplicates it — the append is positional, so re-ingesting an acquisition would add a second slice carrying the same instant rather than replacing the first. `allow_out_of_order=True` opts out, which is what recovering an already-broken cube (the 31TEH case) needs; the existing test for exact `.sel` on a non-monotonic axis now builds its fixture through that flag. The check runs after the ragged-cube assertion, not before: on a half-built cube the last timestamp is not a reliable thing to compare against, and raggedness is the more fundamental fault. It also runs before any array is resized, so a refusal cannot itself leave a ragged cube. D3 — adopt the S2/OLCI store root. `consolidate_s1_store` now calls `utils.write_store_root_geo_metadata`, so the published root carries `proj:code = EPSG:4326` and a lon/lat `spatial:bbox` covering every orbit group, matching what S2 and OLCI publish. `create_s1_store` still writes a complete native-CRS root at creation so a never-consolidated store validates; this refines it once the orbit extents are known. The union is rebuilt from the child groups and never reads the root's own bbox, and all three keys are replaced in one `attrs.update`, so the pair cannot go half-updated. `eopf:writer_schema` survives because that is an update rather than a replace. The helper reprojects with `transform_bounds(..., densify_pts=21)`, which also avoids the near-global bbox that a corner-only conversion produces in UTM zones 1 and 60. Incidentally fixes a latent crash on the new error path: `np.datetime64` accepts a plain int but rejects np.int64 (numpy 2.4), and the timestamps are np.int64, so formatting the message raised "Could not convert object to NumPy datetime" instead of reporting the ordering problem. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011TEYQYdyBLTXuEQJwgMZEC

… paths F11 -- overview `border_mask` contradicted the averaged backscatter. The mask was subsampled (`nearest`) while vv/vh were block-averaged, so on a diagonal swath edge 65 pixels at r20m read `border_mask == 0` over finite `vv`. The docstring calls it the authoritative valid-data mask and STAC advertises `nodata: 0`, so applying it at preview zoom erased real data. Adds a `max` branch over the same ceil+edge-pad grid the `average` branch uses, which guarantees `isfinite(vv) => mask != 0` by construction. NOT the converse: real γ⁰ can be NaN inside the swath, where the two disagree at r10m already and no downsampling rule can repair it. `_downsample_2d` now rejects an unrecognised method instead of falling through to the block mean, which for the uint8 mask silently reproduces the very defect this fixes. F15 -- discovery emitted `ASC`/`DES` while ingest expected `ascending`/`descending`, and never validated. No CLI command consumes discovery, so the library path IS the production path: a caller passing `acq["orbit_dir"]` built `<store>/ASC/` and only failed at STAC, after a multi-hour ingest. Discovery now also emits `orbit_direction`, and the ingester validates before any I/O. Incomplete bundles were also appended unconditionally after the warning, so a VV-only bundle died on `KeyError: 'vh'`; completeness is now the keys actually consumed (`vh_mask` never is), each bundle carries `complete`, and `skip_incomplete=True` drops them. B3 -- one untagged GeoTIFF aborted discovery of an entire archive, while an unparseable *filename* was silently skipped. Now best-effort. The stamp is resolved once PER BUNDLE rather than per file: it is part of the grouping key, and derived `_BorderMask` products commonly carry no ACQUISITION_DATETIME, so per-file resolution split one acquisition into two half-bundles that `skip_incomplete` then discarded -- discovery returning `[]` with every file present on disk and the ingest loop reporting success having written nothing. B7 -- `_normalise_s1tiling_datetime` passed unrecognised forms through to fail ~560 lines later inside `np.datetime64`. It now validates and returns the parser's own canonical spelling. Returning the raw text would have been worse than the bug: `fromisoformat` and `np.datetime64` are different parsers, and `'20230115'` passes the first while the second reads it as the YEAR 20230115, storing 2206-09-06 with no error and permanently poisoning the orbit's out-of-order guard. B8 -- xarray's private `FillValueCoder` was imported at module scope, so any xarray reorganisation raised from `import eopf_geozarr` itself, breaking the S2 and OLCI paths that never touch S1. Guarding the import only covered a rename; the value is needed at import time either way. Now eight bytes of `struct`, byte-identical to what xarray produces, with a test pinning the two together. Also: a conditions group written before x/y existed had no repair path. Backfilled -- after the grid checks, per coordinate rather than both-or-nothing, refusing when the group records no `spatial:shape` to check against. The conditions path now opens the orbit with its consolidated block bypassed and strips the stale blocks afterwards, as the acquisition path does: every membership test there was being answered from a snapshot, so the repair was invisible to every reader that rejected the store, the next ordinary call raised `ContainsArrayError` over an array on disk, and condition arrays written in the same call silently never received `grid_mapping`. D5 note: `skip_incomplete=True` changes the list `discover_s1tiling_acquisitions` returns, which data-pipeline's `scripts/ingest_v1_s1_rtc.py` consumes. Adding `orbit_direction`/`complete` is backward-compatible; the default is not. Coordinate with the pin bump. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EzfLgkdrTVoY6YQKDDNuQr

…the F11 review Two decisions the F11 mask change raised, investigated and closed. Both land as documentation plus one small correctness fix; no behaviour change. Coverage metric -- KEEP block-`max`, keep reading r720m. `slice_coverages` counts valid mask pixels at r720m, and block-max makes that an upper bound on the native valid fraction. That is the correct input for the only decision it feeds. Overview masks are block-max while vv/vh are `nanmean`, and the two agree exactly: `(mask != 0) == isfinite(vv)` is array-identical at every overview level, so the number is the fraction of the preview image that renders as data, which is precisely what "which slice should the thumbnail default to" needs. The speckled-mask pathology that motivated the question cannot occur: S1Tiling's `SmoothBorderMask` applies a binary morphological opening with a ball of radius 5, so the valid set is a union of ~110 m disks (a 50% random-speckle mask opens to 0.0000). Residual fine invalidity is interior radar-shadow holes, which the opening preserves. On a solid swath edge the bias is 0.8749 -> 0.8783. Blast radius is small either way: nothing in this library calls `slice_coverages`, the sole consumer is data-pipeline's preview picker (wrapped in try/except, falling back to the default slice), and the number is never written to STAC nor used to filter or gate registration. `pick_slice` has two regimes -- above the gate selection is purely by recency and the value stops mattering; below it selection is purely relative, where a shared bias cancels. So the fix is to stop calling it "tile coverage" and document what it is, including that it must not be reused as a quality metric and must not be "fixed" by reading a finer level (36x the bytes, and further from the quantity the decision needs). The equivalence is now pinned by a test rather than left as a claim. Writer stamp -- DO NOT bump `WRITER_SCHEMA`; 3 stays reserved for the v0.12.0 layout change. The stamp is written only by `create_s1_store`, never on append, so bumping it would mark cubes built fresh under the new writer (uniformly new semantics, not mixed) and leave every cube that actually becomes mixed by appending unmarked -- inverted relative to the hazard. Spending 3 here would also force v0.12.0's planned hard refusal to fire over a mask difference with no layout consequence, which would brick appends to the whole archive. The mix belongs in the conformance migration, which can recompute overview masks from r10m: idempotent, so it needs no generation marker at all. Alongside: the orbit group declares `multiscales.resampling_method: "average"`, which is true for vv/vh and has never been true for the mask -- `nearest` before F11, block-`max` after. The real rule is now declared on the array itself. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EzfLgkdrTVoY6YQKDDNuQr

…ts them (#261) * fix(s1): correct the emitted STAC items and the data model that rejects them Stacked on `s1-tiling`; this is the STAC-builder and data-model half of the #216 remediation, split out so the catalogue-facing changes can be reviewed on their own. ## The data model rejected the stores this repo's own writer produces `S1RtcRoot.from_zarr` failed with 23 `extra_forbidden` errors on any real store: the member TypedDicts are `closed=True` and omitted `x`/`y`/`spatial_ref` at every level, plus `time` on the overviews (written since #192). Every model test stayed green because they only ever validated a static fixture that predated both. Adding the keys is not the fix on its own. Both TypedDicts are `closed=True, total=False`, so every key is NotRequired: adding keys only relaxes `extra_forbidden` and could never detect a writer that *stops* emitting a coordinate — exactly the #192 regression class. So each dataset class also gets a `model_validator` asserting the coordinates are present. `Required[...]` was not used; it would break the documented incremental-construction pattern. The fixture is regenerated from real writer output by a committed script (`tests/_test_data/s1_rtc_examples/regenerate.py`) that dumps with the *generic* `GroupSpec`, never `S1RtcRoot` — dumping through the model would record only what the model already accepts and close the loop falsely. `TestWriterRoundTrip` validates a freshly ingested store in both directions and is the check that cannot go stale; a further test asserts the fixture still matches writer output, so a hand-edit cannot reopen the drift one level down. Also relaxes the conditions group from requiring `gamma_area_*` to requiring any condition array: `ingest_s1tiling_conditions` gates only on "at least one condition path" and all three CLI flags default to None, so a lia-only group is writable today and was then unopenable. ## Every emitted STAC item failed the render extension it declared `_rgb_render` omitted `assets`, the one required field of a Render Object — and the block was emitted under `properties`, while render v1.0.0's Item branch requires `renders` at the item ROOT (its own examples/item-landsat8.json agrees; only the README's "Item Properties" table says otherwise). Both are fixed. Verified by running stac-validator against emitted items; no test pins it, because `jsonschema` is not installed and is in no dependency group — a test that imported it would error rather than skip, and marking it `network` would be silently deselected by CI's `-m "not network"`. The tests assert the required fields and their placement directly. `sat:relative_orbit` / `sat:absolute_orbit` are declared `{"type": "integer", "minimum": 1}`, and both coordinates are created with `fill_value=0`, so a torn append — or the known upstream bug putting 0 on live slices — made a validating STAC API refuse the item. `0` now reads as "not recorded" rather than orbit zero. A cube may only claim a single-valued orbit field when every slice agrees AND every slice is recorded. Values are pooled only from orbit groups that carry the coordinate, so a dual-orbit cube with one group written before those coordinates existed would otherwise assert one track for acquisitions from both orbit directions — while correctly omitting `sat:orbit_state`. Other fixes: a malformed tile id is now rejected rather than minted into `grid:code`, which is what tile-filtered searches and the cube-to-acquisition cross-links join on; `providers` and `created` are emitted (a STAC API search returns bare items, so an item without `providers` loses its attribution); the inert timestamps extension is dropped; `_utm_to_wgs84` densifies with `transform_bounds(densify_pts=21)`, so a UTM zone 1/60 tile gets a 1.75-degree crossing bbox instead of a 358-degree one that spanned the globe, with `_bbox_to_geometry` splitting it into a two-part MultiPolygon at the antimeridian and `_union_wgs84` unwrapping before taking extremes; and `to_epsg()` returning None falls back to WKT instead of emitting a null reference system. ## Cross-repo follow-ups (data-pipeline, not fixed here) - `tests/unit/test_build_s1_rtc_collections.py:184` compares `_rgb_render` output with `assets` stripped from one side only; it breaks on the next pin bump. - `_reorient_item_to_orbit` (`scripts/register_v1_s1_rtc.py`) rewrites the render `expression` but not the new `assets`, so a dual-orbit cube pinned to a descending preview ships a self-contradicting render block. Rendering is unaffected (`_render_to_query` allow-lists fields), so it is silent metadata corruption. - `run_ingest_register.py` builds the store path from an unvalidated `--tile-id` and registers after ingesting, so a typo now leaves an orphaned S3 cube plus a failed step. Validate the tile id at argparse time. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011TEYQYdyBLTXuEQJwgMZEC * fix(s1): mirror `renders` into properties for pinned consumers The move of `renders` from `properties` to the item root is correct -- render v1.0.0's Item branch requires it at the root -- but it is a silent breaking change for three data-pipeline consumers on that repo's `main`: register_per_acquisition.py:132 d["properties"]["renders"]["rgb"] KeyError register_v1_s1_rtc.py:57 props.get("renders", {}).get("rgb") -> None register_v1.py:191 item.properties.get("renders") -> None The two `.get` sites fail silently, which turns `_reorient_item_to_orbit` into a no-op: every dual-orbit cube would ship the preferred orbit's render regardless of which slice the preview is pinned to. All three break the moment the data-model pin bumps. Emit `renders` in both places until those consumers read the root. The root copy stays authoritative; the mirror is built from the same object so the two cannot drift. On per-acquisition items both copies must be re-pointed: `props` is cloned from the cube base, so mirroring without the override would leave the *preferred* orbit's render in `properties` and the run orbit's at the root -- reintroducing the same defect one level down. Test added for exactly that (verified failing without the fix). Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EzfLgkdrTVoY6YQKDDNuQr --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]>

`URLError`, `TimeoutError`, `ConnectionResetError` and `ssl.SSLError` are all `OSError` subclasses, so `(OSError, HTTPException)` subsumes the previous tuple and covers the rest of the failure modes of an import-time download. The same block lands verbatim on #262's branch, which rewrote this `except` from the other direction: identical text on both sides means whichever PR merges second merges cleanly instead of conflicting. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01CHpM7QqgsF4jzhPxUxdU7V

`URLError`, `TimeoutError`, `ConnectionResetError` and `ssl.SSLError` are all `OSError` subclasses, so `(OSError, HTTPException)` subsumes the previous tuple and covers the rest of the failure modes of an import-time download. Also warn instead of degrading silently: with validation off, `check_standard_name` accepts anything, so a caller that believes it is validating is not. The same block lands verbatim on the `s1-tiling` branch (#216), which rewrote this `except` from the other direction: identical text on both sides means whichever PR merges second merges cleanly instead of conflicting. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01CHpM7QqgsF4jzhPxUxdU7V

Both comments told a future maintainer to delete `properties.renders` once data-pipeline read the item root. Following that would have removed every render/viewer/tilejson/thumbnail link the pipeline derives, and made `register_per_acquisition` raise. The mirror exists because of the API, not because of data-pipeline. The deployed STAC API models its write body as `Item-Input`: ten root fields, `renders` not among them, and no `additionalProperties` — while `ItemProperties` and `Asset` in the same OpenAPI document do allow extras. Pydantic drops what it does not model, so only `properties.renders` is expected to survive a POST/PUT. Both copies are therefore load-bearing: the root one makes the item satisfy the extension it declares, the mirror is the one that comes back out of the catalogue. Basis: the deployed API's published OpenAPI (`POST /collections/{collection_id}/items` takes `anyOf[Item-Input, ItemCollection-Input]`, `PUT …/items/{item_id}` takes `Item-Input`), plus an offline repro through `stac_pydantic.Item`. Not confirmed against a real POST — and note that live items showing only `properties.renders` is NOT evidence, since no item carrying a root copy has ever been registered. Consumers should read the root first and fall back to `properties`, which EOPF-Explorer/data-pipeline#413 does. Comments only — no behaviour change. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01CHpM7QqgsF4jzhPxUxdU7V

lhoupert · 2026-09-14T08:19:51Z

@d-v-b ok for me to merge this big one?

d-v-b · 2026-09-14T09:08:49Z

go for it!

emmanuelmathot and others added 20 commits April 2, 2026 14:45

refactor: improve Pydantic model definitions and streamline imports i…

dcf0dab

…n S1 RTC module

docs: update Phase 1 checklist, add Phase 1 findings, link tracking i…

319611b

…ssue #139

fix: standardize spatial dimensions to lowercase in S1 RTC models and…

1dd5685

… test cases

fix: add 1D spatial coordinate arrays and use lowercase dimension nam…

5c7ce91

…es in S1 GRD RTC prototype

feat: add detailed implementation plan for Phase 2 GeoTIFF ingestion …

cdfd48b

…pipeline

fix: update Phase 2 GeoTIFF ingestion plan to include 1D spatial coor…

de5f563

…dinate arrays and enforce lowercase dimension names

fix: configure ruff TC001 for Pydantic runtime-evaluated base classes

78fba35

- Add [tool.ruff.lint.flake8-type-checking] runtime-evaluated-base-classes for pydantic.BaseModel so Pydantic field type imports aren't flagged - Remove 4 stale noqa comments auto-fixed by ruff

feat: implement S1Tiling conditions ingestion and discovery commands …

1e891e5

…in CLI

Revert "feat(stac): add build_s1_rtc_stac_item and generate-stac-s1 C…

3e7a3fd

…LI (#173)" (#179) This reverts commit f749368.

lhoupert requested a review from d-v-b July 14, 2026 12:35

lhoupert mentioned this pull request Jul 14, 2026

refactor(s2): validate CRS/EPSG and bbox attrs via typed constructors (#180 parity) #217

Closed

1 task done

Merge branch 'main' into s1-tiling

1fd2d56

lhoupert requested a review from emmanuelmathot August 21, 2026 16:10

emmanuelmathot approved these changes Sep 9, 2026

View reviewed changes

lhoupert and others added 3 commits September 9, 2026 13:08

lhoupert and others added 5 commits September 9, 2026 13:11

refactor(s1): fold the duplicated write-mode open into one helper

4db6330

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011TEYQYdyBLTXuEQJwgMZEC

lhoupert mentioned this pull request Sep 10, 2026

fix(s1): correct the emitted STAC items and the data model that rejects them #261

Merged

1 task done

lhoupert and others added 4 commits September 10, 2026 11:32

This was referenced Sep 11, 2026

fix(s2): write geo metadata for conditions/mask groups so SCL is readable #262

Closed

fix(s1-rtc): prepare the registration and validation paths for the data-model v0.11.0 pin EOPF-Explorer/data-pipeline#413

Merged

lhoupert force-pushed the s1-tiling branch from c28e722 to b869072 Compare September 11, 2026 12:22

lhoupert merged commit f19f423 into main Sep 18, 2026
5 checks passed

github-actions Bot mentioned this pull request Sep 18, 2026

chore: release 0.11.0 #235

Merged

This was referenced Sep 24, 2026

feat!: reintegrate cpm-geozarr into main #277

Merged

Extension of functionalities to all other modes #248

Open

Sentinel-1 GRD γ0T RTC GeoZarr V3 — Implementation Tracker #139

Closed

Navigation Menu

Repository navigation

feat(s1): Sentinel-1 GRD RTC GeoZarr ingestion, data model and STAC builders - #216

feat(s1): Sentinel-1 GRD RTC GeoZarr ingestion, data model and STAC builders#216
lhoupert merged 35 commits into
mainfrom
s1-tiling

lhoupert commented Jul 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 9, 2026 •

edited

Loading

Uh oh!

lhoupert commented Sep 14, 2026

Uh oh!

d-v-b commented Sep 14, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

lhoupert commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Breaking changes for data-pipeline — coordinate with the pin bump

For reviewers

Author attestation

Uh oh!

github-actions Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Security Audit Report

Bandit — Static Security Analysis (Security tab)

pip-audit — Dependency Vulnerabilities (Security tab)

Uh oh!

lhoupert commented Sep 14, 2026

Uh oh!

d-v-b commented Sep 14, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lhoupert commented Jul 14, 2026 •

edited

Loading

github-actions Bot commented Sep 9, 2026 •

edited

Loading