Repository navigation
feat(s1-rtc): STAC item builders + ingest fixes for TiTiler rendering - #180
Conversation
Adds eopf_geozarr.stac.s1_rtc.build_s1_rtc_stac_item, which reads a consolidated S1 GRD RTC Zarr V3 store and returns a pystac.Item with SAR/SAT/projection extensions, WGS84 bbox derived via pyproj, and vv/vh asset sub-paths. Ascending orbit is preferred when both are present. Also adds the generate-stac-s1 CLI subcommand that prints the item as JSON. 8 unit tests cover all acceptance criteria from the plan. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Four functions in s1_ingest.py silently failed for s3:// paths:
- discover_s1tiling_acquisitions / discover_s1tiling_conditions used
pathlib.Path.glob(), which normalises s3:// to s3:/ and returns 0 matches
- ingest_s1tiling_acquisition / ingest_s1tiling_conditions coerced paths via
Path(), corrupting s3:// URIs, and called .exists() which always returns
False for S3 (existence checks now use path_exists() from fs_utils)
Adds three private helpers:
- _list_tifs(): uses s3fs.S3FileSystem.glob for s3:// prefixes
- _coerce_input_path(): preserves str for s3://, returns Path otherwise
- _rasterio_env(): rasterio.Env(AWSSession) context for s3:// paths;
strips https:// from AWS_ENDPOINT_URL since GDAL expects hostname only
Tested: rasterio.open("s3://...") confirmed working on OVH S3 with
AWS_S3_ENDPOINT or AWS_ENDPOINT_URL set; hostname-only required by GDAL.
Refs #139
Co-authored-by: Claude Sonnet 4.6 <[email protected]>
…ix_set Per Emmanuel's review on #173: - vv/vh asset hrefs now point to {store}/{orbit} (the zarr group that carries multiscales), matching how S2 reflectance assets point to measurements/reflectance. TiTiler reads tile_matrix_set from that group's multiscales attributes. - create_s1_store now writes tile_matrix_set into the orbit group's multiscales so TiTiler can discover the tiling scheme without error. - Updated test_asset_hrefs to assert the new orbit-group href. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…atibility TiTiler's _validate_zarr checks ds.rio.crs on the native resolution group (r10m). Without proj:code in the resolution group's attrs, rioxarray returns None and the group is excluded → groups=[] → bounds unpack error. S2 zarr already has proj:code at the resolution group level; mirror that for S1. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…176) The deployed titiler-eopf reader resolves a group's CRS via rioxarray, which reads a CF spatial_ref/crs_wkt coordinate -- not the GeoZarr proj:code attr. S1 GRD RTC stores carried only proj:code, so rio.crs was None and every multiscale group was rejected (HTTP 500 on render/info). The S2 converter already writes a spatial_ref grid-mapping; this brings S1 in line. Also stop writing tile_matrix_set into the orbit-group multiscales: the data-model owner confirmed it is not part of the S1 GRD RTC data model, and the data_api reader already treats it as optional (MISSING default). - add _add_grid_mapping() using pyproj CRS.to_cf() (the same source rioxarray uses, so no hard-coded projection) and call it for every resolution level (both store-creation paths) and the conditions group - remove _create_tile_matrix_set() and its multiscales entry - tests: assert no tile_matrix_set, spatial_ref present + rio.crs resolves to the native EPSG, conditions group carries grid_mapping This fixes at the source what data-pipeline currently works around in ingest_v1_s1_rtc.py (_patch_cf_grid_mapping / _patch_tile_matrix_limits). Co-authored-by: Claude Opus 4.8 <[email protected]>
#178) * chore: release 0.9.0 * feat: implement scale-offset and data type casting via codecs * fix: fix dependency declaration * chore: use latest version of cast value * chore: make cast-value a project dependency * test: expand test coverage * fix: upgrade pytest to 9.0.3 (CVE-2025-71176) Fixes insecure /tmp directory handling on UNIX systems. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * - Use zarr-python's presumptive implementation of scale-offset - use scalar map for handling NaN - ensure that downsampled arrays use scalar map + cast value - improve tests across parametrization of relelvant functions * feat: add store-root spatial:bbox and tighten minispec requirements (#164) * feat: add store-root spatial:bbox and tighten minispec requirements Introduces a GeoZarr "Store Root" layer in the minispec so clients can read a summary footprint without walking into child groups, and tightens the multiscale profile so `spatial:bbox` at the root and `spatial:transform` + `spatial:shape` on every layout entry are mandatory. Adds a new `geozarr.store` pydantic module (`GeoZarrStoreAttrs`, `GeoZarrMultiscaleGroupAttrs`, `GeoZarr`) enforcing the tightened profile, and updates the S2 converter to union child-group bboxes into an EPSG:4326 footprint written at the store root. Closes #156. Addresses the clear parts of #163; the array-level and non-multiscale-group parts of that issue need further clarification. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat: update GeoZarr store model and optimize S2 converter for improved bbox handling * Addresses review feedback on #164: - Adds a "GeoZarr Hierarchy & Identification" subsection to the Store Root section of the minispec, codifying d-v-b's proposed rules: single root, root prefix ends with `.zarr`, suffix occurs at most once in the hierarchy, and explicit terminal-path conditions. - Promotes the store-root CRS to mandatory: at least one of `proj:code`, `proj:wkt2`, `proj:projjson` MUST be set; there is no implicit EPSG:4326 default. Per @vincentsarago's review. - Promotes `zarr_conventions` declaration at the store root from RECOMMENDED to required. - Converter now writes `proj:code: "EPSG:4326"` at the root alongside `spatial:bbox`. - Pydantic `GeoZarrStoreAttrs` enforces the new CRS-required rule. Cross-links the new hierarchy + spatial:extent follow-ups upstream: zarr-developers/geozarr-spec#132 (hierarchy & root identification, also addresses #124 URL parsing) and #133 (STAC-style spatial:extent). --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> * chore: uv.lock * refactor: use zarr-python 3.2.0 * chore: lockfile * fix: pin fill value based on data type instead of relying on xarray * test: update tests to consistently use nan fill value for floats * chore: bump urllib3 to 2.7.0 * chore: skip quicklook groups (#165) * chore: group dependabot updates for actions and pip (#160) * chore: group dependabot updates for actions and pip - Group all GitHub Actions bumps (minor, patch, major) into one PR - Group minor and patch pip bumps into one PR; major bumps remain ungrouped for individual review Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * chore: change dependabot schedule day to wednesday * ci: add comment_on option to security auditing action * ci: switch dependabot Python ecosystem from pip to uv uv ecosystem reads uv.lock directly, enabling Dependabot to raise PRs for lockfile-pinned versions including security patches. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> --------- Co-authored-by: Claude Sonnet 4.6 <[email protected]> * fix: derive coarse spatial transforms from coordinates (#168) * fix: derive coarse spatial transforms from coordinates * refactor: improve function definitions for clarity and consistency * chore: release 0.10.0 (#162) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix(converter): align convert layout to S2, strip _eopf_attrs, fix _FillValue (#171) - Remove TMS layout and /0 /1 /2 numeric overview groups from the general 'convert' CLI; produce S2-style sibling r{2**level} overviews instead. 'convert-s2-optimized' continues to emit the same layout. - Drop the '--tile-width' CLI option. - Sanitize array attrs in all converters: strip stale '_eopf_attrs' and source-only '_FillValue' / 'dtype' / 'valid_min' / 'valid_max' on decoded floats; rewrite 'units: digital_counts' -> 'units: 1'. The sanitizer now returns a new dict and callers reassign 'attrs', so stale keys are actually removed (a previous '.update()' pattern left them in place). - Sanitize coord attrs too (datetime coords in /conditions/meteorology were leaking '_eopf_decode_datetime64' via '_eopf_attrs'). - Set '_FillValue' properly for float measurements so xarray's CF encoder produces the base64-NaN representation needed for 'use_zarr_fill_value_as_mask=True' round-trip (xarray#11345). Fixes #171. - Remove dead 'utils.encode_cf_fill_value' (no callers). - Tests: open each r{N} multiscale level separately; migrate 'test_multiscales_round_trip' from deprecated tms.Multiscales to zcm.Multiscales (with tuplify_json to handle extra='allow' fields). - Regenerate snapshot fixtures; verified 0 '_eopf_attrs', 'tile_matrix*', or 'digital_counts' markers across S2A/S2B/S2C. * test: add guardrails for converter output attrs (#171) Walks the snapshotted GroupSpec JSON fixtures and asserts: - no '_eopf_attrs' anywhere; - no 'tile_matrix*' markers (TMS removal); - no 'units: digital_counts' on float arrays (decoded scale/offset); - every float array under '/measurements/' has '_FillValue' set (required for CF NaN-mask round-trip, xarray#11345). * docs: deprecate v0 references, document r{N} overview layout (#171) - Drop 'tile_width' parameter from API examples (option removed from CLI/API). - Replace '/measurements/r10m/{0,1,2}' nested-pyramid examples with the current flat 'r{2**level}' sibling-group layout. - Remove the V0 vs V1 split in converter.md / examples.md / quickstart.md; the general 'convert' command now produces the same flat layout as 'convert-s2-optimized'. The S2-optimized section is reframed as a feature description rather than a 'V1 vs V0' comparison. - Update architecture.md multiscales snippet to the new model ('layout' / 'asset' / 'derived_from' / 'transform'). - Update faq.md inspection snippet to walk the new layout. * refactor: clean up code formatting and enhance attribute handling in geozarr conversion * fix(cli-e2e): drop deprecated --tile-width flag from tests and example * refactor: remove deprecated v0 layout, sanitize attributes, and fix _FillValue handling * test: add _FillValue masking roundtrip test (#172) Addresses @vincentsarago's PR review: assert float arrays written with the converter's _FillValue convention round-trip through xarray's use_zarr_fill_value_as_mask=True so NaN cells come back masked. * test: drive _FillValue masking test through create_geozarr_dataset (#172) Replace the standalone xarray write with a real converter run: build a small float DataTree, invoke create_geozarr_dataset, then reopen the output with use_zarr_fill_value_as_mask=True and assert masking on the nodata patch. Exercises the converter's _FillValue + encoding path end-to-end, per @vincentsarago's review. * fix: green CI for #172 — S2 convert flat layout + idna CVE (#177) * fix(security): bump idna 3.13 -> 3.17 to clear CVE-2026-45409 pip-audit in the Security Audit workflow blocks on idna 3.13 (CVE-2026-45409, incomplete fix of CVE-2024-3651; fixed in 3.15). Bump the transitive pin via uv lock. pip-audit now reports no known vulnerabilities. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(cli): route Sentinel-2 convert through the optimized flat layout Generic `convert` emitted GeoZarr-0.4 multiscales with native data at the group root and overviews as nested r{2**N} sub-groups. That tree cannot be opened by xr.open_datatree (an overview child's x/y conflict with the parent's inherited x/y), so the `info` and `validate` CLI commands crashed on the converter's own Sentinel-2 output. Detect Sentinel-2 inputs and delegate to the existing, tested convert_s2_optimized path, which emits flat sibling r{N}m levels (the layout PR #172 documents). Non-S2 inputs keep the generic create_geozarr_dataset path. Detection is best-effort and never aborts conversion. Regenerate the geozarr_examples structure snapshots to the flat layout. Co-Authored-By: Claude Opus 4.8 <[email protected]> * test: make CLI e2e assertions debuggable and fix multiscale round-trip - The CLI logs errors to stdout via structlog; assert on stdout+stderr in test_cli_e2e so a non-zero exit shows the actual error instead of an empty stderr. - test_multiscale_attrs_round_trip tuplified only one side of its equality, so flat-layout level groups carrying list-valued attrs (e.g. spatial:bbox) broke a check that is only about JSON list/tuple normalisation. Tuplify both sides; the model still round-trips losslessly. Co-Authored-By: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Davis Vann Bennett <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: Emmanuel Mathot <[email protected]>
The previous S1 RTC preview rendered a single VH band as grayscale with an incorrect rescale (0,219), unsuitable for linear gamma0 RTC values. Declare a render-extension `renders.rgb` config producing the standard dual-pol false- colour composite (R=VV, G=VH, B=VV/VH ratio) with rescale 0–0.1 and tilesize 256, referencing the preferred orbit group. Downstream titiler-based services consume this to generate previews/tiles/tilejson. Co-Authored-By: Claude Opus 4.8 <[email protected]>
A single [min, max] pair applies to all bands in titiler, so emitting three identical pairs was redundant (and forced collapse logic in the consumer). Same rendered output. Co-Authored-By: Claude Opus 4.8 <[email protected]>
The registrar hardcoded the Sentinel-1 preview as a single VH band with an incorrect rescale (0,219), producing a near-black grayscale thumbnail unsuited to linear gamma0 RTC values. Make `add_visualization_links` and `add_thumbnail_asset` prefer a producer- declared `renders` config (STAC render extension) when present, falling back to the existing mission defaults otherwise. New `_select_render` picks the preferred render (rgb/visual/thumbnail/default) and `_render_to_query` converts it into a titiler query string (expression, rescale, bidx, tilesize, ...). Paired with EOPF-Explorer/data-model#180, which emits a `renders.rgb` dual-pol RGB composite (R=VV, G=VH, B=VV/VH ratio, rescale 0-0.1) on S1 RTC items, this yields the correct preview/tiles/tilejson without any per-mission code here. Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
Tracking a known follow-up surfaced by the 32TLR S1 RTC e2e (data-pipeline #226, item F1):
Not required to land this PR — captured here so it's tracked alongside the S1 RTC builder work. |
Brings in #184 (multi-frame masked-timestamp discovery) and the #173→#179 build_s1_rtc_stac_item revert from the s1-tiling line. Conflict in tests/test_s1_rtc_ingest.py resolved by keeping both newly-added discovery tests (s3:// listing + masked-stamp resolution); s1_ingest.py auto-merged (s3fs _list_tifs + the masked-stamp branch are compatible). Full test_s1_rtc_ingest.py suite passes (50).
…ender path (#246) titiler-eopf reconstructs the store path as s3://{bucket}/tests-output/{collection}/{item_id}.zarr and ignores the STAC asset href, so the GeoZarr cube must be named after the item id (s1-rtc-{tile}) for new tiles to preview. Parse the tile from the s1-rtc- prefix (was s1-grd-rtc-); the item id (s1-rtc-{tile}) is unchanged, so filename == item id by construction. Pairs with the data-pipeline direct-write change (store written at the tests-output path) which replaces the #250 auto-copy. Dev-phase, no consumers — disposable test data. Revert to s1-grd-rtc- when titiler-eopf#108 (resolve store from href) lands. Co-Authored-By: Claude Opus 4.8 <[email protected]>
build_s1_rtc_stac_item opened the cube with zarr.open_consolidated, which raises `ValueError: Consolidated metadata ... not found` when the store lacks root consolidated metadata. A per-tile cube grown by appending a time-slice to an existing same-orbit group ends up in exactly that state — re-consolidating an append on the S3 store is unreliable — so live STAC registration failed on the 2nd+ same-orbit acquisition for a tile (it broke the Pyrenees S1 RTC soak). The builder must not require consolidated metadata: titiler reads these stores fine without it. Fall back to a direct zarr.open_group(mode="r") read when the consolidated metadata is absent; re-raise any other ValueError. Adds a regression test: build from a 2-slice same-orbit store created with consolidate=False (fails before the fix in open_consolidated, passes after). Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
Commit |
…-builder # Conflicts: # .release-please-manifest.json # CHANGELOG.md # pyproject.toml # uv.lock
…ring (#192) (#193) Per-acquisition previews render via titiler `sel=time={INTEGER index}` — a positional index that breaks once a cube's time axis goes non-monotonic (a cross-run append of an earlier-dated scene; proved on 31TEH: `[06-08, 06-07]` → the 06-07 item has no preview). Keeping positional indices correct would need cube reorder + full re-registration on every append (not scalable). The deployed titiler (v0.5.0) already does label-based `da.sel(..., method=...)` via `open_datatree(decode_times=True)`; the only gap is the cube's `time` array, which is a bare int64 with no CF datetime metadata, so it can't be a datetime index. Encoding it lets previews select by exact datetime, which works even on a non-monotonic axis — no reorder, register only the new item. titiler picks a multiscale level by zoom (previews use a coarse level), so `time` must resolve there. Putting it only at the orbit-group level and inheriting fails to open while r10m stays bare int64 (AlignmentError: int64 vs datetime64 on the shared dim). So encode `time` consistently at EVERY level: - new `_create_time_coordinate_array` + `TIME_CF_ATTRS` (units/calendar/ standard_name); created at every level in create + append-group paths. - the append write resizes/writes `time` at every level (was r10m only). - absolute_orbit/relative_orbit/platform stay at r10m (not selected on). - stored dtype stays int64 ns, so register_per_acquisition's raw read is unaffected. Adds TestTimeCFDatetime incl. exact datetime `.sel` on a non-monotonic axis at the native and a coarse level. 55 passed. Refs #192 Co-authored-by: Claude Opus 4.8 <[email protected]>
|
Nudge to land this: the S1 RTC datetime-rendering work (validated end-to-end, 10/10 stable renders by exact datetime) currently pins eopf-geozarr to |
…brary (#195) (#196) * fix(s1-rtc): enrich cube STAC item + fix vv/vh asset model (#195) The S1 RTC cube STAC item carried duplicate vv/vh assets with byte-identical hrefs (bug #1), exposed no asset for the descending orbit group of a dual-orbit cube (bug #2), and was missing most of the metadata the S2 L2A reference carries. Rework `build_s1_rtc_stac_item`: - Asset model: replace the two indistinguishable vv/vh assets with one `gamma0-rtc-backscatter-{asc,desc}` asset per present orbit group, exposing VV/VH as named STAC 1.1 `bands` (no more duplicate hrefs), plus a `border-mask-{orbit}` asset for the valid-data mask. Each γ⁰ asset carries data_type/nodata/unit/gsd. - Identity: add `constellation`, `instruments: [c-sar]`, `gsd` (platform stays per-acquisition — a cube can mix S1A/S1C). - Projection: add proj:bbox, and proj:shape/proj:transform (best-effort from the r10m group attrs). - Datacube extension: cube:dimensions (time as a bounded extent — not a values list, the cube grows by appending — plus x/y) + cube:variables. - Descriptive: title/description; created (earliest acquisition, stable across rebuilds) / updated (build time) via the timestamps extension. - Fold the corrected render rescale (0.0,0.2) into `_rgb_render`. Out of scope (tracked follow-ups): statistics (#157), processing:software / DEM lineage, and per-product CDSE fields are not in the store; per-acquisition item construction is moved in a follow-up commit. Co-Authored-By: Claude Opus 4.8 <[email protected]> * feat(s1-rtc): move per-acquisition + coverage STAC construction into the library Consolidate the *pure* S1 RTC STAC construction in eopf_geozarr.stac.s1_rtc so item shape has a single owner (the library); registration (STAC-API upsert, S3 alternates, gateway rewrite, TiTiler links, CDSE discovery) stays in data-pipeline. Adds, ported from the data-pipeline scripts (deployment URLs stripped): - acquisition_id(tile_id, when) - build_s1_rtc_per_acquisition_items(store, *, orbit, collection_id) -> list[pystac.Item]: one single-datetime item per cube time slice of an orbit, reoriented to that orbit (sat:orbit_state + render expression + only that orbit's γ⁰/mask assets), per-slice platform, datacube structure dropped (a single acquisition is not a cube). Deployment-agnostic: carries the render config + datetime; the registration layer derives the cube-endpoint TiTiler links (sel=time={datetime}) from it. - Slice / pick_slice / slice_coverages: coverage-based preview-slice selection (reads border_mask at r720m). Ported the construction unit tests (pick_slice, slice_coverages, and the per-acquisition construction subset); the link/thumbnail/alternate-asset and rescale-override tests stay in data-pipeline as registration concerns. Follow-up (separate data-pipeline PR): bump the data-model pin, delete the moved functions from register_per_acquisition.py / register_v1_s1_rtc.py, and rewire them to consume this library output. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): per-acquisition items carry their own orbit footprint Review follow-up. A per-acquisition item cloned the cube base, so on a dual-orbit cube every item inherited the WGS84 union bbox/geometry of both orbits and the preferred (ascending) orbit's proj:bbox — advertising a footprint wider than, and a data bbox from the wrong orbit relative to, the acquisition's own orbit. Recompute bbox/geometry/proj:bbox from the run orbit's spatial:bbox (factored the bbox->polygon construction into `_bbox_to_geometry`, shared with the cube builder), and give per-acquisition items a single-acquisition description rather than the cube's "datacube" wording. Tests: per-acq footprint = run orbit (not the union); `created` = earliest acquisition (idempotent across rebuilds); ValueError on an invalid or absent orbit. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): omit orbit_state on dual-orbit cubes; expose datacube dimension sizes Two review observations from inspecting the live items: - sat:orbit_state is single-valued, so on a dual-orbit cube (both ascending and descending groups present) a single value mislabels half the slices. Set it (and declare the SAT extension) only for a single-orbit cube; per-acquisition items, which are single-orbit, still carry the real per-orbit value. Ensure per-acq items declare the SAT extension even when cloned from a dual-orbit base that omitted it. - cube:dimensions previously carried only `extent`, so the number of elements per dimension wasn't visible. The time axis is irregularly sampled, so list its discrete `values` (count = number of acquisitions); the regular x/y axes get a `step` (size derivable from extent; exact pixel count is also in proj:shape). Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): datacube time axis carries extent only (drop values enumeration) Enumerating every acquisition as cube:dimensions.time.values does not scale as the cube is appended over the mission. No STAC extension provides a scalable temporal element-count field (datacube temporal_dimension offers only values/step/extent, and step is null for S1's irregular sampling; projection proj:shape is spatial-only; raster has no dimension shape). So the time axis carries a bounded `extent` only. x/y element counts remain available via proj:shape (exact) and the datacube x/y extent+step. The number of acquisitions is obtained from the per-acquisition collection (STAC search count), which stays correct as the cube grows. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): correct datacube variable_type, platform value, created, time values Review of the live items surfaced several metadata issues: - cube:variables used `type` (a non-standard, silently-ignored key) instead of the datacube extension's `variable_type`; border_mask is now `auxiliary`, vv/vh `data`. - per-acquisition `platform` carried the store's short code (e.g. `s1a`); normalize to the STAC convention `sentinel-1a` (mirrors the Sentinel-2 reference). - `created` (timestamps ext) was set to an acquisition time, which misuses the field (it means the item's creation instant); omit it and keep `updated` = build time. - the cube's irregular time axis now lists its discrete `values` (the acquisition instants) so the number of time steps is visible; the regular x/y axes stay extent + step (their pixel count is in proj:shape). Co-Authored-By: Claude Opus 4.8 <[email protected]> * refactor(s1-rtc): robust EPSG parse for datacube reference_system + clarify per-acq proj inheritance Review follow-ups (no behaviour change for the EPSG:NNNN stores in use): - derive the datacube reference_system via pyproj.CRS(...).to_epsg() instead of splitting the proj:code string, so non-"EPSG:NNNN" CRS forms don't break it. - comment why per-acquisition items recompute proj:bbox from their orbit but inherit proj:code/shape/transform (the MGRS grid is shared across orbits). Co-Authored-By: Claude Opus 4.8 <[email protected]> * docs(s1-rtc): note the dual-orbit time axis spans both orbits in the datacube A dual-orbit cube merges two per-orbit sub-cubes (disjoint time axes) onto a shared grid. Modelling orbit as a dimension would imply a sparse, mostly-empty orbit×time grid; instead orbit stays an attribute of each acquisition, conveyed via the per-orbit assets. Add a `description` on the time dimension so the merged axis isn't misread. Single-orbit cubes get no note. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): distinct per-acquisition titles; clearer asset titles - per-acquisition items inherited the cube's "… — tile {id}" title, so every scene in the acquisitions collection was titled identically. Give each a title with its datetime + orbit so siblings are distinguishable. - simplify the border-mask asset title ("Valid-data mask") and make the zarr-store title say "Sentinel-1" for naming consistency. Co-Authored-By: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: Claude Opus 4.8 <[email protected]>
…v/vh pyramid (#197) * perf(s1-rtc): shard the conditions arrays like the vv/vh pyramid A real S1 RTC cube is 3807 objects / 3.5 GB, of which 3604 (94.7%) are the conditions/gamma_area_<relorbit> arrays: [10980,10980] float32, inner chunk 366², NO sharding_indexed -> ~900 tiny chunk objects each. They are time-invariant yet dominate the object count, which dominated the ingest's S3 upload wall-time (a live pod sat ~34 min in "Uploading store" at 9 millicores). The vv/vh/border_mask display pyramid is already sharded (one shard per time slice over the full spatial extent, inner 366²). Apply that same existing layout to the condition arrays: add shards=(h, w) to the one create_array in ingest_s1tiling_conditions. All condition arrays (gamma_area, lia, incidence_angle) share that write path and the same 2D full-resolution shape, so all collapse from ~900 chunk objects to 1 shard object (cube ~3807 -> ~210). calculate_aligned_chunk_size returns a divisor of the dimension, so (h, w) is a clean multiple of the inner chunk (Zarr v3 shard-divisibility). conditions arrays are NOT in TiTiler's render path (vv/vh/border_mask), so this does not touch the web-render layout; it only makes a client read a condition array in one ranged GET instead of ~900. Values are byte-identical. Tests: +2 (sharding codec present; 9 inner chunks -> 1 on-disk shard object + byte-identical roundtrip). 57 passed. Spec: claude-docs/specs/s1_gamma_area_sharding.md. Cross-repo Task T5 of data-pipeline/claude-docs/plans/s1_ingest_upload_perf.md. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_011LsWkVvRfkRzjqAMrzfmRP * docs(s1-rtc): record real-S3 sharding benchmark in the T5 spec Validated on the live OVH bucket (laptop->DE): object collapse 100->1 (prod ~900->1), PUT 1.7x faster even with batched concurrency on, divisibility valid at the production 10980² (aligned 366 divides 10980), reads byte-identical. Honest caveat recorded: a full-array sequential read is NOT faster sharded (same bytes, one un-parallelizable object) — the win is object-count (upload + listing) and windowed/partial cloud reads, not full-read throughput. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_011LsWkVvRfkRzjqAMrzfmRP * refactor(s1-rtc): trim the conditions-sharding comment to match vv/vh style Comment-only. The surrounding vv/vh sharding has no inline explainer; the long cloud-access rationale now lives in claude-docs/specs/s1_gamma_area_sharding.md. Keep only the non-obvious bits: why one shard, and the Zarr v3 shard-divisibility invariant. No behavior change (20 condition/shard tests green). Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_011LsWkVvRfkRzjqAMrzfmRP * chore(s1-rtc): drop the claude-docs spec from this PR The data-model repo has no claude-docs/specs convention; the spec was noise for this PR's reviewers. The problem statement, real-S3 benchmark and migration note live in the data-pipeline plan + tracking issue EOPF-Explorer/data-pipeline#288 and PR #197's description. Also drop the now-dangling spec path from the code comment (the rationale stays inline). No behavior change (20 condition/shard tests green). Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_011LsWkVvRfkRzjqAMrzfmRP --------- Co-authored-by: Claude Opus 4.8 <[email protected]>
… writer) (#200) * perf(s1-rtc): shard the conditions arrays like the vv/vh pyramid A real S1 RTC cube is 3807 objects / 3.5 GB, of which 3604 (94.7%) are the conditions/gamma_area_<relorbit> arrays: [10980,10980] float32, inner chunk 366², NO sharding_indexed -> ~900 tiny chunk objects each. They are time-invariant yet dominate the object count, which dominated the ingest's S3 upload wall-time (a live pod sat ~34 min in "Uploading store" at 9 millicores). The vv/vh/border_mask display pyramid is already sharded (one shard per time slice over the full spatial extent, inner 366²). Apply that same existing layout to the condition arrays: add shards=(h, w) to the one create_array in ingest_s1tiling_conditions. All condition arrays (gamma_area, lia, incidence_angle) share that write path and the same 2D full-resolution shape, so all collapse from ~900 chunk objects to 1 shard object (cube ~3807 -> ~210). calculate_aligned_chunk_size returns a divisor of the dimension, so (h, w) is a clean multiple of the inner chunk (Zarr v3 shard-divisibility). conditions arrays are NOT in TiTiler's render path (vv/vh/border_mask), so this does not touch the web-render layout; it only makes a client read a condition array in one ranged GET instead of ~900. Values are byte-identical. Tests: +2 (sharding codec present; 9 inner chunks -> 1 on-disk shard object + byte-identical roundtrip). 57 passed. Spec: claude-docs/specs/s1_gamma_area_sharding.md. Cross-repo Task T5 of data-pipeline/claude-docs/plans/s1_ingest_upload_perf.md. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_011LsWkVvRfkRzjqAMrzfmRP * docs(s1-rtc): record real-S3 sharding benchmark in the T5 spec Validated on the live OVH bucket (laptop->DE): object collapse 100->1 (prod ~900->1), PUT 1.7x faster even with batched concurrency on, divisibility valid at the production 10980² (aligned 366 divides 10980), reads byte-identical. Honest caveat recorded: a full-array sequential read is NOT faster sharded (same bytes, one un-parallelizable object) — the win is object-count (upload + listing) and windowed/partial cloud reads, not full-read throughput. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_011LsWkVvRfkRzjqAMrzfmRP * refactor(s1-rtc): trim the conditions-sharding comment to match vv/vh style Comment-only. The surrounding vv/vh sharding has no inline explainer; the long cloud-access rationale now lives in claude-docs/specs/s1_gamma_area_sharding.md. Keep only the non-obvious bits: why one shard, and the Zarr v3 shard-divisibility invariant. No behavior change (20 condition/shard tests green). Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_011LsWkVvRfkRzjqAMrzfmRP * chore(s1-rtc): drop the claude-docs spec from this PR The data-model repo has no claude-docs/specs convention; the spec was noise for this PR's reviewers. The problem statement, real-S3 benchmark and migration note live in the data-pipeline plan + tracking issue EOPF-Explorer/data-pipeline#288 and PR #197's description. Also drop the now-dangling spec path from the code comment (the rationale stays inline). No behavior change (20 condition/shard tests green). Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_011LsWkVvRfkRzjqAMrzfmRP * fix(s1-rtc): heal a multiscale level missing `time` on append (robust writer) `ingest_s1tiling_acquisition` resized `level["time"]` on every multiscale level, assuming a fresh build created `time` at each level (#192). A cube built before #192 -- or left half-built by an interrupted append -- can carry `r10m/time` yet lack it at a coarser level, so the resize raised `KeyError: 'time'` and, because the append consistency check validated only CRS + shape, the ingest was non-convergent (observed on 30TWM). Before the per-level write loop, recreate any missing-level `time` from `r10m/time` (backfilling the existing slices so prior timestamps are preserved), or raise a clear error when the cube is inconsistent in a way a backfill cannot fix (a level's length disagrees with `r10m/time`, or `r10m` has slices but no `time`). This is the durable upstream counterpart to the data-pipeline guard (data-pipeline #294), making that orchestration-side mitigation belt-and-suspenders. Tests: 4 new cases (heal, no-op when healthy, raise on half-built, raise on missing r10m/time); full s1_ingest suite 61 passed. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01V3qS75byrUuCSHFqcWi26B --------- Co-authored-by: Claude Opus 4.8 <[email protected]>
* fix(s1-rtc): write CF `_FillValue` on float arrays (data-model #172 parity) The S1 RTC ingest writer (`s1_ingest.py`) was never touched by #172, which added the CF `_FillValue` attribute to the S2 and S1-GRD paths so xarray can mask NaN nodata via `use_zarr_fill_value_as_mask=True` despite xarray #11345 — the zarr-level `fill_value` field alone is not surfaced through xarray's encoding. Consequently S1 RTC cubes carried only `grid_mapping` on their float bands, while S2 cubes carry `_FillValue` (`AAAAAAAA+H8=`) + standard_name + units, leaving S1 nodata unmaskable for generic xarray/CF consumers. Set the CF `_FillValue` (FillValueCoder-encoded, matching S2) plus standard_name and units on the float backscatter bands (vv/vh) at every multiscale level, and `_FillValue` on the float condition arrays (gamma_area, lia). The two near-identical vv/vh creation loops — new store vs. new orbit on an existing store — are unified into a shared `_create_band_arrays` helper so they can't drift again (the inline path previously lacked the metadata). Not ported (S2-specific or N/A to the RTC writer): the scale/offset + CastValue codec (S1 vv/vh are float32, unpacked), v0 deprecation (RTC is v1-only), and `sanitize_array_attrs` (s1_ingest writes controlled attrs zarr-direct and never copies source EOPF attrs). Tests: create-store unit, end-to-end ingest across both orbit-creation paths, and float-conditions coverage. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_011sBdb17MWSAn7VEPSnqKJU * refactor(s1-rtc): address review — round-trip test, orbit-builder dedup, typed attrs - Add an end-to-end NaN-masking round-trip test (open with `use_zarr_fill_value_as_mask=True`, `to_masked_array()`), mirroring the S2 guarantee in `test_array_attrs.py`. Proves `_FillValue` actually masks nodata, not just that the attribute is present. - Unify the two orbit-creation paths — new store (`create_s1_store`) and new orbit added to an existing store (`ingest_s1tiling_acquisition`) — into a shared `_build_orbit_group` helper. Removes ~60 lines of duplication and fixes the inline path's missing `proj:code` on level groups (guarded by a new test). Net: `s1_ingest.py` shrinks while gaining the fix. - Model the backscatter band CF attrs as a `S1BackscatterAttrsJSON` TypedDict (PR review request), alongside the existing `Standard*CoordAttrsJSON` types. - Document the internal `xarray.backends.zarr.FillValueCoder` import. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_011sBdb17MWSAn7VEPSnqKJU --------- Co-authored-by: Claude Opus 4.8 <[email protected]>
| if TYPE_CHECKING: | ||
| from pathlib import Path | ||
|
|
||
| CRS = "EPSG:32631" |
There was a problem hiding this comment.
would be good to associate special values like this with types. I wonder if we need a NewType for CRS strings.
There was a problem hiding this comment.
Agreed! I added CRSCode = NewType("CRSCode", str) with a validating make_crs_code
| from pathlib import Path | ||
|
|
||
| CRS = "EPSG:32631" | ||
| UTM_BBOX = [300000.0, 4900000.0, 400000.0, 5000000.0] |
There was a problem hiding this comment.
this pattern:
from typing import NewType
BoundingBox2D = NewType("BoundingBox2D", tuple[float, float, float, float])
def make_bounding_box(value: object) -> BoundingBox2D: ...ensures that we never confuse a bounding box with some other collection of numbers
There was a problem hiding this comment.
I adopted this pattern! thanks
…ent (#202) S1 RTC previews render the out-of-swath region as opaque black while the S2 reference renders it transparent. The cause is the *data*, not the metadata: #201 already gave vv/vh the same dtype/`fill_value`=NaN/ `_FillValue`/`grid_mapping` encoding as S2, but those are inert because no NaN is ever written — s1tiling stores `0.0` out of swath, and titiler treats `0` as valid data and paints it black. Mask nodata to NaN at the writer: - vv/vh: `np.where(border_mask == 0, NaN, ...)` — border_mask is the authoritative valid-data mask (0 = no-data). `_downsample_2d` already uses `np.nanmean` for floats, so NaN propagates to every overview level. - float conditions (gamma_area/lia): masked read off the GeoTIFF's declared nodata (border_mask is N/A for static conditions); a no-op when no nodata is declared. Only newly re-ingested cubes get NaN; existing cubes are remediated separately. Tests: NaN ⟺ border_mask==0 at native + overview levels, masking round-trips via `use_zarr_fill_value_as_mask`, conditions declared-nodata → NaN. Co-authored-by: Claude Opus 4.8 <[email protected]>
…d one (#203) S1 RTC cubes ended up with only one orbit consolidated on disk (staging: 30TWM asc✓/desc✗, 32TNN/30UVU asc✗/desc✓). Root cause: the pipeline ingests acquisitions one orbit at a time in separate pods, each of which strips all consolidated metadata (so `time` can resize), ingests its orbit, then calls `consolidate_s1_store(store, orbit_direction)` — which only re-consolidated the single orbit passed. So whichever orbit was ingested last is the only one left with on-disk consolidated metadata. Consolidate every orbit group present (iterate `root.groups()`), then the root. The minimal append-fetch already pulls all `zarr.json`, so both orbits' metadata is local — same reason the root consolidation already works. Signature unchanged (`orbit_direction` kept for logging/callers). Impact is cosmetic: the root consolidation is complete for both orbits and readers opening at the root get a synthesized child view (titiler renders the unconsolidated orbit fine). The fix matters for clients opening a single orbit group standalone (the STAC `<cube>.zarr/<orbit>` hrefs) with `consolidated=True`, which otherwise fall back to a listing. Test asserts each orbit group is consolidated *standalone* — checking via `root[orbit]` is a false-green because a consolidated root synthesizes the child's view. Existing cubes self-heal on their next re-ingest. Co-authored-by: Claude Opus 4.8 <[email protected]>
The render expression emits 3 bands (vv; vh; vv/vh) but supplied a single rescale pair [0,0.2] applied to all three. The vv/vh ratio (natural range ~1-15) saturated under that stretch -> flat blue/purple wash, and low cross-pol water dropped to transparent, which made (correctly geolocated) ascending swaths look mislocated. Give each band its own pair: vv [0,0.4], vh [0,0.1], ratio [1,15]. Cosmetic preview-only change; pixel placement is unchanged. Verified live against the raster API and the S1 STAC tests. Co-authored-by: Claude Opus 4.8 <[email protected]>
#205) Both the cube item and (by inheritance) its per-acquisition items now carry the STAC grid extension with grid:code = "MGRS-{tile}". This makes the tile a first-class queryable property: the acquisitions collection becomes natively tile-filterable in STAC Browser, and cube↔acquisition cross-links can filter on grid:code instead of an id-prefix LIKE. Cube-builder only — per-acquisition items inherit it via the existing build_s1_rtc_per_acquisition_items copy ({**base_dict} carries stac_extensions; the per-acq property denylist excludes grid:code). Claude-Session: https://claude.ai/code/session_019z3eVtkSNHhN9vHd8QqGcf Co-authored-by: Claude Opus 4.8 <[email protected]>
Aligns the S1 RTC STAC-builder branch with main via merge (no force-push). Resolved to exactly match the verified stac-builder-rebased tree: main's #199 content + the S1 tiling + RTC/STAC work, uv.lock regenerated with pystac. Co-Authored-By: Claude Fable 5 <[email protected]>
) PR #199 migrated the project from mypy to pyright, but the S1 RTC code was written under mypy with zarr type-checking disabled (ignore_missing_imports). Under pyright, zarr 3.x is fully typed, so every `group[name]` is a `zarr.Array | zarr.Group` union and the S1 code + tests surfaced ~1390 type errors (index/argument/attribute-access on the union). Resolve all of them with runtime narrowing, matching #199's idioms (no new `Any`, no blanket ignores): - s1_ingest.py: add `_child_group`/`_child_array` helpers (isinstance-or-raise, mirroring geozarr.py / s2_multiscale.py) used at every store member access; narrow `arr.metadata` to ArrayV3Metadata for `dimension_names`; guard the append-validation attrs chains with isinstance; give `_rasterio_env` a real return type instead of `# type: ignore[return]`. - s1_rtc.py: narrow the NotRequired TypedDict member accessors with `.get()` + explicit `raise KeyError` (behavior-identical); drop mypy-era `# type: ignore` comments pyright flags as unnecessary. - tests: add isinstance-narrowing `_group`/`_array` helpers and `np.asarray` around zarr reads; no assertion weakened. - pyz/v3.py: add `# type: ignore[type-var]` on the GroupSpec base (parity with pyz/v2.py:43). The S1 models are the first to subclass pyz.v3.GroupSpec under a type checker, so this intentional variance now needs the same suppression v2 already carries. Full-project pyright: 0 errors. No runtime behavior or on-disk output changes; full test suite green. Co-authored-by: Claude Fable 5 <[email protected]>
…180) Address the three open review threads from d-v-b on #180 (all typing): - "typeddictify this" (TIME_CF_ATTRS): add S1TimeCoordAttrsJSON next to its sibling S1BackscatterAttrsJSON and annotate the constant Final[...]. The call-site spread needs the same cast("dict", ...) idiom the backscatter attrs already use (zarr's attrs.update JSON union vs TypedDict). - NewType for CRS strings / BoundingBox2D: add CRSCode and BoundingBox2D NewTypes with validating make_crs_code / make_bounding_box constructors (accept object, validate at runtime) in eopf_geozarr.types, replacing the unchecked cast() reads of store attrs — corrupt attrs now fail loudly. Threaded through stac/s1_rtc.py (incl. a _OrbitInfo NamedTuple replacing the untyped per-orbit dict and its downstream cast cluster), conversion/s1_ingest.py (S1TilingMetadata.crs/bounds, _add_grid_mapping, conditions path), and the test constants the review anchored on. CRS validation is intentionally light (non-empty str); pyproj raises CRSError downstream on unparseable values. Bbox tuples convert via list(...) at pystac/JSON boundaries. - tests/test_types.py: unit tests for both constructors. Also fix four pre-existing ruff findings in s1_ingest.py that #215 (merged via GitHub, bypassing local hooks) left behind: RUF002, TC003, SIM108, RUF059 — required for pre-commit to pass on this file. No runtime behavior or on-disk output changes. Full-project pyright: 0 errors; full non-network suite: 531 passed. Co-Authored-By: Claude Fable 5 <[email protected]>
|
@emmanuelmathot are you happy for me to merge this in s1-tiling? so then I can prepare the PR to merge in main branch? |
…uilders (#216) * feat: Phase 0 — S1 GRD RTC design, S1Tiling integration, and real-data GeoTIFF→GeoZarr V3 prototype - Implementation plan with Zarr V3 store structure, conventions, and phased roadmap - S1Tiling Docker instructions with EODAG 4.0.0 patch (5 breaking changes fixed) - GeoTIFF inspector script for S1Tiling output validation - Synthetic prototype validating sharding, resize/append, overview generation - Real-data conversion: 3 acquisitions (31TCH) → 1.8 GB Zarr store with 6 overview levels - Key findings: chunk divisibility constraint, datetime format, coverage variability * phase 0: integrate all lessons learned into plan and docs - Update §2 array metadata example to actual zarr-python 3.1.1 on-disk format (sharding codec in codecs array, chunk_grid holds shard shape, inner chunks 366) - Update conditions group: per-orbit naming (gamma_area_{orbit}), mark LIA and incidence_angle as aspirational (not confirmed as S1Tiling outputs) - Add Operational Findings subsection: EODAG 4.0.0 patch, DEM tile extent, GAMMA_AREA filename format - Fix docker instructions: CALIBRATION tag is GammaNaughtRTC (not gamma_naught_rtc) - Mark Phase 0 checklist fully complete - Update §9 API corrections to reflect plan is now corrected * phase 1: S1 RTC Pydantic models aligned with S2 pattern - Add src/eopf_geozarr/data_api/s1_rtc.py — Zarr V3 Pydantic models for S1 GRD γ0T RTC GeoZarr stores, using pyz.v3 GroupSpec/ArraySpec with TypedDict members (same pattern as s2.py uses pyz.v2) - Models: S1RtcRoot, S1RtcOrbitGroup, S1RtcNativeResolutionDataset, S1RtcOverviewResolutionDataset, S1RtcConditionsGroup - Validation: convention UUIDs, spatial:dimensions, multiscales layout, required data arrays (vv/vh/border_mask), gamma_area presence - Add tests/_test_data/s1_rtc_examples/s1-grd-rtc-31TCH.json — realistic fixture with 3 timesteps, 6 overview levels, 3 gamma_area conditions - Add tests/test_data_api/test_s1_rtc.py — 11 tests: round-trip, structure validation, negative cases (missing orbit, r10m, UUIDs, etc.) - Add conftest fixture s1_rtc_json_example parametrized over all fixtures * refactor: improve Pydantic model definitions and streamline imports in S1 RTC module * docs: update Phase 1 checklist, add Phase 1 findings, link tracking issue #139 * fix: standardize spatial dimensions to lowercase in S1 RTC models and test cases * fix: add 1D spatial coordinate arrays and use lowercase dimension names in S1 GRD RTC prototype * feat: add detailed implementation plan for Phase 2 GeoTIFF ingestion pipeline * fix: update Phase 2 GeoTIFF ingestion plan to include 1D spatial coordinate arrays and enforce lowercase dimension names * feat: Phase 2 — S1 GRD RTC GeoTIFF ingestion pipeline Productionise the GeoTIFF → Zarr V3 ingestion pipeline for S1Tiling gamma-naught RTC outputs. New module: src/eopf_geozarr/conversion/s1_ingest.py (~500 lines) - extract_geotiff_metadata(): rasterio-based metadata extraction with tag validation and S1Tiling datetime normalisation - create_s1_store(): Zarr V3 store creation with full GeoZarr conventions (multiscales, proj:, spatial:), sharded arrays, 1D spatial coordinate arrays at every resolution level - ingest_s1tiling_acquisition(): main API — create-or-open store, read GeoTIFFs, generate overviews (average/nearest), write all levels, append coordinate variables. CRS/shape consistency validation on append. - consolidate_s1_store(): post-batch metadata consolidation - discover_s1tiling_acquisitions(): file discovery with grouping Tests: tests/test_s1_rtc_ingest.py (27 tests) - Metadata extraction, store creation, ingestion (create + append), data integrity, xarray roundtrip, CRS/shape mismatch rejection, consolidation, file discovery Uses zarr_cm CMO dicts for convention metadata (not hardcoded UUIDs). Reuses calculate_aligned_chunk_size() from utils.py. Exports wired in conversion/__init__.py. Refs: #139 * refactor: use zcm.Multiscales typed model per reviewer feedback - Replace dict[str, Any] multiscales field with zcm.Multiscales import - Remove inline MultiscalesTransform/ScaleLevel/Multiscales classes - Update spatial:dimensions validator to lowercase ['y', 'x'] (Python 3.12) - Remove typing_extensions import (native in Python 3.12) - Update test assertions for Pydantic model attribute access - Auto-format fixes from ruff (imports, fixture parens, line length) * fix: configure ruff TC001 for Pydantic runtime-evaluated base classes - Add [tool.ruff.lint.flake8-type-checking] runtime-evaluated-base-classes for pydantic.BaseModel so Pydantic field type imports aren't flagged - Remove 4 stale noqa comments auto-fixed by ruff * feat: implement S1Tiling conditions ingestion and discovery commands in CLI * feat(stac): add build_s1_rtc_stac_item and generate-stac-s1 CLI (#173) * feat(stac): add build_s1_rtc_stac_item and generate-stac-s1 CLI command Adds eopf_geozarr.stac.s1_rtc.build_s1_rtc_stac_item, which reads a consolidated S1 GRD RTC Zarr V3 store and returns a pystac.Item with SAR/SAT/projection extensions, WGS84 bbox derived via pyproj, and vv/vh asset sub-paths. Ascending orbit is preferred when both are present. Also adds the generate-stac-s1 CLI subcommand that prints the item as JSON. 8 unit tests cover all acceptance criteria from the plan. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix: support s3:// URIs in S1Tiling discovery and ingest (#175) Four functions in s1_ingest.py silently failed for s3:// paths: - discover_s1tiling_acquisitions / discover_s1tiling_conditions used pathlib.Path.glob(), which normalises s3:// to s3:/ and returns 0 matches - ingest_s1tiling_acquisition / ingest_s1tiling_conditions coerced paths via Path(), corrupting s3:// URIs, and called .exists() which always returns False for S3 (existence checks now use path_exists() from fs_utils) Adds three private helpers: - _list_tifs(): uses s3fs.S3FileSystem.glob for s3:// prefixes - _coerce_input_path(): preserves str for s3://, returns Path otherwise - _rasterio_env(): rasterio.Env(AWSSession) context for s3:// paths; strips https:// from AWS_ENDPOINT_URL since GDAL expects hostname only Tested: rasterio.open("s3://...") confirmed working on OVH S3 with AWS_S3_ENDPOINT or AWS_ENDPOINT_URL set; hostname-only required by GDAL. Refs #139 Co-authored-by: Claude Sonnet 4.6 <[email protected]> * fix(stac): point vv/vh asset hrefs to orbit group root; add tile_matrix_set Per Emmanuel's review on #173: - vv/vh asset hrefs now point to {store}/{orbit} (the zarr group that carries multiscales), matching how S2 reflectance assets point to measurements/reflectance. TiTiler reads tile_matrix_set from that group's multiscales attributes. - create_s1_store now writes tile_matrix_set into the orbit group's multiscales so TiTiler can discover the tiling scheme without error. - Updated test_asset_hrefs to assert the new orbit-group href. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(ingest): add proj:code to resolution group attrs for TiTiler compatibility TiTiler's _validate_zarr checks ds.rio.crs on the native resolution group (r10m). Without proj:code in the resolution group's attrs, rioxarray returns None and the group is excluded → groups=[] → bounds unpack error. S2 zarr already has proj:code at the resolution group level; mirror that for S1. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(s1): write CF grid_mapping, drop tile_matrix_set from RTC store (#176) The deployed titiler-eopf reader resolves a group's CRS via rioxarray, which reads a CF spatial_ref/crs_wkt coordinate -- not the GeoZarr proj:code attr. S1 GRD RTC stores carried only proj:code, so rio.crs was None and every multiscale group was rejected (HTTP 500 on render/info). The S2 converter already writes a spatial_ref grid-mapping; this brings S1 in line. Also stop writing tile_matrix_set into the orbit-group multiscales: the data-model owner confirmed it is not part of the S1 GRD RTC data model, and the data_api reader already treats it as optional (MISSING default). - add _add_grid_mapping() using pyproj CRS.to_cf() (the same source rioxarray uses, so no hard-coded projection) and call it for every resolution level (both store-creation paths) and the conditions group - remove _create_tile_matrix_set() and its multiscales entry - tests: assert no tile_matrix_set, spatial_ref present + rio.crs resolves to the native EPSG, conditions group carries grid_mapping This fixes at the source what data-pipeline currently works around in ingest_v1_s1_rtc.py (_patch_cf_grid_mapping / _patch_tile_matrix_limits). Co-authored-by: Claude Opus 4.8 <[email protected]> * Merge #172 (deprecate v0, fix fill values, sanitize attrs) into s1-rtc (#178) * chore: release 0.9.0 * feat: implement scale-offset and data type casting via codecs * fix: fix dependency declaration * chore: use latest version of cast value * chore: make cast-value a project dependency * test: expand test coverage * fix: upgrade pytest to 9.0.3 (CVE-2025-71176) Fixes insecure /tmp directory handling on UNIX systems. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * - Use zarr-python's presumptive implementation of scale-offset - use scalar map for handling NaN - ensure that downsampled arrays use scalar map + cast value - improve tests across parametrization of relelvant functions * feat: add store-root spatial:bbox and tighten minispec requirements (#164) * feat: add store-root spatial:bbox and tighten minispec requirements Introduces a GeoZarr "Store Root" layer in the minispec so clients can read a summary footprint without walking into child groups, and tightens the multiscale profile so `spatial:bbox` at the root and `spatial:transform` + `spatial:shape` on every layout entry are mandatory. Adds a new `geozarr.store` pydantic module (`GeoZarrStoreAttrs`, `GeoZarrMultiscaleGroupAttrs`, `GeoZarr`) enforcing the tightened profile, and updates the S2 converter to union child-group bboxes into an EPSG:4326 footprint written at the store root. Closes #156. Addresses the clear parts of #163; the array-level and non-multiscale-group parts of that issue need further clarification. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat: update GeoZarr store model and optimize S2 converter for improved bbox handling * Addresses review feedback on #164: - Adds a "GeoZarr Hierarchy & Identification" subsection to the Store Root section of the minispec, codifying d-v-b's proposed rules: single root, root prefix ends with `.zarr`, suffix occurs at most once in the hierarchy, and explicit terminal-path conditions. - Promotes the store-root CRS to mandatory: at least one of `proj:code`, `proj:wkt2`, `proj:projjson` MUST be set; there is no implicit EPSG:4326 default. Per @vincentsarago's review. - Promotes `zarr_conventions` declaration at the store root from RECOMMENDED to required. - Converter now writes `proj:code: "EPSG:4326"` at the root alongside `spatial:bbox`. - Pydantic `GeoZarrStoreAttrs` enforces the new CRS-required rule. Cross-links the new hierarchy + spatial:extent follow-ups upstream: zarr-developers/geozarr-spec#132 (hierarchy & root identification, also addresses #124 URL parsing) and #133 (STAC-style spatial:extent). --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> * chore: uv.lock * refactor: use zarr-python 3.2.0 * chore: lockfile * fix: pin fill value based on data type instead of relying on xarray * test: update tests to consistently use nan fill value for floats * chore: bump urllib3 to 2.7.0 * chore: skip quicklook groups (#165) * chore: group dependabot updates for actions and pip (#160) * chore: group dependabot updates for actions and pip - Group all GitHub Actions bumps (minor, patch, major) into one PR - Group minor and patch pip bumps into one PR; major bumps remain ungrouped for individual review Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * chore: change dependabot schedule day to wednesday * ci: add comment_on option to security auditing action * ci: switch dependabot Python ecosystem from pip to uv uv ecosystem reads uv.lock directly, enabling Dependabot to raise PRs for lockfile-pinned versions including security patches. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> --------- Co-authored-by: Claude Sonnet 4.6 <[email protected]> * fix: derive coarse spatial transforms from coordinates (#168) * fix: derive coarse spatial transforms from coordinates * refactor: improve function definitions for clarity and consistency * chore: release 0.10.0 (#162) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix(converter): align convert layout to S2, strip _eopf_attrs, fix _FillValue (#171) - Remove TMS layout and /0 /1 /2 numeric overview groups from the general 'convert' CLI; produce S2-style sibling r{2**level} overviews instead. 'convert-s2-optimized' continues to emit the same layout. - Drop the '--tile-width' CLI option. - Sanitize array attrs in all converters: strip stale '_eopf_attrs' and source-only '_FillValue' / 'dtype' / 'valid_min' / 'valid_max' on decoded floats; rewrite 'units: digital_counts' -> 'units: 1'. The sanitizer now returns a new dict and callers reassign 'attrs', so stale keys are actually removed (a previous '.update()' pattern left them in place). - Sanitize coord attrs too (datetime coords in /conditions/meteorology were leaking '_eopf_decode_datetime64' via '_eopf_attrs'). - Set '_FillValue' properly for float measurements so xarray's CF encoder produces the base64-NaN representation needed for 'use_zarr_fill_value_as_mask=True' round-trip (xarray#11345). Fixes #171. - Remove dead 'utils.encode_cf_fill_value' (no callers). - Tests: open each r{N} multiscale level separately; migrate 'test_multiscales_round_trip' from deprecated tms.Multiscales to zcm.Multiscales (with tuplify_json to handle extra='allow' fields). - Regenerate snapshot fixtures; verified 0 '_eopf_attrs', 'tile_matrix*', or 'digital_counts' markers across S2A/S2B/S2C. * test: add guardrails for converter output attrs (#171) Walks the snapshotted GroupSpec JSON fixtures and asserts: - no '_eopf_attrs' anywhere; - no 'tile_matrix*' markers (TMS removal); - no 'units: digital_counts' on float arrays (decoded scale/offset); - every float array under '/measurements/' has '_FillValue' set (required for CF NaN-mask round-trip, xarray#11345). * docs: deprecate v0 references, document r{N} overview layout (#171) - Drop 'tile_width' parameter from API examples (option removed from CLI/API). - Replace '/measurements/r10m/{0,1,2}' nested-pyramid examples with the current flat 'r{2**level}' sibling-group layout. - Remove the V0 vs V1 split in converter.md / examples.md / quickstart.md; the general 'convert' command now produces the same flat layout as 'convert-s2-optimized'. The S2-optimized section is reframed as a feature description rather than a 'V1 vs V0' comparison. - Update architecture.md multiscales snippet to the new model ('layout' / 'asset' / 'derived_from' / 'transform'). - Update faq.md inspection snippet to walk the new layout. * refactor: clean up code formatting and enhance attribute handling in geozarr conversion * fix(cli-e2e): drop deprecated --tile-width flag from tests and example * refactor: remove deprecated v0 layout, sanitize attributes, and fix _FillValue handling * test: add _FillValue masking roundtrip test (#172) Addresses @vincentsarago's PR review: assert float arrays written with the converter's _FillValue convention round-trip through xarray's use_zarr_fill_value_as_mask=True so NaN cells come back masked. * test: drive _FillValue masking test through create_geozarr_dataset (#172) Replace the standalone xarray write with a real converter run: build a small float DataTree, invoke create_geozarr_dataset, then reopen the output with use_zarr_fill_value_as_mask=True and assert masking on the nodata patch. Exercises the converter's _FillValue + encoding path end-to-end, per @vincentsarago's review. * fix: green CI for #172 — S2 convert flat layout + idna CVE (#177) * fix(security): bump idna 3.13 -> 3.17 to clear CVE-2026-45409 pip-audit in the Security Audit workflow blocks on idna 3.13 (CVE-2026-45409, incomplete fix of CVE-2024-3651; fixed in 3.15). Bump the transitive pin via uv lock. pip-audit now reports no known vulnerabilities. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(cli): route Sentinel-2 convert through the optimized flat layout Generic `convert` emitted GeoZarr-0.4 multiscales with native data at the group root and overviews as nested r{2**N} sub-groups. That tree cannot be opened by xr.open_datatree (an overview child's x/y conflict with the parent's inherited x/y), so the `info` and `validate` CLI commands crashed on the converter's own Sentinel-2 output. Detect Sentinel-2 inputs and delegate to the existing, tested convert_s2_optimized path, which emits flat sibling r{N}m levels (the layout PR #172 documents). Non-S2 inputs keep the generic create_geozarr_dataset path. Detection is best-effort and never aborts conversion. Regenerate the geozarr_examples structure snapshots to the flat layout. Co-Authored-By: Claude Opus 4.8 <[email protected]> * test: make CLI e2e assertions debuggable and fix multiscale round-trip - The CLI logs errors to stdout via structlog; assert on stdout+stderr in test_cli_e2e so a non-zero exit shows the actual error instead of an empty stderr. - test_multiscale_attrs_round_trip tuplified only one side of its equality, so flat-layout level groups carrying list-valued attrs (e.g. spatial:bbox) broke a check that is only about JSON list/tuple normalisation. Tuplify both sides; the model still round-trips losslessly. Co-Authored-By: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Davis Vann Bennett <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: Emmanuel Mathot <[email protected]> --------- Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Davis Vann Bennett <[email protected]> Co-authored-by: Emmanuel Mathot <[email protected]> * Revert "feat(stac): add build_s1_rtc_stac_item and generate-stac-s1 CLI (#173)" (#179) This reverts commit f749368ac3870790a69b250926f0af1aa6b58d0e. * fix(s1-ingest): discover multi-frame acquisitions with a masked timestamp (#183) (#184) discover_s1tiling_acquisitions returned 0 acquisitions for multi-frame S1Tiling products: their GeoTIFF filenames mask the acquisition time as '…txxxxxx', which the filename regex (\d{8}t\d{6}) rejected, so parse_s1tiling_filename returned None and every file was skipped — silently, with no error. - Relax the acq_stamp pattern to accept a masked time (\d{6}|x{6}), so masked files are no longer dropped. - In discover, when the stamp is masked, resolve the real YYYYMMDDtHHMMSS stamp from the GeoTIFF ACQUISITION_DATETIME tag (via extract_geotiff_metadata) and use it for grouping + downstream STAC datetime. This is the durable upstream fix for the data-pipeline #237 workaround (which renamed masked products before discovery); that workaround can now be retired. Closes #183. Co-authored-by: Claude Opus 4.8 <[email protected]> * chore: merge main into s1-tiling (#190) * chore: release 0.9.0 * feat: implement scale-offset and data type casting via codecs * fix: fix dependency declaration * chore: use latest version of cast value * chore: make cast-value a project dependency * test: expand test coverage * fix: upgrade pytest to 9.0.3 (CVE-2025-71176) Fixes insecure /tmp directory handling on UNIX systems. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * - Use zarr-python's presumptive implementation of scale-offset - use scalar map for handling NaN - ensure that downsampled arrays use scalar map + cast value - improve tests across parametrization of relelvant functions * feat: add store-root spatial:bbox and tighten minispec requirements (#164) * feat: add store-root spatial:bbox and tighten minispec requirements Introduces a GeoZarr "Store Root" layer in the minispec so clients can read a summary footprint without walking into child groups, and tightens the multiscale profile so `spatial:bbox` at the root and `spatial:transform` + `spatial:shape` on every layout entry are mandatory. Adds a new `geozarr.store` pydantic module (`GeoZarrStoreAttrs`, `GeoZarrMultiscaleGroupAttrs`, `GeoZarr`) enforcing the tightened profile, and updates the S2 converter to union child-group bboxes into an EPSG:4326 footprint written at the store root. Closes #156. Addresses the clear parts of #163; the array-level and non-multiscale-group parts of that issue need further clarification. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat: update GeoZarr store model and optimize S2 converter for improved bbox handling * Addresses review feedback on #164: - Adds a "GeoZarr Hierarchy & Identification" subsection to the Store Root section of the minispec, codifying d-v-b's proposed rules: single root, root prefix ends with `.zarr`, suffix occurs at most once in the hierarchy, and explicit terminal-path conditions. - Promotes the store-root CRS to mandatory: at least one of `proj:code`, `proj:wkt2`, `proj:projjson` MUST be set; there is no implicit EPSG:4326 default. Per @vincentsarago's review. - Promotes `zarr_conventions` declaration at the store root from RECOMMENDED to required. - Converter now writes `proj:code: "EPSG:4326"` at the root alongside `spatial:bbox`. - Pydantic `GeoZarrStoreAttrs` enforces the new CRS-required rule. Cross-links the new hierarchy + spatial:extent follow-ups upstream: zarr-developers/geozarr-spec#132 (hierarchy & root identification, also addresses #124 URL parsing) and #133 (STAC-style spatial:extent). --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> * chore: uv.lock * refactor: use zarr-python 3.2.0 * chore: lockfile * fix: pin fill value based on data type instead of relying on xarray * test: update tests to consistently use nan fill value for floats * chore: bump urllib3 to 2.7.0 * chore: skip quicklook groups (#165) * chore: group dependabot updates for actions and pip (#160) * chore: group dependabot updates for actions and pip - Group all GitHub Actions bumps (minor, patch, major) into one PR - Group minor and patch pip bumps into one PR; major bumps remain ungrouped for individual review Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * chore: change dependabot schedule day to wednesday * ci: add comment_on option to security auditing action * ci: switch dependabot Python ecosystem from pip to uv uv ecosystem reads uv.lock directly, enabling Dependabot to raise PRs for lockfile-pinned versions including security patches. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> --------- Co-authored-by: Claude Sonnet 4.6 <[email protected]> * fix: derive coarse spatial transforms from coordinates (#168) * fix: derive coarse spatial transforms from coordinates * refactor: improve function definitions for clarity and consistency * chore: release 0.10.0 (#162) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * enh: deprecate v0, fix fill values and sanitize attributes (#172) * fix(converter): align convert layout to S2, strip _eopf_attrs, fix _FillValue (#171) - Remove TMS layout and /0 /1 /2 numeric overview groups from the general 'convert' CLI; produce S2-style sibling r{2**level} overviews instead. 'convert-s2-optimized' continues to emit the same layout. - Drop the '--tile-width' CLI option. - Sanitize array attrs in all converters: strip stale '_eopf_attrs' and source-only '_FillValue' / 'dtype' / 'valid_min' / 'valid_max' on decoded floats; rewrite 'units: digital_counts' -> 'units: 1'. The sanitizer now returns a new dict and callers reassign 'attrs', so stale keys are actually removed (a previous '.update()' pattern left them in place). - Sanitize coord attrs too (datetime coords in /conditions/meteorology were leaking '_eopf_decode_datetime64' via '_eopf_attrs'). - Set '_FillValue' properly for float measurements so xarray's CF encoder produces the base64-NaN representation needed for 'use_zarr_fill_value_as_mask=True' round-trip (xarray#11345). Fixes #171. - Remove dead 'utils.encode_cf_fill_value' (no callers). - Tests: open each r{N} multiscale level separately; migrate 'test_multiscales_round_trip' from deprecated tms.Multiscales to zcm.Multiscales (with tuplify_json to handle extra='allow' fields). - Regenerate snapshot fixtures; verified 0 '_eopf_attrs', 'tile_matrix*', or 'digital_counts' markers across S2A/S2B/S2C. * test: add guardrails for converter output attrs (#171) Walks the snapshotted GroupSpec JSON fixtures and asserts: - no '_eopf_attrs' anywhere; - no 'tile_matrix*' markers (TMS removal); - no 'units: digital_counts' on float arrays (decoded scale/offset); - every float array under '/measurements/' has '_FillValue' set (required for CF NaN-mask round-trip, xarray#11345). * docs: deprecate v0 references, document r{N} overview layout (#171) - Drop 'tile_width' parameter from API examples (option removed from CLI/API). - Replace '/measurements/r10m/{0,1,2}' nested-pyramid examples with the current flat 'r{2**level}' sibling-group layout. - Remove the V0 vs V1 split in converter.md / examples.md / quickstart.md; the general 'convert' command now produces the same flat layout as 'convert-s2-optimized'. The S2-optimized section is reframed as a feature description rather than a 'V1 vs V0' comparison. - Update architecture.md multiscales snippet to the new model ('layout' / 'asset' / 'derived_from' / 'transform'). - Update faq.md inspection snippet to walk the new layout. * refactor: clean up code formatting and enhance attribute handling in geozarr conversion * fix(cli-e2e): drop deprecated --tile-width flag from tests and example * refactor: remove deprecated v0 layout, sanitize attributes, and fix _FillValue handling * test: add _FillValue masking roundtrip test (#172) Addresses @vincentsarago's PR review: assert float arrays written with the converter's _FillValue convention round-trip through xarray's use_zarr_fill_value_as_mask=True so NaN cells come back masked. * test: drive _FillValue masking test through create_geozarr_dataset (#172) Replace the standalone xarray write with a real converter run: build a small float DataTree, invoke create_geozarr_dataset, then reopen the output with use_zarr_fill_value_as_mask=True and assert masking on the nodata patch. Exercises the converter's _FillValue + encoding path end-to-end, per @vincentsarago's review. * fix: green CI for #172 — S2 convert flat layout + idna CVE (#177) * fix(security): bump idna 3.13 -> 3.17 to clear CVE-2026-45409 pip-audit in the Security Audit workflow blocks on idna 3.13 (CVE-2026-45409, incomplete fix of CVE-2024-3651; fixed in 3.15). Bump the transitive pin via uv lock. pip-audit now reports no known vulnerabilities. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(cli): route Sentinel-2 convert through the optimized flat layout Generic `convert` emitted GeoZarr-0.4 multiscales with native data at the group root and overviews as nested r{2**N} sub-groups. That tree cannot be opened by xr.open_datatree (an overview child's x/y conflict with the parent's inherited x/y), so the `info` and `validate` CLI commands crashed on the converter's own Sentinel-2 output. Detect Sentinel-2 inputs and delegate to the existing, tested convert_s2_optimized path, which emits flat sibling r{N}m levels (the layout PR #172 documents). Non-S2 inputs keep the generic create_geozarr_dataset path. Detection is best-effort and never aborts conversion. Regenerate the geozarr_examples structure snapshots to the flat layout. Co-Authored-By: Claude Opus 4.8 <[email protected]> * test: make CLI e2e assertions debuggable and fix multiscale round-trip - The CLI logs errors to stdout via structlog; assert on stdout+stderr in test_cli_e2e so a non-zero exit shows the actual error instead of an empty stderr. - test_multiscale_attrs_round_trip tuplified only one side of its equality, so flat-layout level groups carrying list-valued attrs (e.g. spatial:bbox) broke a check that is only about JSON list/tuple normalisation. Tuplify both sides; the model still round-trips losslessly. Co-Authored-By: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: Loïc Houpert <[email protected]> Co-authored-by: Claude Opus 4.8 <[email protected]> * ci(deps): bump the actions group across 1 directory with 8 updates (#167) Bumps the actions group with 8 updates in the / directory: | Package | From | To | | --- | --- | --- | | [actions/checkout](https://github.com/actions/checkout) | `5.0.1` | `6.0.2` | | [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) | `7.6.0` | `8.1.0` | | [actions/cache](https://github.com/actions/cache) | `5.0.4` | `5.0.5` | | [codecov/codecov-action](https://github.com/codecov/codecov-action) | `5.5.3` | `6.0.0` | | [googleapis/release-please-action](https://github.com/googleapis/release-please-action) | `4.4.0` | `5.0.0` | | [actions/upload-artifact](https://github.com/actions/upload-artifact) | `6.0.0` | `7.0.1` | | [actions/download-artifact](https://github.com/actions/download-artifact) | `7.0.0` | `8.0.1` | | [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) | `1.13.0` | `1.14.0` | Updates `actions/checkout` from 5.0.1 to 6.0.2 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/93cb6efe18208431cddfb8368fd83d5badbf9bfd...de0fac2e4500dabe0009e67214ff5f5447ce83dd) Updates `astral-sh/setup-uv` from 7.6.0 to 8.1.0 - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](https://github.com/astral-sh/setup-uv/compare/37802adc94f370d6bfd71619e3f0bf239e1f3b78...08807647e7069bb48b6ef5acd8ec9567f424441b) Updates `actions/cache` from 5.0.4 to 5.0.5 - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/668228422ae6a00e4ad889ee87cd7109ec5666a7...27d5ce7f107fe9357f9df03efb73ab90386fccae) Updates `codecov/codecov-action` from 5.5.3 to 6.0.0 - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/codecov/codecov-action/compare/1af58845a975a7985b0beb0cbe6fbbb71a41dbad...57e3a136b779b570ffcdbf80b3bdc90e7fab3de2) Updates `googleapis/release-please-action` from 4.4.0 to 5.0.0 - [Release notes](https://github.com/googleapis/release-please-action/releases) - [Changelog](https://github.com/googleapis/release-please-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/googleapis/release-please-action/compare/16a9c90856f42705d54a6fda1823352bdc62cf38...45996ed1f6d02564a971a2fa1b5860e934307cf7) Updates `actions/upload-artifact` from 6.0.0 to 7.0.1 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/b7c566a772e6b6bfb58ed0dc250532a479d7789f...043fb46d1a93c77aae656e7c1c64a875d1fc6a0a) Updates `actions/download-artifact` from 7.0.0 to 8.0.1 - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](https://github.com/actions/download-artifact/compare/37930b1c2abaa49bbe596cd826c3c89aef350131...3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c) Updates `pypa/gh-action-pypi-publish` from 1.13.0 to 1.14.0 - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](https://github.com/pypa/gh-action-pypi-publish/compare/ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e...cef221092ed1bacb1cc03d23a2d87d1d172e277b) --- updated-dependencies: - dependency-name: actions/cache dependency-version: 5.0.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: actions/checkout dependency-version: 6.0.2 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/download-artifact dependency-version: 8.0.1 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/upload-artifact dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: astral-sh/setup-uv dependency-version: 8.1.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: codecov/codecov-action dependency-version: 6.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: googleapis/release-please-action dependency-version: 5.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: pypa/gh-action-pypi-publish dependency-version: 1.14.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * deps(deps-dev): bump mypy from 1.20.2 to 2.1.0 (#170) Bumps [mypy](https://github.com/python/mypy) from 1.20.2 to 2.1.0. - [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md) - [Commits](https://github.com/python/mypy/compare/v1.20.2...v2.1.0) --- updated-dependencies: - dependency-name: mypy dependency-version: 2.1.0 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * deps(deps): bump the uv-minor-patch group across 1 directory with 4 updates (#174) Bumps the uv-minor-patch group with 4 updates in the / directory: [pydantic](https://github.com/pydantic/pydantic), [numpy](https://github.com/numpy/numpy), [cf-xarray](https://github.com/xarray-contrib/cf-xarray) and [pymdown-extensions](https://github.com/facelessuser/pymdown-extensions). Updates `pydantic` from 2.13.3 to 2.13.4 - [Release notes](https://github.com/pydantic/pydantic/releases) - [Changelog](https://github.com/pydantic/pydantic/blob/main/HISTORY.md) - [Commits](https://github.com/pydantic/pydantic/compare/v2.13.3...v2.13.4) Updates `numpy` from 2.4.4 to 2.4.6 - [Release notes](https://github.com/numpy/numpy/releases) - [Changelog](https://github.com/numpy/numpy/blob/main/doc/RELEASE_WALKTHROUGH.rst) - [Commits](https://github.com/numpy/numpy/compare/v2.4.4...v2.4.6) Updates `cf-xarray` from 0.11.0 to 0.11.1 - [Release notes](https://github.com/xarray-contrib/cf-xarray/releases) - [Commits](https://github.com/xarray-contrib/cf-xarray/compare/v0.11.0...v0.11.1) Updates `pymdown-extensions` from 10.21.2 to 10.21.3 - [Release notes](https://github.com/facelessuser/pymdown-extensions/releases) - [Commits](https://github.com/facelessuser/pymdown-extensions/compare/10.21.2...10.21.3) --- updated-dependencies: - dependency-name: pydantic dependency-version: 2.13.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: uv-minor-patch - dependency-name: numpy dependency-version: 2.4.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: uv-minor-patch - dependency-name: cf-xarray dependency-version: 0.11.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: uv-minor-patch - dependency-name: pymdown-extensions dependency-version: 10.21.3 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: uv-minor-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(deps): bump aiohttp to >=3.14.0 to resolve CVE-2026-34993 and CVE-2026-47265 (#182) The Security Audit job's pip-audit step flagged aiohttp 3.13.5 for CVE-2026-34993 and CVE-2026-47265 (both fixed in 3.14.0). Raise the floor and refresh the lock to 3.14.1. Co-authored-by: Claude Opus 4.8 <[email protected]> * chore: release 0.10.1 (#181) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * ci(deps): bump the actions group with 3 updates (#185) Bumps the actions group with 3 updates: [actions/checkout](https://github.com/actions/checkout), [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) and [codecov/codecov-action](https://github.com/codecov/codecov-action). Updates `actions/checkout` from 6.0.2 to 6.0.3 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/de0fac2e4500dabe0009e67214ff5f5447ce83dd...df4cb1c069e1874edd31b4311f1884172cec0e10) Updates `astral-sh/setup-uv` from 8.1.0 to 8.2.0 - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](https://github.com/astral-sh/setup-uv/compare/08807647e7069bb48b6ef5acd8ec9567f424441b...fac544c07dec837d0ccb6301d7b5580bf5edae39) Updates `codecov/codecov-action` from 6.0.0 to 6.0.1 - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/codecov/codecov-action/compare/57e3a136b779b570ffcdbf80b3bdc90e7fab3de2...e79a6962e0d4c0c17b229090214935d2e33f8354) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 6.0.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: astral-sh/setup-uv dependency-version: 8.2.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: codecov/codecov-action dependency-version: 6.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Davis Vann Bennett <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: Emmanuel Mathot <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(s1-rtc): STAC item builders + ingest fixes for TiTiler rendering (#180) * feat(stac): add build_s1_rtc_stac_item and generate-stac-s1 CLI command Adds eopf_geozarr.stac.s1_rtc.build_s1_rtc_stac_item, which reads a consolidated S1 GRD RTC Zarr V3 store and returns a pystac.Item with SAR/SAT/projection extensions, WGS84 bbox derived via pyproj, and vv/vh asset sub-paths. Ascending orbit is preferred when both are present. Also adds the generate-stac-s1 CLI subcommand that prints the item as JSON. 8 unit tests cover all acceptance criteria from the plan. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix: support s3:// URIs in S1Tiling discovery and ingest (#175) Four functions in s1_ingest.py silently failed for s3:// paths: - discover_s1tiling_acquisitions / discover_s1tiling_conditions used pathlib.Path.glob(), which normalises s3:// to s3:/ and returns 0 matches - ingest_s1tiling_acquisition / ingest_s1tiling_conditions coerced paths via Path(), corrupting s3:// URIs, and called .exists() which always returns False for S3 (existence checks now use path_exists() from fs_utils) Adds three private helpers: - _list_tifs(): uses s3fs.S3FileSystem.glob for s3:// prefixes - _coerce_input_path(): preserves str for s3://, returns Path otherwise - _rasterio_env(): rasterio.Env(AWSSession) context for s3:// paths; strips https:// from AWS_ENDPOINT_URL since GDAL expects hostname only Tested: rasterio.open("s3://...") confirmed working on OVH S3 with AWS_S3_ENDPOINT or AWS_ENDPOINT_URL set; hostname-only required by GDAL. Refs #139 Co-authored-by: Claude Sonnet 4.6 <[email protected]> * fix(stac): point vv/vh asset hrefs to orbit group root; add tile_matrix_set Per Emmanuel's review on #173: - vv/vh asset hrefs now point to {store}/{orbit} (the zarr group that carries multiscales), matching how S2 reflectance assets point to measurements/reflectance. TiTiler reads tile_matrix_set from that group's multiscales attributes. - create_s1_store now writes tile_matrix_set into the orbit group's multiscales so TiTiler can discover the tiling scheme without error. - Updated test_asset_hrefs to assert the new orbit-group href. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(ingest): add proj:code to resolution group attrs for TiTiler compatibility TiTiler's _validate_zarr checks ds.rio.crs on the native resolution group (r10m). Without proj:code in the resolution group's attrs, rioxarray returns None and the group is excluded → groups=[] → bounds unpack error. S2 zarr already has proj:code at the resolution group level; mirror that for S1. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(s1): write CF grid_mapping, drop tile_matrix_set from RTC store (#176) The deployed titiler-eopf reader resolves a group's CRS via rioxarray, which reads a CF spatial_ref/crs_wkt coordinate -- not the GeoZarr proj:code attr. S1 GRD RTC stores carried only proj:code, so rio.crs was None and every multiscale group was rejected (HTTP 500 on render/info). The S2 converter already writes a spatial_ref grid-mapping; this brings S1 in line. Also stop writing tile_matrix_set into the orbit-group multiscales: the data-model owner confirmed it is not part of the S1 GRD RTC data model, and the data_api reader already treats it as optional (MISSING default). - add _add_grid_mapping() using pyproj CRS.to_cf() (the same source rioxarray uses, so no hard-coded projection) and call it for every resolution level (both store-creation paths) and the conditions group - remove _create_tile_matrix_set() and its multiscales entry - tests: assert no tile_matrix_set, spatial_ref present + rio.crs resolves to the native EPSG, conditions group carries grid_mapping This fixes at the source what data-pipeline currently works around in ingest_v1_s1_rtc.py (_patch_cf_grid_mapping / _patch_tile_matrix_limits). Co-authored-by: Claude Opus 4.8 <[email protected]> * Merge #172 (deprecate v0, fix fill values, sanitize attrs) into s1-rtc (#178) * chore: release 0.9.0 * feat: implement scale-offset and data type casting via codecs * fix: fix dependency declaration * chore: use latest version of cast value * chore: make cast-value a project dependency * test: expand test coverage * fix: upgrade pytest to 9.0.3 (CVE-2025-71176) Fixes insecure /tmp directory handling on UNIX systems. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * - Use zarr-python's presumptive implementation of scale-offset - use scalar map for handling NaN - ensure that downsampled arrays use scalar map + cast value - improve tests across parametrization of relelvant functions * feat: add store-root spatial:bbox and tighten minispec requirements (#164) * feat: add store-root spatial:bbox and tighten minispec requirements Introduces a GeoZarr "Store Root" layer in the minispec so clients can read a summary footprint without walking into child groups, and tightens the multiscale profile so `spatial:bbox` at the root and `spatial:transform` + `spatial:shape` on every layout entry are mandatory. Adds a new `geozarr.store` pydantic module (`GeoZarrStoreAttrs`, `GeoZarrMultiscaleGroupAttrs`, `GeoZarr`) enforcing the tightened profile, and updates the S2 converter to union child-group bboxes into an EPSG:4326 footprint written at the store root. Closes #156. Addresses the clear parts of #163; the array-level and non-multiscale-group parts of that issue need further clarification. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat: update GeoZarr store model and optimize S2 converter for improved bbox handling * Addresses review feedback on #164: - Adds a "GeoZarr Hierarchy & Identification" subsection to the Store Root section of the minispec, codifying d-v-b's proposed rules: single root, root prefix ends with `.zarr`, suffix occurs at most once in the hierarchy, and explicit terminal-path conditions. - Promotes the store-root CRS to mandatory: at least one of `proj:code`, `proj:wkt2`, `proj:projjson` MUST be set; there is no implicit EPSG:4326 default. Per @vincentsarago's review. - Promotes `zarr_conventions` declaration at the store root from RECOMMENDED to required. - Converter now writes `proj:code: "EPSG:4326"` at the root alongside `spatial:bbox`. - Pydantic `GeoZarrStoreAttrs` enforces the new CRS-required rule. Cross-links the new hierarchy + spatial:extent follow-ups upstream: zarr-developers/geozarr-spec#132 (hierarchy & root identification, also addresses #124 URL parsing) and #133 (STAC-style spatial:extent). --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> * chore: uv.lock * refactor: use zarr-python 3.2.0 * chore: lockfile * fix: pin fill value based on data type instead of relying on xarray * test: update tests to consistently use nan fill value for floats * chore: bump urllib3 to 2.7.0 * chore: skip quicklook groups (#165) * chore: group dependabot updates for actions and pip (#160) * chore: group dependabot updates for actions and pip - Group all GitHub Actions bumps (minor, patch, major) into one PR - Group minor and patch pip bumps into one PR; major bumps remain ungrouped for individual review Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * chore: change dependabot schedule day to wednesday * ci: add comment_on option to security auditing action * ci: switch dependabot Python ecosystem from pip to uv uv ecosystem reads uv.lock directly, enabling Dependabot to raise PRs for lockfile-pinned versions including security patches. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> --------- Co-authored-by: Claude Sonnet 4.6 <[email protected]> * fix: derive coarse spatial transforms from coordinates (#168) * fix: derive coarse spatial transforms from coordinates * refactor: improve function definitions for clarity and consistency * chore: release 0.10.0 (#162) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix(converter): align convert layout to S2, strip _eopf_attrs, fix _FillValue (#171) - Remove TMS layout and /0 /1 /2 numeric overview groups from the general 'convert' CLI; produce S2-style sibling r{2**level} overviews instead. 'convert-s2-optimized' continues to emit the same layout. - Drop the '--tile-width' CLI option. - Sanitize array attrs in all converters: strip stale '_eopf_attrs' and source-only '_FillValue' / 'dtype' / 'valid_min' / 'valid_max' on decoded floats; rewrite 'units: digital_counts' -> 'units: 1'. The sanitizer now returns a new dict and callers reassign 'attrs', so stale keys are actually removed (a previous '.update()' pattern left them in place). - Sanitize coord attrs too (datetime coords in /conditions/meteorology were leaking '_eopf_decode_datetime64' via '_eopf_attrs'). - Set '_FillValue' properly for float measurements so xarray's CF encoder produces the base64-NaN representation needed for 'use_zarr_fill_value_as_mask=True' round-trip (xarray#11345). Fixes #171. - Remove dead 'utils.encode_cf_fill_value' (no callers). - Tests: open each r{N} multiscale level separately; migrate 'test_multiscales_round_trip' from deprecated tms.Multiscales to zcm.Multiscales (with tuplify_json to handle extra='allow' fields). - Regenerate snapshot fixtures; verified 0 '_eopf_attrs', 'tile_matrix*', or 'digital_counts' markers across S2A/S2B/S2C. * test: add guardrails for converter output attrs (#171) Walks the snapshotted GroupSpec JSON fixtures and asserts: - no '_eopf_attrs' anywhere; - no 'tile_matrix*' markers (TMS removal); - no 'units: digital_counts' on float arrays (decoded scale/offset); - every float array under '/measurements/' has '_FillValue' set (required for CF NaN-mask round-trip, xarray#11345). * docs: deprecate v0 references, document r{N} overview layout (#171) - Drop 'tile_width' parameter from API examples (option removed from CLI/API). - Replace '/measurements/r10m/{0,1,2}' nested-pyramid examples with the current flat 'r{2**level}' sibling-group layout. - Remove the V0 vs V1 split in converter.md / examples.md / quickstart.md; the general 'convert' command now produces the same flat layout as 'convert-s2-optimized'. The S2-optimized section is reframed as a feature description rather than a 'V1 vs V0' comparison. - Update architecture.md multiscales snippet to the new model ('layout' / 'asset' / 'derived_from' / 'transform'). - Update faq.md inspection snippet to walk the new layout. * refactor: clean up code formatting and enhance attribute handling in geozarr conversion * fix(cli-e2e): drop deprecated --tile-width flag from tests and example * refactor: remove deprecated v0 layout, sanitize attributes, and fix _FillValue handling * test: add _FillValue masking roundtrip test (#172) Addresses @vincentsarago's PR review: assert float arrays written with the converter's _FillValue convention round-trip through xarray's use_zarr_fill_value_as_mask=True so NaN cells come back masked. * test: drive _FillValue masking test through create_geozarr_dataset (#172) Replace the standalone xarray write with a real converter run: build a small float DataTree, invoke create_geozarr_dataset, then reopen the output with use_zarr_fill_value_as_mask=True and assert masking on the nodata patch. Exercises the converter's _FillValue + encoding path end-to-end, per @vincentsarago's review. * fix: green CI for #172 — S2 convert flat layout + idna CVE (#177) * fix(security): bump idna 3.13 -> 3.17 to clear CVE-2026-45409 pip-audit in the Security Audit workflow blocks on idna 3.13 (CVE-2026-45409, incomplete fix of CVE-2024-3651; fixed in 3.15). Bump the transitive pin via uv lock. pip-audit now reports no known vulnerabilities. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(cli): route Sentinel-2 convert through the optimized flat layout Generic `convert` emitted GeoZarr-0.4 multiscales with native data at the group root and overviews as nested r{2**N} sub-groups. That tree cannot be opened by xr.open_datatree (an overview child's x/y conflict with the parent's inherited x/y), so the `info` and `validate` CLI commands crashed on the converter's own Sentinel-2 output. Detect Sentinel-2 inputs and delegate to the existing, tested convert_s2_optimized path, which emits flat sibling r{N}m levels (the layout PR #172 documents). Non-S2 inputs keep the generic create_geozarr_dataset path. Detection is best-effort and never aborts conversion. Regenerate the geozarr_examples structure snapshots to the flat layout. Co-Authored-By: Claude Opus 4.8 <[email protected]> * test: make CLI e2e assertions debuggable and fix multiscale round-trip - The CLI logs errors to stdout via structlog; assert on stdout+stderr in test_cli_e2e so a non-zero exit shows the actual error instead of an empty stderr. - test_multiscale_attrs_round_trip tuplified only one side of its equality, so flat-layout level groups carrying list-valued attrs (e.g. spatial:bbox) broke a check that is only about JSON list/tuple normalisation. Tuplify both sides; the model still round-trips losslessly. Co-Authored-By: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Davis Vann Bennett <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: Emmanuel Mathot <[email protected]> * feat(s1-stac): add render extension with dual-pol RGB composite preview The previous S1 RTC preview rendered a single VH band as grayscale with an incorrect rescale (0,219), unsuitable for linear gamma0 RTC values. Declare a render-extension `renders.rgb` config producing the standard dual-pol false- colour composite (R=VV, G=VH, B=VV/VH ratio) with rescale 0–0.1 and tilesize 256, referencing the preferred orbit group. Downstream titiler-based services consume this to generate previews/tiles/tilejson. Co-Authored-By: Claude Opus 4.8 <[email protected]> * refactor(s1-stac): emit a single rescale pair in the RGB render A single [min, max] pair applies to all bands in titiler, so emitting three identical pairs was redundant (and forced collapse logic in the consumer). Same rendered output. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): TEMPORARY name store as s1-rtc-{tile} to match titiler render path (#246) titiler-eopf reconstructs the store path as s3://{bucket}/tests-output/{collection}/{item_id}.zarr and ignores the STAC asset href, so the GeoZarr cube must be named after the item id (s1-rtc-{tile}) for new tiles to preview. Parse the tile from the s1-rtc- prefix (was s1-grd-rtc-); the item id (s1-rtc-{tile}) is unchanged, so filename == item id by construction. Pairs with the data-pipeline direct-write change (store written at the tests-output path) which replaces the #250 auto-copy. Dev-phase, no consumers — disposable test data. Revert to s1-grd-rtc- when titiler-eopf#108 (resolve store from href) lands. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): build STAC item from a non-consolidated cube store build_s1_rtc_stac_item opened the cube with zarr.open_consolidated, which raises `ValueError: Consolidated metadata ... not found` when the store lacks root consolidated metadata. A per-tile cube grown by appending a time-slice to an existing same-orbit group ends up in exactly that state — re-consolidating an append on the S3 store is unreliable — so live STAC registration failed on the 2nd+ same-orbit acquisition for a tile (it broke the Pyrenees S1 RTC soak). The builder must not require consolidated metadata: titiler reads these stores fine without it. Fall back to a direct zarr.open_group(mode="r") read when the consolidated metadata is absent; re-raise any other ValueError. Adds a regression test: build from a 2-slice same-orbit store created with consolidate=False (fails before the fix in open_consolidated, passes after). Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): CF-encode `time` at every level for datetime-based rendering (#192) (#193) Per-acquisition previews render via titiler `sel=time={INTEGER index}` — a positional index that breaks once a cube's time axis goes non-monotonic (a cross-run append of an earlier-dated scene; proved on 31TEH: `[06-08, 06-07]` → the 06-07 item has no preview). Keeping positional indices correct would need cube reorder + full re-registration on every append (not scalable). The deployed titiler (v0.5.0) already does label-based `da.sel(..., method=...)` via `open_datatree(decode_times=True)`; the only gap is the cube's `time` array, which is a bare int64 with no CF datetime metadata, so it can't be a datetime index. Encoding it lets previews select by exact datetime, which works even on a non-monotonic axis — no reorder, register only the new item. titiler picks a multiscale level by zoom (previews use a coarse level), so `time` must resolve there. Putting it only at the orbit-group level and inheriting fails to open while r10m stays bare int64 (AlignmentError: int64 vs datetime64 on the shared dim). So encode `time` consistently at EVERY level: - new `_create_time_coordinate_array` + `TIME_CF_ATTRS` (units/calendar/ standard_name); created at every level in create + append-group paths. - the append write resizes/writes `time` at every level (was r10m only). - absolute_orbit/relative_orbit/platform stay at r10m (not selected on). - stored dtype stays int64 ns, so register_per_acquisition's raw read is unaffected. Adds TestTimeCFDatetime incl. exact datetime `.sel` on a non-monotonic axis at the native and a coarse level. 55 passed. Refs EOPF-Explorer/data-model#192 Co-authored-by: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): enrich STAC items & consolidate construction into the library (#195) (#196) * fix(s1-rtc): enrich cube STAC item + fix vv/vh asset model (#195) The S1 RTC cube STAC item carried duplicate vv/vh assets with byte-identical hrefs (bug #1), exposed no asset for the descending orbit group of a dual-orbit cube (bug #2), and was missing most of the metadata the S2 L2A reference carries. Rework `build_s1_rtc_stac_item`: - Asset model: replace the two indistinguishable vv/vh assets with one `gamma0-rtc-backscatter-{asc,desc}` asset per present orbit group, exposing VV/VH as named STAC 1.1 `bands` (no more duplicate hrefs), plus a `border-mask-{orbit}` asset for the valid-data mask. Each γ⁰ asset carries data_type/nodata/unit/gsd. - Identity: add `constellation`, `instruments: [c-sar]`, `gsd` (platform stays per-acquisition — a cube can mix S1A/S1C). - Projection: add proj:bbox, and proj:shape/proj:transform (best-effort from the r10m group attrs). - Datacube extension: cube:dimensions (time as a bounded extent — not a values list, the cube grows by appending — plus x/y) + cube:variables. - Descriptive: title/description; created (earliest acquisition, stable across rebuilds) / updated (build time) via the timestamps extension. - Fold the corrected render rescale (0.0,0.2) into `_rgb_render`. Out of scope (tracked follow-ups): statistics (#157), processing:software / DEM lineage, and per-product CDSE fields are not in the store; per-acquisition item construction is moved in a follow-up commit. Co-Authored-By: Claude Opus 4.8 <[email protected]> * feat(s1-rtc): move per-acquisition + coverage STAC construction into the library Consolidate the *pure* S1 RTC STAC construction in eopf_geozarr.stac.s1_rtc so item shape has a single owner (the library); registration (STAC-API upsert, S3 alternates, gateway rewrite, TiTiler links, CDSE discovery) stays in data-pipeline. Adds, ported from the data-pipeline scripts (deployment URLs stripped): - acquisition_id(tile_id, when) - build_s1_rtc_per_acquisition_items(store, *, orbit, collection_id) -> list[pystac.Item]: one single-datetime item per cube time slice of an orbit, reoriented to that orbit (sat:orbit_state + render expression + only that orbit's γ⁰/mask assets), per-slice platform, datacube structure dropped (a single acquisition is not a cube). Deployment-agnostic: carries the render config + datetime; the registration layer derives the cube-endpoint TiTiler links (sel=time={datetime}) from it. - Slice / pick_slice / slice_coverages: coverage-based preview-slice selection (reads border_mask at r720m). Ported the construction unit tests (pick_slice, slice_coverages, and the per-acquisition construction subset); the link/thumbnail/alternate-asset and rescale-override tests stay in data-pipeline as registration concerns. Follow-up (separate data-pipeline PR): bump the data-model pin, delete the moved functions from register_per_acquisition.py / register_v1_s1_rtc.py, and rewire them to consume this library output. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): per-acquisition items carry their own orbit footprint Review follow-up. A per-acquisition item cloned the cube base, so on a dual-orbit cube every item inherited the WGS84 union bbox/geometry of both orbits and the preferred (ascending) orbit's proj:bbox — advertising a footprint wider than, and a data bbox from the wrong orbit relative to, the acquisition's own orbit. Recompute bbox/geometry/proj:bbox from the run orbit's spatial:bbox (factored the bbox->polygon construction into `_bbox_to_geometry`, shared with the cube builder), and give per-acquisition items a single-acquisition description rather than the cube's "datacube" wording. Tests: per-acq footprint = run orbit (not the union); `created` = earliest acquisition (idempotent across rebuilds); ValueError on an invalid or absent orbit. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): omit orbit_state on dual-orbit cubes; expose datacube dimension sizes Two review observations from inspecting the live items: - sat:orbit_state is single-valued, so on a dual-orbit cube (both ascending and descending groups present) a single value mislabels half the slices. Set it (and declare the SAT extension) only for a single-orbit cube; per-acquisition items, which are single-orbit, still carry the real per-orbit value. Ensure per-acq items declare the SAT extension even when cloned from a dual-orbit base that omitted it. - cube:dimensions previously carried only `extent`, so the number of elements per dimension wasn't visible. The time axis is irregularly sampled, so list its discrete `values` (count = number of acquisitions); the regular x/y axes get a `step` (size derivable from extent; exact pixel count is also in proj:shape). Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): datacube time axis carries extent only (drop values enumeration) Enumerating every acquisition as cube:dimensions.time.values does not scale as the cube is appended over the mission. No STAC extension provides a scalable temporal element-count field (datacube temporal_dimension offers only values/step/extent, and step is null for S1's irregular sampling; projection proj:shape is spatial-only; raster has no dimension shape). So the time axis carries a bounded `extent` only. x/y element counts remain available via proj:shape (exact) and the datacube x/y extent+step. The number of acquisitions is obtained from the per-acquisition collection (STAC search count), which stays correct as the cube grows. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(s1-rtc): correct datacube variable_type, platform value, created, time values Review of the live items surfaced several metadata issues: - cube:variables used `type` (a non-standard, silently-ignored key) instead of the datacube extension's `variable_type`; border_mask is now `auxiliary`, vv/vh `data`. - per-acquisition `platform` carried the store's short code (e.g. `s1a`); normalize to the STAC convention `sentinel-1a` (mirrors the Sentinel-2 reference). - `created` (timestamps ext) was set to an acquisition time, which misuses the field (it means the item's creation instant); omit it and keep `updated` = build time. - the cube's irregular time axis now lists its discrete `values` (the acquisition instants) so the number of time steps is visible; the regular x/y axes stay extent + step (their pixel count is in proj:shape). Co-Authored-By: Claude Opus 4.8 <[email protected]> * refactor(s1-rtc): robust EPSG parse for datacube reference_system + clarify per-acq proj inheritance Review follow-ups (no behaviour change for the EPSG:NNNN stores in use): - derive the datacube reference_system via pyproj.CRS(...).to_epsg() instead of splitting the proj:code string, so non-"EPSG:NNNN" CRS forms don't break it. - comment why per-acquisition items recompute proj:bbox from their orbit but inherit proj:code/shape/transform (the MGRS grid is shared across orbits). Co-Authored-By: Claude Opus 4.8 <[email protected]> * docs(s1-rtc): note the dual-orbit time axis spans both orbits in the datacube A dual-orbit cube merges two per-orbit sub-cubes (disjoint time axes) onto a shared grid. Modelling orbit as a dimension would imply a sparse, mostly-empty orbit×time grid; instead orbit stays an attribute of each acquisition, conveyed via the per-orbit assets. Add a `description` on the time dimension so the merged axis isn't misread. Single-orbit cubes …
reopen #173
Summary
S1 GRD RTC STAC item generation, plus the ingest fixes needed for those items to render correctly in TiTiler.
STAC builders (
eopf_geozarr/stac/s1_rtc.py)build_s1_rtc_stac_item(zarr_store, collection_id): opens consolidated or non-consolidated cubes, unions the WGS84 bbox across orbit groups (ascending preferred for projection/render defaults), and applies the SAR, Projection, Render, Datacube, Timestamps and Grid extensions (SAT only when single-orbit).gamma0-rtc-backscatter-asc/-descassets with named VV/VHbands, a border-mask asset, and the store asset (fix(s1-rtc): enrich STAC items & consolidate construction into the library (#195) #196).grid:codeMGRS tile + Grid extension (feat(s1-rtc): grid:code (MGRS tile) + grid extension on S1 RTC STAC items #205).build_s1_rtc_per_acquisition_items,pick_slice,slice_coverages(S1 RTC STAC items: metadata incoherences & gaps vs the S2 L2A reference (cube + per-acquisition) #195/fix(s1-rtc): enrich STAC items & consolidate construction into the library (#195) #196) — registration stays in data-pipeline.s1-rtc-{tile}until titiler-eopf#108 lands.generate-stac-s1 --store <path> [--collection <id>].Ingest fixes (
conversion/s1_ingest.py)s3://URI support in discovery and ingest; CFgrid_mappingwritten andtile_matrix_setdropped;proj:codeon resolution groups; masked multi-frame timestamps handled (S1 RTC: discover_s1tiling_acquisitions returns 0 for multi-frame products (masked acq timestamp) #183).time(stored int64 ns) at every multiscale level so TiTiler selects acquisitions by exact datetime instead of a fragile positional index — robust to non-monotonic time axes from cross-run appends, so registration stays incremental (S1 RTC per-acquisition previews break on non-monotonic time axis — render by datetime, not index #192/fix(s1-rtc): CF-encodetimeat every level for datetime-based rendering #193).NaN, not0, so TiTiler masks out-of-swath transparent (fix(s1-rtc): store nodata as NaN, not 0, so titiler masks it transparent #202), with CF_FillValueon float arrays (fix(s1-rtc): write CF_FillValueon float arrays (#172 parity) #201);np.nanmeancarriesNaNto every overview. Affects newly re-ingested cubes only.timeon append (fix(s1-rtc): heal a multiscale level missingtimeon append (robust writer) #200); shards the conditions arrays like the vv/vh pyramid (perf(s1-rtc): shard the conditions arrays (gamma_area/LIA) like the vv/vh pyramid #197).Typing
S1TimeCoordAttrsJSONTypedDict for the CF time attrs;CRSCode/BoundingBox2DNewTypes with validatingmake_*constructors replacing unchecked casts of store attrs.Dependencies: adds
pystac>=1.8.0.For reviewers
9ebad3e(TypedDict + NewTypes as suggested). Behavior-identical; the constructors validate at runtime where the casts didn't. CRS validation is intentionally light (non-empty string) — pyproj raisesCRSErrordownstream on unparseable values.57592cd,9ebad3e) are intended behavior-unchanged: no on-disk store output or STAC JSON changes.Author attestation
Implementation and tests were AI-assisted (Claude Code); every change was reviewed and verified against the full test suite, the pre-commit gates (ruff, pyright), and a live TiTiler end-to-end validation.