Repository navigation
Description
Symptom
build_s1_rtc_stac_item() (src/eopf_geozarr/stac/s1_rtc.py) opens the cube with zarr.open_consolidated(zarr_store, zarr_format=3), which raises:
ValueError: Consolidated metadata requested with 'use_consolidated=True' but not found in ''.
In the live S1 GRD RTC pipeline this fails STAC registration (register_v1_s1_rtc.py / register_per_acquisition.py), so the cube item never registers / the previous item is left deleted.
Trigger (reproduced)
The per-tile cube (s1-rtc-{tile}.zarr) accumulates one time slice per acquisition. The failure occurs specifically when a slice is appended to an existing same-orbit group (e.g. the 2nd+ ascending acquisition for a tile):
| Scenario | Result |
|---|---|
| Fresh single-acquisition store | ✅ builds |
| Append a new orbit group (descending into an ascending store) | ✅ builds |
| Append a slice to an existing same-orbit group (2nd ascending) | ❌ ValueError: Consolidated metadata ... not found |
Observed live on staging: 30TWM (descending appended into an ascending store) built fine; 31TEH (ascending appended into an existing ascending store) failed; a clean-slate 31TEH (fresh store) built fine.
Why it happens
The data-pipeline ingest does call consolidate_s1_store() after every acquisition (ingest_v1_s1_rtc.py:189,195 → s1_ingest.py:688-700, which consolidates the orbit group then the root). Yet after a same-orbit append on the S3 store, the root is read back without consolidated metadata, so open_consolidated raises. titiler reads the very same stores fine (its /info returns 200) because it does not require consolidated metadata.
Fix (done)
The builder must not require consolidated metadata — fall back to a direct zarr.open_group(zarr_store, mode="r", zarr_format=3) when it's absent (re-raising any other ValueError). Shipped in PR #180 (commit bbd18e3) with a regression test (tests/test_s1_stac.py::test_builds_from_non_consolidated_store: a 2-slice same-orbit store built with consolidate=False — fails before the fix, passes after).
Open sub-question (separate, not blocking)
Why does consolidate_s1_store() not leave the root consolidated after an S3 same-orbit append, despite being called? Likely S3 zarr-store consolidation behavior (note geozarr.py:1235 checks store.supports_consolidated_metadata). Worth tracking separately — the builder-tolerant fix unblocks production regardless, but losing consolidated metadata on appends forfeits its read optimization for all consumers that rely on it.
Activity
emmanuelmathot commentedon Sep 28, 2026
Closing: the fix is on
mainsince #180. The STAC builder opens the cube withzarr.open_group(..., use_consolidated=None), so it uses consolidated metadata when present and lists the store otherwise.tests/test_s1_stac.py::test_builds_from_non_consolidated_storecovers the same-orbit append case.@lhoupert, the side question remains: why
consolidate_s1_store()does not leave the root consolidated after a same-orbit append on S3. Do you still see it in production? If yes, please open a separate issue for it.