Skip to content

fix(s1-rtc): prepare the registration and validation paths for the data-model v0.11.0 pin - #413

Merged
lhoupert merged 12 commits into
mainfrom
fix/s1-rtc-pin-prep
Sep 21, 2026
Merged

lhoupert merged 12 commits into
mainfrom
fix/s1-rtc-pin-prep

Conversation

@lhoupert

Copy link
Copy Markdown
Collaborator

Four fixes that must land before the eopf-geozarr pin moves to v0.11.0 (EOPF-Explorer/data-model#216): the collection-render test dropped assets from one side of its comparison only; three registration paths and two catalog migrations still read renders from properties after the render extension moved it to the item root; --tile-id reached the S3 store path unvalidated, so a typo left an orphaned cube behind a multi-hour ingest; and the structural gate tolerated drift by coordinate name, which already makes it report FAIL for every current-generation store.

For reviewers: verified against both library revisions — 1129 unit tests pass under the currently pinned 9ede8c3 and under the #216 tip. The 16 remaining failures are pre-existing (mgrs not installed, gen_aoi_tiles fixtures) and identical on main. Seven further pin-bump hazards surfaced while doing this and are deliberately not addressed here — the sharpest is s1_store_meta.PINNED_EOPF_GEOZARR_VERSION, which both gates the conformance migration and names its completion marker, so a version bump re-derives every already-migrated cube.

Author attestation

  • I am a human, these are my changes, and I have reviewed and understood every change and can explain why each is correct.

AI-assisted: implementation and tests drafted with Claude Code and reviewed by two independent agents; every behavioural claim was checked by running it against real S1 RTC stores built with both library revisions.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CHpM7QqgsF4jzhPxUxdU7V

lhoupert and others added 4 commits September 11, 2026 12:07
…comparison

`test_collection_render_tracks_the_builder` stripped `assets` from our dict only. That held
while upstream omitted it; data-model #216 adds `assets` to every render (the render
extension's one required field), after which the two dicts can never compare equal and the
test fails the moment the pin moves.

Verified against both data-model revisions: fails on the current tree with the new library,
passes with the fix under the pinned 9ede8c3 and the PR #216 tip alike.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01CHpM7QqgsF4jzhPxUxdU7V
The render extension requires `renders` on the item object; data-model #216 moved it there and
writes a temporary `properties` mirror so this repo keeps working across the relocation. Three
consumers read the old location: `register_per_acquisition` subscripted it (a hard KeyError once
the mirror goes), `register_v1_s1_rtc` and `register_v1` used `.get` and would silently lose the
config — which turns `_reorient_item_to_orbit` into a no-op and ships the preferred orbit's render
on every dual-orbit cube.

One helper, `register_v1.renders_blocks`, returns every live copy, root first. Reading the first
survives the mirror's removal and still reads items registered before the move; rewriting *all* of
them is what `_reorient_item_to_orbit` needs, so the root and the mirror cannot disagree.

Also corrects two comments that still described the single `(0.0, 0.2)` rescale the builder
replaced with a per-band stretch.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01CHpM7QqgsF4jzhPxUxdU7V

Two more silent readers of the old location go with it: the catalog migrations
`align_visualization_links` and `add_xyz_link` in operator-tools, which on a miss report
"nothing to align" rather than failing.

`_reorient_item_to_orbit` now also moves `renders.rgb.assets`, which #216 added and which names
one orbit's γ⁰ asset: reorienting the expression alone left the descending preview advertising
the ascending asset on every dual-orbit cube.
`--tile-id` had no validation: the cube path is built straight from it, and the STAC builder only
rejects a bad store name at register time — i.e. after a multi-hour ingest has already written
`s1-rtc-<typo>.zarr`. data-model #216 makes that rejection a hard raise, so a typo leaves an
orphaned cube on S3. Validate it where the collection is already validated, before any subprocess.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01CHpM7QqgsF4jzhPxUxdU7V
…nate name

Two defects in the structural gate, both reproduced against real stores built by the pinned
writer:

* `time` was missing from the tolerated coordinate names, so a CURRENT-generation store reported
  FAIL — five `extra_forbidden` errors, one per overview level. The gate has been crying wolf on
  every store it validates.
* after the pin bump a genuinely pre-#192 cube (no `time` on r20m..r720m — the ~170 published
  ones) fails a different way, `value_error` from the #192 coordinate check, which the name-based
  allowlist cannot see: WARN today, FAIL tomorrow.

Tolerance is now keyed to `eopf:writer_schema`, written at store creation from #216 on. Absent =
pre-#216 cube, so known drift is WARN; present = a generation that emits all of this, so any
validation error is a real defect. Verified across the matrix (current-gen, pre-#192, pre-#192 +
stamp) against both the pinned data-model and the PR #216 tip.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01CHpM7QqgsF4jzhPxUxdU7V

The tolerated coordinate-check failures are matched in full, not by the shared "must contain
coordinate arrays" wording: the library raises that for four group kinds, and a substring match
also swallowed "Native resolution dataset must contain coordinate arrays ['x', 'y',
'spatial_ref']" — a cube with no georeferencing at all — passing it as WARN on every unstamped
store, i.e. the whole published archive.
lhoupert added a commit to EOPF-Explorer/data-model that referenced this pull request Sep 11, 2026
Both comments told a future maintainer to delete `properties.renders` once data-pipeline read the
item root. Following that would have removed every render/viewer/tilejson/thumbnail link the
pipeline derives, and made `register_per_acquisition` raise.

The mirror exists because of the API, not because of data-pipeline. The deployed STAC API models
its write body as `Item-Input`: ten root fields, `renders` not among them, and no
`additionalProperties` — while `ItemProperties` and `Asset` in the same OpenAPI document do allow
extras. Pydantic drops what it does not model, so only `properties.renders` is expected to survive
a POST/PUT. Both copies are therefore load-bearing: the root one makes the item satisfy the
extension it declares, the mirror is the one that comes back out of the catalogue.

Basis: the deployed API's published OpenAPI (`POST /collections/{collection_id}/items` takes
`anyOf[Item-Input, ItemCollection-Input]`, `PUT …/items/{item_id}` takes `Item-Input`), plus an
offline repro through `stac_pydantic.Item`. Not confirmed against a real POST — and note that live
items showing only `properties.renders` is NOT evidence, since no item carrying a root copy has
ever been registered. Consumers should read the root first and fall back to `properties`, which
EOPF-Explorer/data-pipeline#413 does.

Comments only — no behaviour change.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01CHpM7QqgsF4jzhPxUxdU7V
lhoupert and others added 2 commits September 18, 2026 13:07
…rvive it

data-model #216 is merged and v0.11.0 released, so the stand-in SHA pin can go.
Pin the TAG, not the merge SHA: `__version__` comes from `pyproject.toml` and the
migration's writer guard compares against it. `uv.lock` keeps `rev=v0.11.0` plus
the resolved `7d662ed`, so builds stay reproducible.

The bump cannot land on its own -- staging only the pin turns all 13 migration
tests red, because `assert_writer_pinned()` still demands 0.10.2. Two further
things had to move with it.

Writer pin. The earlier re-pins were justified by `s1_ingest.py` being
byte-identical to the previously validated writer. That is no longer true (+679
/-92 lines), so the argument is narrowed to the symbols the re-derive actually
borrows, each checked at v0.11.0: `OVERVIEW_CHAIN` identical,
`FLOAT32_NAN_FILL_VALUE`/`BACKSCATTER_CF_ATTRS` identical, and
`_downsample_2d`'s "average" path byte-identical -- v0.11.0 only *adds* a "max"
branch and a ValueError on unknown methods. Since a version string cannot tell
"added a branch" from "changed the branch I depend on", the guard now also
probes `_downsample_2d` numerically, with a NaN in the fixture so it pins
`nanmean` rather than `mean`.

Completion marker. It was `eopf_geozarr.__version__`, which coupled "has this
store been migrated?" to "which release am I running?". Moving the pin would
have stopped every already-migrated store's marker matching and re-derived the
whole fleet's bulk vv/vh for nothing. The marker is now a stable token, and the
version-valued markers older runs wrote are still accepted.

Store root. From 0.11.0 the library's `consolidate_s1_store` also rewrites the
store ROOT (proj:code -> EPSG:4326, lon/lat spatial:bbox, zarr_conventions).
This script runs against s3:// production cubes and its contract is "re-derive
vv/vh, touch nothing else", so that is now opt-in behind `--rewrite-root-geo`.
Left off, consolidation is plain `zarr.consolidate_metadata` over every orbit
group and the root, which is all #203 ever asked for.

Also corrects the module docstring's "reproduces exactly what a fresh re-ingest
would write": from 0.11.0 the writer builds overview `border_mask` levels with
block-max while migrated cubes keep `nearest`, and this migration deliberately
never touches `border_mask`.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
From eopf-geozarr 0.11.0 `ingest_s1tiling_acquisition` refuses an append whose
`time` precedes the cube's last slice, and it grew `allow_out_of_order` to say
otherwise. `ingest_v1_s1_rtc.py` never passed it, so two legitimate cases became
hard failures of the Argo step:

  - backfilling an older scene into an existing cube;
  - appending at all to a cube built by the OLD discovery order, which sorted by
    (platform, tile, orbit_dir, rel_orbit, acq_stamp). Platform dominated the
    timestamp, so every cube built from a mixed S1A/S1C archive is non-
    chronological and each future append is out-of-order by construction --
    i.e. permanently un-appendable without this flag.

Plumbed as `--allow-out-of-order` through `run_ingest` and `ingest_all`, default
off, so an unexpected inversion in a routine cron run still fails loudly: there
it means the discovery order moved under us, which is worth stopping for. The
existing call-arg assertions now pin the default explicitly.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The RC tag `v0.8.0-s1rtc-rc3` predates the v0.11.0 pin, so it carries neither the
stable completion marker nor `--rewrite-root-geo`; running it now would re-derive
the fleet against the old writer guard. `type=sha` in build.yml publishes
`sha-<short>` for every pushed commit, so the commit-sha tag names this exact
tree without cutting a release. Taken from the build log of run 35343077824, not
constructed by hand.

Also exposes `rewrite_root_geo` as a workflow parameter, defaulting to "false" —
without it the new flag is unreachable from the deployed workflow, which would
make the opt-in gate decorative.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`sha-38966e3` (previous commit) names an image that does not exist. `PUSH_IMAGE`
in .github/workflows/build.yml is false for a plain branch push, so that build's
`sha-<head>` tag is produced inside the runner and discarded — it never reaches
the OVH registry. Verified: the push-event run for this branch skips the "Push
Docker image" step entirely.

Only the pull_request build publishes, and docker/metadata-action's `type=sha`
there names the synthesised MERGE commit (`sha-6163331`), which is not in git
history and is replaced on every push. So no immutable commit-sha tag exists for
this branch at all.

`pr-413` is the one published tag that stays correct as the PR moves. It is a
MOVING tag, which is fine for a dry-run and not acceptable for a real run against
production cubes — the comment says so, and says to pin the main-branch
`sha-<main-sha>` once #413 merges.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
@lhoupert
lhoupert marked this pull request as ready for review September 18, 2026 12:30
lhoupert and others added 3 commits September 21, 2026 11:03
…oldest-first

`--allow-out-of-order` landed on ingest_v1_s1_rtc.py but stopped there, so the only
entry point production ever calls — run_ingest_register.py, driven by the CDSE watcher
and the Argo templates — could not reach it.

That matters because the watcher's ordering is unspecified: the CDSE search sets no
`sortby` (it answers datetime-descending in practice) and Script B appends ONE
acquisition per call under the 0.11.0 monotonicity guard. Newest-first means every
older product in the run is an out-of-order append: each fails, none is marked
processed, and the next run repeats the same order and the same failures — a permanent
stall with no CLI escape hatch.

Two changes, addressing cause and recovery:
- `query_cdse` now returns products oldest-first (stable, so same-date ties keep CDSE's
  order). This is what stops the inversion happening at all.
- `--allow-out-of-order` is plumbed through run_ingest_register.py and the watcher, so a
  cube that is ALREADY inverted can be backfilled. Default OFF either way, so a routine
  cron inversion still fails loudly.

Also hoists the MGRS tile-id check into `check_tile_id` and calls it from `run_watch`,
next to the `check_env_consistency` call that is already hoisted for the same reason:
the guard's own rationale ("a typo writes nothing") did not hold for the watcher, which
runs Script A — hours of orthorectification writing GeoTIFFs to S3 — before Script B
ever validates the id.

Co-Authored-By: Claude Opus 5 <[email protected]>
…the URI

The non-`rewrite_root_geo` consolidation branch inlines what `consolidate_s1_store`
does, but dropped two of its preconditions:

- It consolidated over `report.orbits`, captured from the pre-drop read-only handle —
  i.e. THROUGH the store's consolidated metadata. The library re-enumerates with
  `use_consolidated=False` precisely because an orbit group appended since the last
  consolidation is absent from a stale root block. The re-derive loop walks the fresh
  post-drop handle, so such an orbit was re-derived and then left unconsolidated, and
  readers opening it standalone fall back to a listing — the #203 defect this migration
  exists to fix.
- It passed the raw href-derived URI. `consolidate_s1_store` normalizes first, which
  matters on OVH S3 ("sensitive to double slashes").

The store is now opened `use_consolidated=False` and `report.orbits` is refreshed from
the post-drop listing, so the reported, re-derived and consolidated orbit sets are the
same set, and both consolidate calls take the normalized path.

Also: `--rewrite-root-geo` was a silent no-op on any store already carrying the
completion marker — the early return fires first. Once the fleet is migrated that is
every store, so an operator would get `already-current=N, derived=0` and no root
rewrite, with nothing saying the flag was ignored. It now warns. (Acting on the flag
instead would make an idempotent no-op path write to production roots; that is a
separate decision, not a silent one.)

Tests: a store whose consolidated root omits an orbit still gets that orbit enumerated
and consolidated; the ignored flag is reported.

Co-Authored-By: Claude Opus 5 <[email protected]>
The manifest carried a ⚠ comment saying `pr-413` (and any moving tag) must never drive
a real run. A comment cannot stop a submit: `imagePullPolicy: Always` plus a moving tag
means a `dry_run=false` run can pull a different build than the one the dry run
validated, against production cubes on a bucket with S3 versioning OFF.

The template now exits 1 before the driver starts when the image tag is `pr-*` or
`latest` and `dry_run` is not "true" — the bound is a feature of the tool, not a note in
the header. `PIPELINE_IMAGE_VERSION` is exported so the guard can read it.

The test runs the template's own script body under bash with a stub `python` on PATH,
so the control is exercised rather than assumed: refused on every moving tag with
dry_run=false, allowed for dry runs, out of the way for a `sha-<main-sha>` tag. It also
pins the manifest's safe defaults (dry_run true; allow_no_backup, rewrite_root_geo and
rollback false).

Co-Authored-By: Claude Opus 5 <[email protected]>
@lhoupert
lhoupert merged commit 33f03f8 into main Sep 21, 2026
9 checks passed
@github-actions github-actions Bot mentioned this pull request Sep 21, 2026

This branch was previously deployed

1 inactive deployment
harbor-registry — 166042ed Deployed Sep 21, 2026 by lhoupert via Build and publish image #1833
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant