Repository navigation
S1 RTC ingest is I/O-bound: ~34-min S3 "Uploading store" stall per cube #288
Description
Problem
The S1 RTC ingest (scripts/ingest_v1_s1_rtc.py) spends ~30–40 min per cube just uploading to S3. A live pod sat ~34 min in Uploading store at 9 millicores — CPU idle, no compute happening. At cron parallelism 3, ~17 products ≈ a near-all-day run.
Profiling ruled out compute, scheduling, and nodes — it's serial S3 transfer of too many objects. A real cube (s1-rtc-31TEG, staging) is 3807 objects / 3.5 GB, and:
- 94.7% (3604 objects) are
conditions/gamma_area_*— unsharded[10980,10980]arrays split into ~900 tiny 366² chunk objects each. - The uploader did
rm+ re-uploaded the entire accumulated cube on every append, re-PUTting those ~3600 static objects every run. - Transfers ran in a serial
put_file/get_fileloop → thousands of blocking round-trips.
Key insight: the dominant cost is object COUNT, not per-object speed.
Solution — two complementary fixes
| PR | Lever | Change | Gain |
|---|---|---|---|
| #287 (this repo) | How objects move | Concurrent batched PUT/GET + incremental upload (per append send only new/changed objects; never re-upload the whole cube) | PUT ~26×, append ~13× on real S3 |
| EOPF-Explorer/data-model#197 | How many objects exist | Shard gamma_area like vv/vh already are → ~900 objects → 1 per array (cube 3807 → ~210) |
object count −94% |
They multiply, not overlap: #287 makes today's cubes upload in minutes (works immediately); #197 makes every cube permanently lighter — fewer objects to list each append, to upload cold, and for clients (titiler/xarray) to read. Both validated on the live OVH bucket.
Follow-ups
- In-cluster wall-time + titiler render confirmation (laptop creds are cross-owner-denied on existing prod cubes).
- Old cubes need a re-ingest to pick up the sharded layout — build(deps): bump jupyter-server from 2.17.0 to 2.18.0 #197 doesn't auto-migrate.
Plan: claude-docs/plans/s1_ingest_upload_perf.md.
Activity
chore(s1-rtc): drop the claude-docs spec from this PR
perf(s1-rtc): shard the conditions arrays (gamma_area/LIA) like the v…
fix(s1-rtc): heal a multiscale level missing `time` on append (robust…
lhoupert commentedon Jun 19, 2026
Adding the download-side fix + the in-cluster confirmation this issue's follow-up was waiting on.
The two levers tracked here — #287 (incremental/concurrent upload) and data-model EOPF-Explorer/data-model#197 (shard
gamma_area) — target the upload and the object count. The remaining I/O cost was the download: every append fetched the whole cube (_fetch_store_from_s3, O(cube)) even though it only reads zarr metadata + the small 1-D coordinate arrays and writes one new slice.#294 replaces that with a fetch-minimal append (
_fetch_for_append: everyzarr.json+ onlyndim ≤ 1coordinate chunks; skips the bulkvv/vh/border_mask/gamma_area_*/lia_*chunks) → O(metadata + coords). The_sync_treedeletion is scoped to the same coordinate/metadata set, so the deliberately-skipped bulk chunks are never dropped (no data loss). Safe because the bulk arrays shard with time-extent 1 (a new slice writes entirely new objects).In-cluster confirmation (image
sha-62207b1, devseed-staging — closes the open "in-cluster wall-time + render" follow-up):Incremental upload … N/M object(s) sent, 0 deleted;So upload + object-count + download are all addressed and confirmed in-cluster — the ~34-min stall is gone. Suggest closing once #294 merges.
lhoupert commentedon Jun 30, 2026
done
feat(s1-rtc): STAC item builders + ingest fixes for TiTiler rendering (…