Skip to content

bug(seeders): gdelt-intel / grocery-basket / supply-chain-trade trip the #4786 240s fetch-deadline on legitimate slow-retry runs (retry-budget > deadline); supply-chain loses last-good data #4864

Description

@koala73

Summary

Three seeders — seed-gdelt-intel, seed-grocery-basket, seed-supply-chain-trade — repeatedly hit the #4786 fetch-phase deadline guard (scripts/_seed-utils.mjs raceFetchDeadline), exit 75, and fire a Railway "Deployment crashed" email each time. The diagnose-railway-seeders classifier buckets them GRACEFUL (correct at the exit-code layer), but the root cause is a retry-budget-vs-deadline mismatch, not a transient upstream blip — and for seed-supply-chain-trade the graceful "no data lost" guarantee failed (manual seed required).

Evidence (Railway logs, 2026-07-05)

seed-gdelt-intel  (deploy 48ecec76-…):
  FETCH FAILED: intelligence:gdelt-intel fetch phase exceeded 240000ms deadline (likely a non-settling await — see issue #4786)
  === Failed gracefully (244202ms) ===
  [GDELT] proxy (curl) attempt 1/2 failed (non-retryable): curl: (28) Connection timed out after 15001 ms

seed-grocery-basket  (deploy 1fb4bd25-…):
  FETCH FAILED: economic:grocery-basket fetch phase exceeded 240000ms deadline
  === Failed gracefully (240476ms) ===

seed-supply-chain-trade  (deploy 70e7a8c0-…):
  FETCH FAILED: supply_chain:shipping fetch phase exceeded 240000ms deadline
  === Failed gracefully (240699ms) ===
  WARNING: 2 key(s) were expired/missing — EXPIRE was a no-op; manual seed required

Root cause

raceFetchDeadline deadline defaults to lockTtlMs (120s) + FETCH_PHASE_DEADLINE_MARGIN_MS (120s) = 240s (_seed-utils.mjs:1249,1373). The guard's design invariant is "a healthy seeder never outlives lockTtlMs". These three violate it:

  • gdelt-intel: fetchGdeltJson (_gdelt-fetch.mjs) worst-case for ONE article fetch ≈ direct loop (4×15s timeout + 10+20+30s backoff ≈ 120s) + proxy loop (5×15s + 4×5s ≈ 95s) ≈ 215s. Serialized across 6 topics + timelines (code comment at seed-gdelt-intel.mjs:89 already admits a "~5 min ceiling"), it deterministically exceeds 240s under a GDELT/Decodo throttle storm. The hard deadline aborts fetchAllTopics before its cache-merge fallback (:151) runs, so it exits 75 instead of publishing partial+cached data and exiting 0.
  • grocery-basket: slow EXA/LLM "learned-route" price extraction stacks past 240s (same class — TBD exact budget).
  • supply-chain-trade: hit the deadline AND its last-good keys had already expired → extendExistingTtl was a no-op (_seed-utils.mjs:709) → actual data staleness, not a clean self-heal. The graceful path only preserves data that still exists; a long enough failure window drops it.

Because exit 75 is non-zero, Railway paints red and emails on every trip — so the emails are the visible symptom of the seeders giving up far more often than they should.

Fix direction (per seeder)

  1. Size the fetch-phase budget to the real worst case via lockTtlMs / fetchPhaseTimeoutMs so #4786's guard trips only on a genuine hang, not a legitimate slow-retry sequence.
  2. gdelt-intel: add an internal soft budget so it stops fetching and falls through to its existing cache-merge + validate() (≥3/6 topics) BEFORE the hard deadline — publish partial+cached, exit 0.
  3. supply-chain-trade: ensure last-good survives the failure window (raise TTL / add coverage floor) so a graceful skip actually preserves data instead of manual seed required.
  4. Verify each seeder's retry loops have hard per-attempt AbortSignal.timeout (gdelt does; audit grocery + supply-chain).

Each change lands with a failing-first regression test per the bug-fix protocol.

Not in scope

Exit-75 emails on a genuine multi-tick upstream outage are working-as-designed (warm-ping exit 0 vs publish exit 75 convention is intentional). This issue is about the seeders tripping the deadline on legitimately-slow-but-recoverable runs and about the supply-chain data-loss.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions