Summary
Three seeders — seed-gdelt-intel, seed-grocery-basket, seed-supply-chain-trade — repeatedly hit the #4786 fetch-phase deadline guard (scripts/_seed-utils.mjs raceFetchDeadline), exit 75, and fire a Railway "Deployment crashed" email each time. The diagnose-railway-seeders classifier buckets them GRACEFUL (correct at the exit-code layer), but the root cause is a retry-budget-vs-deadline mismatch, not a transient upstream blip — and for seed-supply-chain-trade the graceful "no data lost" guarantee failed (manual seed required).
Evidence (Railway logs, 2026-07-05)
seed-gdelt-intel (deploy 48ecec76-…):
FETCH FAILED: intelligence:gdelt-intel fetch phase exceeded 240000ms deadline (likely a non-settling await — see issue #4786)
=== Failed gracefully (244202ms) ===
[GDELT] proxy (curl) attempt 1/2 failed (non-retryable): curl: (28) Connection timed out after 15001 ms
seed-grocery-basket (deploy 1fb4bd25-…):
FETCH FAILED: economic:grocery-basket fetch phase exceeded 240000ms deadline
=== Failed gracefully (240476ms) ===
seed-supply-chain-trade (deploy 70e7a8c0-…):
FETCH FAILED: supply_chain:shipping fetch phase exceeded 240000ms deadline
=== Failed gracefully (240699ms) ===
WARNING: 2 key(s) were expired/missing — EXPIRE was a no-op; manual seed required
Root cause
raceFetchDeadline deadline defaults to lockTtlMs (120s) + FETCH_PHASE_DEADLINE_MARGIN_MS (120s) = 240s (_seed-utils.mjs:1249,1373). The guard's design invariant is "a healthy seeder never outlives lockTtlMs". These three violate it:
- gdelt-intel:
fetchGdeltJson (_gdelt-fetch.mjs) worst-case for ONE article fetch ≈ direct loop (4×15s timeout + 10+20+30s backoff ≈ 120s) + proxy loop (5×15s + 4×5s ≈ 95s) ≈ 215s. Serialized across 6 topics + timelines (code comment at seed-gdelt-intel.mjs:89 already admits a "~5 min ceiling"), it deterministically exceeds 240s under a GDELT/Decodo throttle storm. The hard deadline aborts fetchAllTopics before its cache-merge fallback (:151) runs, so it exits 75 instead of publishing partial+cached data and exiting 0.
- grocery-basket: slow EXA/LLM "learned-route" price extraction stacks past 240s (same class — TBD exact budget).
- supply-chain-trade: hit the deadline AND its last-good keys had already expired →
extendExistingTtl was a no-op (_seed-utils.mjs:709) → actual data staleness, not a clean self-heal. The graceful path only preserves data that still exists; a long enough failure window drops it.
Because exit 75 is non-zero, Railway paints red and emails on every trip — so the emails are the visible symptom of the seeders giving up far more often than they should.
Fix direction (per seeder)
- Size the fetch-phase budget to the real worst case via
lockTtlMs / fetchPhaseTimeoutMs so #4786's guard trips only on a genuine hang, not a legitimate slow-retry sequence.
- gdelt-intel: add an internal soft budget so it stops fetching and falls through to its existing cache-merge +
validate() (≥3/6 topics) BEFORE the hard deadline — publish partial+cached, exit 0.
- supply-chain-trade: ensure last-good survives the failure window (raise TTL / add coverage floor) so a graceful skip actually preserves data instead of
manual seed required.
- Verify each seeder's retry loops have hard per-attempt
AbortSignal.timeout (gdelt does; audit grocery + supply-chain).
Each change lands with a failing-first regression test per the bug-fix protocol.
Not in scope
Exit-75 emails on a genuine multi-tick upstream outage are working-as-designed (warm-ping exit 0 vs publish exit 75 convention is intentional). This issue is about the seeders tripping the deadline on legitimately-slow-but-recoverable runs and about the supply-chain data-loss.
Summary
Three seeders —
seed-gdelt-intel,seed-grocery-basket,seed-supply-chain-trade— repeatedly hit the#4786fetch-phase deadline guard (scripts/_seed-utils.mjsraceFetchDeadline), exit 75, and fire a Railway "Deployment crashed" email each time. Thediagnose-railway-seedersclassifier buckets themGRACEFUL(correct at the exit-code layer), but the root cause is a retry-budget-vs-deadline mismatch, not a transient upstream blip — and forseed-supply-chain-tradethe graceful "no data lost" guarantee failed (manual seed required).Evidence (Railway logs, 2026-07-05)
Root cause
raceFetchDeadlinedeadline defaults tolockTtlMs (120s) + FETCH_PHASE_DEADLINE_MARGIN_MS (120s) = 240s(_seed-utils.mjs:1249,1373). The guard's design invariant is "a healthy seeder never outliveslockTtlMs". These three violate it:fetchGdeltJson(_gdelt-fetch.mjs) worst-case for ONE article fetch ≈ direct loop (4×15s timeout + 10+20+30s backoff ≈ 120s) + proxy loop (5×15s + 4×5s ≈ 95s) ≈ 215s. Serialized across 6 topics + timelines (code comment atseed-gdelt-intel.mjs:89already admits a "~5 min ceiling"), it deterministically exceeds 240s under a GDELT/Decodo throttle storm. The hard deadline abortsfetchAllTopicsbefore its cache-merge fallback (:151) runs, so it exits 75 instead of publishing partial+cached data and exiting 0.extendExistingTtlwas a no-op (_seed-utils.mjs:709) → actual data staleness, not a clean self-heal. The graceful path only preserves data that still exists; a long enough failure window drops it.Because exit 75 is non-zero, Railway paints red and emails on every trip — so the emails are the visible symptom of the seeders giving up far more often than they should.
Fix direction (per seeder)
lockTtlMs/fetchPhaseTimeoutMsso#4786's guard trips only on a genuine hang, not a legitimate slow-retry sequence.validate()(≥3/6 topics) BEFORE the hard deadline — publish partial+cached, exit 0.manual seed required.AbortSignal.timeout(gdelt does; audit grocery + supply-chain).Each change lands with a failing-first regression test per the bug-fix protocol.
Not in scope
Exit-75 emails on a genuine multi-tick upstream outage are working-as-designed (
warm-ping exit 0vspublish exit 75convention is intentional). This issue is about the seeders tripping the deadline on legitimately-slow-but-recoverable runs and about the supply-chain data-loss.