Skip to content

fix(seed-bigmac): 50-country sequential Exa loop breaches 240s fetch-phase deadline → recurring exit-75 'Deploy Crashed!' alerts #4994

Description

@koala73

Summary

seed-bigmac fires Railway "Deploy Crashed!" alerts on a recurring basis. The crash is classified GRACEFUL (exit 75, last-good TTL extended, no data lost) — but it is not a random upstream blip. It is a structural fragility: the per-country Exa search loop is sequential and mathematically guaranteed to breach the runner's fetch-phase deadline whenever Exa latency is elevated.

Evidence (live, 2026-07-07, deployment 6fc174d4-d440-408e-889f-bb552acca84e)

FETCH FAILED: economic:bigmac fetch phase exceeded 240000ms deadline (likely a non-settling await — see issue #4786)
Extended TTL on 3 key(s) (864000s)
=== Failed gracefully (240503ms) ===
  Processing 🇿🇦 South Africa (ZAR)...
    Big Mac: 54.9 ZAR = \$3.3763
  Processing 🇳🇬 Nigeria (NGN)...

The run burned the full 240.5s and was killed mid-loop (still processing countries).

Root cause

scripts/seed-bigmac.mjs:

  • fetchBigMacPrices() (line 132) loops sequentially over COUNTRIES50 countries (line 136: for (const country of COUNTRIES)), awaiting searchExa() each iteration. No Promise.all, no concurrency pool.
  • searchExa() (line 105) does one fetch('https://api.exa.ai/search', { signal: AbortSignal.timeout(15_000) }) — a 15s per-call ceiling.
  • The whole phase runs under runSeed(...)'s 240 000 ms deadline (line 272).

The math: 50 sequential calls under a 240s ceiling means the average per-country latency only has to exceed 240s ÷ 50 = 4.8s to breach. A partial Exa slowdown (calls drifting toward their 15s timeout) breaches it deterministically. This is the same non-settling/slow-fetch class flagged in #4786, but here the outer deadline guard converts the hang into a whole-run crash + operator alert instead of exit-13.

Why "graceful" is the wrong disposition

Even though no data is lost (TTL extended), each occurrence:

  1. Fires a Railway "Deploy Crashed!" notification → operator alert fatigue.
  2. Burns up to 240s of compute doing nothing useful.
  3. Is invisible in /api/health (seed-meta stays fresh via the preserve-last-good path), so the only signal is the crash badge the operator is told to ignore.

Proposed fix

Convert the whole-run crash into localized partial-data so the run exits 0:

  • Run the 50-country loop with bounded concurrency (e.g. pool of 5–8) + a per-country time budget, so total wall-clock stays well under 240s even under Exa latency.
  • Treat an individual country's Exa failure/timeout as a per-country skip (keep prior snapshot value for that country), not a whole-run abort.
  • Optionally lower the outer 240s guard once the loop is bounded, so a genuine hang is caught faster.

Result: partial refreshes succeed and exit 0 (no crash alert); only a total-Exa-outage would trip the graceful path.

Meta

Filed off the /diagnose-railway-seeders review — the skill classified this GRACEFUL / "action: None", which is the disposition being re-examined. A recurring graceful crash that alerts the operator is a bug to fix, not noise to ignore.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions