Summary

seed-forecasts intermittently crashes with Node's "unsettled top-level await" (exit 13) — a red Railway badge that is not the usual graceful exit-75 self-heal and not a FATAL: throw. Root cause: the S3-mode branch of getR2JsonObject (scripts/_r2-storage.mjs) has no request timeout / AbortSignal, while its Cloudflare-R2-API sibling branch does. When an S3 body stream stalls mid-transfer, response.Body.transformToString() returns a promise that never settles (never resolves, never rejects), so the caller's try/catch can't catch it and the process's top-level await runSeed(...) hangs until the event loop drains → exit 13.

Rare, self-healing (no data lost — prior tick fresh, TTL valid), but produces a spurious red badge and skips one forecast tick.

Evidence (Railway, deployment 40599b41-0689-47d7-853b-de5c63f1ba69, 2026-07-04)

Three consecutive hourly runs succeed, then the latest run dies:

=== Done (149302ms) ===            # 16:05 run — state=OK
Starting Container
  [Trigger] source=military_chain requester=seed-military-flights
=== forecast:predictions Seed ===
  Run ID:  1783184464575-chreol
  Key:     forecast:predictions:v2
  Mode:    contract (envelope dual-write)
  [Chokepoints] Warm-ping OK
Warning: Detected unsettled top-level await at file:///app/seed-forecasts.mjs:16298
  await runSeed('forecast', 'predictions', CANONICAL_KEY, async () => {
  ^

It dies after [Chokepoints] Warm-ping OK and before Reading input data from Redis... — i.e. inside fetchForecasts(), in the prior-state read block, not the Redis input read.

Root cause

fetchForecasts() (scripts/seed-forecasts.mjs:15159) runs, right after the warm-ping and before the Redis input read:

// scripts/seed-forecasts.mjs:15160-15168
await warmPingChokepoints();                                   // logs "[Chokepoints] Warm-ping OK"
const traceStorageConfig = resolveR2StorageConfig();
const [priorWorldStates, priorWorldStateFallback, priorTracePointer] = traceStorageConfig
  ? await Promise.all([
      readForecastWorldStateHistory(traceStorageConfig, WORLD_STATE_HISTORY_LIMIT),
      readPreviousForecastWorldState(traceStorageConfig),
      readPreviousForecastTracePointer(),
    ])
  : [[], null, null];
console.log('  Reading input data from Redis...');             // never printed on the crashed run

All three prior-state readers wrap their bodies in try/catch that returns a safe fallback ([] / null), and their Redis primitives (redisCommand, redisGet) both use AbortSignal.timeout(10_000). The only unguarded network primitive reached here is getR2JsonObject (called by readForecastWorldStateHistory and readPreviousForecastWorldState).

getR2JsonObject (scripts/_r2-storage.mjs:197) has two branches:

• api-mode (config.mode === 'api') — fetch(...) with signal: AbortSignal.timeout(30_000) (line 206). ✅ Guaranteed to settle.
• s3-mode (else, line 222) — AWS SDK v3:

  const response = await client.send(new GetObjectCommand({ Bucket, Key }));  // line 226 — no abortSignal
  const body = await response.Body?.transformToString?.();                    // line 230 — no timeout

  ❌ No timeout, no AbortSignal.

Production runs s3-mode — confirmed by the log line [Trace] Storage mode=s3 bucket=worldmonitor-data prefix=seed-data/forecast-traces.

Why it dangles with no open handle (→ event loop drains → exit 13, not a hang): client.send() resolves once response headers arrive, returning a streaming Body. transformToString() then consumes that stream. If the body socket goes idle mid-transfer and the keep-alive agent silently reaps/destroys it without emitting 'end' or 'error' to the stream consumer, transformToString()'s promise never settles and there is no active libuv handle keeping the loop alive. Node then prints "Detected unsettled top-level await" and exits 13. Because it's a non-settling promise (not a rejection), the surrounding try/catch in readForecastWorldStateHistory / readPreviousForecastWorldState and any withR2Retry retry-on-rejection logic are all bypassed.

This is the exact asymmetry that makes it a bug: the api-mode branch is protected; the s3-mode branch — the one production actually uses — is not.

Impact

• Severity: low. Rare (observed 1 crash after ≥3 consecutive successes), self-healing on the next trigger, no data loss (canonical forecast:predictions:v2 stays fresh, TTL still valid).
• Cost: one missed forecast refresh tick + a spurious red Railway badge that reads as a real fault to the seeder-triage tooling (classified UNKNOWN, exit code 1).

Proposed fix

Give the s3-mode branch of getR2JsonObject the same hard wall-clock ceiling the api-mode branch already has, so a stalled S3 read rejects instead of dangling:

• Pass { abortSignal: AbortSignal.timeout(30_000) } to client.send(command, opts) (AWS SDK v3 honors abortSignal per-request), and wrap the send + transformToString() pair in a Promise.race against a timeout so the body-streaming phase is also bounded (the request-level abort does not always cover a stalled body stream).

On timeout it throws → caught by getR2JsonObject's inner catch (rethrown, since it's not a 404) → caught by the caller's try/catch → returns the [] / null fallback → fetchForecasts() proceeds normally (these priors are optional). Net effect: a stalled R2 read degrades to "no priors this run" instead of killing the process with exit 13.

Optionally, add a belt-and-suspenders top-level run watchdog in runSeed (force a logged, graceful-ish exit if fetchFn exceeds the lock TTL) so any future non-settling await surfaces as a diagnosable exit rather than a bare exit-13.

Test strategy (Bug Fix Protocol — reproduce first)

1. Failing test: inject a fake S3 client into getR2JsonObject (s3-mode) whose Body.transformToString() returns a never-settling promise; assert the call rejects/settles within ~30s rather than hanging. Without the fix, the test hangs (proving the dangling promise).
2. Add the timeout; assert getR2JsonObject rejects on stall and that readForecastWorldStateHistory returns [] (fallback) rather than propagating.

References

• Deployed code = origin/main @ f1072a9d2 (crash line seed-forecasts.mjs:16298 matches exactly).
• Crash deployment: railway logs 40599b41-0689-47d7-853b-de5c63f1ba69 -d --lines 400
• Fix site: scripts/_r2-storage.mjs:222-231 (s3-mode branch of getR2JsonObject); guarded sibling at scripts/_r2-storage.mjs:199-207.
• Trigger site: scripts/seed-forecasts.mjs:15160-15168 (fetchForecasts prior-state Promise.all).