Symptom
seed-conflict-intel has been red on ~6 of every 7 cron ticks since ~19:30 UTC 2026-07-09 (right after #5134 deployed). Every crash is graceful (exit 75, === Failed gracefully (244438ms) ===), self-heals to green on a later tick, and fires a "Deploy Crashed!" email — classic GRACEFUL·CHRONIC. Recovered-crash sweep (2026-07-10 03:57 UTC): 6 self-healed crashes at 00:45, 01:16, 01:45, 02:15, 02:45, 03:15 UTC vs 1 success.
This is not just alert noise: prod health shows acledIntel: {status: EMPTY, records: 0, maxStaleMin: 38} (crit). conflict:acled:v1:all:0:0 has been empty since the run's keys expired ("2 key(s) were expired/missing — EXPIRE was a no-op"). The GDELT fallback #5134 added to feed the escalation EMA has delivered zero events in prod — even the runs that exit 0 take the "publishing auxiliary feeds only" path. The EMA is exactly as blind as before #5134, now with a crash storm on top.
Root cause — fallback's worst case doesn't fit the fetch deadline
fetchGdeltConflictEvents (scripts/seed-conflict-intel.mjs:211) sweeps ~20 countries at CONCURRENCY=4 (5 batches). Per country (maxRetries: 1, proxyMaxAttempts: 2, seed-conflict-intel.mjs:203), the worst case when GDELT times out is:
- direct: 15s timeout + 10s backoff + 15s timeout = 40s
- proxy: 15s curl + 5s backoff + 15s curl = 35s
- ≈ 75s per country → 5 batches × 75s ≈ 375s worst case, against runSeed's 240s fetch deadline.
The line-202 comment says "keep each call cheap so the whole sweep fits the run window", but the arithmetic never fit for the all-fail case. Whenever GDELT throttles hard, the deadline breach is guaranteed: FETCH FAILED: conflict:acled-intel fetch phase exceeded 240000ms deadline → exit 75 → Railway CRASHED. The coverage floor at :232 never gets to fire (it IS caught at :439–444 and would degrade to aux-only + exit 0 — the run just never reaches it in time).
Whether a given tick exits 0 or 75 is pure timing luck: connection-reset failures fail fast (~1–2s) and squeak under the deadline; timeout failures burn the full budget.
Trigger — GDELT is currently rejecting Railway + Decodo
Every run since #5134 shows the same wall:
- direct from Railway:
fetch failed (resets) / HTTP 429 / timeouts
- Decodo proxy:
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL on ~100% of attempts (TLS handshake killed — GDELT shedding datacenter/proxy IPs)
- GDELT is degraded generally: a single direct probe from a residential IP returned HTTP 200 in 12.3s (normally 1–2s)
So the "expected success ~92%" model in _gdelt-fetch.mjs (5 proxy attempts) doesn't hold during a GDELT brownout — attempts are not independent when the block is IP-class-wide.
Fix (make the graceful path exit 0, per the seed-bigmac #4994 pattern)
- Time-box the whole fallback sweep to a budget that provably fits: race
fetchGdeltConflictEvents against ~120–150s; on expiry, stop launching batches and evaluate the coverage floor with whatever completed. Deadline can then never fire on this path → worst case becomes caught-floor → aux-only publish → exit 0.
- Shrink per-country worst case:
maxRetries: 1 → 0, proxyMaxAttempts: 2 → 1, timeoutMs: 15s → 8–10s for this call site → ≈20–25s/country → ≈100–125s sweep worst case.
- Circuit-breaker: if batch 1 goes 0/4, skip the remaining batches (GDELT is down; grinding through 16 more countries can't reach the 80% floor anyway —
GDELT_MIN_SUCCESSFUL_COUNTRIES = 16/20).
- Any combination of the above; (1) alone is sufficient to stop the crashes.
The actual root cause remains unprovisioned ACLED creds
The fallback only runs because ACLED_* creds were never provisioned (service env has only an empty VITE_ACLED_ACCESS_TOKEN; the seeder reads unprefixed ACLED_* — see #5126). GDELT article volume is a coarse proxy; registering an ACLED account and setting the creds makes this whole path dormant. Worth tracking separately if there's no issue for it.
Evidence
- Crashed run 2026-07-10T03:30Z (deployment
01f52a91-24e9-413f-8f3f-aaacee33a92b): all countries exhaust direct+proxy → FETCH FAILED … 240000ms deadline → === Failed gracefully (244438ms) ===
- "SUCCESS" run 03:45Z (
29e282fd-c4e7-4b5e-93a9-042943571e1d): identical GDELT failures in batch 1; acledIntel still EMPTY after it
- Re-pull:
railway logs 01f52a91-24e9-413f-8f3f-aaacee33a92b -d --lines 400
Related: #5134 (introduced fallback, closed #5099) · #5126 (creds-absent graceful skip) · #5136 (resolution-feed consequence) · #4786 (fetch-deadline mechanism) · #4994 (same class: sequential sweep vs deadline)
Symptom
seed-conflict-intelhas been red on ~6 of every 7 cron ticks since ~19:30 UTC 2026-07-09 (right after #5134 deployed). Every crash is graceful (exit 75,=== Failed gracefully (244438ms) ===), self-heals to green on a later tick, and fires a "Deploy Crashed!" email — classic GRACEFUL·CHRONIC. Recovered-crash sweep (2026-07-10 03:57 UTC): 6 self-healed crashes at 00:45, 01:16, 01:45, 02:15, 02:45, 03:15 UTC vs 1 success.This is not just alert noise: prod health shows
acledIntel: {status: EMPTY, records: 0, maxStaleMin: 38}(crit).conflict:acled:v1:all:0:0has been empty since the run's keys expired ("2 key(s) were expired/missing — EXPIRE was a no-op"). The GDELT fallback #5134 added to feed the escalation EMA has delivered zero events in prod — even the runs that exit 0 take the "publishing auxiliary feeds only" path. The EMA is exactly as blind as before #5134, now with a crash storm on top.Root cause — fallback's worst case doesn't fit the fetch deadline
fetchGdeltConflictEvents(scripts/seed-conflict-intel.mjs:211) sweeps ~20 countries at CONCURRENCY=4 (5 batches). Per country (maxRetries: 1, proxyMaxAttempts: 2, seed-conflict-intel.mjs:203), the worst case when GDELT times out is:The line-202 comment says "keep each call cheap so the whole sweep fits the run window", but the arithmetic never fit for the all-fail case. Whenever GDELT throttles hard, the deadline breach is guaranteed:
FETCH FAILED: conflict:acled-intel fetch phase exceeded 240000ms deadline→ exit 75 → Railway CRASHED. The coverage floor at :232 never gets to fire (it IS caught at :439–444 and would degrade to aux-only + exit 0 — the run just never reaches it in time).Whether a given tick exits 0 or 75 is pure timing luck: connection-reset failures fail fast (~1–2s) and squeak under the deadline; timeout failures burn the full budget.
Trigger — GDELT is currently rejecting Railway + Decodo
Every run since #5134 shows the same wall:
fetch failed(resets) / HTTP 429 / timeoutscurl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALLon ~100% of attempts (TLS handshake killed — GDELT shedding datacenter/proxy IPs)So the "expected success ~92%" model in
_gdelt-fetch.mjs(5 proxy attempts) doesn't hold during a GDELT brownout — attempts are not independent when the block is IP-class-wide.Fix (make the graceful path exit 0, per the seed-bigmac #4994 pattern)
fetchGdeltConflictEventsagainst ~120–150s; on expiry, stop launching batches and evaluate the coverage floor with whatever completed. Deadline can then never fire on this path → worst case becomes caught-floor → aux-only publish → exit 0.maxRetries: 1 → 0,proxyMaxAttempts: 2 → 1,timeoutMs: 15s → 8–10sfor this call site → ≈20–25s/country → ≈100–125s sweep worst case.GDELT_MIN_SUCCESSFUL_COUNTRIES= 16/20).The actual root cause remains unprovisioned ACLED creds
The fallback only runs because
ACLED_*creds were never provisioned (service env has only an emptyVITE_ACLED_ACCESS_TOKEN; the seeder reads unprefixedACLED_*— see #5126). GDELT article volume is a coarse proxy; registering an ACLED account and setting the creds makes this whole path dormant. Worth tracking separately if there's no issue for it.Evidence
01f52a91-24e9-413f-8f3f-aaacee33a92b): all countries exhaust direct+proxy →FETCH FAILED … 240000ms deadline→=== Failed gracefully (244438ms) ===29e282fd-c4e7-4b5e-93a9-042943571e1d): identical GDELT failures in batch 1;acledIntelstill EMPTY after itrailway logs 01f52a91-24e9-413f-8f3f-aaacee33a92b -d --lines 400Related: #5134 (introduced fallback, closed #5099) · #5126 (creds-absent graceful skip) · #5136 (resolution-feed consequence) · #4786 (fetch-deadline mechanism) · #4994 (same class: sequential sweep vs deadline)