Skip to content

seed-conflict-intel GRACEFUL·CHRONIC again: #5134 GDELT fallback worst case (~375s) exceeds 240s fetch deadline during GDELT brownout — crashes ~6/7 ticks, conflict:acled key EMPTY in prod #5140

Description

@koala73

Symptom

seed-conflict-intel has been red on ~6 of every 7 cron ticks since ~19:30 UTC 2026-07-09 (right after #5134 deployed). Every crash is graceful (exit 75, === Failed gracefully (244438ms) ===), self-heals to green on a later tick, and fires a "Deploy Crashed!" email — classic GRACEFUL·CHRONIC. Recovered-crash sweep (2026-07-10 03:57 UTC): 6 self-healed crashes at 00:45, 01:16, 01:45, 02:15, 02:45, 03:15 UTC vs 1 success.

This is not just alert noise: prod health shows acledIntel: {status: EMPTY, records: 0, maxStaleMin: 38} (crit). conflict:acled:v1:all:0:0 has been empty since the run's keys expired ("2 key(s) were expired/missing — EXPIRE was a no-op"). The GDELT fallback #5134 added to feed the escalation EMA has delivered zero events in prod — even the runs that exit 0 take the "publishing auxiliary feeds only" path. The EMA is exactly as blind as before #5134, now with a crash storm on top.

Root cause — fallback's worst case doesn't fit the fetch deadline

fetchGdeltConflictEvents (scripts/seed-conflict-intel.mjs:211) sweeps ~20 countries at CONCURRENCY=4 (5 batches). Per country (maxRetries: 1, proxyMaxAttempts: 2, seed-conflict-intel.mjs:203), the worst case when GDELT times out is:

  • direct: 15s timeout + 10s backoff + 15s timeout = 40s
  • proxy: 15s curl + 5s backoff + 15s curl = 35s
  • 75s per country → 5 batches × 75s ≈ 375s worst case, against runSeed's 240s fetch deadline.

The line-202 comment says "keep each call cheap so the whole sweep fits the run window", but the arithmetic never fit for the all-fail case. Whenever GDELT throttles hard, the deadline breach is guaranteed: FETCH FAILED: conflict:acled-intel fetch phase exceeded 240000ms deadline → exit 75 → Railway CRASHED. The coverage floor at :232 never gets to fire (it IS caught at :439–444 and would degrade to aux-only + exit 0 — the run just never reaches it in time).

Whether a given tick exits 0 or 75 is pure timing luck: connection-reset failures fail fast (~1–2s) and squeak under the deadline; timeout failures burn the full budget.

Trigger — GDELT is currently rejecting Railway + Decodo

Every run since #5134 shows the same wall:

  • direct from Railway: fetch failed (resets) / HTTP 429 / timeouts
  • Decodo proxy: curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL on ~100% of attempts (TLS handshake killed — GDELT shedding datacenter/proxy IPs)
  • GDELT is degraded generally: a single direct probe from a residential IP returned HTTP 200 in 12.3s (normally 1–2s)

So the "expected success ~92%" model in _gdelt-fetch.mjs (5 proxy attempts) doesn't hold during a GDELT brownout — attempts are not independent when the block is IP-class-wide.

Fix (make the graceful path exit 0, per the seed-bigmac #4994 pattern)

  1. Time-box the whole fallback sweep to a budget that provably fits: race fetchGdeltConflictEvents against ~120–150s; on expiry, stop launching batches and evaluate the coverage floor with whatever completed. Deadline can then never fire on this path → worst case becomes caught-floor → aux-only publish → exit 0.
  2. Shrink per-country worst case: maxRetries: 1 → 0, proxyMaxAttempts: 2 → 1, timeoutMs: 15s → 8–10s for this call site → ≈20–25s/country → ≈100–125s sweep worst case.
  3. Circuit-breaker: if batch 1 goes 0/4, skip the remaining batches (GDELT is down; grinding through 16 more countries can't reach the 80% floor anyway — GDELT_MIN_SUCCESSFUL_COUNTRIES = 16/20).
  4. Any combination of the above; (1) alone is sufficient to stop the crashes.

The actual root cause remains unprovisioned ACLED creds

The fallback only runs because ACLED_* creds were never provisioned (service env has only an empty VITE_ACLED_ACCESS_TOKEN; the seeder reads unprefixed ACLED_* — see #5126). GDELT article volume is a coarse proxy; registering an ACLED account and setting the creds makes this whole path dormant. Worth tracking separately if there's no issue for it.

Evidence

  • Crashed run 2026-07-10T03:30Z (deployment 01f52a91-24e9-413f-8f3f-aaacee33a92b): all countries exhaust direct+proxy → FETCH FAILED … 240000ms deadline=== Failed gracefully (244438ms) ===
  • "SUCCESS" run 03:45Z (29e282fd-c4e7-4b5e-93a9-042943571e1d): identical GDELT failures in batch 1; acledIntel still EMPTY after it
  • Re-pull: railway logs 01f52a91-24e9-413f-8f3f-aaacee33a92b -d --lines 400

Related: #5134 (introduced fallback, closed #5099) · #5126 (creds-absent graceful skip) · #5136 (resolution-feed consequence) · #4786 (fetch-deadline mechanism) · #4994 (same class: sequential sweep vs deadline)

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: AI/intelAI analysis, intelligence findings, summarizationbugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions