Symptom
Railway seed-gdelt-intel red again 2026-07-23 (deployment b4114aad-8a5b-4481-8274-b8b7d917fcaa): FATAL: The operation was aborted due to timeout → exit 1 → "Deploy Crashed!". Diagnose-tool sweep counts 3 crashes / 0 terminal successes in the 36h window.
This is after the #5437 fix (PR #5438, merged 2026-07-22 10:37 UTC) reached the service: the crashed run's log carries the fix's fingerprint — Retry 1/2 in 1000ms: The operation was aborted due to timeout immediately preceding the FATAL. The retry wrapper ran, exhausted its budget, and the TimeoutError still escaped as exit 1. Meanwhile the article path holds: the 00:06 and 04:04 UTC runs backfilled 6×10 cached articles via cache-merge and published (recordCount=6, === Done ===).
Three strands
1. #5438's retry is insufficient under a sustained brownout — and the exhaustion path is still exit-1.
#5437 modeled the Redis failures as one-off blips; this run shows the wrapped skip-path op timing out on both attempts back-to-back. Two directions (not mutually exclusive):
2. All 12 tone/vol timeline keys are EXPIRED — dead feature, "manual seed required", and re-seeding is impossible while GDELT is down.
Verified 2026-07-23 08:30 UTC via Upstash REST: gdelt:intel:tone:{military,cyber,nuclear,sanctions,intelligence,maritime} and the 6 gdelt:intel:vol:* siblings all EXISTS=0 / TTL=-2. TIMELINE_TTL=43200 (12h) was sized to survive one missed 6h tick, not a multi-day brownout; every run since expiry logs WARNING: 6 key(s) were expired/missing — EXPIRE was a no-op; manual seed required (once for tone, once for vol) and nothing ever re-seeds an expired key. Consumers of the timelines (escalation signals, #2044) have been dark since ~07-21. Fix ideas: re-WRITE last-good timeline values with a fresh TTL instead of EXPIRE-extending (the timeline analogue of the article cache-merge), and/or raise TIMELINE_TTL to brownout scale. Then backfill manually once GDELT answers again.
3. Content-age blindspot — #5437's "separate concern", never filed.
/api/health reports gdeltIntel fresh (seed-meta fetchedAt 2026-07-23 00:04, fleet 0 stale) while the published articles are the same cached payload republished for days (payloadBytes=23714 identical across runs; per-topic fetchedAt 07-01…07-20 per the #5437 probe). Opt seed-gdelt-intel into the content-age trio (maxContentAgeMin/newestItemAt) so brownout coasting surfaces as STALE_CONTENT instead of hiding behind a fresh envelope.
Environment
GDELT brownout continues from Railway on 07-23: direct fetches abort at 15s; the Decodo proxy path gets curl: (35) SSL_ERROR_SYSCALL, curl: (28) timeouts, curl: (56) CONNECT tunnel failed, response 522, and HTTP 429.
Evidence
Crashed run tail (b4114aad, run id 1784793666314-ffizzt):
Soft budget (150s) reached after 1/6 topic(s) — remaining 5 will fall back to cached snapshot
Extended TTL on 2 key(s) (86400s)
Retry 1/2 in 1000ms: The operation was aborted due to timeout
WARNING: 6 key(s) were expired/missing — EXPIRE was a no-op; manual seed required
[GDELT] proxy (curl) attempt 2/2 failed (last attempt): curl failed: curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to api.gdeltproject.org:443
FATAL: The operation was aborted due to timeout
Note: no no fresh articles — using 10 cached articles backfill lines in the crashed run (present in the 00:06/04:04 OK runs) — the merge produced nothing there, so the run went down the skip path before the FATAL.
Redis probe (08:30 UTC): all 12 gdelt:intel:{tone,vol}:* keys exists=0 ttl=-2; intelligence:gdelt-intel:v1 + seed-meta:intelligence:gdelt-intel healthy, TTL 84976s, seed-meta {"fetchedAt":1784779479007,"recordCount":6}.
Related: #5437 / PR #5438 (previous round), #4864 (soft-budget design), #2044 (timeline feature).
Symptom
Railway
seed-gdelt-intelred again 2026-07-23 (deploymentb4114aad-8a5b-4481-8274-b8b7d917fcaa):FATAL: The operation was aborted due to timeout→ exit 1 → "Deploy Crashed!". Diagnose-tool sweep counts 3 crashes / 0 terminal successes in the 36h window.This is after the #5437 fix (PR #5438, merged 2026-07-22 10:37 UTC) reached the service: the crashed run's log carries the fix's fingerprint —
Retry 1/2 in 1000ms: The operation was aborted due to timeoutimmediately preceding the FATAL. The retry wrapper ran, exhausted its budget, and the TimeoutError still escaped as exit 1. Meanwhile the article path holds: the 00:06 and 04:04 UTC runs backfilled 6×10 cached articles via cache-merge and published (recordCount=6,=== Done ===).Three strands
1. #5438's retry is insufficient under a sustained brownout — and the exhaustion path is still exit-1.
#5437 modeled the Redis failures as one-off blips; this run shows the wrapped skip-path op timing out on both attempts back-to-back. Two directions (not mutually exclusive):
2. All 12 tone/vol timeline keys are EXPIRED — dead feature, "manual seed required", and re-seeding is impossible while GDELT is down.
Verified 2026-07-23 08:30 UTC via Upstash REST:
gdelt:intel:tone:{military,cyber,nuclear,sanctions,intelligence,maritime}and the 6gdelt:intel:vol:*siblings allEXISTS=0 / TTL=-2.TIMELINE_TTL=43200(12h) was sized to survive one missed 6h tick, not a multi-day brownout; every run since expiry logsWARNING: 6 key(s) were expired/missing — EXPIRE was a no-op; manual seed required(once for tone, once for vol) and nothing ever re-seeds an expired key. Consumers of the timelines (escalation signals, #2044) have been dark since ~07-21. Fix ideas: re-WRITE last-good timeline values with a fresh TTL instead of EXPIRE-extending (the timeline analogue of the article cache-merge), and/or raise TIMELINE_TTL to brownout scale. Then backfill manually once GDELT answers again.3. Content-age blindspot — #5437's "separate concern", never filed.
/api/healthreports gdeltIntel fresh (seed-meta fetchedAt 2026-07-23 00:04, fleet 0 stale) while the published articles are the same cached payload republished for days (payloadBytes=23714identical across runs; per-topic fetchedAt 07-01…07-20 per the #5437 probe). Opt seed-gdelt-intel into the content-age trio (maxContentAgeMin/newestItemAt) so brownout coasting surfaces as STALE_CONTENT instead of hiding behind a fresh envelope.Environment
GDELT brownout continues from Railway on 07-23: direct fetches abort at 15s; the Decodo proxy path gets
curl: (35) SSL_ERROR_SYSCALL,curl: (28)timeouts,curl: (56) CONNECT tunnel failed, response 522, andHTTP 429.Evidence
Crashed run tail (b4114aad, run id
1784793666314-ffizzt):Note: no
no fresh articles — using 10 cached articlesbackfill lines in the crashed run (present in the 00:06/04:04 OK runs) — the merge produced nothing there, so the run went down the skip path before the FATAL.Redis probe (08:30 UTC): all 12
gdelt:intel:{tone,vol}:*keysexists=0 ttl=-2;intelligence:gdelt-intel:v1+seed-meta:intelligence:gdelt-intelhealthy, TTL 84976s, seed-meta{"fetchedAt":1784779479007,"recordCount":6}.Related: #5437 / PR #5438 (previous round), #4864 (soft-budget design), #2044 (timeline feature).