fix(seed): time-box seed-conflict-intel GDELT fallback sweep to fit the fetch deadline (#5140) by koala73 · Pull Request #5141 · koala73/worldmonitor

koala73 · 2026-07-10T04:26:55Z

Problem

Since #5134 deployed (2026-07-09 19:25 UTC), seed-conflict-intel crashed on ~6 of 7 cron ticks. The GDELT fallback sweeps 20 countries at concurrency 4; a full-brownout sweep (~375s+) breaches runSeed's fetch deadline → FETCH FAILED → exit 75 → Railway "Crashed" every ~30 min. GDELT is currently rejecting Railway egress (resets/429s) and killing Decodo TLS handshakes (SSL_ERROR_SYSCALL). The existing caught coverage-floor → aux-only → exit 0 path was unreachable in time.

Fix (commit 1 + adversarial-review hardening in commit 2)

Sweep guards — checked before each batch launch, both landing in the caught floor throw → aux-only publish → exit 0:

Phase-anchored launch cutoff (GDELT_SWEEP_BUDGET_MS = 120s): fetchAll anchors the clock at fetch-phase START and passes an absolute deadlineAt down, so slow aux feeds (HAPI is sequential, ~306s worst) shrink the sweep window instead of stacking on top of it.
Floor-unreachable breaker: stop when successes + remaining < 16 — a total brownout stops after 2 batches. When both trip together the stop reason names both (Greptile).

Worst-case bounds made real (adversarial review found the original arithmetic broken by four confirmed mechanisms):

maxRetries: 1 → 0 — a second direct attempt honors GDELT's Retry-After (≤60s sleep, MAX_RETRY_AFTER_MS), voiding any batch bound; the IP-rotating proxy is the designed 429 answer.
proxyMaxAttempts: 2 → 1 — curlFetch is execFileSync (sync, ≤20s): proxy attempts SERIALIZE across the batch, so each attempt adds 4×20s of worst case. Worst batch is now 15s + 4×20s ≈ 95s (92s observed live).
lockTtlMs: 120s → 360s (per the bug(seeders): gdelt-intel / grocery-basket / supply-chain-trade trip the #4786 240s fetch-deadline on legitimate slow-retry runs (retry-budget > deadline); supply-chain loses last-good data #4864 precedent, same failure class): worst single fetch attempt ≈ max(HAPI 306s, cutoff 120s + batch 95s) + writes ≈ 336s fits the lock — restoring runSeed's documented "a healthy seeder never outlives its own lock" invariant (a ~230s run against a 120s lock violated it) — and the deadline becomes lockTtl + 120s margin = 480s, replacing a 10s knife-edge with real margin. Cron is 30min, so a dangling lock costs ≤6 of those minutes.

Known residual (accepted): runSeed wraps fetchAll in withRetry(3) inside ONE shared deadline (fleet-wide, pre-existing), so a transient mid-fetch rejection (e.g. Redis hiccup during extra-key writes) that forces a second full attempt can still overrun in a pathological double-brownout — that yields a one-off graceful exit 75 (self-heals), not the chronic per-tick crash this PR eliminates.

Verification

Failing tests first: behavior tests failed on the unfixed logic (20 launches vs 8), pass after. tests/conflict-gdelt.test.mjs: cutoff stop, zero-launch when aux ate the phase, floor breaker, combined stop reason — 14/14.
Deadline contract moved to tests/seed-fetch-deadline-budget-invariants.test.mjs (the bug(seeders): gdelt-intel / grocery-basket / supply-chain-trade trip the #4786 240s fetch-deadline on legitimate slow-retry runs (retry-budget > deadline); supply-chain loses last-good data #4864 home for this class), deriving from the seeder's exported constants (GDELT_SWEEP_BUDGET_MS, GDELT_COUNTRY_FETCH_OPTS, ACLED_INTEL_LOCK_TTL_MS) instead of literal-vs-literal, and pinning maxRetries === 0 + the lock-invariant inequality + that runSeed actually wires the lock const.
Live probes during the actual brownout (read-only): commit-1 knobs rejected in 92s (budget path); commit-2 knobs rejected in 76s (floor-unreachable after 2 batches) → aux-only exit 0. Before the fix: 244s+ crash.
Sibling suites green (acled-resolution-feed-seed 7/7, forecast-resolution-eval 24/24, invariants 4/4). Biome clean.

Notes

Stops the crash storm; cannot populate acledIntel during a brownout — the durable fix remains provisioning real ACLED_* creds (seed-conflict-intel GRACEFUL·CHRONIC again: #5134 GDELT fallback worst case (~375s) exceeds 240s fetch deadline during GDELT brownout — crashes ~6/7 ticks, conflict:acled key EMPTY in prod #5140 closing note).
Post-merge: verify the next tick logs Done or the early-stop marker instead of FETCH FAILED (service redeploy lagged ~1h on fix(seed): don't crash seed-conflict-intel when ACLED creds are absent #5126 — railway up unblocks).

https://claude.ai/code/session_01LnV7XZ3FuoxC8MuD3gA2UU

…he fetch deadline (#5140) The #5134 fallback swept 20 countries x ~75-92s of direct+proxy retries at concurrency 4 (~375s worst case) against runSeed's 240s fetch deadline, so a GDELT brownout crashed the seeder (exit 75) every ~30-min tick instead of reaching the caught coverage-floor -> aux-only -> exit 0 path. - GDELT_SWEEP_BUDGET_MS=75s wall-clock budget: stop launching batches once spent; 60s aux headroom + 75s budget + 95s in-flight batch = 230s <= 240s (92s in-flight observed live during the 2026-07-10 brownout). - Floor-unreachable breaker: stop sweeping when successes + remaining < 16. - Skipped countries land in failedCountries with a distinct reason so the floor error's failure sample says why. Live probe during the brownout: sweep now rejects in 92s (was 244s+ crash), degrading to the aux-only exit-0 path. Claude-Session: https://claude.ai/code/session_01LnV7XZ3FuoxC8MuD3gA2UU

vercel · 2026-07-10T04:27:03Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
worldmonitor	Ignored	Preview	Jul 10, 2026 4:54am

greptile-apps · 2026-07-10T04:32:48Z

Greptile Summary

This PR adds two early-exit guards to the GDELT fallback sweep in fetchGdeltConflictEvents, preventing the 5-batch worst-case (~375s) from breaching runSeed's 240s fetch deadline during a GDELT brownout. The guards stop launching new batches once either the wall-clock budget (75s) is spent or the 80% coverage floor becomes arithmetically unreachable, falling through to the existing aux-only / exit 0 path instead of crashing with exit 75.

Wall-clock budget guard: budgetSpentAt = now() + 75s is anchored at fetchGdeltConflictEvents entry (after the ~60s aux Promise.allSettled), capping total sweep time so the worst-case in-flight batch (95s observed) still fits the deadline: 60 + 75 + 95 = 230s ≤ 240s.
Floor-reachability guard: stops sweeping as soon as successfulCountries + remaining < 16, avoiding wasted retries when a total brownout makes the floor impossible to reach.
Tests: three new tests verify both guard behaviors and pin the arithmetic contract as a regression guard against future timeout-knob changes.

Confidence Score: 4/5

Safe to merge — the two guards are correctly placed before each batch launch, the arithmetic is sound (230s ≤ 240s), and all three new tests reproduce the failure modes and pin the contract.

The core fix is straightforward and well-tested. The now() injection is correctly wired for both the budget calculation and the per-iteration check. The only gap is a cosmetic one in the reason string when both guards fire simultaneously, which does not affect the outcome.

No files require special attention; the arithmetic test in tests/conflict-gdelt.test.mjs is the key regression guard to keep passing if timeout knobs are ever tuned.

Important Files Changed

Filename	Overview
scripts/seed-conflict-intel.mjs	Adds GDELT_SWEEP_BUDGET_MS constant and two pre-launch guards in fetchGdeltConflictEvents (wall-clock budget and coverage-floor reachability), injecting a now() clock for testability. Logic and arithmetic are correct; one minor ambiguity when both guards fire simultaneously.
tests/conflict-gdelt.test.mjs	Three new tests: budget-guard behavior (8 calls expected), floor-unreachable guard behavior (8 calls expected), and arithmetic contract pinning budget + in-flight + aux ≤ 240s. All traces are correct; fake-time mechanics work as expected with Promise.all + synchronous async functions.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A([fetchGdeltConflictEvents entry]) --> B[budgetSpentAt = now + 75s]
    B --> C{i < 20?}
    C -- No --> G[Loop done]
    G --> H{successfulCountries >= 16?}
    H -- Yes --> I([Return results])
    H -- No --> J([Throw coverage-floor error])
    C -- Yes --> D[remaining = CONFLICT_COUNTRIES.slice i]
    D --> E{overBudget?\nnow >= budgetSpentAt}
    E -- Yes --> K[Mark remaining as sweep budget exhausted]
    K --> L[console.warn early stop]
    L --> G
    E -- No --> F{floorUnreachable?\nsuccessful + remaining < 16}
    F -- Yes --> M[Mark remaining as coverage floor unreachable]
    M --> L
    F -- No --> N[batch = remaining.slice 0-4]
    N --> O[await Promise.all fetchCountryEvents x4]
    O --> P[Accumulate successes and failures]
    P --> Q[await pace 500ms]
    Q --> R[i += 4]
    R --> C

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A([fetchGdeltConflictEvents entry]) --> B[budgetSpentAt = now + 75s]
    B --> C{i < 20?}
    C -- No --> G[Loop done]
    G --> H{successfulCountries >= 16?}
    H -- Yes --> I([Return results])
    H -- No --> J([Throw coverage-floor error])
    C -- Yes --> D[remaining = CONFLICT_COUNTRIES.slice i]
    D --> E{overBudget?\nnow >= budgetSpentAt}
    E -- Yes --> K[Mark remaining as sweep budget exhausted]
    K --> L[console.warn early stop]
    L --> G
    E -- No --> F{floorUnreachable?\nsuccessful + remaining < 16}
    F -- Yes --> M[Mark remaining as coverage floor unreachable]
    M --> L
    F -- No --> N[batch = remaining.slice 0-4]
    N --> O[await Promise.all fetchCountryEvents x4]
    O --> P[Accumulate successes and failures]
    P --> Q[await pace 500ms]
    Q --> R[i += 4]
    R --> C

_{Reviews (1): Last reviewed commit: "fix(seed): time-box seed-conflict-intel ..." | Re-trigger Greptile}

…orst-case runtime (#5140) Adversarial review of the first commit found the 240s-fit arithmetic broken by four confirmed mechanisms; this hardens all of them: - Retry-After exposure: maxRetries 1->0 (a second direct attempt honors GDELT's Retry-After up to 60s, voiding any batch bound; the IP-rotating proxy is the designed 429 answer). - Serialized sync curls: proxyMaxAttempts 2->1 (curlFetch is execFileSync, so proxy attempts block the event loop one at a time -- each attempt adds 4x20s of worst case per batch). - Aux blindness: fetchAll anchors the sweep cutoff (120s) at fetch-phase START and passes an absolute deadlineAt, so slow aux feeds (HAPI sequential, ~306s worst) shrink the sweep window instead of stacking on it. - Knife-edge margin + lock invariant: lockTtlMs 120s->360s per the #4864 precedent -- worst attempt ~336s fits the lock (runSeed's "never outlive your lock" design) and the deadline becomes 480s. Also: combined stop reason when budget and floor trip together (Greptile), and the deadline contract moved to seed-fetch-deadline-budget-invariants.test.mjs deriving from exported constants instead of literal-vs-literal. Live probe during the ongoing brownout: rejects in 76s (floor-unreachable after 2 batches), degrading to aux-only exit 0. Claude-Session: https://claude.ai/code/session_01LnV7XZ3FuoxC8MuD3gA2UU

vercel Bot deployed to Preview July 10, 2026 04:31 View deployment

greptile-apps Bot reviewed Jul 10, 2026

View reviewed changes

Comment thread scripts/seed-conflict-intel.mjs Outdated

koala73 merged commit cd940cc into main Jul 10, 2026
24 checks passed

koala73 deleted the fix/conflict-intel-gdelt-sweep-budget branch July 10, 2026 05:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(seed): time-box seed-conflict-intel GDELT fallback sweep to fit the fetch deadline (#5140)#5141

fix(seed): time-box seed-conflict-intel GDELT fallback sweep to fit the fetch deadline (#5140)#5141
koala73 merged 2 commits into
mainfrom
fix/conflict-intel-gdelt-sweep-budget

koala73 commented Jul 10, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Jul 10, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Jul 10, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

koala73 commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix (commit 1 + adversarial-review hardening in commit 2)

Verification

Notes

Uh oh!

vercel Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Jul 10, 2026

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Flowchart

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

koala73 commented Jul 10, 2026 •

edited

Loading

vercel Bot commented Jul 10, 2026 •

edited

Loading