fix(forecast): stop false SEED_ERROR when market_implications is starved of the shared LLM run budget (#4978) by koala73 · Pull Request #4980 · koala73/worldmonitor

koala73 · 2026-07-06T14:50:40Z

Closes #4978.

Problem

/api/health intermittently reports marketImplications: SEED_ERROR. The market-implications LLM stage is not failing on its own merits — it is being starved of the shared 150s forecast LLM run budget and then mislabels that starve as a producer failure.

market_implications runs LAST (in afterPublish) and shares one run budget (FORECAST_LLM_RUN_BUDGET_MS = 150_000) with every upstream LLM stage. When combined/scenario breach the 25s deepseek/deepseek-v4-flash call timeout (the #4944 model), they drain that budget before the tail stage runs; callForecastLLM throws a budget error and returns null. Pre-fix the caller treated that null identically to a real failure and wrote a status:'error' seed-meta → health SEED_ERROR. It self-heals next tick whenever upstream returns under the timeout.

Evidence (one seed-forecasts container, 2 consecutive runs, 2026-07-06):

14:03:32 (failed): combined+scenario timed out → [LLM:market_implications] budget exhausted after 1ms → writing error seed-meta.
14:15:16 (next tick): upstream fast → market_implications published 5 cards. Self-healed.

Changes

Distinguish budget-starve from real failure. A pre-call guard skips the LLM when the remaining run budget is below what a market-implications call needs (~20s); a null result with the run budget now <=0 is treated the same. On a starve we preserve last-good WITHOUT rewriting seed-meta.fetchedAt, so age-based STALE_SEED (maxStaleMin=120) still escalates if the starve persists past 2h — the gpsjam preserve-last-good design. Genuine provider failures (budget remaining) still surface SEED_ERROR.
Reorder. Run market_implications before the best-effort telemetry in afterPublish (history + deep-forecast snapshots, ~20s R2 trace export) so their wall-clock can't push the tail stage past the run deadline. Recovers the ~20s that tipped the failing run over the 150s deadline. Trace export is unaffected (verified by forecast-trace-export.test.mjs).

Not in this PR (env follow-up)

The root trigger — deepseek-v4-flash latency draining the shared budget — is #4944 territory. A complementary combined-stage groq fallback is a 1-line Railway env change (FORECAST_LLM_COMBINED_PROVIDER_ORDER=openrouter,groq), deliberately left out of code to respect the intentional #4944 provider pin. Recommend applying it so a combined timeout stops wasting ~25s and degrading forecast narratives to fallbackNarratives.

Test plan

tests/market-implications-budget-starve.test.mjs (new): starve → status:'ok' preserved, fetchedAt untouched, no stage-cache write; genuine failure (budget remaining) → still SEED_ERROR. Failing-test-first per Bug Fix Protocol (failed 'error' !== 'ok' pre-fix).
Ordering guard added to market-implications-seed-health.test.mjs.
node --test across market-implications + forecast suites: 336 pass.

https://claude.ai/code/session_019uhnqAKEHTws3QMaUvq58N

…ved of the shared LLM run budget (#4978) market_implications is the LAST forecast LLM stage (afterPublish) and shares the single 150s run budget with every upstream stage. When upstream stages are slow (e.g. deepseek-v4-flash breaching its 25s call timeout on combined/ scenario, #4944), they drain that budget before this tail stage runs; callForecastLLM then throws a budget error and returns null. Pre-fix the caller treated that starve identically to a real LLM failure and wrote a status:'error' seed-meta, so /api/health flipped to SEED_ERROR for benign, self-healing resource contention (observed 2026-07-06 14:03; self-healed 14:15). Two changes: - Distinguish a budget-starve from a real failure. On starve (pre-call guard, or a null result with the run budget now <=0), preserve last-good WITHOUT rewriting seed-meta.fetchedAt, so age-based STALE_SEED (maxStaleMin=120) still escalates if the starve persists past 2h (gpsjam preserve-last-good design). Genuine provider failures with budget remaining still surface SEED_ERROR. - Reorder: run market_implications BEFORE the best-effort telemetry in afterPublish (history + deep-forecast snapshots, ~20s R2 trace export) so that wall-clock can't push the tail stage past the run deadline. This recovers the ~20s that pushed it over the 150s deadline in the failing run. Root trigger (deepseek-v4-flash latency draining the shared budget) is #4944 territory; a complementary combined-stage groq fallback is a 1-line Railway env change (FORECAST_LLM_COMBINED_PROVIDER_ORDER=openrouter,groq), left out of code to respect the intentional #4944 pin. Tests: tests/market-implications-budget-starve.test.mjs (starve preserves last-good + no error meta; genuine failure still errors) + an afterPublish ordering guard in market-implications-seed-health.test.mjs. Failing-test-first per Bug Fix Protocol. Claude-Session: https://claude.ai/code/session_019uhnqAKEHTws3QMaUvq58N

vercel · 2026-07-06T14:50:46Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
worldmonitor	Ignored	Preview	Jul 6, 2026 4:52pm

greptile-apps · 2026-07-06T14:56:48Z

Greptile Summary

This PR fixes a false SEED_ERROR on /api/health caused by market_implications being starved of the shared 150s forecast LLM run budget when upstream stages (e.g. deepseek-v4-flash) consume the budget before the tail stage runs.

Budget guards: a pre-call guard skips the LLM when fewer than 20s of run budget remain, and a post-call guard detects mid-call exhaustion (getRemainingForecastLlmRunBudgetMs() <= 0); both paths call preserveMarketImplicationsLastGoodOnStarve (EXPIRE-refresh only, no seed-meta rewrite) so age-based STALE_SEED still escalates after 2h without suppressing a single starved tick.
Reordering: buildAndSeedMarketImplications is now invoked first in afterPublish, before the ~20s R2 trace export, recovering the wall-clock headroom that was tipping the failing run past the deadline.
Tests: a new market-implications-budget-starve.test.mjs validates the two branches; the "genuine provider failure" test's fetch mock is dead code (all providers are skipped when API keys are absent) — noted in the comment below.

Confidence Score: 4/5

Safe to merge; the budget-guard logic is correct, the reordering is independent of the telemetry it precedes, and the preserve-last-good path avoids touching seed-meta so STALE_SEED escalation remains intact.

The core fix (pre-call guard + post-call discriminator + EXPIRE-refresh) is well-reasoned and matches the described failure evidence. The one test weakness — the genuine provider failure test exercises a no-API-key code path rather than a real fetch failure — means the post-call branch is not fully exercised by the test suite, leaving a small gap in coverage. The production logic itself is sound.

tests/market-implications-budget-starve.test.mjs — the second test's fetch mock is unreachable; the genuine-failure branch in buildAndSeedMarketImplications lacks a test that actually invokes an LLM provider URL.

Important Files Changed

Filename	Overview
scripts/seed-forecasts.mjs	Adds pre-call + post-call run-budget guards to buildAndSeedMarketImplications and reorders the stage before best-effort telemetry in afterPublish; logic is sound and the new getRemainingForecastLlmRunBudgetMs helper is a clean extraction of existing budget-tracking state.
tests/market-implications-budget-starve.test.mjs	New test covers the budget-starve happy path correctly; the genuine provider failure test installs a fetch mock that is never invoked because callForecastLLM skips all providers when API keys are absent.
tests/market-implications-seed-health.test.mjs	Adds an ordering guard that statically verifies market_implications runs before the R2 trace export in afterPublish; straightforward and correct.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant R as afterPublish
    participant MI as buildAndSeedMarketImplications
    participant LLM as callForecastLLM
    participant Redis as Redis

    R->>MI: invoke (now FIRST, before telemetry)
    MI->>MI: check stage cache (fingerprint hit?)
    alt cache hit
        MI->>Redis: redisSet (republish cached cards)
        MI-->>R: return (skipped LLM)
    else cache miss
        MI->>MI: getRemainingForecastLlmRunBudgetMs()
        alt "budget < 20s (pre-call guard)"
            MI->>Redis: EXPIRE data key
            MI->>Redis: EXPIRE meta key
            MI-->>R: return (no SEED_ERROR, last-good preserved)
        else "budget >= 20s"
            MI->>LLM: callForecastLLM(...)
            alt LLM success
                LLM-->>MI: "{ text, model }"
                MI->>Redis: redisSet cards + seed-meta ok
                MI-->>R: return
            else LLM returns null
                LLM-->>MI: null
                MI->>MI: "getRemainingForecastLlmRunBudgetMs() <= 0?"
                alt budget exhausted mid-call (starve)
                    MI->>Redis: EXPIRE data key
                    MI->>Redis: EXPIRE meta key
                    MI-->>R: return (no SEED_ERROR)
                else budget remaining (genuine failure)
                    MI->>Redis: "redisSet seed-meta status=error"
                    MI-->>R: return (SEED_ERROR)
                end
            end
        end
    end
    R->>R: appendHistorySnapshot (best-effort)
    R->>R: writeDeepForecastSnapshot (best-effort)
    R->>R: writeForecastTraceArtifacts ~20s R2 export (best-effort)

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant R as afterPublish
    participant MI as buildAndSeedMarketImplications
    participant LLM as callForecastLLM
    participant Redis as Redis

    R->>MI: invoke (now FIRST, before telemetry)
    MI->>MI: check stage cache (fingerprint hit?)
    alt cache hit
        MI->>Redis: redisSet (republish cached cards)
        MI-->>R: return (skipped LLM)
    else cache miss
        MI->>MI: getRemainingForecastLlmRunBudgetMs()
        alt "budget < 20s (pre-call guard)"
            MI->>Redis: EXPIRE data key
            MI->>Redis: EXPIRE meta key
            MI-->>R: return (no SEED_ERROR, last-good preserved)
        else "budget >= 20s"
            MI->>LLM: callForecastLLM(...)
            alt LLM success
                LLM-->>MI: "{ text, model }"
                MI->>Redis: redisSet cards + seed-meta ok
                MI-->>R: return
            else LLM returns null
                LLM-->>MI: null
                MI->>MI: "getRemainingForecastLlmRunBudgetMs() <= 0?"
                alt budget exhausted mid-call (starve)
                    MI->>Redis: EXPIRE data key
                    MI->>Redis: EXPIRE meta key
                    MI-->>R: return (no SEED_ERROR)
                else budget remaining (genuine failure)
                    MI->>Redis: "redisSet seed-meta status=error"
                    MI-->>R: return (SEED_ERROR)
                end
            end
        end
    end
    R->>R: appendHistorySnapshot (best-effort)
    R->>R: writeDeepForecastSnapshot (best-effort)
    R->>R: writeForecastTraceArtifacts ~20s R2 export (best-effort)

_{Reviews (1): Last reviewed commit: "fix(forecast): stop false SEED_ERROR whe..." | Re-trigger Greptile}

Address PR #4980 review feedback by keeping provider failures distinct from run-budget starvation and restoring stale OK market-implications meta from last-good payloads during starved runs. Also updates the budget-starve tests so the provider-failure regression performs a real provider request.

…vider timeout + harden classification (#4980 review) Addresses the ce-code-review findings on #4980: - #1/#4: raise MARKET_IMPLICATIONS_MIN_RUN_BUDGET_MS 20_000 -> 30_000 (= max provider.timeout 25s + FORECAST_LLM_STAGE_BUDGET_GUARD_MS 5s). At 20s the pre-call guard admitted calls that were then timeout-CAPPED below the provider's own timeout, which is indistinguishable from a genuinely hung provider: a real openrouter timeout in the [20s,30s) band was misclassified as a benign budget starve and its SEED_ERROR suppressed, and 20-25s calls were guaranteed to abort mid-flight (~15s wasted). Admitting only at >=30s gives the provider its full window, so any timeout is attributable and a genuine failure still surfaces SEED_ERROR. - #2: getRemainingForecastLlmBudgetMs now delegates the run-remaining calc to getRemainingForecastLlmRunBudgetMs (was a verbatim duplicate formula). - #3: document the failure-reason classification invariant on the sawProviderFailure / sawBudgetCappedTimeout declarations and the budget-capped-timeout heuristic (comments only, no behavior change). - #5: add tests for the 20-30s guard skip (locks the 30s threshold), the retained mid-call budget_exhausted preserve branch, and its provider_failed symmetry (via the __setForecastLlmCallOverrideForTests seam). - #6: test 1 now sets a real provider key + transport counter so its llmCalls===0 tripwire exercises the guard instead of the !apiKey short-circuit; removed the now-unused budgetStarveFetch helper. test 4 deadline bumped 25s -> 35s to keep passing the new 30s guard. Tests: 605 pass across all seed-forecasts.mjs consumers; biome + unicode clean. Claude-Session: https://claude.ai/code/session_017odw3Pf9ue8RZzxYAQ37P8

koala73 added the bug Something isn't working label Jul 6, 2026

vercel Bot deployed to Preview July 6, 2026 14:53 View deployment

greptile-apps Bot reviewed Jul 6, 2026

View reviewed changes

Comment thread tests/market-implications-budget-starve.test.mjs

koala73 mentioned this pull request Jul 6, 2026

feat(iran-events): flag-gate the whole Iran-events domain, default OFF (war-ended sunset) #4982

Merged

koala73 mentioned this pull request Jul 6, 2026

fix(gpsjam): restore the free gpsjam.org source, retire the Wingbits quota dependency #4987

Merged

koala73 merged commit c3751ba into main Jul 6, 2026
24 checks passed

koala73 deleted the fix/market-implications-budget-starve-4978 branch July 6, 2026 17:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(forecast): stop false SEED_ERROR when market_implications is starved of the shared LLM run budget (#4978)#4980

fix(forecast): stop false SEED_ERROR when market_implications is starved of the shared LLM run budget (#4978)#4980
koala73 merged 3 commits into
mainfrom
fix/market-implications-budget-starve-4978

koala73 commented Jul 6, 2026

Uh oh!

vercel Bot commented Jul 6, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Jul 6, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

koala73 commented Jul 6, 2026

Problem

Changes

Not in this PR (env follow-up)

Test plan

Uh oh!

vercel Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Jul 6, 2026

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Sequence Diagram

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented Jul 6, 2026 •

edited

Loading