fix(forecast): reserve run-budget for the full provider chain so market_implications' groq fallback isn't stranded (#4978 follow-up) by koala73 · Pull Request #5003 · koala73/worldmonitor

koala73 · 2026-07-07T04:34:22Z

Problem

/api/health intermittently flips to WARNING because marketImplications writes SEED_ERROR. Caught live in a seed-forecasts run log:

[LLM:combined]  openrouter success  latencyMs=16069      ← fast
[LLM:scenario]  openrouter success  latencyMs=14200      ← fast
[LLM:market_implications] openrouter invalid response: The operation was aborted due to timeout   ← deepseek-v4-flash timed out (~25s)
[LLM:market_implications] groq llm budget exhausted after 51395ms                                 ← fallback STRANDED
[MarketImplications] LLM returned no response — writing error seed-meta                            ← SEED_ERROR

market_implications is the last forecast LLM stage under the shared 150s run budget, with provider order openrouter → groq. #4978's admission guard (MARKET_IMPLICATIONS_MIN_RUN_BUDGET_MS) reserved budget for only the primary attempt (max(provider.timeout) + guard = 30s). So when deepseek-v4-flash times out (~25s), the run budget is drained and the groq fallback can't run — a recoverable timeout gets misreported as a producer failure (SEED_ERROR).

Fix

Reserve the entire provider chain in the admission guard: sum(provider.timeout) + stage guard = 25s + 20s + 5s = 50s, computed from FORECAST_LLM_PROVIDERS so it tracks the config.

An admitted call can now exhaust the primary and still run the fallback.
Below that, it skips and preserves last-good (health stays green; age-based STALE_SEED still escalates past 2h) rather than attempting a chain it can't finish.
Keeps deepseek-v4-flash primary per the feat(llm): migrate LLM stack to OpenRouter DeepSeek tiers — core off Groq, upgraded fallback, config bug fixes #4944 migration — no model/provider-order change.
A genuine both-providers-failed still writes SEED_ERROR (marketImplications is deliberately a hard health signal).

Tests

New reproduction: a 40s budget (covers the primary + guard = old 30s threshold, but not the 50s chain) with the default openrouter→groq order now skips → preserve last-good, no SEED_ERROR. Verified red on the old 30_000 constant (pass 7/fail 1), green on the fix (8/8).
Existing tests that intended admission were recalibrated from 35–40s to 60s for the new threshold; the "skip band" test reframed to the full-chain reservation. All 8 in market-implications-budget-starve.test.mjs pass, plus market-implications-seed-health (4) and -cache-guard (5).

Note / follow-up

If the tail stage is chronically left < 50s budget, it will preserve-last-good more often and could eventually STALE_SEED — that would indicate the 150s forecast run budget is too tight for a 5-stage chain + fallback, worth a separate look (market_implications runs in afterPublish, so a dedicated post-publish budget is an option).

https://claude.ai/code/session_01XKZZy7bzUNTZeJDVWgXbjj

…et_implications' groq fallback isn't stranded (#4978 follow-up) marketImplications flipped /api/health to WARNING (SEED_ERROR) intermittently. Root cause (confirmed from a live seed-forecasts run log): market_implications is the tail LLM stage under the shared 150s run budget, provider order openrouter→groq. #4978's admission guard reserved budget for only the PRIMARY attempt (max provider timeout + guard = 30s). When deepseek-v4-flash timed out (~25s), the run budget was drained and the groq FALLBACK was stranded ("groq llm budget exhausted") — so a recoverable timeout was misreported as SEED_ERROR. Fix: reserve the ENTIRE provider chain in the admission guard — sum(provider.timeout) + stage guard (25s + 20s + 5s = 50s), computed from FORECAST_LLM_PROVIDERS so it tracks the config. An admitted call can now exhaust the primary AND still run the fallback; below that it skips and preserves last-good (green) rather than attempting a chain it can't finish. Keeps deepseek primary (per the #4944 migration). Genuine both-providers-failed still SEED_ERRORs. Tests: new reproduction (40s budget covers the primary but not the chain → skips, no SEED_ERROR — fails on the old 30s constant, passes on the fix). Existing tests that intended admission were recalibrated from 35-40s to 60s for the new threshold. Claude-Session: https://claude.ai/code/session_01XKZZy7bzUNTZeJDVWgXbjj

vercel · 2026-07-07T04:34:28Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
worldmonitor	Ignored	Preview	Jul 7, 2026 5:04am

greptile-apps · 2026-07-07T04:36:35Z

Greptile Summary

The admission guard for the market_implications LLM tail stage is widened from a hardcoded 30_000 ms (primary provider timeout + stage guard) to a dynamically computed 50_000 ms (sum of all provider timeouts + stage guard), ensuring the entire openrouter → groq fallback chain fits within the remaining run budget before admission.

scripts/seed-forecasts.mjs: MARKET_IMPLICATIONS_MIN_RUN_BUDGET_MS is now FORECAST_LLM_PROVIDERS.reduce(…timeout…) + FORECAST_LLM_STAGE_BUDGET_GUARD_MS (= 25s + 20s + 5s = 50s). One inline comment at the budget-capped-timeout attribution logic still names the old 30s value.
tests/market-implications-budget-starve.test.mjs: Adds a direct regression test for the stranded-fallback scenario (40s budget, default openrouter→groq chain → skip, status ok, no SEED_ERROR); existing tests recalibrated from 35–40s to 60s to clear the new 50s threshold.

Confidence Score: 4/5

Safe to merge — the change correctly widens the admission window to cover the full provider chain and is accompanied by a focused regression test for the exact production failure scenario.

The core logic change is a one-liner reduce over a small, stable array, and the calculation matches the values stated in the PR description. The new regression test exercises the previously broken path end-to-end. The only blemish is one inline comment inside callForecastLLM that still refers to the old 30s figure, which could mislead future readers of that attribution logic.

The stale >= 30s comment at line 14712 of scripts/seed-forecasts.mjs is the only line that needs a second look.

Important Files Changed

Filename	Overview
scripts/seed-forecasts.mjs	Changes MARKET_IMPLICATIONS_MIN_RUN_BUDGET_MS from a hardcoded 30s to a dynamic sum of all provider timeouts (25s + 20s) plus the 5s stage guard = 50s; one inline comment still names the old 30s value.
tests/market-implications-budget-starve.test.mjs	Adds a targeted regression test for the reported stranded-fallback scenario (40s budget, default openrouter→groq chain → skip, no SEED_ERROR); recalibrates existing admission thresholds from 35–40s to 60s to match the new 50s guard.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[buildAndSeedMarketImplications] --> B{runBudgetRemainingMs\n>= MIN_RUN_BUDGET_MS\n50s full-chain reservation?}
    B -- No --> C[preserveLastGood\nno SEED_ERROR\nhealth stays green]
    B -- Yes --> D[callForecastLLM\nopenrouter deepseek-v4-flash\ntimeout=25s]
    D -- success --> E[write market_implications seed\nstatus=ok]
    D -- timeout/fail --> F[groq llama-3.3-70b\ntimeout=20s]
    F -- success --> E
    F -- fail --> G[write SEED_ERROR\nhealth WARNING]
    C --> H[age-based STALE_SEED\nescalates after 2h]

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[buildAndSeedMarketImplications] --> B{runBudgetRemainingMs\n>= MIN_RUN_BUDGET_MS\n50s full-chain reservation?}
    B -- No --> C[preserveLastGood\nno SEED_ERROR\nhealth stays green]
    B -- Yes --> D[callForecastLLM\nopenrouter deepseek-v4-flash\ntimeout=25s]
    D -- success --> E[write market_implications seed\nstatus=ok]
    D -- timeout/fail --> F[groq llama-3.3-70b\ntimeout=20s]
    F -- success --> E
    F -- fail --> G[write SEED_ERROR\nhealth WARNING]
    C --> H[age-based STALE_SEED\nescalates after 2h]

Comments Outside Diff (1)

scripts/seed-forecasts.mjs, line 14712-14713 (link)

The inline comment still references the old 30s threshold. The admission guard is now 50s (full-chain reservation), so the guarantee described here applies at the new value.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

_{Reviews (1): Last reviewed commit: "fix(forecast): reserve run-budget for th..." | Re-trigger Greptile}

…-aware (PR #5003 review) Addresses the review of PR #5003 (both P1s + both P2s): #1 (P1) retries strand the fallback: each provider runs under withRetry(3) = 4 attempts, so openrouter's retries drained the run budget and groq was never reached (reviewer reproduced: OpenRouter called twice, Groq never, SEED_ERROR). A static reservation can't cover the retry chain (4×25 + 4×20 = 180s > the 150s budget). Fix: add a per-call `maxRetries` option to callForecastLLM and pass `maxRetries: 0` for market_implications — one attempt per provider, so a slow primary falls straight through to the fallback. New regression asserts openrouter=1, groq=1 (red without it). #2 (P1) reservation ignored provider-order overrides + runnable keys: replaced the static all-provider sum with getMarketImplicationsMinRunBudgetMs(llmOptions), which sums the RESOLVED, key-filtered chain + stage guard. A single-provider override now reserves 30s (not 50s) so a 40s budget admits it; a no-key chain reserves only the guard so a genuine outage surfaces SEED_ERROR instead of hiding as a starve. New regression covers the single-provider-admitted case. #3 (P2) test hygiene: ENV_KEYS now saves/restores GROQ_API_KEY and the global FORECAST_LLM_PROVIDER_ORDER; default-chain tests clear both order envs. #4 (P2) updated the stale "MIN_RUN_BUDGET >= 30s" comment to the chain reservation. Tests: 10/10 in market-implications-budget-starve (both new regressions red without their respective fix), plus seed-health (4) and cache-guard (5) green. Claude-Session: https://claude.ai/code/session_01XKZZy7bzUNTZeJDVWgXbjj

vercel Bot deployed to Preview July 7, 2026 04:37 View deployment

koala73 merged commit 7d1ca89 into main Jul 7, 2026
24 checks passed

koala73 deleted the fix/market-implications-fallback-budget branch July 7, 2026 07:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(forecast): reserve run-budget for the full provider chain so market_implications' groq fallback isn't stranded (#4978 follow-up)#5003

fix(forecast): reserve run-budget for the full provider chain so market_implications' groq fallback isn't stranded (#4978 follow-up)#5003
koala73 merged 2 commits into
mainfrom
fix/market-implications-fallback-budget

koala73 commented Jul 7, 2026

Uh oh!

vercel Bot commented Jul 7, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Jul 7, 2026 •

edited

Loading

Comments Outside Diff (1)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

koala73 commented Jul 7, 2026

Problem

Fix

Tests

Note / follow-up

Uh oh!

vercel Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Flowchart

Comments Outside Diff (1)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented Jul 7, 2026 •

edited

Loading

greptile-apps Bot commented Jul 7, 2026 •

edited

Loading