epic(forecast): verified probabilistic forecasting — resolution engine, public track record, learned calibration, LLM ensemble

Why this is a bet

The forecast system (MiroFish-inspired) is architecturally ahead of most "AI prediction" products — 10 rule-based detectors fused over ~40 live Redis inputs, Polymarket/Kalshi anchoring, a theater-limited multi-agent simulation that rescores forecasts, and even a self-refining prompt loop with regression rollback. But nothing ever checks whether a forecast came true. Repo-wide: no resolution, no Brier score, no accuracy ledger. forecast:predictions:history:v1 stores 45 days of snapshots that are never scored against reality. The existing "calibration" field is market-price anchoring, not verification; evaluate-forecast-run.mjs / evaluate-forecast-benchmark.mjs are structural/generation QA only.

That missing loop is simultaneously our biggest credibility risk ("AI slop probabilities") and the biggest available selling point: no consumer/prosumer intelligence product publishes a scored, auditable forecasting track record. WorldMonitor can, because — unlike MiroFish or a chat wrapper — we already operate the ground-truth sensors (ACLED/UCDP, PortWatch chokepoints, market/commodity quotes, EIA/JODI energy, outages, GPS jamming, sanctions...). The same feeds that generate a forecast can resolve it automatically.

Context: state of the art moved fast — LLM ensembles now approach superforecaster Brier parity on ForecastBench (projected parity late 2026; AIA Forecaster already matches on market-derived questions). The moat is shifting from "can you forecast" to "can you prove it and wire it into live data." We have the second part.

✅ Status & forward plan — updated 2026-07-09 (supersedes earlier status + comment thread)

The verification loop is live in production. Bets 1–2 (deterministic half) are shipped and deployed; the scorecard is accumulating. What remains is (a) making every day's output fully resolvable and (b) waiting — with instrumentation — for enough resolved forecasts to unlock the learning/product bets.

Live as of 2026-07-09: scorecard entries: 55, scored: 0, pending: 20, pending-judge: 34, void: 1; canonical mix 50% hard / 50% judged (≥80%-hard KPI not yet met — likely supply-limited, needs more judged→hard). scored: 0 because forecasts haven't matured yet, and 62% of the ledger is pending-judge, which cannot score until judged-resolution ships.

The reframe: "waiting for data" gates only three things

Waiting only blocks Bet 3, Bet 4-promote, and Bet 5-launch. Everything else is do-now — and it determines whether the accumulating data is worth anything. Shipping judged-resolution now vs in 30 days is the difference between accumulating at ~90% vs ~50% coverage for a month, and that month is unrecoverable. The wait is a window to make the pipeline good before it fills.

🟢 Lane A — do NOW (makes the accumulating data usable) · tracked: #5091 #5092 #5093 #5094

1. Judged resolution (feat(forecast): Bet 2 — resolution engine + Brier scorecard: score every forecast against ground truth (#4930) #5007) — Batch market and RSS fetching with progressive updates #1 priority. ✅ delivered via fix(forecast): resolve judged forecast outcomes #5087, deployed 07-09. Without it 62% of every day's forecasts never score; waiting cannot fix it, only the judge can.
2. Lock the ≥80%-hard KPI → fix(forecast): resolvability coverage — reach the ≥80%-hard KPI (57% live; #5066 follow-up, Lane A #2) #5091. Live 57% (was 43%); Fix forecast hard resolution coverage #5072 helped but the KPI is still unmet — supply-limited, so promote more detectors judged→hard. Most time-sensitive: forecasts published under-covered are permanently under-scored while the wait runs.
3. Bet 4 ensemble — build the shadow now → feat(forecast): Bet 4 — LLM-ensemble shadow probability on top-K (shadow now, promote on Brier) (Lane A #3) #5093. Shadow-publish an LLM-ensemble probability beside the detector probability so it's scored alongside from day one (promotion is Lane C).
4. Bet 5 — build the page shell now → feat(forecast): Bet 5 — public track-record page shell (build now, launch on N≥100) (Lane A #4) #5092. /forecasts/track-record reading the live get-forecast-scorecard; honestly shows "N resolved, accumulating." Data-independent to build; only the public launch is gated.
5. Bet 6 sim verification / persistent actor memory → feat(forecast): Bet 6 — verifiable sim branches + persistent actor memory + engine bridge (Lane A #5) #5094. No data dependency — fully parallel.
6. Observability. ✅ done 07-09 — scorecard-watch (local script + daily cloud routine) fires the two first-fire milestones (first hard scored / first judged sealed) plus resolver-stall & VOID-spike alerts.

🟡 Lane B — the wait (watch, don't guess dates)

Resolver runs daily 06:00 UTC. Short-horizon at-deadline hard families (price/yesPrice/hexCount) should throw the first real scored YES/NO within ~1–2 weeks; ACLED counts lag by their settlement window. The scorecard resolved/scored count is the progress bar — trigger on it, not the calendar.

🔴 Lane C — wait-gated (unlock on data triggers, not dates)

This week

Ship judged resolution (#5007), confirm coverage climbs toward ≥80%, stand up the scorecard-watch. Then: build Lane A in parallel, watch Lane B fill, trigger Lane C on data.

Also open, off the critical path: #4933 (pre-#4930 hardening residual), #4963 (Brier baseline not LLM-free for market/supply_chain), #5007 remaining deliverables.

Current state (mapped 2026-07-06)

• Generator: scripts/seed-forecasts.mjs (17.8k lines), hourly Railway seeder. Detectors detectConflictScenarios (:982) … detectFromPredictionMarkets (:2163) → hand-tuned probabilities with per-domain caps → market blend 0.4*market + 0.6*internal → hard-coded horizon projection curves → LLM narrative enrichment (Groq llama-3.1-8b-instant → OpenRouter gemini-2.5-flash; LLM never sets probabilities) → publish 10–14 to forecast:predictions:v2.
• MiroFish sim: "Phase 2 — Theater-Limited Simulation Runner" (:16536) — per-theater 2-round LLM sim (3 fixed archetypes: escalation/containment/market_cascade, temp 0) → applySimulationMerge (:11613) adjusts path acceptance (+0.08×conf bucket-channel match, +0.04 actor overlap, −0.12 invalidator, −0.15 stabilizer), applied with one-cycle delay via forecast:sim-decorations:v1.
• Learning that exists: impact-expansion prompt self-refinement (forecast:prompt:impact-expansion:learned, locked scorer + rollback, :15694) — refines prose, not probability truth.
• Separate deterministic scenario engine: run-scenario supply-chain worker (6 templates, exposure×disruption×costShock math) — currently disconnected from forecasts.
• Verification: none (the gap this epic closes).

The roadmap — 6 bets, strict spine 1→2→3

Bet 1 — Resolvability contract (foundation, ~1 wk)

Every published forecast carries a machine-checkable resolution spec, emitted by the same detector that scored it:

resolution {
  kind: hard | judged
  metric_key      // e.g. redis feed + JMESPath: chokepoints v4 hormuz.transitDelta
  operator/threshold, window
  deadline        // derived from time_horizon at publish
  source_feed     // provenance: which WM feed resolves it
}

• Hard-resolvable classes (target ≥80% of published): conflict event counts (ACLED/UCDP), market/commodity levels (quotes feeds), chokepoint transit deltas (PortWatch), outage counts, GPS-jamming intensity, sanctions-list changes, theater posture transitions.
• judged fallback for the rest: resolution question text + retrieval scope, resolved by LLM judge in Bet 2 (dual-model agreement + citations, else VOID).
• Proto: add Resolution to proto/worldmonitor/forecast/v1/forecast.proto; extend scoreForecastReadiness so an unresolvable forecast is publish-penalized. Side effect: forces sharper question titles (today's "Escalation risk in X" vagueness is exactly what makes verification impossible).
• New module, not more monolith — seed-forecasts.mjs is 17.8k lines; resolution spec builders live in a new scripts/_forecast-resolution.mjs imported by the seeder.

Bet 2 — Resolution engine + scorecard (the verification loop, 1–2 wk)

New seeder seed-forecast-resolutions.mjs (daily cron; one runSeed() per process):

• Sweep forecast:predictions:history:v1 for matured forecasts → read metric_key from the same Redis feeds → resolve YES/NO/VOID; judged ones via two independent models over WM news archive with mandatory citations, disagreement → VOID.
• Write per-forecast outcomes to forecast:resolutions:v1 + append-only R2 ledger (audit trail with the evidence snapshot at resolution time).
• Aggregate forecast:scorecard:v1: rolling Brier + log score, per-domain/per-detector calibration curves, vs-market skill (on market-anchored forecasts: did internal beat the Polymarket close?), sample counts.
• Anti-gaming invariants: every published forecast is scored (no cherry-picking); VOID rate reported; scorer code path locked + tested like the existing benchmark suite.
• Start this clock ASAP — public credibility (Bet 5) needs ≥100 resolved forecasts (~30–60 days at 10–14/hr-run with dedup across runs).

Bet 3 — Learned calibration replaces hand-tuned constants (after ~30d of resolutions, ~1 wk)

Repurpose the existing locked-scorer/rollback discipline from prompt-learning to probability-learning:

• Per-detector reliability: isotonic/Platt post-calibration with low-N shrinkage → replaces static domain caps (cyber 0.72, conflict 0.90...) as data accumulates.
• Market-blend weight learned per domain (global 0.4/0.6 today).
• Sim-merge weights (+0.08/−0.12/−0.15) re-fit against which sim-backed paths actually materialized.
• Horizon projection curves (today hard-coded per domain) replaced by empirical decay measured from resolutions.
• Guardrail: frozen-baseline shadow scoring; auto-rollback if rolling Brier regresses (same pattern as PROMPT_LEARNED_KEY rollback).

Bet 4 — LLM forecaster ensemble on the top stories (after Bet 2, 1–2 wk, flag-gated)

Today the LLM never sets a probability — that leaves the strongest known accuracy lever unused (retrieval-grounded LLM ensembles approach superforecaster parity on ForecastBench). For the top K≈6 forecasts per run:

• 3 diverse reasoning passes over the already-built evidence table + news context + theater-sim paths: outside-view/base-rate, inside-view/signal-weighing, adversarial refuter.
• Trimmed-mean aggregate → three-way blend: detector prior + LLM ensemble + market anchor (weights learned in Bet 3).
• Stronger model for this stage only (~18 calls/run, prompt-hash cached like every other stage; respects the per-run LLM budget).
• Prove it before promoting: shadow-publish LLM-blended probability alongside detector probability; Bet 2 scores both; promote when the blend wins on rolling Brier.

Bet 5 — Track record as product (parallel after Bet 2, 1–2 wk build, publish when N≥100)

The selling point:

• Public /forecasts/track-record: rolling Brier + calibration plot, per-domain breakdown, vs-market skill, and a resolved-forecast ledger with receipts (what we said, when, at what probability → what happened → resolving source). "The only intelligence dashboard that publishes its Brier score."
• Forecast cards: detector reliability badge + family resolution history; "Resolved ✓/✗" chips.
• API/MCP: resolution fields on get-forecasts, new get_forecast_track_record tool — agents can audit accuracy before trusting (agent-readiness/orank alignment).
• Brief/digest hook: "forecast movements & resolutions" section (reuses annotateForecastChanges).
• Gating: panel stays free-trial; track-record summary public (it IS the marketing); deep sim, conditional scenarios, per-detector analytics → Pro.

Bet 6 — Simulation that earns its keep (parallel after Bet 1, incremental)

• Verifiable sim: theater invalidators/stabilizers/timingMarkers are already structured — bind them to feed metrics so branch predictions get resolved and scored like forecasts (sim gets its own track record; kills "narrative theater" criticism).
• Persistent actor memory: actor postures/objectives evolve across runs (continuity records exist; today each run mostly resets) — the genuinely valuable MiroFish idea.
• Bridge the two engines: a forecast with routeFacilityKey auto-runs the deterministic supply-chain worker → "if this lands: top exposed countries/sectors" card with real trade math. Demo-able differentiator; zero new data needed.
• Pro custom scenarios: parameterized user-triggered theater sims via the existing trigger-simulation queue (bounded templates, not free-form).

KPIs

• ≥80% of published forecasts carry hard resolution specs (Bet 1)
• 100% of matured forecasts resolved ≤24h after deadline; VOID rate <15% (Bet 2)
• Rolling 90d Brier <0.20 at publish, target <0.15; calibration slope→1 (Bets 3–4)
• Vs-market skill ≥ parity on anchored forecasts (beating Polymarket is the headline if achieved)
• ≥100 resolved forecasts before public track-record launch (Bet 5)

Risks / gotchas

• Self-grading bias: prefer hard metric resolution; judged resolutions need dual-model agreement + citations + audit ledger; VOID over guessing.
• Question quality: resolvability forces title sharpening — expect detector title rewrites (good, but touches published copy).
• Monolith growth: all new stages as separate modules/seeders; seed-forecasts.mjs must not grow another 5k lines. One runSeed() per process (seeder gotcha).
• Cost: resolver is deterministic reads; LLM ensemble bounded to top-K with prompt-hash caching per existing cache-identity discipline.
• Public API exposure: track-record endpoints are anon-read → cache headers + client-caller sweep per the premium-gating 4xx-flood pattern.

References (state of the art)

• ForecastBench (dynamic AI forecasting benchmark; superforecaster Brier 0.081 vs best-LLM ~0.10, parity projected late 2026)
• AIA Forecaster: matches superforecasters on market-derived questions (FB-Market 0.0753 vs human 0.0740)
• Eternis: calibration-aware reward design (asymmetric penalty for confident-wrong) as the training/eval principle — we apply the same asymmetry in scorecard weighting
• MiroFish (github.com/666ghj/MiroFish): GraphRAG + agent personas + parallel simulation + report agent — our theater sim is the pragmatic subset; Bet 6 adopts its persistent-memory idea without the thousand-agent cost