You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
⚠️ SCOPE UPDATE (2026-07-07) — Bet 1 narrowed at plan/review time. Read before implementing.
Two items below are deferred/changed for Bet 1 (full rationale: see the pinned decision comment on this issue, and docs/plans/2026-07-07-001-feat-forecast-resolvability-contract-plan.md):
Publish penalty (scoreForecastReadiness term) → DEFERRED to Bet 2. Bet 1 makes no publish-selection change. The penalty has no floor-free hook in the current pipeline (computeAnalysisPriority and readiness.overall both feed drop/suppression gates), and its editorial cost should land with the Bet-2 payoff.
≥80% hard KPI → EMPIRICAL target. Replaced with the observed hard-spec rate over a real batch; only 4 of 7 domains are hard and downgrade filters bound coverage below the domain-share ceiling. 100% spec coverage (hard-or-judged) is preserved.
The deliverables/KPI checkboxes below are left as originally written for history; treat the two items above as superseded.
feat(forecast): Bet 1 — resolvability contract: every published forecast carries a machine-checkable resolution spec (#4930)
Bet 1 of the verified-forecasting epic #4930. Foundation for the Bet-2 resolver, the Brier scorecard, and the public track record. This bet starts the track-record clock: the ~30–60 days of accumulation needed for a credible public record (~100+ resolutions) begins the day resolution specs enter the 45-day history — not the day the resolver ships. The Bet-2 resolver will retroactively score everything accumulated from this bet's deploy onward.
Prerequisites: ✅ all merged — #4939 (word-boundary matching), #4969 (non-finite guards), #4973 (stable semantic-slot ids). The probability spine is clean and every forecast has a scoreable identity.
Contract
Today a forecast is prose with a number ("Escalation risk: Mali — 72%, 30d"). Nothing defines what "came true" means, so no one can score it. New rule: every published forecast carries a resolution spec emitted by the same detector that scored it — it scored from a feed, so it knows its own ground truth.
New Resolution message on proto/worldmonitor/forecast/v1/forecast.proto:
resolution {
kind: hard | judged
metric_key // WM feed + path that resolves it (e.g. chokepoints-v4 hormuz transitDelta)
operator, threshold, window // e.g. ">= 0.20 decline sustained 48h"
deadline // epoch ms, derived from time_horizon at publish
source_feed // provenance: exact Redis key that will judge it
question // judged-kind only: resolution question + retrieval scope
}
hard (target ≥80% of published)
Auto-resolvable from feeds WM already operates — per detector:
Detector
Resolution metric
conflict / UCDP zones
ACLED/UCDP event count in region ≥ N within window
Resolution question + retrieval scope; resolved in Bet 2 by an LLM judge over the WM news archive — dual-model agreement required, citations mandatory, disagreement → VOID (never guess).
Runtime operation
Emission — new module scripts/_forecast-resolution.mjs (NOT more lines in the 17.8k seed-forecasts.mjs) with per-detector spec builders, invoked where detectors build signals. Zero LLM calls; deterministic metadata.
Publish gate — extend scoreForecastReadiness so an unresolvable forecast is publish-penalized. Intended side effect: forces sharper, falsifiable titles (better product copy).
Persistence — spec rides into forecast:predictions:v2 AND buildHistoryForecastEntry (45d history), enabling retroactive scoring. Also add projections (h24/d7/d30) to the history payload in the same PR (audit: they are not persisted today; needed for multi-horizon Brier).
API — expose resolution fields on get-forecasts (proto → generated types → handler passthrough), so agents and the future track-record page can see what each forecast is committed to.
Bet-2 scoring recomputes probability deltas from consecutive history snapshots — never trust stored trend/priorProbability (PRIOR_KEY ttl 7200s writes false 'stable' into history after run gaps).
Score stored forecasts only; never re-run detectors over historical inputs (military inputs are wall-clock-gated — non-deterministic on replay).
feat(forecast): Bet 1 — resolvability contract: every published forecast carries a machine-checkable resolution spec (#4930)
Bet 1 of the verified-forecasting epic #4930. Foundation for the Bet-2 resolver, the Brier scorecard, and the public track record. This bet starts the track-record clock: the ~30–60 days of accumulation needed for a credible public record (~100+ resolutions) begins the day resolution specs enter the 45-day history — not the day the resolver ships. The Bet-2 resolver will retroactively score everything accumulated from this bet's deploy onward.
Prerequisites: ✅ all merged — #4939 (word-boundary matching), #4969 (non-finite guards), #4973 (stable semantic-slot ids). The probability spine is clean and every forecast has a scoreable identity.
Contract
Today a forecast is prose with a number ("Escalation risk: Mali — 72%, 30d"). Nothing defines what "came true" means, so no one can score it. New rule: every published forecast carries a resolution spec emitted by the same detector that scored it — it scored from a feed, so it knows its own ground truth.
New
Resolutionmessage onproto/worldmonitor/forecast/v1/forecast.proto:hard(target ≥80% of published)Auto-resolvable from feeds WM already operates — per detector:
judged(fallback)Resolution question + retrieval scope; resolved in Bet 2 by an LLM judge over the WM news archive — dual-model agreement required, citations mandatory, disagreement → VOID (never guess).
Runtime operation
scripts/_forecast-resolution.mjs(NOT more lines in the 17.8k seed-forecasts.mjs) with per-detector spec builders, invoked where detectors build signals. Zero LLM calls; deterministic metadata.scoreForecastReadinessso an unresolvable forecast is publish-penalized. Intended side effect: forces sharper, falsifiable titles (better product copy).forecast:predictions:v2ANDbuildHistoryForecastEntry(45d history), enabling retroactive scoring. Also addprojections(h24/d7/d30) to the history payload in the same PR (audit: they are not persisted today; needed for multi-horizon Brier).get-forecasts(proto → generated types → handler passthrough), so agents and the future track-record page can see what each forecast is committed to.Audit-inherited constraints (from #4933)
trend/priorProbability(PRIOR_KEY ttl 7200s writes false 'stable' into history after run gaps).theater:<theaterId>,state:<bucketId>:<stateUnit.id>, template-title hashes elsewhere).generatedAt + horizon(24h/7d/30d); history snapshotgeneratedAtis the anchor (no per-forecast createdAt persisted).Deliverables
Resolutionproto message + regen (docs-stats/api-contract gates will fire — regen + commit)scripts/_forecast-resolution.mjswith spec builders for all 10 detectors + state-derived familyscoreForecastReadinessresolvability term + publish penaltyget-forecastsexposuredocs/panels/forecast.mdxcalibration section updateKPI
hardspecs; 100% carry at leastjudgedBlocks: Bet 2 (resolver seeder
seed-forecast-resolutions.mjs, scorecard, R2 audit ledger). Part of #4930. Constraints source: #4933.