Skip to content

feat(forecast): Bet 1 — resolvability contract: every published forecast carries a machine-checkable resolution spec (#4930) #4976

Description

@koala73

⚠️ SCOPE UPDATE (2026-07-07) — Bet 1 narrowed at plan/review time. Read before implementing.
Two items below are deferred/changed for Bet 1 (full rationale: see the pinned decision comment on this issue, and docs/plans/2026-07-07-001-feat-forecast-resolvability-contract-plan.md):

  • Publish penalty (scoreForecastReadiness term) → DEFERRED to Bet 2. Bet 1 makes no publish-selection change. The penalty has no floor-free hook in the current pipeline (computeAnalysisPriority and readiness.overall both feed drop/suppression gates), and its editorial cost should land with the Bet-2 payoff.
  • ≥80% hard KPI → EMPIRICAL target. Replaced with the observed hard-spec rate over a real batch; only 4 of 7 domains are hard and downgrade filters bound coverage below the domain-share ceiling. 100% spec coverage (hard-or-judged) is preserved.

The deliverables/KPI checkboxes below are left as originally written for history; treat the two items above as superseded.


feat(forecast): Bet 1 — resolvability contract: every published forecast carries a machine-checkable resolution spec (#4930)

Bet 1 of the verified-forecasting epic #4930. Foundation for the Bet-2 resolver, the Brier scorecard, and the public track record. This bet starts the track-record clock: the ~30–60 days of accumulation needed for a credible public record (~100+ resolutions) begins the day resolution specs enter the 45-day history — not the day the resolver ships. The Bet-2 resolver will retroactively score everything accumulated from this bet's deploy onward.

Prerequisites: ✅ all merged — #4939 (word-boundary matching), #4969 (non-finite guards), #4973 (stable semantic-slot ids). The probability spine is clean and every forecast has a scoreable identity.

Contract

Today a forecast is prose with a number ("Escalation risk: Mali — 72%, 30d"). Nothing defines what "came true" means, so no one can score it. New rule: every published forecast carries a resolution spec emitted by the same detector that scored it — it scored from a feed, so it knows its own ground truth.

New Resolution message on proto/worldmonitor/forecast/v1/forecast.proto:

resolution {
  kind: hard | judged
  metric_key            // WM feed + path that resolves it (e.g. chokepoints-v4 hormuz transitDelta)
  operator, threshold, window   // e.g. ">= 0.20 decline sustained 48h"
  deadline              // epoch ms, derived from time_horizon at publish
  source_feed           // provenance: exact Redis key that will judge it
  question              // judged-kind only: resolution question + retrieval scope
}

hard (target ≥80% of published)

Auto-resolvable from feeds WM already operates — per detector:

Detector Resolution metric
conflict / UCDP zones ACLED/UCDP event count in region ≥ N within window
market / prediction-market quote/commodity level vs threshold (market feeds)
supply_chain / GPS PortWatch transit delta; chokepoint status; jamming intensity
infrastructure outage count/severity (Cloudflare Radar, service statuses)
military theater posture transition (CII/theater feeds)
political unrest event count / CII unrest component threshold
cyber threat-feed volume/severity threshold

judged (fallback)

Resolution question + retrieval scope; resolved in Bet 2 by an LLM judge over the WM news archive — dual-model agreement required, citations mandatory, disagreement → VOID (never guess).

Runtime operation

  1. Emission — new module scripts/_forecast-resolution.mjs (NOT more lines in the 17.8k seed-forecasts.mjs) with per-detector spec builders, invoked where detectors build signals. Zero LLM calls; deterministic metadata.
  2. Publish gate — extend scoreForecastReadiness so an unresolvable forecast is publish-penalized. Intended side effect: forces sharper, falsifiable titles (better product copy).
  3. Persistence — spec rides into forecast:predictions:v2 AND buildHistoryForecastEntry (45d history), enabling retroactive scoring. Also add projections (h24/d7/d30) to the history payload in the same PR (audit: they are not persisted today; needed for multi-horizon Brier).
  4. API — expose resolution fields on get-forecasts (proto → generated types → handler passthrough), so agents and the future track-record page can see what each forecast is committed to.

Audit-inherited constraints (from #4933)

  • Bet-2 scoring recomputes probability deltas from consecutive history snapshots — never trust stored trend/priorProbability (PRIOR_KEY ttl 7200s writes false 'stable' into history after run gaps).
  • Score stored forecasts only; never re-run detectors over historical inputs (military inputs are wall-clock-gated — non-deterministic on replay).
  • Outcome ledger keys on the fix(forecast): stable semantic-slot ids for military and state-derived forecasts (#4933) #4973 stable ids (theater:<theaterId>, state:<bucketId>:<stateUnit.id>, template-title hashes elsewhere).
  • Deadline derivation: generatedAt + horizon (24h/7d/30d); history snapshot generatedAt is the anchor (no per-forecast createdAt persisted).

Deliverables

  • Resolution proto message + regen (docs-stats/api-contract gates will fire — regen + commit)
  • scripts/_forecast-resolution.mjs with spec builders for all 10 detectors + state-derived family
  • scoreForecastReadiness resolvability term + publish penalty
  • History payload: resolution spec + projections persisted
  • get-forecasts exposure
  • Tests per detector class (spec emitted, fields well-formed, deadline math) + readiness-gate tests + history round-trip test
  • docs: docs/panels/forecast.mdx calibration section update

KPI

  • ≥80% of published forecasts carry hard specs; 100% carry at least judged
  • Spec well-formedness enforced by tests (metric_key must reference an existing feed key)
  • Zero additional LLM cost per run

Blocks: Bet 2 (resolver seeder seed-forecast-resolutions.mjs, scorecard, R2 audit ledger). Part of #4930. Constraints source: #4933.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: AI/intelAI analysis, intelligence findings, summarizationenhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions