Skip to content

fix(forecast): resolvability coverage — 43% hard vs ≥80% KPI + UCDP count specs unresolvable off lagged feed (#4930) #5066

Description

@koala73

fix(forecast): resolvability coverage — 43% hard vs ≥80% KPI, and UCDP count specs can never resolve off the lagged feed (Bet-1/Bet-2 follow-up, #4930)

Now that the resolver is deployed (#5062) and the count-family false-NO is fixed (#5063), the binding constraint on the track record is coverage — what fraction of published forecasts can ever produce a scored YES/NO. Measured on prod 2026-07-09, it's far below the epic's ≥80%-hard KPI, and the largest "hard" family is effectively unresolvable with its current source.

Evidence (prod, 2026-07-09)

So the effective trustworthy scored coverage at deploy is ~29%

(4 point-window hard families — price / yesPrice / hexCount / riskScore — minus judged, minus the perpetual-pending UCDP counts.) That's not enough for a credible public Brier (Bet 5 needs N≥100 resolved), and it's why the scorecard reads scored: 0 today.

Options (not mutually exclusive)

  1. Repoint conflict counts to a timely, near-real-time feed (ACLED-style current events) whose maxTs actually advances past the deadline within the resolution window. UCDP is an academic lagged dataset — wrong tool for "events between now and 30d out." This is the single biggest hard-coverage win.
  2. Raise the hard-ratio — audit which detectors emit judged when a hard metric_key was available (Bet 1 D3 deferred political/military/cyber to judged as a cost decision; some may be promotable to hard now). Each promotion removes an LLM-judge cost and adds a deterministic resolution.
  3. Ship judged resolution (already scoped in feat(forecast): Bet 2 — resolution engine + Brier scorecard: score every forecast against ground truth (#4930) #5007) — the only path to scoring the genuinely-non-hard 57%. Dual-model + citations + VOID-on-disagreement.
  4. Recompute the Bet-5 N≥100 timeline under realistic coverage — at ~29% trustworthy hard, minus VOID rate and settlement lags, time-to-100 is materially longer than the epic's "30–60 days."

KPI to restore

  • ≥80% of published forecasts carry a resolvable spec (hard with a feed that covers the window, or judged with a working judge) — not just a well-formed one.

Related: #4930, #5007 (judged resolver), #5024, #5062 (deploy), #5063 (count-coverage guard), #4976 (Bet 1 emission).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions