Skip to content

fix(forecast): resolvability coverage — reach the ≥80%-hard KPI (57% live; #5066 follow-up, Lane A #2) #5091

Description

@koala73

Follow-up to #5066 (closed via #5072). Part of #4930Lane A #2.

#5072 moved the hard-resolvable ratio from 43% → 57% (UCDP→ACLED repoint, political-unrest + cyber-volume promoted to hard, publish-selection boost), but the epic's ≥80%-hard KPI is still not met. #5066 was closed on the fix landing, not on the KPI being reached.

Why this is the most time-sensitive Lane-A item

The resolver is live and accumulating (clock started 2026-07-09). Every forecast published at 57% coverage instead of ≥80% is permanently under-scored — you can't retro-fix the coverage of a forecast that already matured. Closing this gap now directly raises the quality of the data the whole track record is built on, during the wait.

Approach

  • Measure the freshest-snapshot hard-ratio (not the full-window mix) as the KPI signal.
  • The publish-selection boost is supply-limited: you can only publish hard specs if detectors emit them. So audit which detectors still emit judged when a hard metric_key was available, and promote them to hard (Bet-1 D3 deferred political/military/cyber to judged as a cost decision — some are now promotable, each removing an LLM-judge cost).
  • Re-run the coverage measurement over several seed-forecasts cycles to confirm it holds ≥80%, not just spikes.

Done when

≥80% of freshly-published forecasts carry a resolvable hard spec (feed that covers the window), sustained across runs.

Related: #5066, #5072, #4976 (Bet 1 emission), #5007, #4930.


🔄 Reframed 2026-07-10 — the KPI is wrong, and "hard %" is hiding unresolvable specs

Investigating whether #5136/#5138 (conflict→judged) closed this surfaced two things: (a) the "≥80%-hard" target is the wrong metric, and (b) the live hard-ratio is inflated by hard specs that resolve against empty feeds.

Evidence (prod, 2026-07-10)

Live canonical (14 forecasts): 12 hard / 2 judged = 86% nominal hard — looks like the KPI is met. It is not. The count-resolution feeds the hard families point at:

  • conflict:acled-resolution:v1:all:0:0❌ EMPTY (needs ACLED creds we don't have)
  • unrest:events-resolution:v1❌ EMPTYsame root cause (see below)
  • cyber:threats-bootstrap:v2 → ✅ populated
  • supply_chain:chokepoints:v4, intelligence:gpsjam:v2, infra:outages:v1, prediction:markets-bootstrap:v1 → ✅ populated

So of the 12 "hard" specs, 2 conflict-hard + 1 political/unrest-hard resolve against empty feeds → real resolvable-hard ≈ 9/14 = 64%, well below 80%. Raw hard % overstates coverage.

The KPI reframe (do this)

Change the DoD from "≥80% hard" to:

≥80% of freshly-published forecasts carry a resolvable spec — a hard spec whose source feed is populated, OR a judged spec — sustained across runs.

A resolvable judged spec beats an unresolvable hard one. This also makes #5138 (conflict→judged) count toward the KPI instead of appearing to lower it.

Folded in: fix unrest (twin of conflict) — reclassify to judged

unrest:events-resolution:v1 is empty for the identical reason as conflict: seed-unrest-events.mjs:460 only writes it when its ACLED resolution fetch returns data (if (acledResolution?.events?.length)), and we have no ACLED credentials. So unrest hard-count specs resolve against nothing — the same bug #5138 fixed for conflict.

Remaining tasks for this issue

Status 2026-07-10: the code is done and deployed; re-measure shows 100% resolvable at emission. What's left is data-gated verification (~Jul 14), not code.

  • Reframe the KPI to resolvable coverage.
  • Reclassify conflict → judged (fix(forecast): reclassify conflict/ucdp_zone count specs to judged (#5136) #5138, merged + deployed).
  • Reclassify unrest → judged (fix(forecast): reclassify unrest count specs to judged (#5091) #5142, merged + deployed; incl. pending-ledger migration).
  • Deploy to the generator + re-measure — manually deployed seed-forecasts/seed-forecast-resolutions (auto-deploy was blocked by the checkSuites skip). Fresh canonical: conflict/unrest now judged, resolvable coverage = 14/14 = 100% (0 unresolvable-hard), hard% 86→79.
  • Verify judged actually resolves (DATA-GATED, ~Jul 14): 38 pending-judge, 0 due; soonest judged deadline 2026-07-14. The fix(forecast): resolve judged forecast outcomes #5087 resolver reads only a 48h archive window — if the maturing conflict/unrest judged specs stall in pending-judge, the 100%-resolvable is hollow (unresolvable-hard → unresolvable-judged). Scorecard-watch M2 fires on the first judged resolution.
  • ⏳ Confirm resolvable coverage holds ≥80% sustained across several cycles (first measure = 100%).

De-risk (startable now, not data-gated) → tracked in #5183: pre-empt the judged-resolver 48h-archive-window (deadline-centric window + decouple truncated from completeness) before the Jul-14 maturity, so the newly-judged conflict/unrest specs actually resolve instead of forced-VOID. This is the gate that makes the 100%-resolvable real.

Not a problem (audited clear)

cyber (cyber:threats-bootstrap:v2 ✅), supply_chain, gps, infra, prediction_market — populated resolution feeds.

Related: #5136/#5138 (conflict→judged, the template for the unrest fix), #5087 (judged resolver + its 48h-window caveat), #5134 (generator ACLED→GDELT), #4930.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: AI/intelAI analysis, intelligence findings, summarizationenhancementNew feature or requestlane-a#4930 forecast epic — Lane A (do-now, data-independent parallel work)

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions