Skip to content

fix(forecast): conflict/ucdp_zone count-resolution reads empty conflict:acled-resolution:v1 (no ACLED creds) — specs unresolvable (P2, #5097) #5136

Description

@koala73

Part of #5097. The resolution-side twin of #5099 (surfaced while merging #5106 into PR #5134). Verified against main + prod 2026-07-09.

Problem

Conflict count-resolution reads a feed that requires ACLED credentials we don't have, so it's empty — those specs can't resolve.

  • CONFLICT_COUNT_SOURCE_FEED = 'conflict:acled-resolution:v1:all:0:0' (scripts/_forecast-resolution.mjs:44).
  • Both conflict and ucdp_zone count families resolve against it (:190-191), and the emitted spec uses it directly (:393-394: ${CONFLICT_COUNT_SOURCE_FEED}|count(country==…)).
  • It's produced only by seed-conflict-intel's paginated ACLED resolution fetch (fix(forecast): fail ACLED seed when primary feed is unavailable #5106, ACLED_RESOLUTION_CACHE_KEY), which — like the display fetch — no-ops without ACLED creds.
  • Prod: EXISTS conflict:acled-resolution:v1:all:0:0 = 0.

Consequence

Every conflict / ucdp_zone hard count spec resolves against an empty feed → the #5063 coverage guard keeps them pending/VOID (never a real YES/NO). So the flagship conflict domain's hard-count forecasts are unresolvable, silently. This is a coverage gap (not Brier corruption — the guard fails safe), but it caps conflict hard-coverage at ~0 for the count family.

Note: PR #5134 does NOT fix this

#5134 (ACLED-or-GDELT) populates the display key conflict:acled:v1:all:0:0 (the generator EMA input), not the resolution key conflict:acled-resolution:v1:all:0:0. Different feed, different consumer.

Options (decide)

  1. Reclassify conflict count specs → judged — now that judged resolution is live (fix(forecast): resolve judged forecast outcomes #5087), "did conflict escalate in X by the deadline" is a natural judged question. Cleanest; no ACLED needed. Tradeoff: lowers hard-ratio (counter to fix(forecast): resolvability coverage — reach the ≥80%-hard KPI (57% live; #5066 follow-up, Lane A #2) #5091), adds LLM-judge cost.
  2. Extend the GDELT fallback to also populate conflict:acled-resolution:v1 from the same article-volume events. Keyless, reuses fix(forecast): fall back to GDELT for conflict events when ACLED creds absent (#5099) #5134. Fidelity caveat: the count thresholds (e.g. ≥66 events) were calibrated for ACLED event counts — article volume is a different scale, so thresholds would need re-derivation or the specs would mis-resolve.
  3. Repoint conflict count-resolution to UCDP (conflict:ucdp-events:v1) — real event counts, but Fix forecast hard resolution coverage #5072 deliberately moved OFF UCDP because its ~months lag can't resolve within the forecast window (→ perpetual count_source_lags_deadline). Least attractive.

Recommendation: (1) for correctness (judged is live and honest), or (2) only with re-derived thresholds. Confirm before implementing.

Related: #5097, #5099, #5134, #5106, #5072, #5063, #5087, #5091.


✅ Recommendation — Option 1: reclassify conflict + ucdp_zone to judged

Route these two families to judged resolution (live since #5087) instead of hard-count against the empty ACLED feed. Rationale:

  • Option 2 (GDELT → count threshold) is unsound. The thresholds are derived from an event tally × CONFLICT_ESCALATION_RATIO (_forecast-resolution.mjs:388, e.g. "≥66 events in-horizon"). GDELT article volume is a different scale entirely — one event yields dozens of articles, and volume tracks media attention, not event count — so count(articles) >= 66 would false-YES constantly (Ukraine alone has thousands of articles/day). Re-deriving thresholds against article volume has no clean grounding.
  • Option 3 (UCDP) is why Fix forecast hard resolution coverage #5072 exists — UCDP's ~months lag → perpetual count_source_lags_deadline, never resolves in-window.
  • Judged works todayfix(forecast): resolve judged forecast outcomes #5087 resolves "did conflict escalate in X by the deadline" from the WM news archive with dual-model agreement + citations. An unresolvable hard spec is strictly worse than a resolvable judged one: coverage that resolves beats coverage that pends forever.
  • Aligns with fix(forecast): resolvability coverage — reach the ≥80%-hard KPI (57% live; #5066 follow-up, Lane A #2) #5091's reframing that the KPI should measure resolvable coverage, not nominally-hard coverage.

Tradeoffs (accept): lowers the hard-ratio (interacts with #5091 — see step 4); adds ~2 LLM calls per conflict forecast at resolution (bounded by the judge run budget — see step 3). Fully reversible if ACLED creds are ever obtained (step 5).

Implementation plan

1. Emission change — scripts/_forecast-resolution.mjs, deriveHardMetrics.
Make the case 'conflict': case 'ucdp_zone': branch return null. buildHardSpec already falls back to buildJudgedSpec when metrics are null (:538-541), so the forecast emits { kind: 'judged', question: buildQuestion(pred), deadline: deriveDeadline(...) }. Keep the threshold-derivation code beneath a short-circuit + comment (// conflict:acled-resolution:v1 is unpopulated without ACLED credentials (#5136); revert this return to re-enable hard-count once a real event-count feed exists) so re-enabling is a one-line revert. (Alternative: drop conflict/ucdp_zone from FAMILY_FEED + DOMAIN_TO_HARD_FAMILIES — cleaner semantically but touches the family maps; the return null is the smallest reversible change.)

2. Sharpen the judged question (recommended, small).
buildQuestion is generic (Will "<title>" … resolve YES within its horizon?). A family-aware question resolves better: for conflict/ucdp_zone → "Did experience a materially escalated level of armed conflict (vs its recent baseline) by ?" Gate on family so other domains keep the generic question. Optional fast-follow if we want to ship step 1 first.

3. Judge run budget.
Conflict is a high-volume domain, so pending-judge inflow rises. Confirm FORECAST_RESOLUTION_JUDGE_MAX_PER_RUN (default 12) drains conflict + existing judged inflow; bump if the backlog grows monotonically (watch the scorecard pendingJudge trend — the #5062 scorecard-watch already tracks it).

4. Re-measure #5091 coverage after deploy.
This moves conflict/ucdp_zone from hard→judged, lowering the raw hard-ratio. Update #5091's KPI framing to resolvable coverage (hard-with-a-live-feed OR judged), and re-measure — reclassifying an unresolvable family raises real resolvable coverage even as it lowers nominal hard-ratio.

5. Leave the ACLED-resolution path dormant, not deleted.
#5106's conflict:acled-resolution:v1 seeding (paginated ACLED fetch) becomes unused but harmless (already produces nothing). Leave it + a comment so obtaining ACLED creds later = revert step 1 and it lights up. Don't rip it out.

6. Tests (RED-first) + verify on deploy.

  • Unit: a conflict/ucdp_zone pred with a ucdp/conflict_events signal now yields kind: 'judged' with a non-empty question (was hard-count); unrest/cyber/supply_chain still emit hard (change is scoped).
  • Deploy: get-forecasts shows conflict specs as judged; scorecard shows them moving pending-judge → resolved via the judge loop instead of sitting hard-pending forever.

Scope: ~1 file for the core change (step 1), +1 small edit (step 2), +tests. No new infra. Judged resolver (#5087) already deployed.

Related: #5087 (judged resolver this leans on), #5091 (KPI reframe), #5106 (ACLED-resolution feed left dormant), #5134.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: AI/intelAI analysis, intelligence findings, summarizationbugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions