You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Both conflict and ucdp_zone count families resolve against it (:190-191), and the emitted spec uses it directly (:393-394: ${CONFLICT_COUNT_SOURCE_FEED}|count(country==…)).
Every conflict / ucdp_zone hard count spec resolves against an empty feed → the #5063 coverage guard keeps them pending/VOID (never a real YES/NO). So the flagship conflict domain's hard-count forecasts are unresolvable, silently. This is a coverage gap (not Brier corruption — the guard fails safe), but it caps conflict hard-coverage at ~0 for the count family.
#5134 (ACLED-or-GDELT) populates the display key conflict:acled:v1:all:0:0 (the generator EMA input), not the resolution key conflict:acled-resolution:v1:all:0:0. Different feed, different consumer.
Extend the GDELT fallback to also populate conflict:acled-resolution:v1 from the same article-volume events. Keyless, reuses fix(forecast): fall back to GDELT for conflict events when ACLED creds absent (#5099) #5134. Fidelity caveat: the count thresholds (e.g. ≥66 events) were calibrated for ACLED event counts — article volume is a different scale, so thresholds would need re-derivation or the specs would mis-resolve.
Repoint conflict count-resolution to UCDP (conflict:ucdp-events:v1) — real event counts, but Fix forecast hard resolution coverage #5072 deliberately moved OFF UCDP because its ~months lag can't resolve within the forecast window (→ perpetual count_source_lags_deadline). Least attractive.
Recommendation: (1) for correctness (judged is live and honest), or (2) only with re-derived thresholds. Confirm before implementing.
Route these two families to judged resolution (live since #5087) instead of hard-count against the empty ACLED feed. Rationale:
Option 2 (GDELT → count threshold) is unsound. The thresholds are derived from an event tally × CONFLICT_ESCALATION_RATIO (_forecast-resolution.mjs:388, e.g. "≥66 events in-horizon"). GDELT article volume is a different scale entirely — one event yields dozens of articles, and volume tracks media attention, not event count — so count(articles) >= 66 would false-YES constantly (Ukraine alone has thousands of articles/day). Re-deriving thresholds against article volume has no clean grounding.
Judged works today — fix(forecast): resolve judged forecast outcomes #5087 resolves "did conflict escalate in X by the deadline" from the WM news archive with dual-model agreement + citations. An unresolvable hard spec is strictly worse than a resolvable judged one: coverage that resolves beats coverage that pends forever.
Tradeoffs (accept): lowers the hard-ratio (interacts with #5091 — see step 4); adds ~2 LLM calls per conflict forecast at resolution (bounded by the judge run budget — see step 3). Fully reversible if ACLED creds are ever obtained (step 5).
Implementation plan
1. Emission change — scripts/_forecast-resolution.mjs, deriveHardMetrics.
Make the case 'conflict': case 'ucdp_zone': branch return null. buildHardSpec already falls back to buildJudgedSpec when metrics are null (:538-541), so the forecast emits { kind: 'judged', question: buildQuestion(pred), deadline: deriveDeadline(...) }. Keep the threshold-derivation code beneath a short-circuit + comment (// conflict:acled-resolution:v1 is unpopulated without ACLED credentials (#5136); revert this return to re-enable hard-count once a real event-count feed exists) so re-enabling is a one-line revert. (Alternative: drop conflict/ucdp_zone from FAMILY_FEED + DOMAIN_TO_HARD_FAMILIES — cleaner semantically but touches the family maps; the return null is the smallest reversible change.)
2. Sharpen the judged question (recommended, small). buildQuestion is generic (Will "<title>" … resolve YES within its horizon?). A family-aware question resolves better: for conflict/ucdp_zone → "Did experience a materially escalated level of armed conflict (vs its recent baseline) by ?" Gate on family so other domains keep the generic question. Optional fast-follow if we want to ship step 1 first.
3. Judge run budget.
Conflict is a high-volume domain, so pending-judge inflow rises. Confirm FORECAST_RESOLUTION_JUDGE_MAX_PER_RUN (default 12) drains conflict + existing judged inflow; bump if the backlog grows monotonically (watch the scorecard pendingJudge trend — the #5062 scorecard-watch already tracks it).
4. Re-measure #5091 coverage after deploy.
This moves conflict/ucdp_zone from hard→judged, lowering the raw hard-ratio. Update #5091's KPI framing to resolvable coverage (hard-with-a-live-feed OR judged), and re-measure — reclassifying an unresolvable family raises real resolvable coverage even as it lowers nominal hard-ratio.
5. Leave the ACLED-resolution path dormant, not deleted. #5106's conflict:acled-resolution:v1 seeding (paginated ACLED fetch) becomes unused but harmless (already produces nothing). Leave it + a comment so obtaining ACLED creds later = revert step 1 and it lights up. Don't rip it out.
6. Tests (RED-first) + verify on deploy.
Unit: a conflict/ucdp_zone pred with a ucdp/conflict_events signal now yields kind: 'judged' with a non-empty question (was hard-count); unrest/cyber/supply_chain still emit hard (change is scoped).
Deploy: get-forecasts shows conflict specs as judged; scorecard shows them moving pending-judge → resolved via the judge loop instead of sitting hard-pending forever.
Scope: ~1 file for the core change (step 1), +1 small edit (step 2), +tests. No new infra. Judged resolver (#5087) already deployed.
Related: #5087 (judged resolver this leans on), #5091 (KPI reframe), #5106 (ACLED-resolution feed left dormant), #5134.
Part of #5097. The resolution-side twin of #5099 (surfaced while merging #5106 into PR #5134). Verified against main + prod 2026-07-09.
Problem
Conflict count-resolution reads a feed that requires ACLED credentials we don't have, so it's empty — those specs can't resolve.
CONFLICT_COUNT_SOURCE_FEED = 'conflict:acled-resolution:v1:all:0:0'(scripts/_forecast-resolution.mjs:44).conflictanducdp_zonecount families resolve against it (:190-191), and the emitted spec uses it directly (:393-394:${CONFLICT_COUNT_SOURCE_FEED}|count(country==…)).seed-conflict-intel's paginated ACLED resolution fetch (fix(forecast): fail ACLED seed when primary feed is unavailable #5106,ACLED_RESOLUTION_CACHE_KEY), which — like the display fetch — no-ops without ACLED creds.EXISTS conflict:acled-resolution:v1:all:0:0= 0.Consequence
Every conflict / ucdp_zone hard count spec resolves against an empty feed → the #5063 coverage guard keeps them
pending/VOID(never a real YES/NO). So the flagship conflict domain's hard-count forecasts are unresolvable, silently. This is a coverage gap (not Brier corruption — the guard fails safe), but it caps conflict hard-coverage at ~0 for the count family.Note: PR #5134 does NOT fix this
#5134 (ACLED-or-GDELT) populates the display key
conflict:acled:v1:all:0:0(the generator EMA input), not the resolution keyconflict:acled-resolution:v1:all:0:0. Different feed, different consumer.Options (decide)
judged— now that judged resolution is live (fix(forecast): resolve judged forecast outcomes #5087), "did conflict escalate in X by the deadline" is a natural judged question. Cleanest; no ACLED needed. Tradeoff: lowers hard-ratio (counter to fix(forecast): resolvability coverage — reach the ≥80%-hard KPI (57% live; #5066 follow-up, Lane A #2) #5091), adds LLM-judge cost.conflict:acled-resolution:v1from the same article-volume events. Keyless, reuses fix(forecast): fall back to GDELT for conflict events when ACLED creds absent (#5099) #5134. Fidelity caveat: the count thresholds (e.g.≥66 events) were calibrated for ACLED event counts — article volume is a different scale, so thresholds would need re-derivation or the specs would mis-resolve.conflict:ucdp-events:v1) — real event counts, but Fix forecast hard resolution coverage #5072 deliberately moved OFF UCDP because its ~months lag can't resolve within the forecast window (→ perpetualcount_source_lags_deadline). Least attractive.Recommendation: (1) for correctness (judged is live and honest), or (2) only with re-derived thresholds. Confirm before implementing.
Related: #5097, #5099, #5134, #5106, #5072, #5063, #5087, #5091.
✅ Recommendation — Option 1: reclassify conflict + ucdp_zone to
judgedRoute these two families to judged resolution (live since #5087) instead of hard-count against the empty ACLED feed. Rationale:
CONFLICT_ESCALATION_RATIO(_forecast-resolution.mjs:388, e.g. "≥66 events in-horizon"). GDELT article volume is a different scale entirely — one event yields dozens of articles, and volume tracks media attention, not event count — socount(articles) >= 66would false-YES constantly (Ukraine alone has thousands of articles/day). Re-deriving thresholds against article volume has no clean grounding.count_source_lags_deadline, never resolves in-window.Tradeoffs (accept): lowers the hard-ratio (interacts with #5091 — see step 4); adds ~2 LLM calls per conflict forecast at resolution (bounded by the judge run budget — see step 3). Fully reversible if ACLED creds are ever obtained (step 5).
Implementation plan
1. Emission change —
scripts/_forecast-resolution.mjs,deriveHardMetrics.Make the
case 'conflict': case 'ucdp_zone':branchreturn null.buildHardSpecalready falls back tobuildJudgedSpecwhen metrics are null (:538-541), so the forecast emits{ kind: 'judged', question: buildQuestion(pred), deadline: deriveDeadline(...) }. Keep the threshold-derivation code beneath a short-circuit + comment (// conflict:acled-resolution:v1 is unpopulated without ACLED credentials (#5136); revert this return to re-enable hard-count once a real event-count feed exists) so re-enabling is a one-line revert. (Alternative: dropconflict/ucdp_zonefromFAMILY_FEED+DOMAIN_TO_HARD_FAMILIES— cleaner semantically but touches the family maps; thereturn nullis the smallest reversible change.)2. Sharpen the judged question (recommended, small).
buildQuestionis generic (Will "<title>" … resolve YES within its horizon?). A family-aware question resolves better: for conflict/ucdp_zone → "Did experience a materially escalated level of armed conflict (vs its recent baseline) by ?" Gate on family so other domains keep the generic question. Optional fast-follow if we want to ship step 1 first.3. Judge run budget.
Conflict is a high-volume domain, so pending-judge inflow rises. Confirm
FORECAST_RESOLUTION_JUDGE_MAX_PER_RUN(default 12) drains conflict + existing judged inflow; bump if the backlog grows monotonically (watch the scorecardpendingJudgetrend — the #5062 scorecard-watch already tracks it).4. Re-measure #5091 coverage after deploy.
This moves conflict/ucdp_zone from hard→judged, lowering the raw hard-ratio. Update #5091's KPI framing to resolvable coverage (hard-with-a-live-feed OR judged), and re-measure — reclassifying an unresolvable family raises real resolvable coverage even as it lowers nominal hard-ratio.
5. Leave the ACLED-resolution path dormant, not deleted.
#5106's
conflict:acled-resolution:v1seeding (paginated ACLED fetch) becomes unused but harmless (already produces nothing). Leave it + a comment so obtaining ACLED creds later = revert step 1 and it lights up. Don't rip it out.6. Tests (RED-first) + verify on deploy.
kind: 'judged'with a non-emptyquestion(was hard-count); unrest/cyber/supply_chain still emit hard (change is scoped).get-forecastsshows conflict specs asjudged; scorecard shows them movingpending-judge → resolvedvia the judge loop instead of sitting hard-pendingforever.Scope: ~1 file for the core change (step 1), +1 small edit (step 2), +tests. No new infra. Judged resolver (#5087) already deployed.
Related: #5087 (judged resolver this leans on), #5091 (KPI reframe), #5106 (ACLED-resolution feed left dormant), #5134.