fix(forecast): reclassify conflict/ucdp_zone count specs to judged (#5136) by koala73 · Pull Request #5138 · koala73/worldmonitor

koala73 · 2026-07-09T20:09:10Z

Summary

Implements the #5136 recommendation: reclassify conflict / ucdp_zone count specs to judged, since their hard-count feed (conflict:acled-resolution:v1) is empty without ACLED credentials — every conflict count spec was otherwise unresolvable (pending/VOID forever). Judged resolution is live (#5087).

Why judged (not GDELT-count or UCDP)

The count thresholds are derived from an event tally × CONFLICT_ESCALATION_RATIO (e.g. "≥66 events in-horizon"). GDELT article volume is a different scale (one event → dozens of articles), so count(articles) >= 66 would false-YES constantly.
UCDP's ~months lag → perpetual count_source_lags_deadline (why Fix forecast hard resolution coverage #5072 moved off it).
Judged resolves "did conflict escalate in X by the deadline" from the WM news archive with dual-model + citations, and works today. A resolvable judged spec beats an unresolvable hard one.

What changed (`scripts/_forecast-resolution.mjs`)

deriveHardMetrics: conflict/ucdp_zone return null when the count feed is unavailable → buildHardSpec falls back to buildJudgedSpec. Gated by CONFLICT_COUNT_FEED_AVAILABLE (default false); the horizon-commensurable threshold logic (fix(forecast): resolution-spec emission amendment — horizon-commensurable count + resolvable supply/gps windows (Bet-1 follow-up, unblocks #5007) #5010) is preserved, not deleted — flip the flag to re-enable once a populated event-count feed exists.
buildQuestion: family-aware, escalation-framed question for conflict forecasts.
Threaded an optional conflictCountFeedAvailable override through buildResolutionSpec / attachResolutionSpecs (a real config with a production default, and the seam that lets the preserved-hard-logic tests force the hard path — same pattern as _gdelt-fetch.mjs's seams).

Tests

New: conflict now emits judged with an escalation/region-anchored question; scoped-regression that unrest still emits hard (change is conflict/ucdp_zone-only).
Existing conflict-hard tests (feed mapping, fix(forecast): resolution-spec emission amendment — horizon-commensurable count + resolvable supply/gps windows (Bet-1 follow-up, unblocks #5007) #5010 threshold scaling, metricKey grammar, R4 fixture, history round-trip) now pass { conflictCountFeedAvailable: true } — so they keep validating the preserved logic.
405 forecast tests green (forecast-resolution 58, forecast-history, forecast-detectors, forecast-resolution-eval, forecast-resolutions-seeder, forecast-scorecard, + .mts scorecard/get-forecasts).

Follow-ups (not in this PR)

fix(forecast): resolvability coverage — reach the ≥80%-hard KPI (57% live; #5066 follow-up, Lane A #2) #5091: reframe the coverage KPI to resolvable coverage — this lowers nominal hard-ratio but raises real resolvable coverage.
Judge budget: conflict is high-volume; watch pendingJudge on the scorecard-watch and bump FORECAST_RESOLUTION_JUDGE_MAX_PER_RUN if the backlog grows.
Deploy verify: get-forecasts shows conflict specs as judged; scorecard shows them moving pending-judge → resolved.

Related: #5136, #5097, #5087, #5091.

https://claude.ai/code/session_01StNurp4TGC3JLHbTtKJhbp

…5136) Conflict/ucdp_zone hard-count specs resolve against CONFLICT_COUNT_SOURCE_FEED (conflict:acled-resolution:v1), which only populates with ACLED credentials we don't have -> EXISTS=0 in prod -> every conflict count spec is unresolvable (sits pending/VOID forever). Route these families to judged resolution (live via #5087) instead of emitting a dead hard spec. - deriveHardMetrics: conflict/ucdp_zone return null when the count feed is unavailable -> buildHardSpec falls back to buildJudgedSpec (LLM judge over the news archive). Gated by CONFLICT_COUNT_FEED_AVAILABLE (default false); flip to re-enable hard-count once a populated event-count feed exists. - buildQuestion: family-aware, escalation-framed question for conflict forecasts (resolves more reliably against the archive than the generic phrasing). - Threaded an optional `conflictCountFeedAvailable` override through buildResolutionSpec/attachResolutionSpec so the preserved #5010 horizon-commensurable threshold logic stays under test (the tests that lock it force the hard path); production emits judged by default. - GDELT article-volume is NOT a substitute: its scale is article count, not event count, so the thresholds would mis-resolve. Tests: conflict now judged with an escalation question; scoped-regression that unrest still emits hard; existing conflict-hard tests force the feed-available path (preserves #5010 coverage). 405 forecast tests green. Related: #5136, #5097, #5087 (judged resolver), #5091 (KPI reframe follow-up) Claude-Session: https://claude.ai/code/session_01StNurp4TGC3JLHbTtKJhbp

vercel · 2026-07-09T20:09:16Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
worldmonitor	Ignored	Preview	Jul 10, 2026 4:28am

greptile-apps · 2026-07-09T20:13:49Z

Greptile Summary

This PR reclassifies conflict and ucdp_zone count specs from hard to judged resolution, because the underlying ACLED feed (conflict:acled-resolution:v1) is empty without credentials and leaves every conflict spec permanently unresolvable. The existing threshold-scaling logic is preserved behind a CONFLICT_COUNT_FEED_AVAILABLE flag (default false) so it can be re-enabled once a populated event-count feed exists.

deriveHardMetrics returns null early for conflict/ucdp_zone when the feed flag is off, triggering the existing buildHardSpec → buildJudgedSpec fallback path; a new escalation-framed buildQuestion branch supplies a sharper resolution question for the LLM judge.
Options are threaded as an optional last argument through buildResolutionSpec / attachResolutionSpecs / buildHardSpec / deriveHardMetrics, following the same seam pattern as _gdelt-fetch.mjs; test-only overrides use { conflictCountFeedAvailable: true } to keep all existing hard-spec fixture tests green.
Three new tests cover the judged-path behavior (spec shape, question content, scope isolation to conflict-only); all 15+ existing conflict hard-spec tests are preserved via the override option.

Confidence Score: 4/5

Safe to merge — the change converts an always-unresolvable path into a working judged path; no data is lost, and the hard-count path is fully preserved behind the flag.

The option-threading, feed-guard fallback, and test coverage are all well-executed. The one forward-looking concern is that the conflict question template hardcodes escalation framing, which will misfire if a de-escalation detector is ever added to the conflict domain — but all current emitters are escalation-shaped so there is no present defect.

The buildQuestion branch in scripts/_forecast-resolution.mjs (line 539) will need revisiting before any de-escalation or ceasefire-themed conflict detector is added.

Important Files Changed

Filename	Overview
scripts/_forecast-resolution.mjs	Adds CONFLICT_COUNT_FEED_AVAILABLE flag (default false), returns null from deriveHardMetrics for conflict/ucdp_zone when feed unavailable, escalation-framed buildQuestion for conflict domain, and threads an optional options object through the full call stack.
tests/forecast-resolution.test.mjs	Replaces old hard-path happy-path test with three new judged-path tests; retrofits all existing hard-path conflict tests with { conflictCountFeedAvailable: true } to preserve coverage of the preserved threshold logic.
tests/forecast-history.test.mjs	Adds { conflictCountFeedAvailable: true } override to three history round-trip tests that require a hard conflict spec; no logic changed.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[buildResolutionSpec] --> B{state_derived?}
    B -- yes --> J[buildJudgedSpec]
    B -- no --> C{prediction_market signal?}
    C -- yes --> H[buildHardSpec / prediction_market]
    C -- no --> D{JUDGED_DOMAINS?}
    D -- yes --> J
    D -- no --> E[resolveHardFamily]
    E -- null --> J
    E -- family --> F[buildHardSpec]
    F --> G[deriveHardMetrics]
    G -- conflict or ucdp_zone --> K{conflictCountFeedAvailable?}
    K -- false default --> L[return null]
    L --> J
    K -- true override --> M[compute threshold]
    M --> N[hard spec]
    G -- other family --> N
    J --> O[buildQuestion]
    O -- domain == conflict --> P[escalation-framed question]
    O -- other domain --> Q[generic resolve-YES question]

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[buildResolutionSpec] --> B{state_derived?}
    B -- yes --> J[buildJudgedSpec]
    B -- no --> C{prediction_market signal?}
    C -- yes --> H[buildHardSpec / prediction_market]
    C -- no --> D{JUDGED_DOMAINS?}
    D -- yes --> J
    D -- no --> E[resolveHardFamily]
    E -- null --> J
    E -- family --> F[buildHardSpec]
    F --> G[deriveHardMetrics]
    G -- conflict or ucdp_zone --> K{conflictCountFeedAvailable?}
    K -- false default --> L[return null]
    L --> J
    K -- true override --> M[compute threshold]
    M --> N[hard spec]
    G -- other family --> N
    J --> O[buildQuestion]
    O -- domain == conflict --> P[escalation-framed question]
    O -- other domain --> Q[generic resolve-YES question]

_{Reviews (1): Last reviewed commit: "fix(forecast): reclassify conflict/ucdp_..." | Re-trigger Greptile}

#5091) #5138 added a pending-ledger migration for conflict, but the unrest reclassification (this PR) only changed new-forecast emission — existing pending unrest count rows would stay stuck as unresolvable hard specs. Generalize the migration to cover both families. - migratePendingConflictCountEntryToJudged -> table-driven migratePendingCountEntryToJudged over UNAVAILABLE_COUNT_FEED_MIGRATIONS (conflict + unrest), each with its own flag gate + judged-question builder. - unrest question mirrors the generator-side political/instability phrasing. - Tests: new unrest-migration test; the two existing conflict+unrest ledger tests now assert unrest rows migrate to judged (they encoded the old unrest-resolves premise). 150 forecast tests green. Related: #5091, #5138 Claude-Session: https://claude.ai/code/session_01StNurp4TGC3JLHbTtKJhbp

* fix(forecast): reclassify unrest count specs to judged (#5091) The twin of #5136/#5138: unrest hard-count specs resolve against UNREST_COUNT_SOURCE_FEED (unrest:events-resolution:v1), which is EMPTY in prod — seed-unrest-events only writes it from an ACLED resolution fetch, and we have no ACLED credentials. So unrest counts were unresolvable, sitting pending/VOID forever, while inflating the nominal hard-ratio. - deriveHardMetrics: the `unrest` family returns null when the count feed is unavailable -> buildJudgedSpec (LLM judge, #5087). Gated by the new UNREST_COUNT_FEED_AVAILABLE flag (default false) + an `unrestCountFeedAvailable` override; the horizon-scaled threshold logic is preserved for re-enable. - buildQuestion: family-aware, instability-framed question for the political (unrest) domain, mirroring the conflict question. - Not GDELT-for-count: article volume != event count, so the thresholds would mis-resolve (same reason it was rejected for conflict). Tests: unrest now judged by default with an instability question; the scoped regression now uses cyber (a populated feed) as the still-hard exemplar; the unrest feed-mapping + R4 fixtures force the feed-available path. 404 forecast tests green. Related: #5091 (KPI reframe + empty-feed audit), #5136/#5138 (conflict template), #5087 (judged resolver), #4930 Claude-Session: https://claude.ai/code/session_01StNurp4TGC3JLHbTtKJhbp * test(forecast): address Greptile — assert horizon in unrest question + cover tally-null guard under override (#5142) - unrest judged test now asserts the horizon (7d) appears in the question, matching the conflict test's rigor (Greptile P2). - add a test for the unrest tally-null path under unrestCountFeedAvailable:true (a non-numeric count signal still falls back to judged) — the path the flag early-return shadowed by default (Greptile P2). Claude-Session: https://claude.ai/code/session_01StNurp4TGC3JLHbTtKJhbp * fix(forecast): migrate existing pending unrest count entries to judged (#5091) #5138 added a pending-ledger migration for conflict, but the unrest reclassification (this PR) only changed new-forecast emission — existing pending unrest count rows would stay stuck as unresolvable hard specs. Generalize the migration to cover both families. - migratePendingConflictCountEntryToJudged -> table-driven migratePendingCountEntryToJudged over UNAVAILABLE_COUNT_FEED_MIGRATIONS (conflict + unrest), each with its own flag gate + judged-question builder. - unrest question mirrors the generator-side political/instability phrasing. - Tests: new unrest-migration test; the two existing conflict+unrest ledger tests now assert unrest rows migrate to judged (they encoded the old unrest-resolves premise). 150 forecast tests green. Related: #5091, #5138 Claude-Session: https://claude.ai/code/session_01StNurp4TGC3JLHbTtKJhbp * test(forecast): lock unrest threshold scaling

vercel Bot deployed to Preview July 9, 2026 20:12 View deployment

greptile-apps Bot reviewed Jul 9, 2026

View reviewed changes

Comment thread scripts/_forecast-resolution.mjs

fix(forecast): migrate pending conflict counts to judged

90222d3

koala73 merged commit fd35b2c into main Jul 10, 2026
23 checks passed

koala73 deleted the fix/conflict-counts-to-judged branch July 10, 2026 04:39

This was referenced Jul 10, 2026

fix(forecast): resolvability coverage — reach the ≥80%-hard KPI (57% live; #5066 follow-up, Lane A #2) #5091

Open

fix(forecast): reclassify unrest count specs to judged (#5091) #5142

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(forecast): reclassify conflict/ucdp_zone count specs to judged (#5136)#5138

fix(forecast): reclassify conflict/ucdp_zone count specs to judged (#5136)#5138
koala73 merged 2 commits into
mainfrom
fix/conflict-counts-to-judged

koala73 commented Jul 9, 2026

Uh oh!

vercel Bot commented Jul 9, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Jul 9, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

koala73 commented Jul 9, 2026

Summary

Why judged (not GDELT-count or UCDP)

What changed (scripts/_forecast-resolution.mjs)

Tests

Follow-ups (not in this PR)

Uh oh!

vercel Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Jul 9, 2026

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Flowchart

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

What changed (`scripts/_forecast-resolution.mjs`)

vercel Bot commented Jul 9, 2026 •

edited

Loading