Skip to content

fix(forecast): reclassify unrest count specs to judged (#5091)#5142

Merged
koala73 merged 4 commits into
mainfrom
fix/unrest-counts-to-judged
Jul 10, 2026
Merged

fix(forecast): reclassify unrest count specs to judged (#5091)#5142
koala73 merged 4 commits into
mainfrom
fix/unrest-counts-to-judged

Conversation

@koala73

@koala73 koala73 commented Jul 10, 2026

Copy link
Copy Markdown
Owner

Summary

The twin of #5136/#5138, folded into #5091. Unrest hard-count specs resolve against unrest:events-resolution:v1, which is empty in prodseed-unrest-events.mjs:460 only writes it from an ACLED resolution fetch (if (acledResolution?.events?.length)), and we have no ACLED credentials. So unrest counts were unresolvable (pending/VOID forever) while inflating the nominal hard-ratio — exactly the "hard % hides unresolvable specs" problem #5091 documents.

What changed (scripts/_forecast-resolution.mjs)

  • deriveHardMetrics: the unrest family return nulls when the count feed is unavailable → buildJudgedSpec (LLM judge, live via fix(forecast): resolve judged forecast outcomes #5087). Gated by a new UNREST_COUNT_FEED_AVAILABLE flag (default false) + an unrestCountFeedAvailable override; the horizon-scaled threshold logic is preserved for re-enable once a populated feed exists.
  • buildQuestion: family-aware, instability-framed question for the political (unrest) domain, mirroring the conflict question.
  • Not GDELT-for-count: article volume ≠ event count, so the thresholds would mis-resolve (same reason rejected for conflict).

Tests

  • Unrest now emits judged by default with an instability/region-anchored question.
  • The #5138 scoped-regression exemplar switched from unrest → cyber (a genuinely populated feed) so it still proves the change is scoped.
  • The unrest feed-mapping + R4 fixtures force the feed-available path (preserves the threshold coverage).
  • 404 forecast .mjs tests + 9 .mts green.

Note on impact (see #5091)

This lowers the nominal hard-ratio further (unrest leaves the hard column) but raises resolvable coverage — which is why #5091's KPI was reframed to "≥80% resolvable (populated-hard OR judged)." The open verification remains: confirm the judged resolver actually resolves these (not perpetual-pending on its 48h archive window, per the #5087 review) once #5138 + this deploy.

Related: #5091, #5136, #5138, #5087, #4930.

https://claude.ai/code/session_01StNurp4TGC3JLHbTtKJhbp

The twin of #5136/#5138: unrest hard-count specs resolve against
UNREST_COUNT_SOURCE_FEED (unrest:events-resolution:v1), which is EMPTY in prod —
seed-unrest-events only writes it from an ACLED resolution fetch, and we have no
ACLED credentials. So unrest counts were unresolvable, sitting pending/VOID
forever, while inflating the nominal hard-ratio.

- deriveHardMetrics: the `unrest` family returns null when the count feed is
  unavailable -> buildJudgedSpec (LLM judge, #5087). Gated by the new
  UNREST_COUNT_FEED_AVAILABLE flag (default false) + an `unrestCountFeedAvailable`
  override; the horizon-scaled threshold logic is preserved for re-enable.
- buildQuestion: family-aware, instability-framed question for the political
  (unrest) domain, mirroring the conflict question.
- Not GDELT-for-count: article volume != event count, so the thresholds would
  mis-resolve (same reason it was rejected for conflict).

Tests: unrest now judged by default with an instability question; the scoped
regression now uses cyber (a populated feed) as the still-hard exemplar; the
unrest feed-mapping + R4 fixtures force the feed-available path. 404 forecast
tests green.

Related: #5091 (KPI reframe + empty-feed audit), #5136/#5138 (conflict template),
#5087 (judged resolver), #4930

Claude-Session: https://claude.ai/code/session_01StNurp4TGC3JLHbTtKJhbp
@vercel

vercel Bot commented Jul 10, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
worldmonitor Ignored Ignored Preview Jul 10, 2026 5:51am

Request Review

@greptile-apps

greptile-apps Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR reclassifies unrest count resolution specs from hard to judged, applying the same fix already made for conflict specs in #5136. The root cause is identical: unrest:events-resolution:v1 is empty without ACLED credentials, so any hard spec pointing at it would sit pending/VOID forever while inflating the nominal hard-ratio metric.

  • Adds UNREST_COUNT_FEED_AVAILABLE = false flag (mirroring CONFLICT_COUNT_FEED_AVAILABLE) and gates the case 'unrest' path in deriveHardMetrics behind it, preserving the horizon-scaled threshold logic for future re-enablement.
  • Adds a domain === 'political' branch to buildQuestion, emitting an instability-framed question for judged political specs, consistent with the existing conflict escalation-framed question.
  • Test suite updated: the former scoped-regression test is split into two tests — one proving unrest is now judged by default, one proving cyber remains hard; feed-mapping and R4 fixture tests pass { unrestCountFeedAvailable: true } to exercise the preserved threshold path.

Confidence Score: 4/5

Safe to merge; the reclassification is a straightforward flag-gate following the proven conflict pattern and all 404 tests pass.

The implementation faithfully mirrors the CONFLICT_COUNT_FEED_AVAILABLE pattern — same early-return structure, same options-override escape hatch, same threshold logic preserved for re-enablement. The only gaps are a missing horizon assertion in the new unrest judged test and a pre-existing test whose implied code path is now shadowed by the early flag return, leaving the tally-null case for unrest untested under unrestCountFeedAvailable: true. Neither gap affects production behavior.

tests/forecast-resolution.test.mjs — the two test-coverage gaps noted; scripts/_forecast-resolution.mjs is clean.

Important Files Changed

Filename Overview
scripts/_forecast-resolution.mjs Adds UNREST_COUNT_FEED_AVAILABLE flag and early-return null in deriveHardMetrics 'unrest' case; adds political-domain question branch in buildQuestion. Logic is consistent with the existing conflict pattern.
tests/forecast-resolution.test.mjs Scoped-regression test replaced with unrest-judged + cyber-still-hard tests; feed-mapping and R4 tests updated with unrestCountFeedAvailable override. Minor gaps: missing horizon assertion in new unrest test, and a pre-existing test now exercises the flag-gate path rather than the tally-null path its name implies.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[buildResolutionSpec] --> B{state_derived?}
    B -- yes --> J[buildJudgedSpec]
    B -- no --> C{prediction_market signal?}
    C -- yes --> H[buildHardSpec / prediction_market]
    C -- no --> D{JUDGED_DOMAINS?}
    D -- yes --> J
    D -- no --> E[resolveHardFamily]
    E -- no family --> J
    E -- family found --> F[buildHardSpec]
    F --> G[deriveHardMetrics]
    G -- conflict/ucdp_zone --> I1{CONFLICT_COUNT_FEED_AVAILABLE?}
    I1 -- false default --> J
    I1 -- true --> M1[hard conflict spec]
    G -- unrest --> I2{UNREST_COUNT_FEED_AVAILABLE? NEW}
    I2 -- false default --> J
    I2 -- true --> M2[hard unrest spec]
    G -- cyber/other --> M3[hard spec]
    J --> Q[buildQuestion]
    Q -- domain==conflict --> Q1[escalation-framed question]
    Q -- domain==political --> Q2[instability-framed question NEW]
    Q -- other --> Q3[generic question]
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[buildResolutionSpec] --> B{state_derived?}
    B -- yes --> J[buildJudgedSpec]
    B -- no --> C{prediction_market signal?}
    C -- yes --> H[buildHardSpec / prediction_market]
    C -- no --> D{JUDGED_DOMAINS?}
    D -- yes --> J
    D -- no --> E[resolveHardFamily]
    E -- no family --> J
    E -- family found --> F[buildHardSpec]
    F --> G[deriveHardMetrics]
    G -- conflict/ucdp_zone --> I1{CONFLICT_COUNT_FEED_AVAILABLE?}
    I1 -- false default --> J
    I1 -- true --> M1[hard conflict spec]
    G -- unrest --> I2{UNREST_COUNT_FEED_AVAILABLE? NEW}
    I2 -- false default --> J
    I2 -- true --> M2[hard unrest spec]
    G -- cyber/other --> M3[hard spec]
    J --> Q[buildQuestion]
    Q -- domain==conflict --> Q1[escalation-framed question]
    Q -- domain==political --> Q2[instability-framed question NEW]
    Q -- other --> Q3[generic question]
Loading

Comments Outside Diff (1)

  1. tests/forecast-resolution.test.mjs, line 449-461 (link)

    P2 Pre-existing test now covers a different code path. This test's name says "no unrest event count yields judged," implying the tally-extraction path is what produces judged. After this PR, UNREST_COUNT_FEED_AVAILABLE = false fires the early return null in deriveHardMetrics before the tally check is ever reached — so the test now validates the flag gate, not the missing-count fallback. The tally-null path for unrest (a forecast that has the feed available but no numeric count in its signals) is no longer covered by any test. Consider adding a companion test that passes { unrestCountFeedAvailable: true } with a signal that carries no parseable count, mirroring the conflict version at the end of the FIX 1 block.

Reviews (1): Last reviewed commit: "fix(forecast): reclassify unrest count s..." | Re-trigger Greptile

Comment thread tests/forecast-resolution.test.mjs
…+ cover tally-null guard under override (#5142)

- unrest judged test now asserts the horizon (7d) appears in the question,
  matching the conflict test's rigor (Greptile P2).
- add a test for the unrest tally-null path under unrestCountFeedAvailable:true
  (a non-numeric count signal still falls back to judged) — the path the flag
  early-return shadowed by default (Greptile P2).

Claude-Session: https://claude.ai/code/session_01StNurp4TGC3JLHbTtKJhbp
@koala73

koala73 commented Jul 10, 2026

Copy link
Copy Markdown
Owner Author

Addressed both Greptile P2 test-coverage gaps (commit e3587c6):

  • Missing horizon assertion: the unrest judged test now asserts spec.question matches /7d/, matching the conflict test's rigor.
  • Shadowed tally-null path: added a test that forces unrestCountFeedAvailable: true with a non-numeric count signal → still falls back to judged (the preserved tally-null guard the flag early-return shadows by default).

60/60 in forecast-resolution.test.mjs, 404 forecast .mjs overall.

#5091)

#5138 added a pending-ledger migration for conflict, but the unrest
reclassification (this PR) only changed new-forecast emission — existing pending
unrest count rows would stay stuck as unresolvable hard specs. Generalize the
migration to cover both families.

- migratePendingConflictCountEntryToJudged -> table-driven
  migratePendingCountEntryToJudged over UNAVAILABLE_COUNT_FEED_MIGRATIONS
  (conflict + unrest), each with its own flag gate + judged-question builder.
- unrest question mirrors the generator-side political/instability phrasing.
- Tests: new unrest-migration test; the two existing conflict+unrest ledger
  tests now assert unrest rows migrate to judged (they encoded the old
  unrest-resolves premise). 150 forecast tests green.

Related: #5091, #5138

Claude-Session: https://claude.ai/code/session_01StNurp4TGC3JLHbTtKJhbp
@koala73

koala73 commented Jul 10, 2026

Copy link
Copy Markdown
Owner Author

Added the missing pending-ledger migration (commit 08074fd). #5138 migrated existing pending conflict count entries to judged, but this PR originally only changed new-forecast emission — existing pending unrest count rows would stay stuck. Generalized migratePendingConflictCountEntryToJudged → table-driven migratePendingCountEntryToJudged over conflict + unrest (each with its own flag gate + question builder). New unrest-migration test + updated the two existing conflict-migration tests (they encoded the old 'unrest resolves' premise). 150 forecast tests green. #5142 is now the complete unrest fix — emission + backlog, parallel to #5138.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant