fix(forecast): reclassify unrest count specs to judged (#5091)#5142
Conversation
The twin of #5136/#5138: unrest hard-count specs resolve against UNREST_COUNT_SOURCE_FEED (unrest:events-resolution:v1), which is EMPTY in prod — seed-unrest-events only writes it from an ACLED resolution fetch, and we have no ACLED credentials. So unrest counts were unresolvable, sitting pending/VOID forever, while inflating the nominal hard-ratio. - deriveHardMetrics: the `unrest` family returns null when the count feed is unavailable -> buildJudgedSpec (LLM judge, #5087). Gated by the new UNREST_COUNT_FEED_AVAILABLE flag (default false) + an `unrestCountFeedAvailable` override; the horizon-scaled threshold logic is preserved for re-enable. - buildQuestion: family-aware, instability-framed question for the political (unrest) domain, mirroring the conflict question. - Not GDELT-for-count: article volume != event count, so the thresholds would mis-resolve (same reason it was rejected for conflict). Tests: unrest now judged by default with an instability question; the scoped regression now uses cyber (a populated feed) as the still-hard exemplar; the unrest feed-mapping + R4 fixtures force the feed-available path. 404 forecast tests green. Related: #5091 (KPI reframe + empty-feed audit), #5136/#5138 (conflict template), #5087 (judged resolver), #4930 Claude-Session: https://claude.ai/code/session_01StNurp4TGC3JLHbTtKJhbp
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Greptile SummaryThis PR reclassifies unrest count resolution specs from
Confidence Score: 4/5Safe to merge; the reclassification is a straightforward flag-gate following the proven conflict pattern and all 404 tests pass. The implementation faithfully mirrors the CONFLICT_COUNT_FEED_AVAILABLE pattern — same early-return structure, same options-override escape hatch, same threshold logic preserved for re-enablement. The only gaps are a missing horizon assertion in the new unrest judged test and a pre-existing test whose implied code path is now shadowed by the early flag return, leaving the tally-null case for unrest untested under unrestCountFeedAvailable: true. Neither gap affects production behavior. tests/forecast-resolution.test.mjs — the two test-coverage gaps noted; scripts/_forecast-resolution.mjs is clean. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[buildResolutionSpec] --> B{state_derived?}
B -- yes --> J[buildJudgedSpec]
B -- no --> C{prediction_market signal?}
C -- yes --> H[buildHardSpec / prediction_market]
C -- no --> D{JUDGED_DOMAINS?}
D -- yes --> J
D -- no --> E[resolveHardFamily]
E -- no family --> J
E -- family found --> F[buildHardSpec]
F --> G[deriveHardMetrics]
G -- conflict/ucdp_zone --> I1{CONFLICT_COUNT_FEED_AVAILABLE?}
I1 -- false default --> J
I1 -- true --> M1[hard conflict spec]
G -- unrest --> I2{UNREST_COUNT_FEED_AVAILABLE? NEW}
I2 -- false default --> J
I2 -- true --> M2[hard unrest spec]
G -- cyber/other --> M3[hard spec]
J --> Q[buildQuestion]
Q -- domain==conflict --> Q1[escalation-framed question]
Q -- domain==political --> Q2[instability-framed question NEW]
Q -- other --> Q3[generic question]
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
A[buildResolutionSpec] --> B{state_derived?}
B -- yes --> J[buildJudgedSpec]
B -- no --> C{prediction_market signal?}
C -- yes --> H[buildHardSpec / prediction_market]
C -- no --> D{JUDGED_DOMAINS?}
D -- yes --> J
D -- no --> E[resolveHardFamily]
E -- no family --> J
E -- family found --> F[buildHardSpec]
F --> G[deriveHardMetrics]
G -- conflict/ucdp_zone --> I1{CONFLICT_COUNT_FEED_AVAILABLE?}
I1 -- false default --> J
I1 -- true --> M1[hard conflict spec]
G -- unrest --> I2{UNREST_COUNT_FEED_AVAILABLE? NEW}
I2 -- false default --> J
I2 -- true --> M2[hard unrest spec]
G -- cyber/other --> M3[hard spec]
J --> Q[buildQuestion]
Q -- domain==conflict --> Q1[escalation-framed question]
Q -- domain==political --> Q2[instability-framed question NEW]
Q -- other --> Q3[generic question]
|
…+ cover tally-null guard under override (#5142) - unrest judged test now asserts the horizon (7d) appears in the question, matching the conflict test's rigor (Greptile P2). - add a test for the unrest tally-null path under unrestCountFeedAvailable:true (a non-numeric count signal still falls back to judged) — the path the flag early-return shadowed by default (Greptile P2). Claude-Session: https://claude.ai/code/session_01StNurp4TGC3JLHbTtKJhbp
|
Addressed both Greptile P2 test-coverage gaps (commit e3587c6):
60/60 in |
#5091) #5138 added a pending-ledger migration for conflict, but the unrest reclassification (this PR) only changed new-forecast emission — existing pending unrest count rows would stay stuck as unresolvable hard specs. Generalize the migration to cover both families. - migratePendingConflictCountEntryToJudged -> table-driven migratePendingCountEntryToJudged over UNAVAILABLE_COUNT_FEED_MIGRATIONS (conflict + unrest), each with its own flag gate + judged-question builder. - unrest question mirrors the generator-side political/instability phrasing. - Tests: new unrest-migration test; the two existing conflict+unrest ledger tests now assert unrest rows migrate to judged (they encoded the old unrest-resolves premise). 150 forecast tests green. Related: #5091, #5138 Claude-Session: https://claude.ai/code/session_01StNurp4TGC3JLHbTtKJhbp
|
Added the missing pending-ledger migration (commit 08074fd). #5138 migrated existing pending conflict count entries to judged, but this PR originally only changed new-forecast emission — existing pending unrest count rows would stay stuck. Generalized |
Summary
The twin of #5136/#5138, folded into #5091. Unrest hard-count specs resolve against
unrest:events-resolution:v1, which is empty in prod —seed-unrest-events.mjs:460only writes it from an ACLED resolution fetch (if (acledResolution?.events?.length)), and we have no ACLED credentials. So unrest counts were unresolvable (pending/VOID forever) while inflating the nominal hard-ratio — exactly the "hard % hides unresolvable specs" problem #5091 documents.What changed (
scripts/_forecast-resolution.mjs)deriveHardMetrics: theunrestfamilyreturn nulls when the count feed is unavailable →buildJudgedSpec(LLM judge, live via fix(forecast): resolve judged forecast outcomes #5087). Gated by a newUNREST_COUNT_FEED_AVAILABLEflag (defaultfalse) + anunrestCountFeedAvailableoverride; the horizon-scaled threshold logic is preserved for re-enable once a populated feed exists.buildQuestion: family-aware, instability-framed question for thepolitical(unrest) domain, mirroring the conflict question.Tests
judgedby default with an instability/region-anchored question.#5138scoped-regression exemplar switched from unrest → cyber (a genuinely populated feed) so it still proves the change is scoped..mjstests + 9.mtsgreen.Note on impact (see #5091)
This lowers the nominal hard-ratio further (unrest leaves the hard column) but raises resolvable coverage — which is why #5091's KPI was reframed to "≥80% resolvable (populated-hard OR judged)." The open verification remains: confirm the judged resolver actually resolves these (not perpetual-pending on its 48h archive window, per the #5087 review) once #5138 + this deploy.
Related: #5091, #5136, #5138, #5087, #4930.
https://claude.ai/code/session_01StNurp4TGC3JLHbTtKJhbp