fix(forecast): avoid stale UCDP count resolutions#5063
Merged
Conversation
Keep UCDP count specs pending until the source feed has reached the forecast deadline, and VOID pruned below-threshold windows instead of scoring a false NO. Add evaluator and seeder regressions for stale UCDP count snapshots.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Contributor
Greptile SummaryThis PR guards count-family forecast specs against stale UCDP feed snapshots by adding two new pending/void paths before any resolution is attempted: (1) if the feed has no dated records, or its latest dated record pre-dates the deadline, resolution stays pending; (2) if the feed's oldest record post-dates
Confidence Score: 4/5The change is safe to merge; the new pending/void gates only fire for count-family specs and leave all other resolution paths untouched. The two new helper functions are logically sound and well-tested for their main scenarios. The coverage calculation implicitly assumes the feed updates all sub-series in lockstep, which holds for UCDP global snapshots but is not documented; a heterogeneous feed could slip past both safety nets and produce a stale count resolution. The complementary happy path through partialCountEstablishesOutcome (truncated window, partial count already meets threshold) also lacks test coverage. scripts/_forecast-resolution-eval.mjs — the coverage sweep and its interaction with the minTs VOID guard; tests/forecast-resolution-eval.test.mjs — missing test for the truncated-window resolve path. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[resolveHardSpec called\nfn == 'count'] --> B{nowMs < deadline?}
B -- yes --> C[pending: deadline_not_reached]
B -- no --> D{nowMs < deadline\n+ UCDP_SETTLEMENT_LAG?}
D -- yes --> E[pending: count_settlement_lag]
D -- no --> F{feedData == null?}
F -- yes --> G[pending: source_feed_unavailable]
F -- no --> H[summarizeRecordCoverage\ncompute minTs / maxTs]
H --> I{count == 0 or\nmaxTs not finite?}
I -- yes --> J[pending: count_source_no_dated_records]
I -- no --> K{coverage.maxTs\n< deadline?}
K -- yes --> L[pending: count_source_lags_deadline]
K -- no --> M[countMatchingRecords\ngeneratedAt to deadline]
M --> N{coverage.minTs\n> generatedAt?}
N -- no --> P[compareResult: YES or NO]
N -- yes --> O{partialCountEstablishesOutcome?}
O -- yes --> P
O -- no --> Q[VOID: count_source_window_not_retained]
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
A[resolveHardSpec called\nfn == 'count'] --> B{nowMs < deadline?}
B -- yes --> C[pending: deadline_not_reached]
B -- no --> D{nowMs < deadline\n+ UCDP_SETTLEMENT_LAG?}
D -- yes --> E[pending: count_settlement_lag]
D -- no --> F{feedData == null?}
F -- yes --> G[pending: source_feed_unavailable]
F -- no --> H[summarizeRecordCoverage\ncompute minTs / maxTs]
H --> I{count == 0 or\nmaxTs not finite?}
I -- yes --> J[pending: count_source_no_dated_records]
I -- no --> K{coverage.maxTs\n< deadline?}
K -- yes --> L[pending: count_source_lags_deadline]
K -- no --> M[countMatchingRecords\ngeneratedAt to deadline]
M --> N{coverage.minTs\n> generatedAt?}
N -- no --> P[compareResult: YES or NO]
N -- yes --> O{partialCountEstablishesOutcome?}
O -- yes --> P
O -- no --> Q[VOID: count_source_window_not_retained]
Reviews (1): Last reviewed commit: "fix(forecast): guard stale count resolut..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Document the UCDP coverage assumption and add the missing truncated-window YES regression for count specs.
koala73
added a commit
that referenced
this pull request
Jul 9, 2026
Resolve the tests/forecast-resolution-eval.test.mjs conflict (both sides added count tests) and reconcile #5063's source-coverage gate with this PR's new live feeds: - Union this PR's ACLED-lag + cyber-ISO tests with #5063's UCDP source-lag + void/truncated-count tests. - Scope #5063's summarizeRecordCoverage gate to lagged dated-snapshot feeds (settlementLagMs > 0). The naive merge ran it against the new live cyber feed, whose max firstSeenAt trails a future deadline, stranding cyber counts perpetually pending. Review fixes found while resolving: - P1: bridge UCDP<->ACLED country naming. Conflict regions are UCDP-named ("DR Congo (Zaire)") but now resolve against the ACLED feed ("Democratic Republic of Congo"); country-names.json bridged neither, scoring a false NO on active zones. Canonicalize both toward a shared token (drop parenthetical alternate + "the"), verified collision-free. - P2: give unrest:events:v1 the 2-day ACLED settlement lag. It is ACLED "Protests" data dated by event_date, so lag 0 sealed before ingestion and scored false NOs. Cyber stays live (firstSeenAt is real-time). - Correct a stale "No publish-selection change" comment. Adds a UCDP-parenthetical-region regression test. Full forecast suite 335/335. Claude-Session: https://claude.ai/code/session_01LDPvW28NvqVyyYx13a2QE8
koala73
added a commit
that referenced
this pull request
Jul 9, 2026
Main advanced with #5071 (which independently landed the same eval-layer count-resolution coverage: ACLED settlement lag, ISO<->name country aliases, extractRecordTime fields) and #5073 (resolution-ledger pruning). Resolve the overlap in scripts/_forecast-resolution-eval.mjs and its test by taking this branch's version: it is a strict superset of #5071's eval-layer code plus the fixes from review — - unrest:events:v1 gets the 2-day ACLED settlement lag (not 0), - #5063's source-coverage gate is scoped to lagged feeds (settlementLagMs>0) so the new live cyber feed is not stranded pending, - country matching canonicalizes UCDP parenthetical / ACLED article-dropped names so a UCDP-named region resolves against the ACLED feed, - the ACLED test uses the production occurredAt shape, - plus the UCDP-parenthetical-region regression test. Confirmed no unique #5071 test/code is dropped (its titles are a subset of this branch's, modulo one renamed test). This PR's remaining unique surface is the spec-building side (_forecast-resolution.mjs: unrest/cyber hard families, JUDGED_DOMAINS -> {military}) and the tuned publish-selection lift. Full forecast suite 343/343. Claude-Session: https://claude.ai/code/session_01LDPvW28NvqVyyYx13a2QE8
This was referenced Jul 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stale UCDP count feeds can no longer turn conflict forecasts into confident false NO resolutions. Count-family specs now wait until the dated source has reached the forecast deadline, and capped feeds that have already lost the start of the window VOID unsafe below-threshold outcomes instead of entering Brier math.
This preserves the existing point-window behavior for price, yesPrice, hexCount, and riskScore families while making the count resolver prove its source coverage before scoring.
Validation
node --test tests/forecast-resolution-eval.test.mjs tests/forecast-resolutions-seeder.test.mjsgit diff --checkRelated
Related: #5024