Fix forecast hard resolution coverage#5072
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThis PR improves forecast hard-resolution coverage by routing conflict/UCDP count specs to a fresher ACLED feed (2-day lag instead of 14 days), adding hard-resolvability for political unrest-event and cyber-volume forecasts with ISO/name country alias matching, and boosting hard-resolvable forecasts during publication scoring to help hit the ≥80% resolvability KPI.
Confidence Score: 4/5Safe to merge; the core resolution logic and feed routing are well-tested and the new ISO/name alias matching is correct. The
Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[buildResolutionSpec] --> B{domain in JUDGED_DOMAINS?}
B -- military --> J[judged spec]
B -- no --> C{prediction_market signal?}
C -- yes --> D[hard: prediction_market family]
C -- no --> E{domain in DOMAIN_TO_HARD_FAMILIES?}
E -- political → unrest family --> F{unrest_events tally finite?}
F -- yes --> G[hard: unrest:events:v1 lag=0ms]
F -- no --> J
E -- cyber → cyber family --> H{cyber tally finite?}
H -- yes --> I[hard: cyber:threats-bootstrap:v2 lag=0ms ISO↔name matching]
H -- no --> J
E -- conflict/ucdp_zone family --> K{ucdp/conflict_events tally finite?}
K -- yes --> L[hard: conflict:acled:v1:all:0:0 lag=2 days]
K -- no --> J
E -- other families --> M[hard or judged per signal type]
L --> N[computePublishSelectionScore +0.25 resolvabilityLift]
G --> N
I --> N
D --> N
J --> O[score: no lift]
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
A[buildResolutionSpec] --> B{domain in JUDGED_DOMAINS?}
B -- military --> J[judged spec]
B -- no --> C{prediction_market signal?}
C -- yes --> D[hard: prediction_market family]
C -- no --> E{domain in DOMAIN_TO_HARD_FAMILIES?}
E -- political → unrest family --> F{unrest_events tally finite?}
F -- yes --> G[hard: unrest:events:v1 lag=0ms]
F -- no --> J
E -- cyber → cyber family --> H{cyber tally finite?}
H -- yes --> I[hard: cyber:threats-bootstrap:v2 lag=0ms ISO↔name matching]
H -- no --> J
E -- conflict/ucdp_zone family --> K{ucdp/conflict_events tally finite?}
K -- yes --> L[hard: conflict:acled:v1:all:0:0 lag=2 days]
K -- no --> J
E -- other families --> M[hard or judged per signal type]
L --> N[computePublishSelectionScore +0.25 resolvabilityLift]
G --> N
I --> N
D --> N
J --> O[score: no lift]
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| record.timestamp, | ||
| record.generatedAt, | ||
| record.firstSeenAt, | ||
| record.lastSeenAt, |
There was a problem hiding this comment.
lastSeenAt can misattribute persistent threats to the wrong window
extractRecordTime tries firstSeenAt before lastSeenAt, so for records that carry both fields the correct start time wins. But for records that carry only lastSeenAt — a threat active across the forecast boundary — the window check uses the observation end time, not the start time. A long-running threat first seen before generatedAt would be counted in the [generatedAt, deadline] window if its lastSeenAt falls within it, inflating the event count.
Resolve the tests/forecast-resolution-eval.test.mjs conflict (both sides added count tests) and reconcile #5063's source-coverage gate with this PR's new live feeds: - Union this PR's ACLED-lag + cyber-ISO tests with #5063's UCDP source-lag + void/truncated-count tests. - Scope #5063's summarizeRecordCoverage gate to lagged dated-snapshot feeds (settlementLagMs > 0). The naive merge ran it against the new live cyber feed, whose max firstSeenAt trails a future deadline, stranding cyber counts perpetually pending. Review fixes found while resolving: - P1: bridge UCDP<->ACLED country naming. Conflict regions are UCDP-named ("DR Congo (Zaire)") but now resolve against the ACLED feed ("Democratic Republic of Congo"); country-names.json bridged neither, scoring a false NO on active zones. Canonicalize both toward a shared token (drop parenthetical alternate + "the"), verified collision-free. - P2: give unrest:events:v1 the 2-day ACLED settlement lag. It is ACLED "Protests" data dated by event_date, so lag 0 sealed before ingestion and scored false NOs. Cyber stays live (firstSeenAt is real-time). - Correct a stale "No publish-selection change" comment. Adds a UCDP-parenthetical-region regression test. Full forecast suite 335/335. Claude-Session: https://claude.ai/code/session_01LDPvW28NvqVyyYx13a2QE8
…LED test - Reduce RESOLVABLE_HARD_SELECTION_LIFT 0.25 -> 0.12. The flat 0.25 was the second-largest term in computePublishSelectionScore, dwarfing readiness (x0.2), probability (x0.15) and the memory lift (~0.17) and flatly overriding large quality gaps. 0.12 keeps a strong preference for measurable forecasts (the >=80% resolvability KPI) without dominating the primary quality signals; tunable upward if the KPI is missed. - Reshape the ACLED conflict-count test to the production feed shape: seed-conflict-intel.mjs emits numeric occurredAt, not event_date strings. Keeps one event_date record so that fallback branch stays covered. Full forecast suite 335/335. Claude-Session: https://claude.ai/code/session_01LDPvW28NvqVyyYx13a2QE8
Main advanced with #5071 (which independently landed the same eval-layer count-resolution coverage: ACLED settlement lag, ISO<->name country aliases, extractRecordTime fields) and #5073 (resolution-ledger pruning). Resolve the overlap in scripts/_forecast-resolution-eval.mjs and its test by taking this branch's version: it is a strict superset of #5071's eval-layer code plus the fixes from review — - unrest:events:v1 gets the 2-day ACLED settlement lag (not 0), - #5063's source-coverage gate is scoped to lagged feeds (settlementLagMs>0) so the new live cyber feed is not stranded pending, - country matching canonicalizes UCDP parenthetical / ACLED article-dropped names so a UCDP-named region resolves against the ACLED feed, - the ACLED test uses the production occurredAt shape, - plus the UCDP-parenthetical-region regression test. Confirmed no unique #5071 test/code is dropped (its titles are a subset of this branch's, modulo one renamed test). This PR's remaining unique surface is the spec-building side (_forecast-resolution.mjs: unrest/cyber hard families, JUDGED_DOMAINS -> {military}) and the tuned publish-selection lift. Full forecast suite 343/343. Claude-Session: https://claude.ai/code/session_01LDPvW28NvqVyyYx13a2QE8
1 participant
Summary
Fixes #5066
Tests