fix(forecast): word-boundary term matching — stop substring country false positives corrupting probabilities (#4933)#4939
Conversation
…alse positives corrupting probabilities (#4933) Bare String.includes let a country name nested inside another contaminate the probability spine: a Mali forecast was market-calibrated against a Somalia market (0.70→0.54, "somalia".includes("mali")), Niger boosted by Nigeria protest anomalies, Mali outages boosted by Somalia cyber threats, and "Iran" title tokens hit inside "Tirana". Only detectConflictScenarios ever received the word-boundary guard. Add textIncludesTerm (cached word-boundary regex, boundary class mirrors the tokenizeText delimiter) and use it at all nine bare-includes sites: countTermMatches (shared: market calibration + headline relevance), computeMarketMatchScore domain hints + title tokens, and the political / infra(x2) / supply-chain(x2) / military detector signal filters. 7 new regression tests (live-reproduced cases) + 6 positive controls; forecast suites 543/543 green, evaluate-forecast-benchmark 6/6. Claude-Session: https://claude.ai/code/session_01XhvRdRjkLqtkRBZi9ZYAu2
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Greptile SummaryThis PR introduces a
Confidence Score: 3/5The fix is real and correct for the nine patched sites, but the news-corroboration path in Four unpatched
Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A["Incoming text + term"] --> B{"textIncludesTerm"}
B --> C{"Cache hit?"}
C -- yes --> D["Return cached RegExp"]
C -- no --> E{"Cache full >= 4000?"}
E -- yes --> F["cache.clear full flush"]
E -- no --> G["Build word-boundary regex"]
F --> G
G --> H["Store in cache"]
H --> D
D --> I["re.test lowerText"]
subgraph Fixed["Fixed 9 sites"]
J["countTermMatches"]
K["computeMarketMatchScore domain hints + title tokens"]
L["detectPoliticalScenarios"]
M["detectInfraScenarios"]
N["detectSupplyChainScenarios"]
O["detectMilitaryScenarios"]
end
subgraph Unfixed["Still bare includes"]
P["computeHeadlineRelevance domain hints 2130+2138 title tokens 2134+2139"]
end
J --> B
K --> B
L --> B
M --> B
N --> B
O --> B
P -.-> Q["String.includes substring FP risk"]
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
A["Incoming text + term"] --> B{"textIncludesTerm"}
B --> C{"Cache hit?"}
C -- yes --> D["Return cached RegExp"]
C -- no --> E{"Cache full >= 4000?"}
E -- yes --> F["cache.clear full flush"]
E -- no --> G["Build word-boundary regex"]
F --> G
G --> H["Store in cache"]
H --> D
D --> I["re.test lowerText"]
subgraph Fixed["Fixed 9 sites"]
J["countTermMatches"]
K["computeMarketMatchScore domain hints + title tokens"]
L["detectPoliticalScenarios"]
M["detectInfraScenarios"]
N["detectSupplyChainScenarios"]
O["detectMilitaryScenarios"]
end
subgraph Unfixed["Still bare includes"]
P["computeHeadlineRelevance domain hints 2130+2138 title tokens 2134+2139"]
end
J --> B
K --> B
L --> B
M --> B
N --> B
O --> B
P -.-> Q["String.includes substring FP risk"]
|
| if (!lowerText || !lowerTerm) return false; | ||
| let re = termMatchRegexCache.get(lowerTerm); | ||
| if (!re) { | ||
| if (termMatchRegexCache.size >= TERM_MATCH_REGEX_CACHE_MAX) termMatchRegexCache.clear(); |
There was a problem hiding this comment.
Cache flush strategy drops all 4000 compiled regexes at once instead of evicting a single entry. In a seed run with high term diversity this could cause a burst of recompilation. Deleting the oldest entry (
termMatchRegexCache.delete(termMatchRegexCache.keys().next().value)) would avoid the thundering-recompile with very little overhead.
| if (termMatchRegexCache.size >= TERM_MATCH_REGEX_CACHE_MAX) termMatchRegexCache.clear(); | |
| if (termMatchRegexCache.size >= TERM_MATCH_REGEX_CACHE_MAX) { | |
| // Evict the oldest entry instead of flushing everything at once. | |
| termMatchRegexCache.delete(termMatchRegexCache.keys().next().value); | |
| } |
…ine-relevance substring hits, FIFO cache eviction Finding 1 (P1): getSearchTermsForRegion reverse lookup used regionLower.includes(nameLower), so Somalia inherited Mali's terms and — because the loop breaks on first hit — LOST its own keywords entirely (probe: Somalia => [Somalia, Mali, mali, bamako]; now => [Somalia, somalia, mogadishu]). Word-boundary match fixes injection + displacement. Finding 2 (P1): computeHeadlineRelevance had four raw .includes sites (domain hints, title tokens, and both requireSemantic counters) — 'war' matched inside 'award', 'iran' inside 'Tirana', producing false news_corroboration. Now textIncludesTerm. Finding 3 (P2): regex cache evicts oldest entry (FIFO) instead of clearing all 4000 compiled matchers at capacity. 6 new tests (3 confirmed RED pre-fix); forecast suites 545/545, benchmark 6/6. Claude-Session: https://claude.ai/code/session_01XhvRdRjkLqtkRBZi9ZYAu2
|
Review findings addressed in 4813c35 (all three verified at head 5b617ec before fixing): F1 (P1) reverse-lookup poisoning — fixed + worse than reported. F2 (P1) computeHeadlineRelevance — fixed. All four raw F3 (P2) cache eviction — fixed. FIFO oldest-key eviction instead of Verification: forecast-detectors 206/206 (3 new tests confirmed RED pre-fix), forecast-trace-export 310/310, history/integrity/resilience/narrative-cache suites green, evaluate-forecast-benchmark 6/6. |
…ame reverse lookup Finding 1 (P1): exact token boundaries dropped plural matches that bare includes gave for free — 'elections'/'attacks' no longer hit the singular DOMAIN_HINTS. The term regex now accepts an optional s/es suffix on the text side; a term nested inside a DIFFERENT word still never matches (nigers vs nigeria, malis vs somalia — boundary still required). Finding 2 (P1): reverse lookup broke on the FIRST boundary containment, so compounds resolved to the wrong country even though exact entries exist (DR Congo => Congo/brazzaville, Guinea-Bissau and Papua New Guinea => Guinea/conakry). Exact name match now wins outright; the containment fallback picks the longest contained name. 5 new tests (all RED at prior head); detector suite 211/211, trace-export 310/310, benchmark 6/6. Claude-Session: https://claude.ai/code/session_01XhvRdRjkLqtkRBZi9ZYAu2
|
Re-review findings addressed in d453a64 (both reproduced at head 4813c35 before fixing): F1 (P1) plural regression — fixed. The term regex now accepts an optional F2 (P1) compound-name reverse lookup — fixed. Exact name match now wins outright (the JSON has exact entries for all three repro cases); the containment fallback selects the LONGEST contained name instead of breaking on the first. Probes: Verification: forecast-detectors 211/211 (5 new tests RED at prior head), forecast-trace-export 310/310, history/integrity/resilience/narrative suites green, evaluate-forecast-benchmark 6/6. |
… 'war' must not match 'wares' Round-3 review finding (P2): the global (?:s|es)? suffix let 'war' match 'wares' — an Iran conflict forecast calibrated against 'Will Iran export more wares in 2026?' (0.70→0.54, live-reproduced). 'es' is only an English plural after sibilants, so the suffix is now conditional: terms ending in s/x/z/ch/sh allow (?:es)?, everything else s?. gas→gases and tax→taxes keep matching; attacks/elections unchanged. 4 new tests (2 RED at prior head, gases positive control); detector suite 214/214, adjacent suites green, benchmark 6/6. Claude-Session: https://claude.ai/code/session_01XhvRdRjkLqtkRBZi9ZYAu2
|
Round-3 finding addressed in fd1543d (reproduced at head d453a64 first: the wares market calibrated the Iran conflict forecast 0.70→0.54). P2 global 'es' suffix — fixed. The suffix is now conditional on the term's ending: sibilants (s/x/z/ch/sh) allow Verification: forecast-detectors 214/214, trace-export 310/310, adjacent suites green, benchmark 6/6. |
What
Fixes the HIGH finding from the #4933 adversarial audit: bare
String.includesterm matching let a country name nested inside another silently contaminate the forecast probability spine. OnlydetectConflictScenariosever received the word-boundary guard ("prevent substring false positives (IL matching Chile)"); the shared matcher and the other detectors did not.Live-reproduced corruption this PR eliminates (all asserted by new regression tests):
calibrationsetcybersignaloutageonlyanomalyboostais_gapsignalmil_flightssignaltitleHits: 1How
New
textIncludesTerm(lowerText, lowerTerm)helper — cached word-boundary regex, boundary class[^a-z0-9]mirroring thetokenizeTextdelimiter so both matchers agree (possessives like "Mali's" still match; multi-word terms like "strait of hormuz" still match). Swapped in at all nine bare-.includessites:countTermMatches— shared bycomputeMarketMatchScore(market calibration regionHits) andcomputeHeadlineRelevance/attachNewsContext(news corroboration)computeMarketMatchScore— domain hints + title tokensdetectPoliticalScenariosprotestAnomalies,detectInfraScenariosrelatedCyber + nearbyJam,detectSupplyChainScenariosaisGaps + nearbyJam,detectMilitaryScenariosmilFlightsTests
Bug-fix protocol: 13 new tests written first, 7 confirmed RED against the old code (exactly the negative cases), 6 positive controls green before and after (same-country matches, possessive boundary, multi-word terms).
tests/forecast-detectors.test.mjs: 200/200 (was 193 + 7 red)scripts/evaluate-forecast-benchmark.mjs: 6/6 (generation-quality fixtures unaffected)Why it matters
This is fix 1 of 3 P1 prerequisites for the #4930 verified-forecasting epic — the blend corruption writes wrong probabilities into
forecast:predictions:v2and the 45-day history that the future resolution/Brier system will score.Part of #4933.
https://claude.ai/code/session_01XhvRdRjkLqtkRBZi9ZYAu2