Skip to content

fix(forecast): word-boundary term matching — stop substring country false positives corrupting probabilities (#4933)#4939

Merged
koala73 merged 4 commits into
mainfrom
fix/forecast-word-boundary-matching
Jul 6, 2026
Merged

fix(forecast): word-boundary term matching — stop substring country false positives corrupting probabilities (#4933)#4939
koala73 merged 4 commits into
mainfrom
fix/forecast-word-boundary-matching

Conversation

@koala73

@koala73 koala73 commented Jul 6, 2026

Copy link
Copy Markdown
Owner

What

Fixes the HIGH finding from the #4933 adversarial audit: bare String.includes term matching let a country name nested inside another silently contaminate the forecast probability spine. Only detectConflictScenarios ever received the word-boundary guard ("prevent substring false positives (IL matching Chile)"); the shared matcher and the other detectors did not.

Live-reproduced corruption this PR eliminates (all asserted by new regression tests):

Path Before After
Mali political 0.70 vs "Will Somalia's government collapse…" market-blended to 0.54, calibration set untouched 0.70, no calibration
Niger political 0.70 vs "Will Nigeria hold elections…" blended to 0.74 untouched 0.70
Mali major outage + Somalia cyber threat 0.40 → 0.55 + bogus cyber signal 0.40, outage only
Niger CII + Nigeria protest anomaly +0.10 bogus anomaly boost no boost
"Suez" route vs "Suezmax anchorage" AIS anomaly bogus ais_gap signal none
Mali theater vs "Somalia border strip" flight anomaly bogus mil_flights signal none
"Iran" title token vs "Tirana host…" market titleHits: 1 0

How

New textIncludesTerm(lowerText, lowerTerm) helper — cached word-boundary regex, boundary class [^a-z0-9] mirroring the tokenizeText delimiter so both matchers agree (possessives like "Mali's" still match; multi-word terms like "strait of hormuz" still match). Swapped in at all nine bare-.includes sites:

  • countTermMatches — shared by computeMarketMatchScore (market calibration regionHits) and computeHeadlineRelevance/attachNewsContext (news corroboration)
  • computeMarketMatchScore — domain hints + title tokens
  • detectPoliticalScenarios protestAnomalies, detectInfraScenarios relatedCyber + nearbyJam, detectSupplyChainScenarios aisGaps + nearbyJam, detectMilitaryScenarios milFlights

Tests

Bug-fix protocol: 13 new tests written first, 7 confirmed RED against the old code (exactly the negative cases), 6 positive controls green before and after (same-country matches, possessive boundary, multi-word terms).

  • tests/forecast-detectors.test.mjs: 200/200 (was 193 + 7 red)
  • Adjacent suites: forecast-trace-export 310/310, forecast-history 7/7, forecast-integrity-provenance 6/6, forecast-seed-resilience 11/11, forecast-llm-telemetry 4/4, forecast-narrative-cache-hash 5/5, forecasts-ticker-set-envelope-unwrap 5/5
  • scripts/evaluate-forecast-benchmark.mjs: 6/6 (generation-quality fixtures unaffected)

Why it matters

This is fix 1 of 3 P1 prerequisites for the #4930 verified-forecasting epic — the blend corruption writes wrong probabilities into forecast:predictions:v2 and the 45-day history that the future resolution/Brier system will score.

Part of #4933.

https://claude.ai/code/session_01XhvRdRjkLqtkRBZi9ZYAu2

…alse positives corrupting probabilities (#4933)

Bare String.includes let a country name nested inside another contaminate
the probability spine: a Mali forecast was market-calibrated against a
Somalia market (0.70→0.54, "somalia".includes("mali")), Niger boosted by
Nigeria protest anomalies, Mali outages boosted by Somalia cyber threats,
and "Iran" title tokens hit inside "Tirana". Only detectConflictScenarios
ever received the word-boundary guard.

Add textIncludesTerm (cached word-boundary regex, boundary class mirrors
the tokenizeText delimiter) and use it at all nine bare-includes sites:
countTermMatches (shared: market calibration + headline relevance),
computeMarketMatchScore domain hints + title tokens, and the political /
infra(x2) / supply-chain(x2) / military detector signal filters.

7 new regression tests (live-reproduced cases) + 6 positive controls;
forecast suites 543/543 green, evaluate-forecast-benchmark 6/6.

Claude-Session: https://claude.ai/code/session_01XhvRdRjkLqtkRBZi9ZYAu2
@vercel

vercel Bot commented Jul 6, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
worldmonitor Ignored Ignored Preview Jul 6, 2026 10:34am

Request Review

@greptile-apps

greptile-apps Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR introduces a textIncludesTerm word-boundary helper (regex-backed, 4000-entry compiled-regex cache) and replaces bare String.includes at nine call sites across six detector functions and the shared countTermMatches/computeMarketMatchScore paths, eliminating country-substring false positives that were corrupting forecast probabilities. Thirteen new regression tests (7 confirmed-red, 6 positive controls) are added to forecast-detectors.test.mjs.

  • Incomplete fix in computeHeadlineRelevance: the domain-hint scoring loop (line 2130), title-token scoring loop (line 2134), and both counters in the requireSemantic guard (lines 2138\u20132139) still use bare .includes. The PR asserts this function is covered through countTermMatches, but that only handles region terms; the domain and title-token paths are independent and remain vulnerable to the same substring false positive.
  • Cache eviction: the regex cache uses a full Map.clear() flush on overflow rather than single-entry eviction, which could cause a burst of regex recompilation if many distinct terms are encountered in one seed run.

Confidence Score: 3/5

The fix is real and correct for the nine patched sites, but the news-corroboration path in computeHeadlineRelevance retains the same bare .includes loops for domain hints and title tokens.

Four unpatched .includes calls remain in computeHeadlineRelevance for the same class of substring false positive the PR was written to fix. A headline about Tirana can still incorrectly score against an Iran prediction title-token through this path, and the requireSemantic guard counts are also affected.

scripts/seed-forecasts.mjs lines 2129–2140 — computeHeadlineRelevance domain-hint and title-token loops need the same textIncludesTerm swap applied in computeMarketMatchScore.

Important Files Changed

Filename Overview
scripts/seed-forecasts.mjs Introduces textIncludesTerm word-boundary helper and applies it at 9 bare .includes sites; however, the parallel domain-hint and title-token loops inside computeHeadlineRelevance (lines 2130, 2134, 2138–2139) were not patched, leaving the news-corroboration path with the same substring false-positive vulnerability.
tests/forecast-detectors.test.mjs Adds 13 new regression tests covering the 7 confirmed red cases and 6 positive-control scenarios; test structure is clean and mirrors the PR's fix table precisely.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["Incoming text + term"] --> B{"textIncludesTerm"}
    B --> C{"Cache hit?"}
    C -- yes --> D["Return cached RegExp"]
    C -- no --> E{"Cache full >= 4000?"}
    E -- yes --> F["cache.clear full flush"]
    E -- no --> G["Build word-boundary regex"]
    F --> G
    G --> H["Store in cache"]
    H --> D
    D --> I["re.test lowerText"]

    subgraph Fixed["Fixed 9 sites"]
        J["countTermMatches"]
        K["computeMarketMatchScore domain hints + title tokens"]
        L["detectPoliticalScenarios"]
        M["detectInfraScenarios"]
        N["detectSupplyChainScenarios"]
        O["detectMilitaryScenarios"]
    end

    subgraph Unfixed["Still bare includes"]
        P["computeHeadlineRelevance domain hints 2130+2138 title tokens 2134+2139"]
    end

    J --> B
    K --> B
    L --> B
    M --> B
    N --> B
    O --> B
    P -.-> Q["String.includes substring FP risk"]
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A["Incoming text + term"] --> B{"textIncludesTerm"}
    B --> C{"Cache hit?"}
    C -- yes --> D["Return cached RegExp"]
    C -- no --> E{"Cache full >= 4000?"}
    E -- yes --> F["cache.clear full flush"]
    E -- no --> G["Build word-boundary regex"]
    F --> G
    G --> H["Store in cache"]
    H --> D
    D --> I["re.test lowerText"]

    subgraph Fixed["Fixed 9 sites"]
        J["countTermMatches"]
        K["computeMarketMatchScore domain hints + title tokens"]
        L["detectPoliticalScenarios"]
        M["detectInfraScenarios"]
        N["detectSupplyChainScenarios"]
        O["detectMilitaryScenarios"]
    end

    subgraph Unfixed["Still bare includes"]
        P["computeHeadlineRelevance domain hints 2130+2138 title tokens 2134+2139"]
    end

    J --> B
    K --> B
    L --> B
    M --> B
    N --> B
    O --> B
    P -.-> Q["String.includes substring FP risk"]
Loading

Comments Outside Diff (1)

  1. scripts/seed-forecasts.mjs, line 2129-2140 (link)

    P1 Bare .includes calls in computeHeadlineRelevance not patched

    computeHeadlineRelevance has four raw .includes calls — domain-hint scoring (line 2130), title-token scoring (line 2134), and both counters inside the requireSemantic guard (lines 2138–2139) — all untouched by this PR. The PR fixed the identical loops in computeMarketMatchScore (lines 2157/2165) and states that computeHeadlineRelevance is covered "through countTermMatches", but countTermMatches only handles region-term matching; the domain-hint and title-token paths are independent and still vulnerable to the same substring false positive (e.g., a headline about "Tirana" incorrectly scoring +2 on an "Iran" title token). This leaves the news-corroboration path with the same bug the PR targeted.

    Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Reviews (1): Last reviewed commit: "fix(forecast): word-boundary term matchi..." | Re-trigger Greptile

Comment thread scripts/seed-forecasts.mjs Outdated
if (!lowerText || !lowerTerm) return false;
let re = termMatchRegexCache.get(lowerTerm);
if (!re) {
if (termMatchRegexCache.size >= TERM_MATCH_REGEX_CACHE_MAX) termMatchRegexCache.clear();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Cache flush strategy drops all 4000 compiled regexes at once instead of evicting a single entry. In a seed run with high term diversity this could cause a burst of recompilation. Deleting the oldest entry (termMatchRegexCache.delete(termMatchRegexCache.keys().next().value)) would avoid the thundering-recompile with very little overhead.

Suggested change
if (termMatchRegexCache.size >= TERM_MATCH_REGEX_CACHE_MAX) termMatchRegexCache.clear();
if (termMatchRegexCache.size >= TERM_MATCH_REGEX_CACHE_MAX) {
// Evict the oldest entry instead of flushing everything at once.
termMatchRegexCache.delete(termMatchRegexCache.keys().next().value);
}

…ine-relevance substring hits, FIFO cache eviction

Finding 1 (P1): getSearchTermsForRegion reverse lookup used
regionLower.includes(nameLower), so Somalia inherited Mali's terms and
— because the loop breaks on first hit — LOST its own keywords entirely
(probe: Somalia => [Somalia, Mali, mali, bamako]; now => [Somalia,
somalia, mogadishu]). Word-boundary match fixes injection + displacement.

Finding 2 (P1): computeHeadlineRelevance had four raw .includes sites
(domain hints, title tokens, and both requireSemantic counters) — 'war'
matched inside 'award', 'iran' inside 'Tirana', producing false
news_corroboration. Now textIncludesTerm.

Finding 3 (P2): regex cache evicts oldest entry (FIFO) instead of
clearing all 4000 compiled matchers at capacity.

6 new tests (3 confirmed RED pre-fix); forecast suites 545/545,
benchmark 6/6.

Claude-Session: https://claude.ai/code/session_01XhvRdRjkLqtkRBZi9ZYAu2
@koala73

koala73 commented Jul 6, 2026

Copy link
Copy Markdown
Owner Author

Review findings addressed in 4813c35 (all three verified at head 5b617ec before fixing):

F1 (P1) reverse-lookup poisoning — fixed + worse than reported. getSearchTermsForRegion reverse lookup now uses word-boundary matching. Live probe before: Somalia => [Somalia, Mali, mali, bamako] — the substring hit not only injected Mali's terms, the loop's break displaced Somalia's own keywords entirely. After: Somalia => [Somalia, somalia, mogadishu], Nigeria => [Nigeria, nigeria, lagos, abuja]. Regression tests assert both no-leak and own-keywords-present, plus a Myanmar (Burma) parenthetical positive control and a Nigeria-vs-Niger-market calibration guard.

F2 (P1) computeHeadlineRelevance — fixed. All four raw .includes sites (domain-hint loop, title-token loop, both requireSemantic counters) now use textIncludesTerm. RED-first test: 'Award season kicks off in Tirana' with conflict hints + iran token scores 0 under requireSemantic; 'War fears grow as Iran mobilizes' still scores (positive control).

F3 (P2) cache eviction — fixed. FIFO oldest-key eviction instead of clear() at capacity.

Verification: forecast-detectors 206/206 (3 new tests confirmed RED pre-fix), forecast-trace-export 310/310, history/integrity/resilience/narrative-cache suites green, evaluate-forecast-benchmark 6/6.

…ame reverse lookup

Finding 1 (P1): exact token boundaries dropped plural matches that bare
includes gave for free — 'elections'/'attacks' no longer hit the singular
DOMAIN_HINTS. The term regex now accepts an optional s/es suffix on the
text side; a term nested inside a DIFFERENT word still never matches
(nigers vs nigeria, malis vs somalia — boundary still required).

Finding 2 (P1): reverse lookup broke on the FIRST boundary containment,
so compounds resolved to the wrong country even though exact entries
exist (DR Congo => Congo/brazzaville, Guinea-Bissau and Papua New
Guinea => Guinea/conakry). Exact name match now wins outright; the
containment fallback picks the longest contained name.

5 new tests (all RED at prior head); detector suite 211/211,
trace-export 310/310, benchmark 6/6.

Claude-Session: https://claude.ai/code/session_01XhvRdRjkLqtkRBZi9ZYAu2
@koala73

koala73 commented Jul 6, 2026

Copy link
Copy Markdown
Owner Author

Re-review findings addressed in d453a64 (both reproduced at head 4813c35 before fixing):

F1 (P1) plural regression — fixed. The term regex now accepts an optional s/es suffix on the text side: (?:^|[^a-z0-9])term(?:s|es)?(?:[^a-z0-9]|$). 'elections'/'attacks' match the singular hints again; the original false-positive geometry stays closed ('niger'+suffix still cannot match inside 'nigeria', 'mali' still cannot match inside 'somalia' — the leading/trailing boundary is still required, only a plural suffix on the SAME word is tolerated). RED-first tests: Nigeria pred + 'peaceful elections' market now calibrates at the exact 0.4/0.6 blend; 'Missile attacks intensify' scores under requireSemantic.

F2 (P1) compound-name reverse lookup — fixed. Exact name match now wins outright (the JSON has exact entries for all three repro cases); the containment fallback selects the LONGEST contained name instead of breaking on the first. Probes: DR Congo => [DR Congo, congo, kinshasa, drc], Guinea-Bissau => [Guinea-Bissau, guinea-bissau], Papua New Guinea => [Papua New Guinea, papua new guinea]; round-1 results unchanged (Somalia => [Somalia, somalia, mogadishu], Myanmar parenthetical control intact).

Verification: forecast-detectors 211/211 (5 new tests RED at prior head), forecast-trace-export 310/310, history/integrity/resilience/narrative suites green, evaluate-forecast-benchmark 6/6.

… 'war' must not match 'wares'

Round-3 review finding (P2): the global (?:s|es)? suffix let 'war' match
'wares' — an Iran conflict forecast calibrated against 'Will Iran export
more wares in 2026?' (0.70→0.54, live-reproduced). 'es' is only an
English plural after sibilants, so the suffix is now conditional:
terms ending in s/x/z/ch/sh allow (?:es)?, everything else s?.
gas→gases and tax→taxes keep matching; attacks/elections unchanged.

4 new tests (2 RED at prior head, gases positive control); detector
suite 214/214, adjacent suites green, benchmark 6/6.

Claude-Session: https://claude.ai/code/session_01XhvRdRjkLqtkRBZi9ZYAu2
@koala73

koala73 commented Jul 6, 2026

Copy link
Copy Markdown
Owner Author

Round-3 finding addressed in fd1543d (reproduced at head d453a64 first: the wares market calibrated the Iran conflict forecast 0.70→0.54).

P2 global 'es' suffix — fixed. The suffix is now conditional on the term's ending: sibilants (s/x/z/ch/sh) allow (?:es)? (gas→gases, tax→taxes), everything else plain s? (attack→attacks, election→elections). 'war' can no longer match 'wares'. RED-first tests: wares market yields domainHits 0 and no calibration; greenhouse-gases positive control keeps the sibilant path.

Verification: forecast-detectors 214/214, trace-export 310/310, adjacent suites green, benchmark 6/6.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant