fix(forecast): non-finite probability guards — NaN can no longer enter or anchor the probability spine (#4933)#4969
Conversation
…r or anchor the probability spine (#4933) Three guard sites from the adversarial audit (MEDIUM): - makePrediction: NaN/undefined probability or confidence coerces to 0 instead of surviving the clamp chain as NaN and serializing to null (previously the forecast silently vanished at the publish filter). - detectFromPredictionMarkets: a market row with a non-numeric yesPrice is skipped; previously ('oops'||50)/100 = NaN passed the band gate (NaN comparisons are false) and emitted a null-probability forecast. - calibrateWithMarkets: a matching market without a finite yesPrice is excluded from candidates instead of anchoring the blend at a fabricated 50%; a second-best priced market can still match. 5 new tests (4 RED pre-fix); detector suite 219/219, adjacent suites green, benchmark 6/6. Claude-Session: https://claude.ai/code/session_01XhvRdRjkLqtkRBZi9ZYAu2
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Greptile SummaryThis PR plugs three sites in the forecast pipeline where a malformed Redis market row could inject NaN into the probability spine or anchor a blend at a fabricated 50%, by adding
Confidence Score: 4/5The three guards are each correctly placed, the old || 50 fallback is fully removed, and five new tests cover both the failure paths and the positive control. Safe to merge. All three bug sites are fixed correctly and tests confirm the expected behavior. One small edge case — Infinity input to makePrediction now becomes 0 instead of clamping to 1 — is unlikely to occur in practice but is an unannounced behavior change worth a second look. The behavioral change for Infinity inputs in scripts/seed-forecasts.mjs lines 923-924 is the only thing worth double-checking before merge. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Redis market row] --> B{Number.isFinite yesPrice?}
B -- No --> C[skip row / filter out]
B -- Yes --> D[yesPrice / 100]
D --> E[detectFromPredictionMarkets band gate 0.6-0.9]
E -- in band --> F[makePrediction with yesPrice]
E -- out of band --> C
D --> G[calibrateWithMarkets candidate filter]
G -- passes semantic match --> H[0.4 x marketProb + 0.6 x predProb]
G -- filtered --> I[pred.calibration = null, probability unchanged]
F --> J{Number.isFinite probability / confidence?}
J -- Yes --> K[Math.max/min clamp stored]
J -- No / NaN / undefined --> L[coerce to 0 clamp stored]
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
A[Redis market row] --> B{Number.isFinite yesPrice?}
B -- No --> C[skip row / filter out]
B -- Yes --> D[yesPrice / 100]
D --> E[detectFromPredictionMarkets band gate 0.6-0.9]
E -- in band --> F[makePrediction with yesPrice]
E -- out of band --> C
D --> G[calibrateWithMarkets candidate filter]
G -- passes semantic match --> H[0.4 x marketProb + 0.6 x predProb]
G -- filtered --> I[pred.calibration = null, probability unchanged]
F --> J{Number.isFinite probability / confidence?}
J -- Yes --> K[Math.max/min clamp stored]
J -- No / NaN / undefined --> L[coerce to 0 clamp stored]
Reviews (1): Last reviewed commit: "fix(forecast): non-finite probability gu..." | Re-trigger Greptile |
| probability: Math.round(Math.max(0, Math.min(1, Number.isFinite(probability) ? probability : 0)) * 1000) / 1000, | ||
| confidence: Math.round(Math.max(0, Math.min(1, Number.isFinite(confidence) ? confidence : 0)) * 1000) / 1000, |
There was a problem hiding this comment.
Infinity now coerces to 0 instead of clamping to 1
Number.isFinite(Infinity) is false, so a caller passing Infinity (or any non-finite value) is mapped to 0 before the clamp. The original code let Math.max(0, Math.min(1, Infinity)) correctly produce 1. The change is intentional for NaN/undefined, but a +Infinity probability would now silently become 0 instead of 1. In practice, probabilities fed from Redis should never reach Infinity, but if the guard is meant to express "non-numeric inputs get a safe default," a fallback of 0 is defensible — it's just a different semantic than the previous behavior and is worth confirming intentionally.
…0 for unreadable Kalshi prices The downstream finite-only guard was defeated upstream: the Kalshi normalizer converted a non-finite last_price_dollars into a finite 50, which then matched and calibrated forecasts against invented data. New parseKalshiYesPrice mirrors the Polymarket parseYesPrice contract (null for missing/non-numeric/out-of-range → market skipped); a genuine 0 price is kept and filtered downstream, not fabricated. 6 new tests (RED via missing export); prediction-scoring 61/61, forecast-detectors 219/219. Claude-Session: https://claude.ai/code/session_01XhvRdRjkLqtkRBZi9ZYAu2
|
Review finding addressed in cb4e426 — confirmed at source: Fix is upstream where it belongs: new 6 new tests RED-first (missing-export); prediction-scoring 61/61, forecast-detectors 219/219. |
…hole-string price validation
P2: Kalshi selected the top-volume market BEFORE checking price, so an
unpriced top market sank the whole event even when a priced sibling
existed. New selectPricedKalshiMarket picks the highest-volume market
among PRICED ones; the 5000 volume gate applies to the selected market.
P3: parseKalshiYesPrice used parseFloat, which accepts malformed
prefixes ('0.62oops' → 0.62, '0,62' → 0). Now whole-string Number()
validation with an empty/whitespace guard.
8 new tests (RED via missing export); prediction-scoring 66/66,
forecast-detectors 219/219.
Claude-Session: https://claude.ai/code/session_01XhvRdRjkLqtkRBZi9ZYAu2
|
Round-2 findings addressed in 466dba1: P2 priced-sibling fallback — fixed. New P3 whole-string validation — fixed. prediction-scoring 66/66 (8 new tests RED-first), forecast-detectors 219/219. |
What
PR-B of the #4933 hardening batch (MEDIUM audit finding): three sites let a malformed Redis market row inject NaN into the probability spine or anchor the market blend at a fabricated 50%.
makePredictionnull→ forecast silently vanished at the publish filterdetectFromPredictionMarkets('oops'||50)/100 = NaNpassed the band gate (NaN comparisons are false) → null-probability forecast emittedyesPricerow skippedcalibrateWithMarketsTests
Bug-fix protocol: 5 tests written first, 4 confirmed RED (NaN coercion ×2, NaN-emit skip, 50%-anchor skip), 1 positive control (finite in-band price still emits at 0.75).
Part of #4933 (fix 2 of 3 P1/MEDIUM prerequisites for #4930). Branched off main AFTER #4939 merged — no stack.
https://claude.ai/code/session_01XhvRdRjkLqtkRBZi9ZYAu2