🚨 The verification pipeline is measuring a biased, synthetic-heavy input funnel — Brier ≠ real forecasting skill

We built a working resolution + scorecard pipeline (#4930: resolver live, 5 scored, brier 0.28, judged-window fixed via #5196). But the forecasts flowing into it are too narrow and too synthetic to validate whether the pipeline — or the model — is any good. You cannot judge a forecasting system's skill on this sample.

Evidence (prod, 27 unique forecasts over 21 hourly snapshots, 2026-07-12)

• 37% of the funnel is state_derived synthetic backfill (10/27), concentrated in market.
• Only 5 domains ever produce forecasts, and 3 of them are the ones that happen to have live resolution feeds. The geopolitical + macro + energy + health surface — the core of a "world monitor" — is absent.
• The scorecard scores this whole sample, so brier/voidRate/coverage are computed over synthetic-padded, 5-domain data.

Root cause (two coupled, both generator-side)

1. Narrow detector coverage (the missing-signals half of epic(forecast): generator signal integrity & correctness — dead signals, market-calibration polarity, silent no-fire #5097). The signal-integrity bugs are fixed (fix(forecast): FRED macro/rates signals silently dead — shape mismatch kills all 10 (P1, #5097) #5098 FRED, fix(forecast): ACLED conflict signal not produced in prod + health blindspot — conflict EMA blind to near-real-time (P1, #5097) #5099 ACLED, feat(forecast): signal observability — kill silent feed no-fire (per-feed logging + health coverage + temporal keep-warm) (P2, #5097) #5101 observability — all closed). But epic(forecast): generator signal integrity & correctness — dead signals, market-calibration polarity, silent no-fire #5097's missing-signals audit found ~20 feeds collected but consumed by no detector (EIA/JODI energy, FAO food, GDELT, displacement, disease, USNI naval, NOTAM…). No detector → no forecasts for those domains → the funnel is stuck at ~5 domains.
2. Synthetic backfill to a fixed publish target. MIN/MAX_TARGET_PUBLISHED_FORECASTS = 10–14 (seed-forecasts.mjs:116-117); selectDeferredForecastForPublishBackfill (:13862) pads the detector shortfall with state_derived forecasts (:1623) — which is why market/synthetic dominates. These carry awkward auto-titles ("Maritime energy flow disruption from Black Sea maritime disruption state (market)") that also resolve poorly.

Why this is the red alert

Every downstream conclusion — "brier 0.28", "vs-market skill", "≥80% resolvable" — is drawn from a sample that is 37% synthetic and missing the domains the product exists to forecast. We could ship a perfect resolver and still learn nothing about real skill. This blocks the meaning of #4930's KPIs even though the plumbing works.

Required actions

1. Widen detector coverage to the collected-but-unused signals (the epic(forecast): generator signal integrity & correctness — dead signals, market-calibration polarity, silent no-fire #5097 missing-signals list — start with energy: EIA/JODI/inventories/SPR; then food-prices, GDELT, displacement). Each new detector adds a real domain to the funnel.
2. Segregate synthetic from the skill metric. The scorecard already slices byGenerationOrigin — report legacy_detector-only Brier as the headline skill number, exclude/flag state_derived backfill so it doesn't contaminate the measured skill. (Fast, high-value.)
3. Curb / improve the backfill. Don't pad Brier-scored output with low-quality synthetic forecasts to hit a count target; if backfilling for UX, mark them non-scored. Fix the auto-generated titles/claims so they're crisp and resolvable (see companion question-quality issue).
4. Add a funnel-diversity guardrail (log/alert): if a run publishes X% state_derived, that's a signal the funnel has collapsed.

Done when

The scorecard's detector-only skill sample covers ≥6–8 domains with real geopolitical/macro/energy representation, state_derived is excluded from the headline Brier, and no run is majority-synthetic.

Related: #4930 (verification — this defines whether its KPIs mean anything), #5097 (generator integrity — the missing-signals half), #5091 (resolvable coverage), #5092 (Bet-5 track record — must not publish a synthetic-inflated Brier).