Skip to content

feat(forecast): Bet 4 — LLM-ensemble shadow probability on top-K (shadow now, promote on Brier) (Lane A #3) #5093

Description

@koala73

Part of #4930Lane A #3 (Bet 4). Shadow now, promote later.

Today the LLM never sets a probability — the strongest known accuracy lever (retrieval-grounded LLM ensembles approach superforecaster parity on ForecastBench) is unused. Build the ensemble in shadow mode now so comparative Brier data accrues from day one; promotion is Lane C.

Scope (shadow)

  • For the top K≈6 forecasts per run: 3 diverse reasoning passes over the already-built evidence table + news context + theater-sim paths — outside-view/base-rate, inside-view/signal-weighing, adversarial refuter. Trimmed-mean aggregate.
  • Shadow-publish the ensemble probability alongside the detector probability (do NOT replace it). Bet 2's resolver already scores whatever's published — so it will score both and produce a head-to-head Brier.
  • Stronger model for this stage only (~18 calls/run), prompt-hash cached like every other stage, respects the per-run LLM budget (cache-identity discipline).

Promotion (Lane C, not this issue)

Three-way blend (detector prior + LLM ensemble + market anchor, weights learned in Bet 3); promote only when the blend beats the detector on rolling Brier.

Done when

Every top-K forecast carries a shadow ensemble probability that Bet 2 scores separately, with a scorecard slice comparing detector-vs-ensemble Brier.

Related: #4930, #5007 (scores the shadow), Bet 3 (learned blend weights — Lane C).

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: AI/intelAI analysis, intelligence findings, summarizationenhancementNew feature or requestlane-a#4930 forecast epic — Lane A (do-now, data-independent parallel work)

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions