Skip to content

P0(forecast): deploy seed-forecast-resolutions Railway cron — Bet-2 resolver merged (#5024) but never deployed; prod health CRIT, verification clock not started (#4930) #5062

Description

@koala73

P0(forecast): deploy the seed-forecast-resolutions Railway cron — Bet-2 resolver merged but never deployed

The Bet-2 verification loop (PR #5024, merged 2026-07-08) — resolver + scorecard — shipped as code only. The Railway cron service that runs it was never created, so the resolver has never executed in production. Everything downstream of the epic (#4930) that assumes "the clock is ticking since 2026-07-07" is currently false.

Evidence (verified 2026-07-09)

  • No service exists. Railway world-monitor has 77 services; the only forecast ones are deep-forecast-worker and seed-forecasts. There is no seed-forecast-resolutions. The seeder header says the service must be "set up manually via Railway dashboard" — that step never happened.
  • Prod health is DEGRADED, and the only two CRIT problems are this loop: /api/health?compact=1
    • forecastResolutions: { status: EMPTY, records: 0, maxStaleMin: 2160 }
    • forecastScorecard: { status: EMPTY, records: 0, maxStaleMin: 2160 }
      Both Redis keys (forecast:resolutions:v1, forecast:scorecard:v1) have never been written. The forecasts seeder itself is healthy — Bet 1 is emitting specs into a durable nothing.
  • The scorecard endpoint + MCP get_forecast_scorecard return an empty scorecard because the key is absent.

Why P0 — active data loss

  • forecast:predictions:history:v1 is LTRIM'd to ~200 snapshots ≈ 8.3 days (45d is only the TTL). Short-horizon forecasts that mature and roll out of that window before the first resolver run are permanently unresolvable.
  • Zero daily samples are being collected for in-flight at-deadline / within-horizon specs — sampling can only start once the cron runs.
  • Every day undeployed shrinks the recoverable set and the eventual track record.

Recoverable right now

Resolver --dry-run against prod Redis (2026-07-09): first run would immediately capture 54 forecasts into the durable ledger (20 hard-pending, 33 pending-judge, 1 VOID). Deploy today and none of those 54 are lost.

Fix (ops, ~15 min — not engineering)

  1. Create Railway service seed-forecast-resolutions in world-monitor/production.
    • Start command: node scripts/seed-forecast-resolutions.mjs
    • Cron: daily (one runSeed() per process — matches the seeder gotcha).
    • Env: inherit shared seeder env + set CLOUDFLARE_R2_FORECAST_RESOLUTION_PREFIX for receipt archival (R2 archival is best-effort; missing prefix just skips receipts).
    • Watch paths / source: same repo, scripts/seed-forecast-resolutions.mjs + scripts/_forecast-resolution*.mjs + scripts/_forecast-scorecard.mjs.
  2. Trigger one run; confirm forecast:resolutions:v1 + forecast:scorecard:v1 populate and health flips forecastResolutions / forecastScorecard → OK.
  3. Sanity-check the scorecard via get_forecast_scorecard (totals.entries > 0).

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions