The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
worldmonitor	Ignored	Preview	Jul 8, 2026 12:58am

Preview deployment for your docs. Learn more about Mintlify Previews.

💡 Tip: Enable Workflows to automatically generate PRs for you.

Greptile Summary

This PR wires up a full forecast-resolution lifecycle: a daily Railway seeder ingests published forecast specs into a durable Redis ledger, samples hard-spec source feeds, resolves terminal entries, archives R2 receipts, and publishes a Brier/log-score scorecard. A new get-forecast-scorecard API endpoint (with gateway fast caching) exposes the scorecard; the MCP tool registry and health checks are updated to match.

• Resolution engine (seed-forecast-resolutions.mjs): processResolutionCycle is cleanly pure and testable; ingestion, sampling, and resolution are each a single-responsibility step. The settlement-lag guard for UCDP count families and judged-spec handling (pending-judge) is correctly conservative and does not pollute Brier math.
• Scoring (_forecast-scorecard.mjs): Brier/log score, calibration buckets, per-domain/origin slices, and vs-market skill are all computed; VOID/pending entries are correctly excluded from accuracy math.
• Test coverage: Five new test files cover the pure-function helpers, seeder contract, scorecard math, and handler Redis-envelope contract, including idempotent rerun, feed gap, VOID-only input, and per-object R2 failure isolation.

Confidence Score: 4/5

Safe to merge; the resolution engine, scoring math, and handler are all well-tested with no correctness issues found.

The new seeder, evaluator, and scorecard modules are cleanly separated, pure-function exported helpers are fully unit-tested, and the handler covers all three response paths. The only findings are: a redundant computeScorecard call inside buildLedgerForRun whose result is discarded, a misleading publicationCoverage field name (always 0 or 1, not a ratio), and a missing optional chain on entry.spec.sourceFeed in samplePendingEntries that is safe today but inconsistent with the defensive style used elsewhere.

The three concerns are all in scripts/seed-forecast-resolutions.mjs and scripts/_forecast-scorecard.mjs; everything else is straightforward.

Important Files Changed

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Railway as Railway Cron
    participant Seeder as seed-forecast-resolutions.mjs
    participant Redis as Redis (Upstash)
    participant R2 as Cloudflare R2
    participant Gateway as Vercel Gateway
    participant Client as Client / MCP

    Railway->>Seeder: daily trigger
    Seeder->>Redis: LRANGE forecast:predictions:history:v1
    Redis-->>Seeder: history snapshots
    Seeder->>Seeder: ingestHistory
    Seeder->>Redis: GET per-sourceFeed keys
    Redis-->>Seeder: feed data
    Seeder->>Seeder: samplePendingEntries
    Seeder->>Seeder: resolveDueEntries (YES/NO/VOID)
    Seeder->>R2: PUT receipt JSON (best-effort per-object)
    Seeder->>Seeder: markReceiptsArchived
    Seeder->>Redis: SET forecast:resolutions:v1
    Seeder->>Seeder: computeScorecard
    Seeder->>Redis: SET forecast:scorecard:v1
    Seeder->>Redis: SET seed-meta keys
    Client->>Gateway: GET /api/forecast/v1/get-forecast-scorecard
    Gateway->>Redis: GET forecast:scorecard:v1
    Redis-->>Gateway: scorecard envelope
    Gateway-->>Client: GetForecastScorecardResponse

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Railway as Railway Cron
    participant Seeder as seed-forecast-resolutions.mjs
    participant Redis as Redis (Upstash)
    participant R2 as Cloudflare R2
    participant Gateway as Vercel Gateway
    participant Client as Client / MCP

    Railway->>Seeder: daily trigger
    Seeder->>Redis: LRANGE forecast:predictions:history:v1
    Redis-->>Seeder: history snapshots
    Seeder->>Seeder: ingestHistory
    Seeder->>Redis: GET per-sourceFeed keys
    Redis-->>Seeder: feed data
    Seeder->>Seeder: samplePendingEntries
    Seeder->>Seeder: resolveDueEntries (YES/NO/VOID)
    Seeder->>R2: PUT receipt JSON (best-effort per-object)
    Seeder->>Seeder: markReceiptsArchived
    Seeder->>Redis: SET forecast:resolutions:v1
    Seeder->>Seeder: computeScorecard
    Seeder->>Redis: SET forecast:scorecard:v1
    Seeder->>Redis: SET seed-meta keys
    Client->>Gateway: GET /api/forecast/v1/get-forecast-scorecard
    Gateway->>Redis: GET forecast:scorecard:v1
    Redis-->>Gateway: scorecard envelope
    Gateway-->>Client: GetForecastScorecardResponse

_{Reviews (1): Last reviewed commit: "fix(review): harden forecast resolution ..." | Re-trigger Greptile}

Review at head 3943442 against plan rev 2.1 — verdict: NOT ready, two P0s. (Posted as a comment — same-account PR can't take a formal request-changes review.)

The architecture is faithful in most places — the (id, deadline-window) model is correctly implemented (rollover, probability-only pre-deadline updates, one-open-window), the retryable R2 receipt marking is better than the plan asked for, the scorecard math is clean, and the ops wiring (health 36h, railway-services, envelope-aware handler) is right. But two P0s in resolution semantics produce wrong outcomes that freeze permanently into the audit ledger — the worst failure mode this product can have.

P0-1 — Every production prediction-market resolution is inverted

crossesThreshold(value, threshold, baseline): when baseline > threshold it returns YES iff value <= threshold. Production pred-market specs always have baseline > threshold — the detector band-gates emission to yesPrice 60–90, threshold is 50 — so a market that settles YES (→100) resolves NO, and one that settles NO (→0) resolves YES. Executable repro (settled 98, emitted at 72):

outcome: "NO", comparison: "71 crosses 50 from 72"

The at-endDate test passes only because its fixture uses baselineValue: 42 — below 50, a shape production cannot emit (band gate) — the exact fixture-shape trap the plan's omission meta-test exists to prevent, in a different guise. Fix: yesPrice/at-endDate is settlement semantics, not move semantics: outcome = settled value >= threshold, regardless of baseline side. Keep directional crosses for price moves only. Add fixtures: baseline 72 → settle 98 → YES; settle 3 → NO.

P0-2 — at-deadline/at-endDate never uses an at-or-after-deadline read (plan D3 inverted); no-sample entries VOID

Plan D3: resolve as "the first sample/read at-or-after the deadline." The implementation does the opposite, in four mutually-reinforcing places: selectDeadlineSample requires ts === deadline exactly (dead code at daily cadence); selectLastSample(samples, deadline) excludes post-deadline samples; the live feedValue is discarded whenever nowMs > deadline (i.e. on every actual resolve tick); and samplePendingEntries stops sampling once nowMs > deadline, so no post-deadline sample ever exists. Reproduced consequences: an entry maturing between ticks with no samples VOIDs despite the correct value sitting in the live feed (this is every 24h spec ingested at its first tick); entries with samples resolve on values up to 24h stale; pred-market settlement reads pre-settlement prices. Two tests enshrine this ("does not resolve at-deadline from a delayed current feed snapshot") — the drift instinct is understandable, but it contradicts D3, and the evidence already records readTs so lateness is auditable rather than hidden. Fix: at the resolve tick prefer (1) first post-deadline sample, (2) the current live read with readTs: nowMs, (3) optionally last pre-deadline sample as explicitly-degraded evidence; and sample through the resolve tick (or let resolve use the live read resolveDueEntries already passes in).

Both P0s compound with D6 flip-resistance + R2 receipts: once written, wrong outcomes are terminal and archived. On the current schedule the first inverted receipts would be written ~day 2 after deploy.

P1-3 — Plan-mandated adversarial tests missing

No blowup-bound test (many hourly re-emissions of one id → exactly one open entry), no flip-resistance test (terminal entry + drifted feed fixture → unchanged outcome). The rollover/probability-only test exists and is good; the two missing ones are precisely the class that would have caught P0-1's fixture problem.

P2

• publicationCoverage: entries.length ? 1 : 0 is a hardcoded 1 masquerading as a measured metric (R7). Either wire a real denominator (e.g. distinct published forecasts seen in the intake window) or omit the field until it's real — a fake 100% is worse than absence.
• R11 trim not implemented: resolved entries are never trimmed after R2 archive, so the ledger grows unbounded. Trim receiptArchivedAt entries older than the rolling window in the same pass, or record an explicit deferral.

P3

• Unknown operator resolves NO (compare() falls through to false) — should be VOID with reason.
• count entries whose feed permanently vanishes pend forever (no give-up bound past seal + N days).
• metaCritical: true on the scorecard extra key throws (FATAL path) on a transient seed-meta write failure after the canonical publish already succeeded — R9's graceful posture suggests exit 75 / warn instead.
• Q2 (live feed-shape walkthrough) has no artifact in this PR: the alias tables (route→name/label/chokepoint, hexCount→hexes, market feed shape) are unverified guesses; the plan gates extractor freeze on a live walkthrough. Please attach the walkthrough or run it in the review-fix.

Verified good (no action)

Window model incl. multi-pending-window routing; ingest freeze rules; count skips sampling; count settlement lag; omission meta-test; scorecard determinism/nowMs injection/decile buckets/market-skill exclusions; proto optional on maybe-absent scalars; envelope-aware passthrough; generationOrigin added to history entries (needed, tested); _seed-utils extra-key meta support; base is 4 commits behind main but MERGEABLE with no overlapping files.

Full review at head 980b7d063 (supersedes the core-pass comment at 3943442): 8-angle fleet over the entire diff + independent gate sweep. Both prior P0s are verified FIXED at this head (re-ran the executable repros: settlement resolves YES/NO correctly both directions; at-deadline uses post-deadline samples / the live read with honest readTs). Full test sweep green under the correct runners (55+16+29+9 MCP+3 RPC), typecheck, docs-stats, biome all green.

New findings this round

P1 — a null probability scores as a confident 0% and wrecks the headline Brier (CONFIRMED by repro)

createEntry stores probability: Number(forecast.probability) — NaN when the source field is missing/malformed. JSON.stringify converts NaN → null on the Redis round-trip, and on read-back isScoredEntry's Number.isFinite(Number(entry.probability)) passes because Number(null) === 0. Repro: one resolved entry with probability: null → scored: 1, brier: 1, logScore: 13.8155 folded into overall/byDomain/calibration. One malformed forecast visibly corrupts the credibility number. Fix: don't store non-finite probabilities at ingest (Number.isFinite(p) ? p : undefined + pruneUndefined → the entry can pend but never score), AND harden isScoredEntry to typeof entry.probability === 'number' && Number.isFinite(entry.probability). Add the post-round-trip-null test.

P3 — stale is hardcoded false for envelope-less scorecard payloads

When the stored value lacks _seed.fetchedAt (manual write, pre-migration artifact), the handler returns stale: false regardless of age. Still an improvement over get-forecasts (which hardcodes stale: false unconditionally), and the health seed-meta entry catches a dead cron independently — hardening, not blocking.

Adversarial candidates refuted this round (for the record): the "stale spec kept on re-emission" candidate is rev 2.1's frozen-window rule working as designed (probability-only mid-window updates; a changed spec gets its own window at rollover); the UCDP epoch-0 timestamp-shadowing candidate is unreachable (upstream trailing-window filter drops unparseable-date records before the feed is written, and the stored shape retains no string date to fall back to); the same-tick sample-dedup candidate requires a retry path that does not exist in the call graph.

P0 — five of six hard families silently VOID against real production feed shapes (CONFIRMED by repro)

The Q2 feed-shape walkthrough the plan gated on was never done, and it shows — three independent extraction gaps, each verified against real writer shapes:

1. Envelope gap (kills prediction-market, market/commodity, infrastructure): the feed seeders run in runSeed contract mode (declareRecords exported — verified), so their Redis values are {_seed, data} envelopes. The resolver's readRedisJson never unwraps feed reads (only the ledger), and iterateRecords recurses only into ARRAY children — it can never descend through the data object. Repro: a payload shaped exactly like the real prediction-markets output → VOID/no_establishable_metric with a priced matching record present.
2. riskScore alias list is missing the real field (kills supply_chain): the live chokepoints:v4 record carries disruptionScore (this is the exact field the Bet-1 R12 walkthrough resolved against — hormuz 70 ≥ 66); the eval reads riskScore/risk_score/score/risk and returns NaN off a correctly-matched record. Repro: real shape → VOID.
3. hexCount is a field read, not a count (kills gps): gpsjam:v2 is one row per H3 hex with no count-like field; the emission semantics (mirroring count()) require counting matching hex rows for the region. The eval reads a nonexistent field off one hex row → always VOID.

Net: only the UCDP count() family can resolve YES/NO in production. Every unit test passes because every fixture hand-writes a flat, conveniently-named shape no real writer emits — the second fixture-realism failure in this PR (the first masked the settlement inversion with a below-band baseline). Fixes: unwrap envelopes on feed reads (or make iterateRecords descend through plain-object children); add disruptionScore to the riskScore aliases; implement hexCount as a region-filtered row count; and add one fixture per family copied verbatim from the real writer's output shape (that is the Q2 walkthrough, done properly — attach it).

P2 — point-window entries can pend forever, violating R7

When a point-window (at-deadline/at-endDate) spec's feed stays unavailable — or looks available-but-empty, which is exactly what the envelope gap produces — the entry never reaches a terminal state: it resamples an error entry every run indefinitely. R7 requires every entry to terminate or be visibly pending with a path to termination; add a give-up bound (e.g. deadline + N days → VOID with reason feed_never_readable).

P3 (cleanup tier, from the efficiency/reuse angle — non-blocking)

• buildLedgerForRun double-ingests: ingestHistory(...) then processResolutionCycle re-runs ingestHistory(preLedger, []) — two full-ledger deep clones per run for zero new data. Refactor processResolutionCycle to accept a pre-ingested ledger.
• findOpenWindowKey scans all ledger keys per forecast per snapshot (O(snapshots × forecasts × entries)) — pre-index by id before the ingest loop.
• normalizeLedger exists in the seeder (returns keyed object) and the scorecard (returns array) with different semantics — drift risk; extract one canonical helper.
• normalizeSamples handles three phantom shape aliases the seeder never writes.
• One refuted efficiency suggestion for the record: sortLedger is NOT redundant — deterministic key order is what makes ledger round-trips byte-stable (the idempotency property); keep it.

Carried from the prior review, still open at this head

• R11 trim unimplemented — resolved entries are never trimmed after R2 archive; the working ledger grows unbounded (slow). Implement trim of receiptArchivedAt entries older than the rolling window, or record an explicit deferral.
• Plan-mandated tests still missing: blowup-bound (many hourly re-emissions of one id → exactly one open entry) and flip-resistance-under-drifted-feed (terminal entry + changed feed fixture → outcome unchanged). The new yesPrice-50-line test is a different property.
• publicationCoverage now scored/entries — a real ratio (improvement), but it measures scored-share, not published-vs-generated (R7's intent). Rename the field (e.g. scoredShare) or wire the real denominator; a mislabeled metric on a credibility surface invites the exact critique R7 exists to pre-empt.
• P3s: unknown operator → NO instead of VOID; count entries pend forever when a feed permanently disappears; metaCritical: true FATALs on a transient scorecard-meta write blip post-publish (graceful exit 75 fits R9 better).

Verified good this round (beyond the prior list)

Scorecard JSON field names match the generated camelCase type exactly (no silent passthrough drops); both seed-meta: keys are genuinely written (health entries watch real keys); the 39→40 MCP tool count is consistent across all six agent surfaces; proto additive-only with omission-safe pruning (no nulls emitted); prior P0 fixes came with real regression tests (settlement both directions at the 50 line; post-deadline read preference).