The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
worldmonitor	Ignored	Preview	Jul 9, 2026 11:15am

Greptile Summary

This PR adds LLM-judge-based resolution for forecasts that cannot be auto-resolved by metric feeds, routing due pending-judge entries through two competing models (OpenRouter + Groq) over a Redis-backed news archive. Sealed outcomes require dual-model agreement with grounded citations; disagreement, missing citations, or archive gaps conservatively produce VOID or leave entries pending for retry.

• New processResolutionCycleWithJudges orchestrates ingestion, hard-spec resolution, LLM judging, pruning, and scorecard computation in a single async cycle, replacing processResolutionCycle in the live run path.
• readDigestAccumulatorArchive fetches the relevant window of story-track hashes via ZRANGEBYSCORE and pipelines HGETALL lookups, throwing on any empty or errored row (fail-closed design) and flagging truncated reads to prevent false VOIDs.
• Run configuration now includes an explicit lockTtlMs: 180_000 and a widened fetchPhaseTimeoutMs: 150_000 to accommodate the additional LLM latency budget.

Confidence Score: 3/5

Safe to merge with the understanding that any story:track:v1 hash key expiring before its ZSET entry is pruned will silently block all judged resolutions for every subsequent run until the stale ZSET members age out.

The HGETALL fail-closed behavior in readDigestAccumulatorArchive turns a missing Redis key into a blanket archive-unavailable signal that freezes every pending-judge entry indefinitely. The remaining findings are self-contained and do not affect production resolution correctness.

scripts/seed-forecast-resolutions.mjs — specifically the HGETALL empty-result handling in readDigestAccumulatorArchive and whether story track key TTLs are guaranteed to be at least as long as the ZSET accumulator retention window.

Important Files Changed

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Worker as Railway Worker
    participant Redis as Redis (Upstash)
    participant OR as OpenRouter Judge
    participant Groq as Groq Judge
    participant R2 as Cloudflare R2

    Worker->>Redis: LRANGE forecast:predictions:history:v1
    Redis-->>Worker: history snapshots
    Worker->>Worker: ingestHistory / samplePendingEntries / resolveDueEntries
    Worker->>Redis: ZRANGEBYSCORE digest:accumulator:v1:full:en [window]
    Redis-->>Worker: story hashes
    Worker->>Redis: HGETALL pipeline (story:track:v1:hash x N)
    Redis-->>Worker: story track rows (throws on empty/error row)
    loop Each due pending-judge entry
        Worker->>Worker: selectJudgedArchiveItems
        alt No relevant archive items
            Worker->>Worker: archiveCoversEntryWindow check
        else Relevant items found
            Worker->>OR: callForecastLLM
            OR-->>Worker: outcome + citations
            Worker->>Groq: callForecastLLM
            Groq-->>Worker: outcome + citations
            Worker->>Worker: normalizeJudgment x 2
        end
    end
    Worker->>Redis: SET forecast:resolutions:v1
    Worker->>Redis: SET forecast:scorecard:v1
    Worker->>R2: PUT receipt JSON

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Worker as Railway Worker
    participant Redis as Redis (Upstash)
    participant OR as OpenRouter Judge
    participant Groq as Groq Judge
    participant R2 as Cloudflare R2

    Worker->>Redis: LRANGE forecast:predictions:history:v1
    Redis-->>Worker: history snapshots
    Worker->>Worker: ingestHistory / samplePendingEntries / resolveDueEntries
    Worker->>Redis: ZRANGEBYSCORE digest:accumulator:v1:full:en [window]
    Redis-->>Worker: story hashes
    Worker->>Redis: HGETALL pipeline (story:track:v1:hash x N)
    Redis-->>Worker: story track rows (throws on empty/error row)
    loop Each due pending-judge entry
        Worker->>Worker: selectJudgedArchiveItems
        alt No relevant archive items
            Worker->>Worker: archiveCoversEntryWindow check
        else Relevant items found
            Worker->>OR: callForecastLLM
            OR-->>Worker: outcome + citations
            Worker->>Groq: callForecastLLM
            Groq-->>Worker: outcome + citations
            Worker->>Worker: normalizeJudgment x 2
        end
    end
    Worker->>Redis: SET forecast:resolutions:v1
    Worker->>Redis: SET forecast:scorecard:v1
    Worker->>R2: PUT receipt JSON

_{Reviews (1): Last reviewed commit: "fix(forecast): resolve judged forecast o..." | Re-trigger Greptile}

Empty HGETALL result throws and blocks all judged resolutions.
The --dry-run path still calls processResolutionCycle rather than processResolutionCycleWithJudges.

Addressed in b926bb2dd and pushed in 7cfbae48a:

• missing/empty story-track rows are skipped and mark archive coverage incomplete, with regressions for stale ZSET rows
• dry-run now exercises processResolutionCycleWithJudges using no-LLM judge models
• single injected judge lists now fail closed with fewer_than_two_models instead of falling through to live judges

Validation: pre-push hook passed, and the merged-tree focused forecast/script sweep passed.

🔧 Fix plan — judged forecasts perpetual-pend / forced-VOID (48h archive window) — subagent-ready

Now urgent: conflict + unrest were reclassified to judged (#5138/#5142, deployed 07-10). They're 7d/30d-horizon, so they hit this trap head-on. First judged deadlines mature ~2026-07-14 — fix before then or the whole reclassification yields VOIDs, not YES/NO.

The bug (verified against origin/main, prod data)

A judged forecast (gen T0, deadline D=T0+horizon, resolved at now≈D):

1. judgedArchiveWindowForEntry (seed-forecast-resolutions.mjs:408) anchors the required evidence window on min(generatedAt, firstSeenAt, deadline, now) = generatedAt → [T0−6h, now] (the entire 7–30d horizon).
2. readDigestAccumulatorArchive (:1040) clamps the read to coverageStartMs = max(windowStart, now − JUDGED_ARCHIVE_RETENTION_MS) where JUDGED_ARCHIVE_RETENTION_MS = 48h (:34).
3. archiveCoversEntryWindow (:398) requires coverageStart(now−48h) ≤ requiredStart(T0−6h) → false for any horizon > ~48h; it also returns false whenever archiveInput.truncated (the 3,000 hash-cap, :1070).
4. resolveJudgedEntry (:222): dual-agreement (:271) seals YES/NO ignoring completeness — the ONLY resolve path. Otherwise if (!archiveComplete) return pending (:274) → sits pending-judge until maybeExpireJudgedEntry (:197) VOIDs it after 14 attempts + 14d (judge_retry_exhausted).

Net: long-horizon judged forecasts resolve only on dual-agreement from the last 48h; everything else → 14d limbo → VOID. No real Brier signal.

Data (prod, 2026-07-10): accumulator digest:accumulator:v1:full:en retains 93 days (oldest 2026-04-08) — widening is feasible. Story volume: 48h=2,341 (fits 3k cap); 7d=6,989 / 14d=14,398 / 30d=31,577 all exceed the 3,000 hash cap → truncated=true on any window >48h → the completeness gate is guaranteed to fail once widened. So widening the window and fixing the truncated conflation must happen together.

Root cause (two coupled defects)

• R1 — wrong required window: anchored on generation, not the deadline. "Did X happen by D" is answered by news in the run-up to / just after D, not the entire horizon.
• R2 — completeness conflates "reached the window" with "read every story": truncated (hit hash cap) is treated as "incomplete window," so any window with >3k stories can never be "complete," forcing pending → forced-VOID.

The fix (do all three together)

Part 1 — Deadline-centric required window. Rewrite judgedArchiveWindowForEntry:

const deadline = Number(entry?.deadline ?? entry?.spec?.deadline);
const anchor = Number.isFinite(deadline) ? deadline : nowMs;
return { startMs: Math.max(0, anchor - JUDGED_EVIDENCE_LOOKBACK_MS), endMs: nowMs };

New const JUDGED_EVIDENCE_LOOKBACK_MS (default 7d, env FORECAST_RESOLUTION_JUDGE_EVIDENCE_LOOKBACK_MS). Window is now [D−7d, now] — bounded (~8–9d incl. settlement) regardless of horizon. JUDGED_ARCHIVE_LOOKBACK_PAD_MS (6h) is subsumed; keep or fold in.

Part 2 — Read the deadline-centric window; stop clamping to 48h. In readDigestAccumulatorArchive:

• coverageStartMs = max(windowStartMs, nowMs − JUDGED_EVIDENCE_MAX_LOOKBACK_MS) — replace the 48h clamp with a max lookback cap (default 14d, env-tunable) that is ≥ the 7d lookback. The accumulator's 93d retention supports it.
• Raise DEFAULT_JUDGED_ARCHIVE_HASH_LIMIT so a 7–14d window fits without recency-truncation: 15,000 (14d≈14.4k). Pipeline of ~15k HGETALLs is bounded and daily; verify it stays within archiveTimeoutMs (bump to 20–25s) — if too heavy, page the ZSET in chunks. ZREVRANGEBYSCORE is recency-ordered, so if the cap is hit the retained set is the most-recent (deadline neighborhood) — acceptable, but do NOT mark it incomplete (Part 3).

Part 3 — Decouple truncated from completeness; seal matured entries. In archiveCoversEntryWindow:

• Remove if (archiveInput.truncated) return false. Completeness = did the read reach the required deadline-centric start: coverageStartMs ≤ startMs && coverageEndMs ≥ endMs. Hitting the hash cap within the window = sampled-most-recent, not incomplete.
• Keep truncated only as a logged sampling note (already warned at :1072).
• Consequence in resolveJudgedEntry: for any matured entry the window is now covered → agreement seals YES/NO; disagreement / all-VOID / no-evidence → VOID immediately (:277, honest terminal, Brier-excluded) instead of 14d→judge_retry_exhausted. maybeExpireJudgedEntry becomes a rare safety net for genuinely archive-unavailable runs only.

Tests (RED-first, tests/forecast-resolutions-seeder.test.mjs / forecast-resolution-eval — use the judgeModels doubles + injected archive)

1. The trap, fixed: a 30d-horizon judged entry, now = deadline+1h, archive spanning [deadline−7d, now] with truncated:true → archiveCoversEntryWindow is true; disagreeing judges → VOID (judge_disagreement), NOT pending. (Before: pending.)
2. Deadline-centric window: judgedArchiveWindowForEntry for a generatedAt=T0, deadline=T0+30d entry returns startMs = deadline − 7d (not T0−6h).
3. Resolution still works: agreeing judges with a matching citation → YES/NO sealed (unchanged).
4. Genuinely-unavailable still pends: archiveInput.available === false → pending (archive_unavailable), untouched.
5. No-evidence on a covered window → VOID (no_archive_evidence), not pending.
6. readDigestAccumulatorArchive unit: coverageStartMs = deadline-lookback (capped at max), not now−48h.

Validation (post-deploy, before/around 07-14)

• Watch forecast:scorecard:v1: pendingJudge should start falling as due judged entries resolve/VOID; scored/void rise. Scorecard-watch M2 fires on the first judged resolution.
• VOID-rate guardrail: immediate-VOID-on-disagreement can raise VOID rate. If voidRate on judged spikes (>~40%), the wider window isn't producing agreement — consider a 3rd tiebreak judge or majority-lean (follow-up, not v1).
• Confirm judge cost/latency stays within the run budget (FORECAST_RESOLUTION_JUDGE_RUN_BUDGET_MS) with the larger evidence pool + 15k-hash reads.

Config knobs added

FORECAST_RESOLUTION_JUDGE_EVIDENCE_LOOKBACK_MS (7d), FORECAST_RESOLUTION_JUDGE_EVIDENCE_MAX_LOOKBACK_MS (14d), FORECAST_RESOLUTION_JUDGE_ARCHIVE_HASH_LIMIT (→15000). All default-safe; no schema/proto change.

Deploy note

seed-forecast-resolutions auto-deploy is blocked by the checkSuites skip — deploy manually via serviceInstanceDeployV2(serviceId, environmentId, commitSha) then deploymentInstanceExecutionCreate(input:{serviceInstanceId}) (see #5062).

Related: #5091 (this is the open verification gate), #5138/#5142 (conflict/unrest now judged), #5062 (deploy method).

☝️ Tracked as a live work item in #5183 (this is a merged PR — the fix lives there for a subagent to own).