feat(forecast): trigger-simulation endpoint + runId filter (#3734 C1+C2)#3811
Merged
Conversation
Closes #3734. Implements the C1+C2 scope of v3 plan (docs/plans/2026-05-18-003-feat-simulation-trigger-and-runid-filter-plan.md; also tracked as #3798): - POST /api/forecast/v1/trigger-simulation — PRO-gated, mirrors run-scenario shape (isCallerPremium → queue-depth → server-derived runId → idempotency pre-check → enqueue). Returns { queued, runId, pkgFingerprint, reason }. External reason taxonomy collapses to {'', 'no_package', 'already-handled'} — server logs retain the distinction between already-queued vs already-completed-this-cycle to avoid a cron-timing oracle (Sec4). - GET /api/forecast/v1/get-simulation-outcome?runId=X — filter now actually fires. New behavior: by-run hit → return outcome; tombstone hit → fall through to :latest with distinct note; queued-but-not-completed → return processing=true (uses listQueuedSimulationTasks; prevents queued-vs-expired indistinguishability from contaminating the 30-day demand signal); miss → fall through with expiry note. - Worker (writeSimulationOutcome) writes BOTH :latest (canonical, 60d TTL) AND :by-run:{runId} (secondary index, 24h TTL). By-run failure logs + attempts tombstone payload so reads can distinguish transient failure from true expiry (D9). - pkgFingerprint = sha256(pkgKey).slice(0,16) opaque token. Replaces what v2 had as raw R2 path (raw path leaked bucket layout to Pro callers per Sec7). Worker verifies task.pkgFingerprint vs current pointer fingerprint with truthy-guard predicate to avoid spurious package_rotated logs on pre-upgrade in-flight tasks (Feas-F4). - .mjs seeder enqueueSimulationTask backports redis_error reason code so the auto-trigger path can surface transport failures cleanly (Adv-F4 / SG-2). No documented asymmetry between TS and .mjs paths. - Shared shim at server/_shared/_simulation-queue-constants.mjs (with .d.ts) is single source of truth for queue key constants + 24h TTL + pkgFingerprint helper + MAX_QUEUE_DEPTH. Both the seeder and the new TS handler module import natively from it. - PREMIUM_RPC_PATHS entry added so the gateway's Pro-bearer gate fires before per-IP rate-limit (matches run-scenario precedent). - ENDPOINT_RATE_POLICIES entry: { limit: 10, window: '60 s' } per-IP, identical to run-scenario. - Proto: TriggerSimulationRequest carries an optional client_version field (sebuf v0.11.1 emits a typecheck-broken POST client for fully-empty request messages; one field works around it). Server logs but never branches on it. - Plan + tracking metadata: TODO #21 transitioned from pending → complete; TODO #27 work log updated noting the filter is now actually active (the original 2026-03-24 "complete" was docs-only). Tests (26 total, all passing): - tests/simulation-queue-parity.test.mts (7) — TS ↔ .mjs parity for all reason codes (no documented asymmetry); structural lock on the worker's truthy-guard predicate - tests/forecast-trigger-simulation.test.mts (9) — handler test for 403 / 429 / no_package / already-handled (both states collapse to single external code) / happy path with opaque fingerprint / 503 transport on pointer read AND enqueue / regression test that internal codes never leak externally - tests/forecast-get-simulation-outcome.test.mts (6) — all 5 read paths + NOT_FOUND - tests/simulation-outcome-by-run-write.test.mjs (4) — structural locks on :latest canonicality, :by-run TTL + no-NX flag, tombstone-on-failure pattern, return shape Follow-up: #3809 filed for the audit-reframe docs/marketing work (surface "simulations auto-run hourly via cron" in OpenAPI / Pro marketing / future MCP tool descriptions). Closing #3734 in code without that docs change leaves the audit's framing defensible to future reviewers.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
…un-id
Inline operator notes on the script that runs the simulation worker:
- The `--run-id=X` format must match `/^\d{13,}-[a-z0-9-]{1,64}$/i`;
invalid IDs are silently skipped (pre-existing footgun #3734 Adv-F5).
- Force-reprocess via `--run-id` overwrites both :latest and :by-run
keys with no NX, resets the 24h TTL; CDN cache may serve stale for
up to 30min (D6 trade-off documented in v3 plan).
No behavior change — comments only. Discoverable by operators reading
the script they invoke, rather than requiring knowledge of a separate
runbook file.
The local pre-push hook ran 'npm install --prefer-offline' (its safety net when node_modules is missing in a worktree) and that pruned five '[email protected]' optional peer-dep entries that exist for non-macOS platforms. CI's 'npm ci' on Linux then refused the lockfile. Restoring the lockfile to origin/main's version — those entries belong there for cross-platform compatibility.
Contributor
Greptile SummaryThis PR implements two tightly-coupled changes for the forecast simulation pipeline: a new PRO-gated
Confidence Score: 4/5Safe to merge with one targeted fix: the tombstone SET in the worker needs an NX flag before going to production. The tombstone write in writeSimulationOutcome lacks an NX guard. A primary :by-run write that commits server-side but times out at the network layer will be immediately overwritten by the tombstone, permanently marking a successful outcome as a transient failure for the 24h TTL. All other changes are well-designed and covered by 26 new tests. scripts/seed-forecasts.mjs (tombstone write race); server/worldmonitor/forecast/v1/get-simulation-outcome.ts (dual key-source for SIMULATION_OUTCOME_LATEST_KEY) Important Files Changed
Sequence DiagramsequenceDiagram
participant Client
participant Gateway
participant TriggerHandler as trigger-simulation.ts
participant GetOutcomeHandler as get-simulation-outcome.ts
participant Redis
participant Worker as seed-forecasts.mjs
Client->>Gateway: POST /trigger-simulation (Pro key)
Gateway->>TriggerHandler: isCallerPremium, rate-limit check
TriggerHandler->>Redis: ZCARD queue depth
TriggerHandler->>Redis: GET simulation-package:latest
TriggerHandler->>Redis: GET simulation-outcome:latest
TriggerHandler->>Redis: "SET simulation-task:v1:{runId} NX"
TriggerHandler->>Redis: ZADD task-queue
TriggerHandler-->>Client: "queued=true, runId, pkgFingerprint"
Worker->>Redis: ZRANGE drain queue
Worker->>Redis: SET simulation-outcome:latest EX 60d
Worker->>Redis: "SET simulation-outcome:by-run:{runId} EX 24h"
Note over Worker,Redis: tombstone fallback missing NX
Client->>GetOutcomeHandler: "GET /get-simulation-outcome?runId=X"
GetOutcomeHandler->>Redis: "GET by-run:{runId}"
alt by-run hit
GetOutcomeHandler-->>Client: "found=true, processing=false"
else tombstone
GetOutcomeHandler->>Redis: GET :latest
GetOutcomeHandler-->>Client: "found=true, note=by-run failed"
else miss + in-queue
GetOutcomeHandler->>Redis: ZRANGE task queue
GetOutcomeHandler-->>Client: "found=false, processing=true"
else miss + not queued
GetOutcomeHandler->>Redis: GET :latest
GetOutcomeHandler-->>Client: "found=true, note=may have expired"
end
Reviews (1): Last reviewed commit: "docs(simulation): operator notes for pro..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+16974
to
+16979
| try { | ||
| await redisCommand(url, token, [ | ||
| 'SET', byRunKey, | ||
| JSON.stringify({ runId, error: 'by_run_write_failed', tombstoneAt: Date.now() }), | ||
| 'EX', String(SIMULATION_OUTCOME_BY_RUN_TTL_SECONDS), | ||
| ]); |
Contributor
There was a problem hiding this comment.
NX to avoid overwriting a by-run outcome that was committed to Redis but whose response timed out. Without NX, a primary write that succeeds server-side but throws a network error will be silently replaced by the tombstone on the retry path, permanently corrupting the by-run key for the 24h TTL window. Callers querying runId=X would then see the tombstone fallback and receive the "Redis transient" note even though the data was written successfully.
Suggested change
| try { | |
| await redisCommand(url, token, [ | |
| 'SET', byRunKey, | |
| JSON.stringify({ runId, error: 'by_run_write_failed', tombstoneAt: Date.now() }), | |
| 'EX', String(SIMULATION_OUTCOME_BY_RUN_TTL_SECONDS), | |
| ]); | |
| try { | |
| await redisCommand(url, token, [ | |
| 'SET', byRunKey, | |
| JSON.stringify({ runId, error: 'by_run_write_failed', tombstoneAt: Date.now() }), | |
| 'EX', String(SIMULATION_OUTCOME_BY_RUN_TTL_SECONDS), | |
| 'NX', | |
| ]); |
Comment on lines
+9
to
+12
| import { SIMULATION_OUTCOME_LATEST_KEY } from '../../../_shared/cache-keys'; | ||
| import { | ||
| SIMULATION_OUTCOME_BY_RUN_KEY_PREFIX, | ||
| } from '../../../_shared/_simulation-queue-constants.mjs'; |
Contributor
There was a problem hiding this comment.
SIMULATION_OUTCOME_LATEST_KEY is now exported from _simulation-queue-constants.mjs (the designated single source of truth for queue constants) and already imported for SIMULATION_OUTCOME_BY_RUN_KEY_PREFIX just below. Keeping a separate import from cache-keys creates two definitions of the same string — a future schema-version bump that updates one without the other would silently break the :latest fallback path without any TypeScript error.
Suggested change
| import { SIMULATION_OUTCOME_LATEST_KEY } from '../../../_shared/cache-keys'; | |
| import { | |
| SIMULATION_OUTCOME_BY_RUN_KEY_PREFIX, | |
| } from '../../../_shared/_simulation-queue-constants.mjs'; | |
| import { | |
| SIMULATION_OUTCOME_LATEST_KEY, | |
| SIMULATION_OUTCOME_BY_RUN_KEY_PREFIX, | |
| } from '../../../_shared/_simulation-queue-constants.mjs'; |
| const authKind = apiKeyHeader | ||
| ? (apiKeyHeader.startsWith('wm_') ? 'user_api_key' : 'enterprise_api_key') | ||
| : (authHeader ? 'clerk_jwt' : 'unknown'); | ||
| console.log(`[TriggerSimulation] queued runId=${pointer.runId} authKind=${authKind} pkgFingerprint=${pointer.pkgFingerprint}`); |
Contributor
There was a problem hiding this comment.
client_version states "Server LOGS this with the success breadcrumb" but the handler marks _req as unused and clientVersion never appears in the success console.log. Either log it as documented or update the proto comment.
Suggested change
| console.log(`[TriggerSimulation] queued runId=${pointer.runId} authKind=${authKind} pkgFingerprint=${pointer.pkgFingerprint}`); | |
| console.log(`[TriggerSimulation] queued runId=${pointer.runId} authKind=${authKind} pkgFingerprint=${pointer.pkgFingerprint} clientVersion=${_req.clientVersion || ''}`); |
Comment on lines
+33
to
+36
| const resp = await fetch(`${url}/get/${encodeURIComponent(key)}`, { | ||
| headers: { Authorization: `Bearer ${token}` }, | ||
| signal: AbortSignal.timeout(REDIS_READ_TIMEOUT_MS), | ||
| }); |
Contributor
There was a problem hiding this comment.
User-Agent on all server-side fetch calls. redisGetThrowing calls the Upstash REST API without one.
Suggested change
| const resp = await fetch(`${url}/get/${encodeURIComponent(key)}`, { | |
| headers: { Authorization: `Bearer ${token}` }, | |
| signal: AbortSignal.timeout(REDIS_READ_TIMEOUT_MS), | |
| }); | |
| const resp = await fetch(`${url}/get/${encodeURIComponent(key)}`, { | |
| headers: { Authorization: `Bearer ${token}`, 'User-Agent': 'worldmonitor-server/1.0' }, | |
| signal: AbortSignal.timeout(REDIS_READ_TIMEOUT_MS), | |
| }); |
Vercel Edge bundler hoisted `server/_shared/_simulation-queue-constants.mjs`
into Edge function bundles (transitively via cache-keys.ts → various Edge
handlers). The shim's `import { createHash } from 'node:crypto'` is not
available in Edge runtime, breaking deploy with:
The Edge Function "api/discord/oauth/callback" is referencing
unsupported modules:
- __vc__ns__/5/server/_shared/_simulation-queue-constants.mjs:
node:crypto
Switch to `globalThis.crypto.subtle.digest('SHA-256', ...)` — the only
crypto API available in BOTH Edge runtime AND Node 19+. The seeder
(scripts/seed-forecasts.mjs) keeps working without a polyfill.
Web Crypto's subtle.digest is async-only, so pkgFingerprint() is now
async. Both call sites (handler's getSimulationPackagePointer in the TS
module + worker's currentFingerprint in the .mjs seeder) already run
inside async functions; updated to await.
Output is byte-identical to the previous Node-crypto version (16-char
sha256 hex prefix), so payloads and parity tests remain stable.
P1 — Tombstone SET now uses NX (scripts/seed-forecasts.mjs). A by-run primary SET that landed server-side but threw a network error on the response would have been overwritten by the tombstone on the catch path, corrupting the real outcome for the 24h TTL window. NX preserves the real write when it actually landed; if Redis is fully down the tombstone also fails and the read path falls back to :latest with the standard "may have expired" note (acceptable). P2 — Single source of truth for SIMULATION_OUTCOME_LATEST_KEY (server/worldmonitor/forecast/v1/get-simulation-outcome.ts). Was importing the key from both _simulation-queue-constants.mjs (the designated shim) AND cache-keys.ts. Drift on a future schema-version bump could silently break the :latest fallback path. Import only from the shim now; cache-keys still has the const (unchanged, kept for non-simulation consumers). P2 — clientVersion now logged per the proto comment promise (server/worldmonitor/forecast/v1/trigger-simulation.ts). The proto field's doc says "Server LOGS this with the success breadcrumb" but the handler was marking req as unused. Echoes the field in the success log line; sanitized (truncated to 64 chars, restricted charset) since it's a free-text caller-provided string. P2 — User-Agent header on Upstash GET in simulation-queue.ts (server/_shared/simulation-queue.ts). AGENTS.md requires User-Agent on all server-side fetch calls. redisGetThrowing was missing it. Tests: 26/26 simulation tests still pass. Typecheck clean.
Owner
Author
|
Addressed all 4 Greptile findings in commit b17e00c:
All 26 simulation tests still pass. Typecheck clean. |
Fix 1 — processing=true responses can be cached for minutes (P1).
get-simulation-outcome.ts returned {processing: true} on the by-run-miss
+ runId-queued branch without marking no-cache. The gateway's `slow`
cache tier would cache the transient state for up to 30 min, so polling
clients would never see the outcome land after the worker finished.
Added markNoCacheResponse on that branch + regression test asserting
X-No-Cache=1 on the response.
Fix 2 — enqueueSimulationTaskForServer + .mjs enqueueSimulationTask
could return queued:true even when ZADD failed (P1, both sides).
Worker discovers tasks EXCLUSIVELY via ZRANGE on the queue ZSET
(scripts/seed-forecasts.mjs listQueuedSimulationTasks at line 17052,
then processNextSimulationTask iterates queuedRunIds). A task key
without a corresponding ZSET member is invisible to the worker until
the 4h task TTL — silent stuck-invisible failure mode.
Previously labeled "best-effort"; ZADD is actually load-bearing. Now:
- Capture the ZADD result (transport throw OR non-numeric reply both
count as failure)
- On failure: issue compensating DEL on the task key so a future trigger
isn't blocked by SET NX and no stale half-state lingers for the 4h TTL
- Return {queued: false, reason: 'redis_error'}
- EXPIRE failure stays non-fatal (TTL hint only — queue member is
durable; logged but doesn't roll back)
Added parity regression test covering both .mjs and .ts: simulates SET
NX success + ZADD non-numeric reply, asserts both implementations
surface redis_error AND issue the compensating DEL.
All 9223 tests pass; typecheck + typecheck:api green.
Owner
Author
|
Addressed both blocking findings in 2bf6144. Fix 1 — Fix 2 — silent ZADD failure → invisible task (both .mjs and .ts) Both
Parity regression test ( 9223/9223 tests pass; typecheck + typecheck:api green. |
6 tasks
koala73
added a commit
that referenced
this pull request
May 18, 2026
…orkers can resolve it (#3811 hotfix) (#3818) * fix(simulation): move queue-constants shim into scripts/ so Railway workers can resolve it #3811 introduced `_simulation-queue-constants.mjs` at `server/_shared/` and had `scripts/seed-forecasts.mjs` import it via `../server/_shared/...`. That works in dev and on Vercel (esbuild inlines), but the three Railway services that run the seeder — `seed-forecasts`, `simulation-worker`, `deep-forecast-worker` — use the nixpacks build with `root_dir=scripts`, which packages only `scripts/` contents into `/app/` in the container. The relative import resolves to `/server/_shared/_simulation-queue- constants.mjs` at runtime — a path that does not exist in the container — and crashes every worker on startup: Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/server/_shared/_simulation-queue-constants.mjs' imported from /app/seed-forecasts.mjs All three Railway services have been in a crash loop since #3811 merged. Fix: move the shim into `scripts/` (where it ships alongside the seeder) and update the 4 importers: - scripts/seed-forecasts.mjs → './_simulation-queue-constants.mjs' - server/_shared/simulation-queue.ts → '../../scripts/_simulation-queue-constants.mjs' - server/worldmonitor/forecast/v1/trigger-simulation.ts → '../../../../scripts/...' - server/worldmonitor/forecast/v1/get-simulation-outcome.ts → '../../../../scripts/...' esbuild bundles the shim inline at Vercel build time, so the cross- directory `../../scripts/...` path is fine on the server side. The `.d.mts` declaration moves alongside so TS module-resolution still picks it up. The shim's own header now documents WHY it lives in scripts/ to prevent the next contributor from "tidying" it back into server/_shared/. Regression test: tests/scripts-railway-nixpacks-no-escape-import.test.mts. BFS-walks `scripts/seed-forecasts.mjs`, `scripts/process-simulation-tasks.mjs`, and `scripts/process-deep-forecast-tasks.mjs` (the three Railway service entry points), following every relative import inside `scripts/`, and fails if any resolved path escapes `scripts/`. Verified that the test correctly fails with the broken import restored, then passes after fix. Verification: - typecheck (main + api): clean - simulation tests: 30/30 pass (27 existing + 3 new) - docker/build-handlers.mjs: completes with 0 failures - shim resolves at runtime from both old (.mjs entry) and esbuild path Distinct from #3811's Vercel-Edge fix commit (Web Crypto port) — that addressed a build-time bundler error; this addresses a runtime container-packaging error that only fires on Railway. No tests in #3811 exercised a Railway-shipped resolution path, which is how this escaped review. The new test closes that coverage gap for these three workers and any future scripts/ entry added to ENTRY_POINTS. * review(pr-3818): address 2 Greptile P2 findings 1. Shim header missing the Railway-packaging rationale (P2). Three importers (server/_shared/simulation-queue.ts and the two forecast/v1 handlers) added "see the header of scripts/_simulation-queue-constants.mjs for the Railway packaging reason" cross-references in their import comments, but the shim itself was renamed with similarity index 100% — its header was the original Edge-bundling note with no mention of why it lives in scripts/ now. My header edit was made locally but never staged before the previous commit, so the published file shipped without the rationale. Fix: rewrite the header to lead with "DO NOT MOVE THIS FILE BACK TO server/_shared/", spell out the nixpacks root_dir=scripts mechanism, name the three affected Railway services, and link the regression test + incident PRs. A future contributor opening the shim to "tidy" it will now find an in-file warning instead of needing to chase importer comments. 2. isBareOrNode dead code in the BFS loop (P2). collectRelativeImports already pre-filters to `./` and `../` specs only, so isBareOrNode(spec) in the loop body always returned false and the `continue` was never reached. Removed both the unused helper and the loop guard. Tests still 3/3 pass; typecheck:api clean.
This was referenced Jul 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #3734 at C1+C2 scope per the v3 plan (tracked as #3798, file at
docs/plans/2026-05-18-003-feat-simulation-trigger-and-runid-filter-plan.md).POST /api/forecast/v1/trigger-simulation— PRO-gated mutation that enqueues a simulation task for the currentSIMULATION_PACKAGE_LATEST_KEYpointer. Mirrorsrun-scenario.tsshape (Pro gate + per-IP rate-limit 10/60s + queue-depth 429 + ApiError for non-200). Returns{ queued, runId, pkgFingerprint, reason }.GET /api/forecast/v1/get-simulation-outcome?runId=X— the runId filter is now actually active (TODO Expand startup ecosystem coverage with VC blogs, regional news, and unicorn tracking #27 was previously docs-only). Five distinct paths: by-run hit, tombstone-aware fallback,processing=truefor in-queue runIds,:latestfallback with expiry note, no-runId existing behavior.:latest(canonical, 60d) AND:by-run:{runId}(24h secondary index) with tombstone-on-failure + Sentry counter. Adds opaquepkgFingerprintto task payload so worker can detect cron-rotation between handler-time and drain-time..mjsseederenqueueSimulationTaskbackportsredis_errorreason code so the auto-trigger path can surface transport failures cleanly (previously swallowed as 'duplicate').What the audit-review process surfaced (and v3 fixed before shipping)
The v1 plan went through two rounds of six-persona document review (50+ findings across coherence, feasibility, product-lens, security, scope-guardian, adversarial). v3 incorporates the consolidated findings. Notable design corrections vs the audit's literal suggestion:
deduct-situationauth patternrun-scenario.tsinsteadrunIdpkgKeyfor drift detectionpkgFingerprint(sha256:16)ENDPOINT_RATE_POLICIEScheckScopedRateLimitexists as migration path.immutableon by-run hitsimmutablecontradicts worker overwrite-on-rerun.already-queuedvsalready-completed-this-cycleexternal codesalready-handledFiles changed (16, +1860 / -50)
proto/worldmonitor/forecast/v1/trigger_simulation.proto,server/_shared/_simulation-queue-constants.mjs+.d.mts,server/_shared/simulation-queue.ts,server/worldmonitor/forecast/v1/trigger-simulation.ts, 4 test fileshandler.ts,service.proto,get_simulation_outcome.proto(+ regenerated client/server/OpenAPI),get-simulation-outcome.ts,seed-forecasts.mjs(enqueueSimulationTask + worker + writeSimulationOutcome),rate-limit.ts,premium-paths.ts,mcp-api-parity.test.mjs(+1 exclusion),server-handlers.test.mjs(note-text update)Test plan
npm run typecheckcleannpm run typecheck:apiclean (.d.mtsrename was required — TS resolves.mjsimports to.d.mts)make generate)pkgFingerprinttri-state predicate, tombstone semantics, and thealready-handledexternal-code collapse)console.log('queued runId=X authKind=Y') — if fewer than 1 call/week from non-team identities, deprecate C1 + C2 together per the Scope Boundaries coupling noteFollow-ups filed
Out of scope (deferred to C3)
queueDepth+estimatedWaitMsin trigger responseGetSimulationRunStatusRPCmcp-api-parity.test.mjsexclusion.