title: "feat: Simulation trigger endpoint + runId filter (#3734 C1+C2, v3)"
type: feat
status: active
date: 2026-05-18
origin: #3734
supersedes: docs/plans/2026-05-18-002-feat-simulation-trigger-and-runid-filter-plan.md

feat: Simulation trigger endpoint + runId filter (#3734 C1+C2, v3)

Overview

Add POST /api/forecast/v1/trigger-simulation so HTTP-only callers (agents, MCP clients, Pro API users) can initiate a simulation run, and make the existing runId query parameter on GET /api/forecast/v1/get-simulation-outcome actually filter to that specific run instead of silently returning the latest.

This is v3 of the plan. v2 underwent a second six-persona document review that surfaced 4 confidence-100 BLOCKERS and ~24 P1/P2 decisions (mostly real, mostly concrete repo-convention gaps). v3 incorporates the consolidated findings. See "What changed from v2" near the bottom.

The change is additive throughout. The Railway worker, auto-trigger from deep-forecast completion, and existing :latest outcome readers all continue to work unchanged.

Problem Frame

External audit issue #3734 flagged that simulation runs can only be triggered by a human operator running node scripts/process-simulation-tasks.mjs --once directly in Railway. The underlying queue infrastructure exists; the missing piece is an HTTP surface.

A compounding bug: getSimulationOutcome accepts a runId query param but ignores it. TODO #27 was marked complete but only as a docs/OpenAPI-description fix — the filter itself never landed.

Demand signal (honest). No Sentry data, no support tickets, no Pro user requests for an on-demand trigger have been observed. The in-repo TODO #21 has been pending since 2026-03-24 — two months of revealed preference says this is not urgent. Motivation for closing:

1. Audit-recurrence prevention — the audit is public; the gap will be re-flagged until closed.
2. Marginal cost is low — the queue infrastructure already exists.
3. Speculative agent demand — plausible but unverified.

Post-merge instrumentation expectation. The 30-day demand experiment requires a success-path counter the plan now ships explicitly (see U4 — adds console.log('[TriggerSimulation] queued', { authKind, ... }) and a Sentry breadcrumb). The criterion: if non-team Pro identities (authKind ∈ { 'user_api_key', 'enterprise_api_key' }) trigger fewer than 1 successful call/week over 30 days, deprecate the endpoint AND wind down by-run retention (C2 is coupled to C1 — see Scope Boundaries note).

Requirements Trace

Trigger Endpoint (R1–R4)

• R1. Pro callers can fire a trigger via POST /api/forecast/v1/trigger-simulation. Free callers receive 403 via ApiError. Successful response is { queued: true, runId, pkgFingerprint, reason: '' }. Idempotency-state responses use a single opaque external code reason: 'already-handled' (server logs retain the distinction between "already-queued" and "already-completed-this-cycle"). Other reason codes: 'no_package'. There is no error field — errors throw ApiError.
• R2. The runId is derived server-side from SIMULATION_PACKAGE_LATEST_KEY. There is no UUID fallback; pointer absent or no runId → reason: 'no_package'.
• R3. Repeat triggers within the same package cycle are no-ops with reason: 'already-handled'. Whether the underlying state was "queue collision" or "outcome already completed" is logged server-side but not exposed externally (defense against the cron-timing oracle — see Risks).
• R4. Per-IP rate-limit at 10 jobs/min (matches run-scenario). Queue-depth backpressure (LLEN > MAX_QUEUE_DEPTH = 100, also matching run-scenario) returns 429 inside the handler via ApiError.

Read-Path Filter & Retention (R5–R6)

• R5. GET /api/forecast/v1/get-simulation-outcome?runId=X returns the outcome for run X when retained. If X is unknown but currently queued or in-flight, the response includes processing: true (uses listQueuedSimulationTasks). If X is unknown and not queued, falls back to :latest with the existing note field. The note text distinguishes "requested runId not found (may have expired beyond 24h retention)" from "by-run lookup failed (Redis transient)" via the by-run-failure tombstone (U5).
• R6. Multi-outcome retention covers a bounded window (24h, see D3). Force-reprocess via --run-id=X overwrites cleanly AND resets TTL (intentional — see D6).

Backwards Compatibility & Documentation (R7–R8)

• R7. The auto-trigger from deep-forecast completion (scripts/seed-forecasts.mjs:16087) call site is unchanged (signature change in enqueueSimulationTask is backward-compatible via default param). Worker behavior is extended — see U3 — to validate pkgFingerprint when present (auto-triggered tasks default to empty fingerprint and skip verification; HTTP-triggered tasks always have a fingerprint).
• R8. TODO Optimize map rendering layers #21 transitions from pending to complete. TODO Expand startup ecosystem coverage with VC blogs, regional news, and unicorn tracking #27's Work Log notes the filter is now actually active. Follow-up issue filed for the audit-reframe docs/marketing work.

Scope Boundaries

• In scope: Trigger endpoint mirroring run-scenario shape (Pro gate + per-IP rate-limit + queue-depth 429), runId-filter end-to-end including a processing state when the run is queued, multi-outcome Redis retention with bounded TTL, race-protection via opaque pkgFingerprint, success-path instrumentation, all tests, proto + handler wiring, race + monitoring guards.

Deferred to Follow-Up Work

• C3 polish: queueDepth + estimatedWaitMs in trigger response, dedicated GetSimulationRunStatus RPC, MCP tool wrappers. C3 wrappers require an explicit policy decision on whether trigger_simulation should be MCP-exposed at all (see Risks).
• C2 coupling note: if R1 deprecation criterion fires (<1 non-team call/week over 30 days), the by-run retention infrastructure (U5) and the runId filter (U6) should ALSO be deprecated together — they exist primarily to make trigger-and-verify work for the agent caller. The :latest reader continues to serve every existing caller. The deprecation runbook is "remove the C2 keys + the C1 endpoint + the proto entries in one PR; the auto-trigger and :latest path are unaffected."
• Per-Pro-identity rate-limit primitive: deferred. checkScopedRateLimit (server/_shared/rate-limit.ts:196-210) exists as the migration path if per-IP ever proves too coarse.
• Cache-Control optimization on by-run hits: deferred. The route is POST for trigger and GET for get-simulation-outcome; both are handled per existing gateway tier logic. See D6.
• Underlying worker / package logic: unchanged.
• "Phase 3 full per-run history": out of scope. Retention is bounded by TTL.
• Audit-reframe docs/marketing: filed as a separate follow-up issue from the PR Checklist.

Context & Research

Relevant Code and Patterns

Primary precedent — run-scenario.ts:

• server/worldmonitor/scenario/v1/run-scenario.ts — Pro-gated mutation, isCallerPremium + ApiError(403), queue-depth backpressure via LLEN, ApiError(429) at capacity, returns { jobId, status, statusUrl }.
• server/worldmonitor/scenario/v1/get-scenario-status.ts — companion poll endpoint (C3 deferred analog for trigger-simulation).
• server/_shared/rate-limit.ts:99 — '/api/scenario/v1/run-scenario': { limit: 10, window: '60 s' } per-IP policy already in production.
• src/shared/premium-paths.ts:31 — run-scenario is in PREMIUM_RPC_PATHS. trigger-simulation must be added there too (see U4).
• NOT in RPC_CACHE_TIER — POST routes don't go through GET-only cache code; route-cache-tier.test.mjs:40-77 parity test would trip on a stale POST entry. v3 omits this map edit.

Read path to extend:

• server/worldmonitor/forecast/v1/get-simulation-outcome.ts — extends with by-run lookup + processing-state probe + fallback to :latest.

Queue infrastructure (lightly modified for fingerprint capture):

• scripts/seed-forecasts.mjs:16954 — enqueueSimulationTask(runId) becomes enqueueSimulationTask(runId, pkgFingerprint = ''). Backward-compatible default.
• scripts/seed-forecasts.mjs:16951-16999 — queue helpers.
• scripts/seed-forecasts.mjs:17018-17146 — processNextSimulationTask. Predicate becomes if (task.pkgFingerprint && task.pkgFingerprint !== currentFingerprint(pkgPointer)) to avoid spurious warnings on in-flight pre-upgrade tasks.
• scripts/seed-forecasts.mjs:16934 — writeSimulationOutcome extended to write the by-run key + a tombstone on by-run failure (U5).
• scripts/seed-forecasts.mjs:16087 — auto-trigger after writeSimulationPackage. Continues to call enqueueSimulationTask(runId) (no fingerprint), worker skips verification on empty.
• scripts/seed-forecasts.mjs:51 — VALID_RUN_ID_RE = /^\d{13,}-[a-z0-9-]{1,64}$/i.
• scripts/seed-forecasts.mjs redis helper at lines ~593-605 — redisCommand throws on HTTP non-OK (not "swallows as duplicate" as v2 incorrectly claimed). The .mjs 'duplicate' return only fires when Upstash returns 200 + nil body (NX collision). This is the correct reading. U3 backports a redis_error reason to the .mjs by wrapping the SET NX call in try/catch so the auto-trigger path can surface transport failures cleanly.

Auth surface:

• server/_shared/premium-check.ts:17 — isCallerPremium, single entry point.
• src/shared/premium-paths.ts — PREMIUM_RPC_PATHS set. New entry: /api/forecast/v1/trigger-simulation.
• server/gateway.ts:780-844 — legacy Pro-bearer gate (fires before rate-limit when path is in PREMIUM_RPC_PATHS).
• server/gateway.ts:886-898 — per-IP ENDPOINT_RATE_POLICIES rate-limit. Fires AFTER Pro-bearer gate. Both gates fire before the handler.

Identity for the success-path counter:

• server/_shared/usage-identity.ts:42 — buildUsageIdentity returns auth_kind ∈ {'clerk_jwt', 'user_api_key', 'enterprise_api_key', 'widget_key', 'anon'}. U4 success log uses this for the team-vs-external classification (enterprise_api_key with customer_id in the team allowlist → team; everyone else → external).

Cross-tree .mjs import (constraint):

Existing TS-imports-.mjs patterns in the codebase are sibling-only within server/ (e.g., server/worldmonitor/news/v1/_shared.ts:30, server/worldmonitor/supply-chain/v1/get-chokepoint-status.ts:19). There are NO existing TS-imports-from-scripts/ .mjs patterns. v3 honors this by placing the shim at server/_shared/_simulation-queue-constants.mjs (sibling-pattern compliant) and having the scripts/seed-forecasts.mjs seeder import from ../server/_shared/_simulation-queue-constants.mjs (this direction — scripts/ → server/ — has precedent through the JSON-import-from-scripts pattern at 5+ existing sites).

Test harness:

• tests/forecast-*.test.mjs / *.test.mts — existing patterns.
• package.json:51 — "test:data": "tsx --test tests/*.test.mjs tests/*.test.mts". Verification commands MUST use tsx --test when the test imports any .ts module; pure-.mjs tests can use node --test.

Institutional Learnings

• MEMORY.md feedback_fresh_branch_off_origin_main_not_worktree_head — cut the feature branch off origin/main via a fresh worktree.
• Cache-prefix bumps — by-run keys are additive, no migration needed.
• verify-sentry-filter-fires-against-actual-event-shape — test guards against actual production payload shape; applies here to the runId-filter test in U6 and the worker idempotency / fingerprint-verification in U3.

External References

• None. Fully internal to existing project conventions.

Key Technical Decisions

D1. Server-derived runId; no UUID fallback

TriggerSimulationRequest has no run_id field. Handler reads SIMULATION_PACKAGE_LATEST_KEY and uses its runId. If absent or missing runId, response is { queued: false, runId: '', pkgFingerprint: '', reason: 'no_package' }. No crypto.randomUUID() fallback — UUIDs fail validateRunId (^\d{13,}-...).

D2. Rate-limit via existing per-IP ENDPOINT_RATE_POLICIES

Add '/api/forecast/v1/trigger-simulation': { limit: 10, window: '60 s' } to server/_shared/rate-limit.ts. Identical to run-scenario's policy. No new primitive. checkScopedRateLimit exists as the migration path if per-identity ever needed.

D3. Outcome retention TTL: 24 hours; tied to C1 deprecation

New key family: forecast:simulation-outcome:by-run:{runId}. TTL = 24h. The forecast cron cadence (~hourly per general repo convention; not directly cited from a config file in this repo's tree — see Risks) implies ~24 outcomes retained at steady state. The instrumentation hook ("if note: 'requested runId not found...' rate >5% post-merge, bump TTL") remains. If the C1 endpoint is deprecated per the demand-signal criterion, the C2 retention should be deprecated together — see Scope Boundaries.

D4. .mjs ↔ .ts boundary: shim at server/_shared/ (sibling-pattern compliant)

Add server/_shared/_simulation-queue-constants.mjs exporting the queue key constants + TTL + the pkgFingerprint(pkgKey) helper (see D7). Both the seeder and a new TS module (server/_shared/simulation-queue.ts) import constants from this shim.

Important reframe (correcting v2): The shim does NOT solve a cross-language import problem in a way that's invisible to readers. It simply lets each caller use the language-native form: the seeder's .mjs imports another .mjs natively; the TS module imports the .mjs via the existing // @ts-expect-error / .d.ts pattern already used elsewhere in server/_shared/ (e.g., _simulation-queue-constants.d.ts with declared const exports). The "shim is the chosen path" framing means we commit to this bidirectional native-import shape, not that we discovered some magic cross-language import.

Sibling-pattern compliance: shim lives under server/_shared/, matches existing TS-imports-.mjs precedents at server/worldmonitor/news/v1/_shared.ts:30 and similar sites. The seeder at scripts/ imports from server/_shared/ — this direction has 5+ precedents through JSON-import-from-scripts.

D5. Idempotency response taxonomy + race protection via opaque fingerprint

Capture pkgFingerprint = sha256(pkgKey).slice(0, 16) at enqueue time inside the task payload (NOT the raw R2 path — see D7). Worker verifies; mismatch is logged server-side AND (per Sec9 fix in v3) is also tagged into the worker's stored outcome JSON as a _meta.packageRotated: true field (NOT exposed through the user-facing note; see D8 for the response-shape principle).

Why collapse already-queued + already-completed-this-cycle into one external code: A Pro caller polling can observe the transition already-queued → already-completed-this-cycle and measure simulation duration / cron rotation timing as a side channel. Collapsing to already-handled removes the oracle without losing internal diagnosability (server logs keep the distinction). Closes Sec4 from rounds 1+2.

Pre-check semantics: the handler's pre-enqueue check (reads :latest outcome and compares outcome.runId === pkgPointer.runId) is a fast-path optimization only — it avoids consuming a queue slot when the cycle is provably done. The authoritative concurrency primitive is SET NX inside enqueueSimulationTaskForServer. Both layers must coexist; do not remove either.

D6. Cache-Control: rely on gateway tier; no per-handler cache headers

• trigger-simulation is POST → RPC_CACHE_TIER (GET-only logic) does not apply. No entry added (would trip route-cache-tier.test.mjs parity check).
• get-simulation-outcome is GET → existing 'slow' tier (public, max-age=300, s-maxage=1800) is honored by the gateway unchanged. No per-handler cache helper added.

Acknowledged trade-off (Adv-F3 / round 2): the slow tier's 30-min CDN cache + the no-NX overwrite on force-reprocess means an operator who manually runs --run-id=X between minutes 0-30 of a cycle may see CDN-stale outcome for up to 30 min. Same shape as immutable but smaller window. Accepted because force-reprocess is rare (operator/debugging path); documented in Risks.

D7. pkgFingerprint is opaque (not raw R2 path)

pkgFingerprint = sha256(pkgKey).slice(0, 16) (16 hex chars). Stored in the task payload, returned in the trigger response, returned in by-run outcome reads. Not the raw R2 path. Reasons:

1. Security (Sec7 round 2): raw pkgKey is seed-data/forecast-traces/{year}/{month}/{day}/{runId}/simulation-package.json. Returning it discloses the R2 bucket layout to every Pro caller — a latent surface if R2 access policy ever relaxes. An opaque fingerprint reveals nothing about layout.
2. Sufficiency for drift detection (Adv-F1 round 2): a caller comparing the fingerprint returned at trigger time to the fingerprint inside the by-run outcome payload can detect cron rotation. That is the actual capability the field needs to provide; it does not need the raw path.
3. Stable proto type: the field is string pkg_fingerprint = 3 — stays flat string (default ""), no nullable wrapping needed (addresses Feas-F8 round 2).

pkgFingerprint(pkgKey) lives in the shim (D4) so the worker, the handler, and any future caller use the same implementation. The worker compares task.pkgFingerprint against pkgFingerprint(currentPointer.pkgKey).

D8. Outcome-response shape: server-side metadata stays in _meta, user-facing fields are explicit

When the worker tags an outcome with _meta.packageRotated: true (D5), the user-facing read endpoint (get-simulation-outcome) does not surface _meta through the response proto. Server-side tooling that reads from Redis directly can use _meta; HTTP callers get a clean response. This addresses Sec9 round 2 (ambiguity between log signal and response field) by making the principle explicit: _meta is internal, user-facing fields are explicit.

The existing note field continues to carry user-facing messages: "requested runId not found (may have expired beyond 24h retention)" for true expiry, "by-run lookup failed (Redis transient); returned latest" for tombstone hits (U5).

D9. By-run write failures are logged AND tombstoned

When the worker's by-run SET fails (U5), it: (a) logs [Simulation] by-run SET failed runId=X, (b) increments a Sentry counter forecast.simulation.by_run_write_failed, (c) writes a tombstone payload ({ runId, error: 'by_run_write_failed', tombstoneAt: Date.now() }) under the same by-run key. The get-simulation-outcome handler detects the tombstone and surfaces a distinct note text ("by-run lookup failed (Redis transient); returned latest"). Addresses Adv-F6 round 2 — monitoring no longer conflates "expired" with "write-failed."

D10. :latest is canonical; :by-run is a secondary queryable index

Stated explicitly so an implementer doesn't invert the dependency. Worker writes :latest first (await, no swallow); by-run write follows with .catch() per D9. Addresses Coh-C10 round 2.

Open Questions

Resolved During Planning

• runId source? Server-derived; no UUID fallback (D1).
• Rate-limit shape? Per-IP via ENDPOINT_RATE_POLICIES (D2).
• TTL? 24h; coupled to C1 deprecation (D3).
• .mjs/.ts bridge? Shim at server/_shared/_simulation-queue-constants.mjs (D4).
• HTTP status mechanism? ApiError for non-200s; typed message for success/idempotency (D5).
• Cron-rotation race? Capture pkgFingerprint in task payload; worker verifies (D5 + D7).
• pkgKey security? Returned as opaque pkgFingerprint (D7).
• Idempotency external taxonomy? Single already-handled external code; internal logs retain distinction (D5).
• By-run failure semantics? Log + Sentry counter + tombstone with distinct note text (D9).
• Cache headers? No new logic; gateway tier honored (D6).
• Canonical key? :latest; :by-run is secondary (D10).
• What happens to C2 if C1 is deprecated? Wound down together (Scope Boundaries note).

Deferred to Implementation

• Exact Sentry breadcrumb shape for the success-path counter in U4 — pick a key namespace that doesn't collide with existing forecast Sentry tags. Implementer's call.
• Whether to expose pkgFingerprint field in the get-simulation-outcome response — currently planned NO (read endpoint returns only what's in the outcome payload, no fingerprint). If callers need to compare trigger-time fingerprint to read-time fingerprint, add the field in a follow-up.

High-Level Technical Design

Directional guidance for review, not implementation specification.

┌────────────────────────────────────────────────────────────────────────┐
│ GATEWAY ORDER (verified against server/gateway.ts:780-898)             │
└────────────────────────────────────────────────────────────────────────┘

  Pro caller
       │  POST /api/forecast/v1/trigger-simulation
       ▼
  ┌──────────────────┐    ┌──────────────────────┐    ┌──────────────┐
  │ Pro-bearer gate  │───▶│ Per-IP rate-limit    │───▶│ Handler      │
  │ (gateway:780+    │    │ (gateway:886+        │    │ (forecast/   │
  │  uses PREMIUM_   │    │  uses ENDPOINT_      │    │  trigger-    │
  │  RPC_PATHS)      │    │  RATE_POLICIES)      │    │  simulation) │
  └──────────────────┘    └──────────────────────┘    └──────────────┘
       │                       │                            │
   401 if not Pro          429 if >10/60s              isCallerPremium
                                                       defense-in-depth
                                                       (also 403)

┌────────────────────────────────────────────────────────────────────────┐
│ HANDLER BODY                                                           │
└────────────────────────────────────────────────────────────────────────┘

       ▼
  ┌─────────────────────────────────┐
  │ isCallerPremium → ApiError(403) │  (gateway already gated; this is
  └────────────────┬────────────────┘   defense-in-depth)
                   ▼
  ┌─────────────────────────────────┐
  │ getQueueDepth → ApiError(429)   │
  │   if > MAX_QUEUE_DEPTH=100      │
  └────────────────┬────────────────┘
                   ▼
  ┌─────────────────────────────────┐
  │ getSimulationPackagePointer     │
  │  if null/no runId → no_package  │
  │  else compute pkgFingerprint    │
  └────────────────┬────────────────┘
                   ▼
  ┌─────────────────────────────────┐
  │ getSimulationOutcomeLatest      │
  │  if outcome.runId === pointer.  │
  │  runId → log already-completed- │
  │  this-cycle, return external    │
  │  reason='already-handled'       │
  └────────────────┬────────────────┘
                   ▼
  ┌─────────────────────────────────┐
  │ enqueueSimulationTaskForServer  │
  │  (runId, pkgFingerprint)        │
  │  on duplicate → log already-    │
  │   queued, external='already-    │
  │   handled'                      │
  │  on success → log "queued       │
  │   runId=X authKind=Y" + Sentry  │
  │   breadcrumb for demand counter │
  └────────────────┬────────────────┘
                   ▼
  ┌─────────────────────────────────┐
  │ Response: { queued, runId,      │
  │   pkgFingerprint, reason }      │
  └─────────────────────────────────┘

┌────────────────────────────────────────────────────────────────────────┐
│ READ PATH (modified)                                                   │
└────────────────────────────────────────────────────────────────────────┘

  Any caller
       │  GET /api/forecast/v1/get-simulation-outcome?runId=X
       ▼
  ┌─────────────────────────────────┐
  │ if req.runId:                   │
  │   GET :by-run:{runId}           │
  │      ↓ hit                      │
  │   if payload.error===           │
  │     'by_run_write_failed'       │
  │      → tombstone: note='by-run  │
  │        lookup failed (Redis     │
  │        transient); returned     │
  │        latest', fall back       │
  │      else → return outcome      │
  │      ↓ miss                     │
  │   if listQueuedSimulationTasks  │
  │     includes runId → return     │
  │     { processing: true, … }     │
  │      ↓ not queued               │
  │   GET :latest, set note='req    │
  │     runId not found (may have   │
  │     expired beyond 24h)'        │
  │     if outcome.runId !== runId  │
  │ else:                           │
  │   GET :latest (existing path)   │
  └─────────────────────────────────┘

Implementation Units

• U1. Add trigger-simulation proto + service entry

Goal: Define the RPC + emit generated server/client/OpenAPI.

Requirements: R1, R2, R3, R4 (response shape), R8

Dependencies: None

Files:

• Create: proto/worldmonitor/forecast/v1/trigger_simulation.proto
• Modify: proto/worldmonitor/forecast/v1/service.proto
• Generated: src/generated/server/worldmonitor/forecast/v1/service_server.ts, client, OpenAPI yaml

Approach:

• TriggerSimulationRequest: empty body. No run_id.
• TriggerSimulationResponse: bool queued = 1; string run_id = 2; string pkg_fingerprint = 3; string reason = 4;. No error field — errors propagate as HTTP status via ApiError. reason is one of {'', 'no_package', 'already-handled'} (external-facing; internal logs distinguish further).
• service.proto: path: "/trigger-simulation", method: HTTP_METHOD_POST.
• Add proto comment on pkg_fingerprint: "Opaque fingerprint of the simulation package input. Stable identifier for drift detection across trigger/read calls. Do NOT decode."

Patterns to follow:

• proto/worldmonitor/scenario/v1/run_scenario.proto — closest RPC analog
• proto/worldmonitor/forecast/v1/get_simulation_package.proto — message shape

Test scenarios:

• Test expectation: none — proto + generated code. Verification = codegen runs clean.

Verification:

• make generate runs without error
• Generated ForecastServiceHandler includes triggerSimulation
• Pre-push proto gate passes

• U2. Add shared queue constants + fingerprint helper shim (.mjs)

Goal: Single source of truth for constants and the fingerprint computation. Both seeder and TS handler import natively.

Requirements: R1, R6, R7

Dependencies: None

Files:

• Create: server/_shared/_simulation-queue-constants.mjs — exports: SIMULATION_TASK_KEY_PREFIX, SIMULATION_TASK_QUEUE_KEY, SIMULATION_TASK_TTL_SECONDS, SIMULATION_OUTCOME_BY_RUN_KEY_PREFIX = 'forecast:simulation-outcome:by-run', SIMULATION_OUTCOME_BY_RUN_TTL_SECONDS = 24 * 60 * 60, MAX_QUEUE_DEPTH = 100, pkgFingerprint(pkgKey) (returns first 16 hex chars of sha256). Pure-ESM, no Node-specific imports beyond crypto.
• Create: server/_shared/_simulation-queue-constants.d.ts — minimal declaration file so TS callers don't need @ts-expect-error. Declares all consts + function signature.
• Modify: scripts/seed-forecasts.mjs lines ~40-55 — replace local constants with import { ... } from '../server/_shared/_simulation-queue-constants.mjs'.

Approach:

• Sibling pattern: TS modules in server/_shared/ already import .js/.mjs neighbors via // @ts-expect-error or via .d.ts. We use the .d.ts route since it's cleaner for a 5-line const declaration.
• The seeder (in scripts/) imports from server/_shared/ — this direction matches the existing JSON-import-from-scripts pattern (5+ sites).

Patterns to follow:

• server/_shared/feelgood-classifier.js + .d.ts for the .d.ts shim style
• Existing .mjs modules under scripts/_*.mjs for the seeder-side import style

Test scenarios:

• Test expectation: none directly — the parity test in U3 exercises both consumers.

Verification:

• npm run typecheck passes (the .d.ts resolves)
• node scripts/process-simulation-tasks.mjs --once (manual smoke) starts cleanly
• npx tsx -e "import * as c from './server/_shared/_simulation-queue-constants.mjs'; console.log(Object.keys(c))" prints all expected exports including pkgFingerprint

• U3. TS server-side enqueueSimulationTaskForServer + queue-state helpers + parity test + .mjs redis_error backport

Goal: TS module the trigger handler calls. Same Redis schema as the seeder. Worker accepts a pkgFingerprint task field. The .mjs side gains a 'redis_error' reason code so the auto-trigger path can surface transport failures.

Requirements: R1, R3, R4 (queue-depth helper), R7

Dependencies: U2

Files:

• Create: server/_shared/simulation-queue.ts — exports: validateRunId(runId), enqueueSimulationTaskForServer(runId, pkgFingerprint) returning { queued, reason: '' | 'missing_run_id' | 'invalid_run_id_format' | 'duplicate' | 'redis_error' }, getQueueDepth(), getSimulationPackagePointer() (also computes pkgFingerprint), getSimulationOutcomeLatest(), listProcessingRunIds() (wraps LRANGE / ZRANGE to get currently-queued runIds — used by U6 for the processing state).
• Modify: scripts/seed-forecasts.mjs lines ~16954-16967 — update enqueueSimulationTask(runId, pkgFingerprint = ''). Wrap the SET NX call in try/catch; on catch return { queued: false, reason: 'redis_error' } (new — addresses Adv-F4 / SG-2 round 2 by surfacing transport errors in the auto-trigger path).
• Modify: scripts/seed-forecasts.mjs processNextSimulationTask lines ~17043-17051 — predicate becomes if (task.pkgFingerprint && task.pkgFingerprint !== currentFingerprint). The pre-upgrade in-flight tasks (no pkgFingerprint field) silently skip verification. Tag the outcome JSON with _meta.packageRotated: true on mismatch (NOT exposed in user-facing response — D8).
• Test: tests/simulation-queue-parity.test.mts

Approach:

• TS implementation: catches thrown Redis exceptions, classifies as 'redis_error'. The pre-existing seeder threw on transport error and propagated; the v3 backport makes it return 'redis_error' consistently. The parity test enforces identical behavior on identical inputs across both implementations — no asymmetry to pin.
• Worker pkgFingerprint check uses task.pkgFingerprint && task.pkgFingerprint !== currentFingerprint — the truthiness guard handles both pre-upgrade tasks (no field) and auto-trigger empty defaults.

Patterns to follow:

• server/worldmonitor/scenario/v1/run-scenario.ts for ApiError + queue-LLEN pattern
• server/_shared/redis.ts for Redis call style

Test scenarios:

• Happy path: enqueueSimulationTaskForServer('1734567890123-abc', 'a1b2c3d4e5f6g7h8') → { queued: true, reason: '' }; payload has both runId and pkgFingerprint.
• Edge case: empty runId → { queued: false, reason: 'missing_run_id' }.
• Edge case: malformed runId → { queued: false, reason: 'invalid_run_id_format' }.
• Edge case: enqueue same runId twice → second returns 'duplicate'.
• Error path: Redis throws on SET → 'redis_error'. Both implementations classify identically (the .mjs backport).
• Integration: PARITY test — load both implementations in same test process via tsx. For every scenario above, assert identical response shapes, command sequences, args, TTLs. NO documented asymmetry. The redis_error path uses a Redis mock that throws.
• Integration: tri-state pkgFingerprint worker test (addresses Adv-F2 round 2) — three sub-cases:
  • task.pkgFingerprint = '' (auto-trigger / pre-upgrade): worker skips verification, uses latest package, no _meta.packageRotated.
  • task.pkgFingerprint = currentFingerprint (matching): worker processes normally, no _meta.packageRotated.
  • task.pkgFingerprint = 'mismatch-fingerprint-xx': worker still processes using latest package (preserves existing fallback-to-latest behavior at line 17049-17051), logs package_rotated, tags _meta.packageRotated: true in outcome JSON.
• Integration: deployment-rollout window — task enqueued by OLD seeder with no pkgFingerprint field, drained by NEW worker → treated identically to empty fingerprint, no spurious package_rotated log.

Verification:

• npx tsx --test tests/simulation-queue-parity.test.mts green
• npm run typecheck passes
• Seeder still runs: node scripts/process-simulation-tasks.mjs --once

• U4. Implement triggerSimulation handler + wire into service + PREMIUM_RPC_PATHS + success-path log

Goal: The HTTP endpoint mirroring run-scenario.ts. Pro gate via isCallerPremium (defense-in-depth — gateway already gates via PREMIUM_RPC_PATHS); per-IP rate-limit via ENDPOINT_RATE_POLICIES; queue-depth 429; server-derived runId with pkgFingerprint; idempotency taxonomy; success-path log + Sentry breadcrumb for the demand counter.

Requirements: R1, R2, R3, R4, R8

Dependencies: U1, U2, U3

Files:

• Create: server/worldmonitor/forecast/v1/trigger-simulation.ts
• Modify: server/worldmonitor/forecast/v1/handler.ts (one-line wire-up)
• Modify: src/shared/premium-paths.ts — add '/api/forecast/v1/trigger-simulation' to PREMIUM_RPC_PATHS. Mandatory — matches run-scenario precedent at line 31; gateway's legacy Pro-bearer gate (server/gateway.ts:780-844) requires this to fire BEFORE the per-IP rate-limit.
• Modify: server/_shared/rate-limit.ts ENDPOINT_RATE_POLICIES — add '/api/forecast/v1/trigger-simulation': { limit: 10, window: '60 s' }.
• Do NOT modify server/gateway.ts RPC_CACHE_TIER — POST routes don't go through the cache-tier logic; a stale entry would trip tests/route-cache-tier.test.mjs:40-77 (addresses Feas-F1 round 2).
• Test: tests/forecast-trigger-simulation.test.mts

Approach:

• Handler body order:
  1. isCallerPremium(ctx.request) → if false, throw new ApiError(403, 'Pro subscription required', ''). (Gateway already gated, but defense-in-depth.)
  2. getQueueDepth() → if > MAX_QUEUE_DEPTH (100), throw new ApiError(429, 'Simulation queue at capacity, please try again later', '').
  3. getSimulationPackagePointer() → if null or no runId, log + return no_package.
  4. getSimulationOutcomeLatest() → if outcome.runId === pointer.runId, log already-completed-this-cycle (internal code), return external reason: 'already-handled'.
  5. enqueueSimulationTaskForServer(pointer.runId, pointer.pkgFingerprint):
     • 'duplicate' → log already-queued (internal), return external reason: 'already-handled'.
     • 'redis_error' → throw new ApiError(503, 'Simulation queue unavailable', '').
     • 'invalid_run_id_format' → throw new ApiError(500, 'Internal: invalid runId from package pointer', ''). Bug-detector.
     • { queued: true } → console.log('[TriggerSimulation] queued', { runId, authKind: identity.auth_kind }) + Sentry breadcrumb (forecast.trigger_simulation.queued); return { queued: true, runId, pkgFingerprint, reason: '' }.
• All success/idempotency paths apply markNoCacheResponse(ctx.request).
• The success log + Sentry breadcrumb addresses SG-4 round 2 — provides the measurement mechanism for the 30-day demand experiment. Identity classification: auth_kind ∈ {'user_api_key', 'enterprise_api_key'} AND customer_id not in team-allowlist → "non-team external" for the deprecation criterion.

Patterns to follow:

• server/worldmonitor/scenario/v1/run-scenario.ts end-to-end (closest precedent)
• server/worldmonitor/forecast/v1/get-simulation-outcome.ts for error envelope + markNoCacheResponse

Test scenarios:

• Happy path: Pro caller, package present, no existing outcome → { queued: true, runId, pkgFingerprint, reason: '' }. Success log emitted. Sentry breadcrumb captured.
• Happy path (idempotency): same caller calls twice → first queued: true, second queued: false, reason: 'already-handled'. Server logs show already-queued internal code, not exposed.
• Happy path (post-completion): caller fires after worker finishes → queued: false, reason: 'already-handled'. Server logs show already-completed-this-cycle internal code.
• Error path: free caller → ApiError(403), HTTP 403, log 403-no-premium.
• Error path: queue at capacity → ApiError(429), HTTP 429.
• Error path: Redis throws on pointer read → ApiError(503), HTTP 503.
• Edge case: no package pointer → { reason: 'no_package' }, HTTP 200.
• Edge case: package pointer exists but runId field empty → same as no_package.
• Edge case: gateway-side per-IP rate-limit hit (11th call in a minute) → 429 returned by gateway BEFORE handler runs. Test scenario asserts no handler log fires.
• Test that external response NEVER contains internal codes ('already-queued' and 'already-completed-this-cycle' must not appear in the response body — they are server-log-only).
• Integration: full request through createForecastServiceRoutes exercises auto-mount + PREMIUM_RPC_PATHS gating.

Verification:

• npx tsx --test tests/forecast-trigger-simulation.test.mts green
• npm run typecheck passes
• npm run lint:api-contract passes
• node --test tests/route-cache-tier.test.mjs green (verifies the absence of an RPC_CACHE_TIER entry for the POST route)
• Manual smoke (staging): curl -X POST -H 'X-Api-Key: ...' <base>/api/forecast/v1/trigger-simulation returns expected JSON

• U5. Worker writes outcome to :latest + :by-run:{runId} (with tombstone-on-failure, Sentry counter)

Goal: Per-run outcomes queryable for 24h; failures don't silently corrupt the user-facing read path.

Requirements: R5, R6, R7

Dependencies: U2

Files:

• Modify: scripts/seed-forecasts.mjs around line 16934 (writeSimulationOutcome body)
• Test: tests/simulation-outcome-by-run-write.test.mjs

Approach:

• Existing :latest SET preserved as-is (canonical write, D10).
• After :latest SET succeeds, attempt by-run SET:

  await redisCommand(url, token, [
    'SET', `${SIMULATION_OUTCOME_BY_RUN_KEY_PREFIX}:${runId}`,
    JSON.stringify(outcomePayload),
    'EX', String(SIMULATION_OUTCOME_BY_RUN_TTL_SECONDS),
  ]).catch(async (err) => {
    console.warn(`[Simulation] by-run SET failed for ${runId}: ${err.message}`);
    // Sentry counter (defense-in-depth — distinguishes write-failed from expired in monitoring)
    captureCounter('forecast.simulation.by_run_write_failed', { runId });
    // Tombstone (defense-in-depth — read path can distinguish from true expiry)
    await redisCommand(url, token, [
      'SET', `${SIMULATION_OUTCOME_BY_RUN_KEY_PREFIX}:${runId}`,
      JSON.stringify({ runId, error: 'by_run_write_failed', tombstoneAt: Date.now() }),
      'EX', String(SIMULATION_OUTCOME_BY_RUN_TTL_SECONDS),
    ]).catch(() => {});  // best-effort tombstone — if Redis is fully down, no recovery
  });
  

• No NX flag. Re-runs overwrite AND reset TTL (D6 acknowledgement: intentional, surface to ops).

Patterns to follow:

• Existing SET for :latest at scripts/seed-forecasts.mjs:16934

Test scenarios:

• Happy path: both keys written with identical payloads; by-run TTL 86400 ± skew.
• Error path: by-run SET throws → :latest succeeded, tombstone written, Sentry counter incremented; worker returns normally.
• Error path: both by-run SET and tombstone SET fail → log only, no further writes; :latest is still queryable.
• Edge case: re-run with --run-id=X 23h after initial completion → by-run TTL extends another 24h (intentional, D6).
• Integration: full worker run via processNextSimulationTask exercises the by-run write through both HTTP-trigger and auto-trigger paths → asserts R7 (auto-trigger unchanged) holds even with the new SET.

Verification:

• node --test tests/simulation-outcome-by-run-write.test.mjs green
• Manual smoke: after one cron cycle post-deploy, redis-cli KEYS 'forecast:simulation-outcome:by-run:*' | wc -l grows over 24h

• U6. getSimulationOutcome honors runId filter + processing state + tombstone-aware fallback

Goal: Read endpoint distinguishes hit / processing / tombstone / true-expiry / no-runId-supplied states with appropriate user-facing messaging.

Requirements: R5, R8

Dependencies: U2, U3 (listProcessingRunIds), U5

Files:

• Modify: server/worldmonitor/forecast/v1/get-simulation-outcome.ts
• Modify: proto/worldmonitor/forecast/v1/get_simulation_outcome.proto — add bool processing = 10 field; rewrite proto comment on run_id (filter is now active).
• Test: tests/forecast-get-simulation-outcome.test.mts
• Generated: re-run codegen

Approach:

• If req.runId is non-empty:
  1. Read ${SIMULATION_OUTCOME_BY_RUN_KEY_PREFIX}:${req.runId}.
  2. Tombstone hit (payload .error === 'by_run_write_failed'): treat as miss; fall through to step 4 with note text "by-run lookup failed (Redis transient); returned latest" (distinguishes from true expiry).
  3. Real hit: return the outcome with note: ''. (Cache-Control is whatever the gateway tier sets — slow.)
  4. Miss: call listProcessingRunIds(). If req.runId is in queue or in-flight → return { found: false, processing: true, runId: req.runId, ... }.
  5. Miss + not queued: fall back to :latest (existing path); set note to "requested runId not found (may have expired beyond 24h retention)" if outcome.runId !== req.runId.
• If req.runId is empty: existing :latest path unchanged.
• Tombstone-path note: "by-run lookup failed (Redis transient); returned latest available outcome instead". Addresses Adv-F6 round 2 + D9.
• Processing state addresses PL-2 round 2 — prevents the queued-vs-expired indistinguishability from contaminating the 30-day demand signal.

Patterns to follow:

• Current server/worldmonitor/forecast/v1/get-simulation-outcome.ts

Test scenarios:

• Happy path: by-run hit → return outcome, note: '', processing: false.
• Happy path: no runId supplied → existing path, no behavior change.
• Tombstone path: by-run hit with error: 'by_run_write_failed' payload → fall through, note shows tombstone message, processing: false.
• Processing path: runId not in by-run, found in queue → { found: false, processing: true, runId, ... }.
• Edge case: runId in by-run miss, latest matches → return latest, note: '', processing: false.
• Edge case: runId not found anywhere, latest exists with different runId → return latest, note = expiry message, processing: false.
• Edge case: runId not found anywhere, latest empty → NOT_FOUND with markNoCacheResponse, processing: false.
• Error path: Redis fails on by-run read → fall through to :latest; existing error path unchanged.
• Integration: full request via createForecastServiceRoutes with the new processing field correctly serialized.

Verification:

• npx tsx --test tests/forecast-get-simulation-outcome.test.mts green
• Codegen runs clean
• Curl smoke covers all 5 paths

PR Checklist (not implementation units)

• Rename todos/021-pending-p1-simulation-no-http-trigger-endpoint.md → 021-complete-...md; frontmatter status flip; Work Log entry with PR number.
• Update todos/027-complete-p2-simulation-runid-filter-noop-undocumented.md Work Log: filter is now actually active.
• File a follow-up GitHub issue for the audit-reframe docs/marketing work: surface "simulations auto-run hourly via cron" in OpenAPI description, Pro marketing copy, and the trigger_simulation MCP tool description (if C3 lands). Link from PR description.
• Cite the cron cadence used in the D3 retention math (find the cron config or remove the claim — Feas-F6 round 2).
• Note in operator runbook: process-simulation-tasks.mjs --run-id=X requires X to match the runId regex. Invalid IDs are silently skipped — check worker logs for "Skipping invalid runId format" (addresses Adv-F5 round 2 / pre-existing operator footgun).

System-Wide Impact

• Interaction graph:
  • New POST endpoint auto-mounts via createForecastServiceRoutes — no Vercel handler file.
  • Gateway: Pro-bearer gate (PREMIUM_RPC_PATHS) → per-IP rate-limit (ENDPOINT_RATE_POLICIES) → handler. Both gates fire before the handler; the handler's isCallerPremium is defense-in-depth.
  • Worker write path now writes 2 keys (:latest + :by-run) with tombstone-on-failure. Plus Sentry counter for write failures.
  • Read handler now consults 3 sources in order (by-run, processing queue, latest).
• Error propagation:
  • All non-200s on trigger throw ApiError(status, ...) matching run-scenario.ts.
  • Worker by-run write failures: log + counter + tombstone, do not block :latest.
  • Read handler: tombstone routes to fallback with distinct note; processing-state routes to dedicated response.
• State lifecycle risks:
  • Cron rotation race (D5 + D7): pkgFingerprint captured at enqueue, verified at worker drain, mismatch logged + tagged into _meta (NOT user-facing per D8). Pre-upgrade in-flight tasks with no fingerprint field skip verification cleanly (Feas-F4 round 2).
  • Tombstone consistency: :latest is canonical; tombstone is best-effort observability. If both :by-run SET and tombstone SET fail, the read path falls back to :latest with the standard "may have expired" note (same as no tombstone). User-visible behavior degrades gracefully.
  • By-run TTL reset on re-runs (D6): documented as intentional. Operator runbook should note.
• API surface parity:
  • Trigger endpoint mirrors run-scenario.ts: Pro gate + ApiError + per-IP rate-limit + queue-depth backpressure + opaque-token response.
  • get-simulation-outcome adds processing field — proto-additive.
• Integration coverage:
  • U3 parity test (no asymmetry) load-bearing for D4.
  • U3 tri-state pkgFingerprint worker test load-bearing for D7 + Adv-F2.
  • U5 integration via processNextSimulationTask load-bearing for R7.
  • U6 test for tombstone vs expiry vs processing distinction load-bearing for D9 + PL-2.
• Unchanged invariants:
  • :latest Redis key shape, TTL, and write path unchanged.
  • Auto-trigger at scripts/seed-forecasts.mjs:16087 call site unchanged (signature backward-compat).
  • Cron cadence, worker poll interval, gateway cache-tier behavior unchanged.
  • User-facing response shape on get-simulation-outcome extended only with additive processing field; existing fields unchanged.

Risks & Dependencies

Documentation / Operational Notes

• PR description should call out:
  • Net-new POST endpoint mirroring run-scenario shape
  • Behavior change on get-simulation-outcome (filter active + new processing field)
  • Closes AI simulation cannot be triggered via API — human Railway operator required, agents are permanently read-only #3734 and resolves TODOs Optimize map rendering layers #21 + Expand startup ecosystem coverage with VC blogs, regional news, and unicorn tracking #27
  • Links to the follow-up audit-reframe docs/marketing issue
• Monitoring (post-merge):
  • Sentry breadcrumb forecast.trigger_simulation.queued — drives the 30-day demand experiment. Filter by authKind != 'enterprise_api_key:team' for non-team count.
  • Sentry counter forecast.simulation.by_run_write_failed — independent of expiry rate; if >0, investigate Redis transient issues.
  • Sentry: trigger endpoint 429 rate (queue-capacity + per-IP separately).
  • Worker logs: package_rotated warning count → race-fire frequency.
  • Custom: rate of note: 'requested runId not found...' on get-simulation-outcome (drives TTL tuning; distinct from tombstone note rate).
• Operator runbook: --run-id=X requires regex match (^\d{13,}-...); invalid IDs are silently skipped per pre-existing worker behavior.
• Rollback: Pure additive. Revert; by-run keys TTL out in 24h. Trigger endpoint 404s. :latest unchanged.

What changed from v2

Sources & References

• Origin issue: koala73/worldmonitor#3734
• Audit umbrella: #3726
• In-repo TODO precedents: todos/021-pending-p1-simulation-no-http-trigger-endpoint.md, todos/027-complete-p2-simulation-runid-filter-noop-undocumented.md
• Primary structural precedent: server/worldmonitor/scenario/v1/run-scenario.ts + server/worldmonitor/scenario/v1/get-scenario-status.ts
• Per-IP rate-limit precedent: server/_shared/rate-limit.ts:99
• PREMIUM_RPC_PATHS precedent: src/shared/premium-paths.ts:31
• Queue infrastructure: scripts/seed-forecasts.mjs:16951-16999, :17018-17146
• Auto-trigger (unchanged): scripts/seed-forecasts.mjs:16087
• Cache-tier gateway logic (constraint, not changed): server/gateway.ts:1004-1031
• Cache-tier parity test (constraint): tests/route-cache-tier.test.mjs:40-77
• Memory: fresh branch off origin/main: MEMORY.md feedback_fresh_branch_off_origin_main_not_worktree_head
• Superseded plans: docs/plans/2026-05-18-001-..., docs/plans/2026-05-18-002-...