Preview deployment for your docs. Learn more about Mintlify Previews.

💡 Tip: Enable Workflows to automatically generate PRs for you.

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
worldmonitor	Ready	Preview, Comment	Apr 25, 2026 2:34pm

Greptile Summary

This PR implements a per-request Axiom telemetry pipeline wired into createDomainGateway, emitting one request event and one upstream event per outbound fetch, with identity (customer/tier/auth_kind) resolved at the gateway from already-validated auth state and propagated to fetch helpers via AsyncLocalStorage.

The previously-flagged P1 nested-waitUntil issue is correctly resolved: emitRequest now calls deliverUsageEvents (which awaits sendToAxiom directly inside the outer IIFE) rather than emitUsageEvents (which would re-enter ctx.waitUntil). The test recorder.settled quiescence loop is also properly implemented. Two new P2 items remain in usage.ts: deriveRequestId returns '' when x-vercel-id is absent (breaking the upstream ↔ request Axiom join for non-Vercel environments), and getScopeStore is not concurrency-safe on its first call (two simultaneous cold-start requests can produce two AsyncLocalStorage instances, silently discarding scope attribution for one of them).

Confidence Score: 5/5

Safe to merge — no P0/P1 findings in this review round; the previously-flagged P1 nested-waitUntil is correctly fixed.

All findings in this review are P2. The P1 nested-waitUntil from the prior round is properly resolved via deliverUsageEvents. The two new P2s (deriveRequestId empty fallback and getScopeStore race) are low-impact in production (Vercel always supplies x-vercel-id; the ALS race affects at most one early request per cold-start), and telemetry failures are already explicitly allowed to degrade silently.

server/_shared/usage.ts — two new P2s (deriveRequestId fallback and getScopeStore concurrency)

Important Files Changed

Sequence Diagram

sequenceDiagram
    participant V as Vercel Edge (ctx)
    participant G as createDomainGateway
    participant ALS as AsyncLocalStorage (UsageScope)
    participant H as matchedHandler
    participant F as fetchJson / cachedFetchJsonWithMeta
    participant A as Axiom (wm_api_usage)

    V->>G: handler(req, ctx)
    G->>G: auth resolution → usage:UsageIdentityInput
    G->>ALS: runWithUsageScope({ctx, requestId, customerId, route, tier})
    ALS->>H: handlerCall(request)
    H->>F: fetchJson(url, {provider, operation})
    F->>F: finally: ctx.waitUntil(sendToAxiom([upstream event]))
    Note right of F: registered during request phase ✓
    F-->>H: result
    H-->>G: Response
    G->>G: emitRequest(status, reason, cacheTier)
    G->>V: ctx.waitUntil(IIFE)
    Note right of G: IIFE awaits deriveUaHash then awaits deliverUsageEvents directly (no nested waitUntil) ✓
    V->>A: POST /v1/datasets/wm_api_usage/ingest (upstream event)
    V->>A: POST /v1/datasets/wm_api_usage/ingest (request event)

_{Reviews (3): Last reviewed commit: "fix(usage): address koala #3403 review —..." | Re-trigger Greptile}

Greptile review follow-up

@greptileai thanks for the review — second-passed every item against the code and ran an independent Codex audit (codex_review.md). Here's where we landed.

Implemented

P2 — recorder.settled microtask fragility ✅ Fixed in 24d511e

You and Codex both flagged this. Promise.all(pending) snapshotted the array, missing the inner ctx.waitUntil(sendToAxiom(...)) pushed during drain. Replaced with a quiescence loop (while pending.length !== prev) so the helper survives any future async in the emit path. Tests still green.

Not carried forward (with rationale)

@koala73 — flagging the items we consciously decided not to act on, so this is auditable later:

P1 — Nested ctx.waitUntil "may silently drop Axiom delivery"
Not substantiated. The outer waitUntil is registered synchronously inside the request handler at server/gateway.ts:323-329. Cloudflare Workers, MDN, and Vercel docs all describe waitUntil() as the supported mechanism for post-response work, and additional waitUntil() calls are valid while tracked promises are still outstanding. We may simplify the shape in a follow-up for clarity, but it isn't a correctness bug.

P2 — reason: 'ok' on auth rejection paths
This is a schema-design choice, not a defect. The request event already carries status, and the RequestReason union in server/_shared/usage.ts:58 is intentionally narrow. Splitting into auth_401 / auth_403 would make dashboards nicer but isn't required by the documented schema. Open to revisiting if dashboard ergonomics demand it.

P2 — 32-bit FNV-1a "corrupts upstream per-customer attribution"
Rationale is incorrect. Upstream events do not carry principal_id; they inherit customer_id, route, and tier from UsageScope (server/_shared/usage.ts:310-315, server/_shared/fetch-json.ts:55-69). A principal_id collision therefore cannot merge upstream customer attribution as described. Lower collision risk for request-level key analytics can be evaluated separately if it ever matters.

P2 — Uninitialized data in server/_shared/redis.ts
Not actionable. On the rejected-promise path, the function exits via throw after finally, so return { data, source: 'fresh' } is unreachable. Code typechecks and the targeted tests pass.

Verification

• node --import tsx --test tests/usage-telemetry-emission.test.mts — 6/6 ✅
• Full pre-push suite (typecheck, boundaries, rate-limit policies, premium-fetch parity, unit tests) — clean
• Cross-checked waitUntil() semantics against MDN, Cloudflare Workers, and Vercel Functions docs

Aggregated review — telemetry pipeline

Combining my walkthrough, @SebastienMelki's PR-body Greptile-watch list, the Greptile comment, and findings from a fresh local checkout. Two new P1s and two new P2s vs. what Greptile surfaced. The PR is sound architecturally and safe to merge in Phase 1 (telemetry dark), but the two P1s and one of the P2s need to land before flipping USAGE_TELEMETRY=1 in any environment that emits attributable rows.

P1 — Nested ctx.waitUntil silently drops Axiom POSTs

server/gateway.ts:327 (the emitRequest IIFE) + server/_shared/usage.ts:399 (emitUsageEvents).

ctx.waitUntil((async () => {
  const uaHash = await deriveUaHash(originalRequest);   // awaits — drain phase begins
  emitUsageEvents(ctx, [...]);                           // → ctx.waitUntil(sendToAxiom) AFTER drain
})());

In Edge runtimes (Cloudflare Workers semantics, inherited by Vercel Edge), waitUntil registered after the synchronous response phase is unreliable — the runtime may silently ignore it. sendToAxiom runs but its Promise isn't tracked, so the isolate can be torn down mid-POST regardless of the circuit-breaker state. Net effect: the breaker reports healthy while every Axiom POST is best-effort.

Same anti-pattern exists in the upstream emit paths:

• server/_shared/fetch-json.ts:62-79 — import('./usage').then(({ emitUsageEvents }) => emitUsageEvents(ctx, ...)) — the dynamic import resolves outside the synchronous request phase.
• server/_shared/redis.ts:392-419 — same dynamic-import-then-emit pattern in emitUpstreamFromHook.

Fix: export sendToAxiom from usage.ts and call it directly inside the outer IIFE (no re-entry into waitUntil). For the upstream paths, statically import usage.ts so the emit runs synchronously on the call site.

P1 — X-Widget-Key spoofing corrupts per-customer attribution

server/gateway.ts:311 reads request.headers.get('x-widget-key') straight into usage.widgetKey, with no validation. The only real widget-key validation in the repo is in api/widget-agent.ts, not in createDomainGateway.

Combined with server/_shared/usage-identity.ts:73:

if (input.widgetKey) {
  return {
    auth_kind: 'widget_key',
    principal_id: hashKeySync(input.widgetKey),
    customer_id: input.widgetKey,   // ← attacker-controlled
    tier,
  };
}

→ any unauthenticated caller can set X-Widget-Key: <victim-customer> and have telemetry attribute their volume to that customer. This is the exact failure mode the per-customer accounting goal exists to prevent: noisy-anon traffic disguising itself as a paying widget customer's row, or a competitor poisoning a publisher's rate/cost dashboards.

Fix: mirror the validateUserApiKey pattern — only set usage.widgetKey after the gateway has validated it (presence in a registry / signature check / whatever the widget-agent path does today). Until validated, treat the header as anonymous.

P2 — OPTIONS preflight returns without emitRequest()

server/gateway.ts:372-375:

if (request.method === 'OPTIONS') {
  return new Response(null, { status: 204, headers: corsHeaders });
}

No emit. Both PR description and docs/architecture/usage-telemetry.md ("Coverage at every gateway return path — origin block, OPTIONS, …") claim this is covered. It isn't. Add emitRequest(204, 'ok', null); before the return — either keep 'ok' or extend RequestReason with 'preflight' for cleaner dashboards.

P2 — cachedFetchJsonWithMeta reports status=200 for negative-sentinel writes

server/_shared/redis.ts:350 (the .then block):

.then(async (result) => {
  upstreamStatus = 200;            // unconditional
  if (result != null) {
    await setCachedJson(key, result, ttlSeconds);
  } else {
    cacheStatus = 'neg-sentinel';  // ← provider returned empty
    await setCachedJson(key, NEG_SENTINEL, negativeTtlSeconds);
  }
  return result;
})

When the fetcher resolves null (provider returned no data — the whole reason neg-sentinel exists), upstreamStatus is still set to 200. Telemetry will then show "successful provider call" for what was actually an empty/failed upstream. This poisons the cache-hit-ratio recipe in the docs (fresh + neg-sentinel rows under status=200 look like real provider hits) and breaks per-provider error-rate dashboards.

Fix: gate the 200 on result != null — set upstreamStatus = 0 (or define an explicit 'empty' status) for the neg-sentinel branch. Or extend the fetcher contract to return { data, status } so the real upstream HTTP status survives.

P2 — reason: 'ok' on every auth-rejection path (Greptile)

gateway.ts:541, 568, 580, 591, 604 — every 401/403 from auth checks emits reason: 'ok'. Cross-filter on status works, but a reason == "ok" query returns auth failures alongside genuine successes. RequestReason is open; add 'auth_401' | 'auth_403' | 'tier_403' and use them.

P2 — 32-bit FNV-1a collision space (Greptile)

server/_shared/usage-identity.ts:hashKeySync — 4.3 B output space. Birthday-collision expected ~65k widget keys. Two-round XOR-folded FNV-1a is still sync, no extra deps. Worth doing before widget-key adoption picks up (and pairs naturally with the P1 widget-key validation work).

P2 — recorder.settled snapshots pending array (Greptile)

tests/usage-telemetry-emission.test.mts:34-47 — Promise.all(pending) is computed at getter-call time; promises pushed during drain (the inner sendToAxiom) aren't tracked. Tests pass today only because the globalThis.fetch spy is sync up to events.push. Quiescence loop fixes it. (Becomes moot if the P1 nested-waitUntil fix collapses the two-stage emit into one, but the loop is still the right pattern for any future test with drain-phase work.)

P3 — minor

• usage.tier polluted before bearer rejection (gateway.ts:489) — getEntitlements is called and usage.tier set before allowed is checked. A 403 row carries the rejected user's actual tier. Probably OK, slightly confusing in dashboards.
• x-usage-telemetry header is non-prod only — in prod, the only signal that the breaker tripped is the 1%-sampled console.warn. Consider a periodic structured log of breaker state (every 60s while tripped) so on-call can grep without sampling luck.
• Missing upstream event payload assertions — PR body acknowledges this as follow-up. File the issue before merge so it doesn't get lost.
• origin_kind: 'mcp' | 'internal-cron' are in the type but never emitted. Doc table notes this. Fine to leave.

Recommended path

1. Phase 1 merge as-is — emission is dark, P1s are inert.
2. Before Phase 2 preview: fix nested waitUntil (3 sites) and widget-key spoofing (gateway-side validation). Add OPTIONS emit, fix neg-sentinel status, add auth_* reasons.
3. Before Phase 3 prod: 64-bit hash, harden test recorder, file upstream-payload-assertion follow-up.

CI is green (typecheck / biome / unit / markdown). Architecture and rollout plan are right; the fixes above are surgical.

Round-2 review fixes — pushed in 5883976

@koala73 @greptileai — addressed the full set from the aggregated review. Recap:

P1 — fixed

• Nested `ctx.waitUntil` (3 sites) — exported `sendToAxiom`; gateway `emitRequest` IIFE now `await`s `deliverUsageEvents` directly, and `fetch-json.ts` + `redis.ts` use static imports + a single synchronous `ctx.waitUntil(sendToAxiom(...))`. No more two-stage registration that Edge runtimes can drop after the response phase.
• Widget-key spoofing — gateway only populates `usage.widgetKey` after matching the header against `WIDGET_AGENT_KEY` (same check as `api/widget-agent.ts`); unvalidated headers fall through to anon, so attacker-controlled `X-Widget-Key` can no longer poison `customer_id`.

P2 — fixed

• OPTIONS preflight not emitting — added `'preflight'` to `RequestReason` and `emitRequest(204, 'preflight', null)` before the 204 return.
• `cachedFetchJsonWithMeta` neg-sentinel reported as `status=200` — gated `upstreamStatus = 200` on `result != null`; null results now emit `status=0` with `cache_status='neg-sentinel'`, so the cache-hit-ratio recipe and per-provider error-rate dashboards stay honest.
• `reason: 'ok'` on auth/tier rejections — extended union with `auth_401` / `auth_403` / `tier_403`; replaced 6 emit sites in `gateway.ts`. `reason='ok'` now means actual success.
• 32-bit FNV-1a collision space — replaced with a two-round XOR-folded 64-bit variant in `hashKeySync`. Sync, no extra deps. Existing `usage-identity` tests pass unchanged.

Verification

• `tests/usage-telemetry-emission.test.mts` — 6/6 ✅
• `tests/premium-stock-gateway.test.mts` + `tests/gateway-cdn-origin-policy.test.mts` — 15/15 ✅
• `npx vitest run server/tests/usage-identity.test.ts` — 13/13 ✅
• `npx tsc --noEmit` — clean
• Doc table in `docs/architecture/usage-telemetry.md` updated for the new reasons.

P3 — not in this commit

• `usage.tier` populated before bearer rejection: leaving as-is (a 403 row carrying the rejected user's tier is mildly confusing but not wrong).
• Periodic structured breaker-state log: open follow-up if the 1%-sampled `console.warn` proves insufficient in prod.
• `upstream` event payload assertions: filing as follow-up issue.

Ready for re-review.

@koala73 — ready for final review.

Round-2 review fixes verified end-to-end on preview

All P1/P2 items from your aggregated review are landed in d68c089. Pushed an Axiom token to the preview env, hit `/api/market/v1/list-market-quotes` from a logged-in browser, and the event arrived as expected. Per the screenshots I shared with you separately:

```json
{
"event_type": "request",
"domain": "market",
"route": "/api/market/v1/list-market-quotes",
"method": "GET",
"status": 401,
"auth_kind": "anon",
"reason": "auth_401",
"tier": 0,
"duration_ms": 1,
"execution_region": "cdg1:cdg1",
"execution_plane": "vercel-edge",
"country": "LB",
...
}
```

Confirms three things at once:

• Pipeline is live (Axiom `wm_api_usage` dataset, US region) — no nested-`waitUntil` drops.
• `reason: 'auth_401'` lands on the wire (was `'ok'` before this round).
• Domain extraction yields `'market'`, not `'v1'` / `'api'` (the round-1 fix from the prior review still holds after the merge).

What changed since the round-2 push

• All P1/P2 fixes from #issuecomment-4319745129 (pushed in 5883976).
• Merged main and added `'live'` to the `UsageCacheTier` union — main's PR feat(energy-atlas): live tanker map layer + contract (parity PR 3, plan U7-U8) #3402 (energy-atlas) introduced a new cache tier in `gateway.ts` and the synthetic merge with main was failing typecheck:api until I synced it.
• Reverted the temporary unconditional debug logs in `sendToAxiom` now that delivery is confirmed end-to-end on preview.

CI

• `biome` ✅
• `typecheck` ✅
• `unit` ✅
• `gate` ✅

Telemetry is currently scoped to the `feat/usage-telemetry` branch in preview only. Production env vars (`USAGE_TELEMETRY`, `AXIOM_API_TOKEN`) are unset, so emission stays dark on prod regardless of merge timing — Phase 1 contract.

Round-2 verification — all P1/P2 fixes confirmed

Re-pulled feat/usage-telemetry at d68c089f. Verified each finding from the aggregated review against the current source. Posting evidence rather than restating fixes:

CI still green (typecheck / biome / unit / markdown). PR description's deploy-phasing claims now match the code.

Residual non-blockers (file as follow-ups, don't block this PR)

1. Widget-key attribution is effectively flat. WIDGET_AGENT_KEY is a single shared secret, so every validated widget call lands with the same customer_id. Per-publisher granularity would need a key-per-publisher registry (out of scope here). The fix as-landed correctly prevents spoofing; it just means widget telemetry is one row per environment, not per publisher. Worth a separate issue if widget volume grows.
2. Timing-attack-resistant compare for the widget key. === is fine because the existing api/widget-agent.ts uses the same pattern, but a future hardening pass could move both to a constant-time compare.
3. usage.tier populated before bearer rejection (gateway.ts:489 area, earlier P3). Still present. A 403 row carries the actual rejected user's tier. Probably correct for analytics; flag in a comment if you don't want it.
4. Prod breaker visibility — x-usage-telemetry header is non-prod only. In prod the only signal a tripped breaker is the 1%-sampled console.warn. Consider a periodic structured log every ~60s while tripped so an oncall can grep deterministically.
5. Upstream payload assertion test still acknowledged as follow-up in PR body.

Ship recommendation

LGTM for the full deploy plan, not just Phase 1.

• Phase 1 (merge dark) — ship now.
• Phase 2 (Preview env) — go after Phase 1's 24h baseline; nothing blocks it.
• Phase 3 (Prod) — go after Preview's events_per_min matches Vercel request rate within ~2% over 1h.

Round-2 fixes are surgical and the architecture is unchanged. Nice turnaround.

Round-3 review — three new findings, two block Phase 2/3

Re-verified against feat/usage-telemetry HEAD d68c089f. The round-2 widget-key spoofing fix introduced a server-side secret leak; the upstream-event pipeline is wired but never invoked; and request_id is empty outside Vercel. None block Phase 1 merge-dark, but two of three are hard blockers for Phase 2 (Preview emit) and Phase 3 (Prod emit).

P1 — customer_id leaks the validated WIDGET_AGENT_KEY to Axiom

server/_shared/usage-identity.ts:73-79:

if (input.widgetKey) {
  return {
    auth_kind: 'widget_key',
    principal_id: hashKeySync(input.widgetKey),
    customer_id: input.widgetKey,           // ← raw secret
    tier,
  };
}

Per api/widget-agent.ts:10-12 the widget keys are legacy tester/relay keys explicitly described as "never exposed to the browser" — they unlock premium proxy paths to the Railway relay. Emitting the raw key as customer_id ships that secret to Axiom (a third-party log store, not under our control, indexed and persisted) on every validated widget request.

Severity rationale: this is an exfiltration of a server-only credential to an external system. The round-2 fix solved spoofing but introduced a new and arguably worse failure mode — pre-fix the field carried attacker-controlled junk; post-fix it carries the live shared secret.

Fix: keep auth_kind='widget_key' but use a stable non-secret label ('widget-tester', or hash-of-key, or the env name 'WIDGET_AGENT_KEY') for customer_id. principal_id already uses the hash; carry the same hash through to customer_id, or just emit a constant since there's effectively one widget customer today (the residual #1 from round-2).

Audit task before flipping the env var anywhere: confirm Axiom retention isn't longer than the widget-key rotation cadence, and rotate WIDGET_AGENT_KEY once the leak is fixed (the value has presumably already been emitted in preview tests at 3dcf171c / 12d1d06f).

P1 — upstream emission is wired but never invoked

server/_shared/fetch-json.ts:45 and server/_shared/redis.ts:395 both gate on provider / opts.usage.provider:

if (provider) { /* emit */ }              // fetch-json.ts:45
if (!usage?.provider) return;             // redis.ts:393

No production call site passes either. Sampled checks:

PR description Phase-3 success criteria says "Top-N customer_id should match known top accounts" and the docs advertise per-customer × per-provider cost attribution as a primary deliverable — neither is achievable until call sites are instrumented. The upstream event table will be empty in Axiom.

Fix: either (a) instrument each provider-tagged fetch in this PR — at minimum the LLM, GitHub, SEC, HN, intelligence, and infrastructure paths above; or (b) demote upstream emission from a "what this gives us" bullet to a "scaffolding ready, instrumentation TODO" follow-up issue, and update docs/architecture/usage-telemetry.md accordingly so operators don't query an empty table on day one.

P2 — deriveRequestId() returns '' outside Vercel

server/_shared/usage.ts:197-199:

export function deriveRequestId(req: Request): string {
  return req.headers.get('x-vercel-id') ?? '';
}

Empty string breaks every join key in the docs:

• request × upstream join on request_id collapses every non-Vercel request into one bucket.
• Local dev (vercel dev does set x-vercel-id, but plain node test harness doesn't), Railway, Tauri, MCP, internal-cron — all collide on ''.
• The test recorder for the tier=2 path bypasses Vercel headers entirely; their telemetry rows would all share request_id=''.

Fix:

export function deriveRequestId(req: Request): string {
  return (
    req.headers.get('x-request-id') ??
    req.headers.get('x-vercel-id') ??
    crypto.randomUUID()
  );
}

crypto.randomUUID() is available in Edge, Node ≥19, and is in the Web Crypto spec. Existing Vercel calls keep the same id; non-Vercel emits become joinable.

Combined ship state

Recommend:

1. Fix P1 widget customer_id (one-line: replace input.widgetKey with hashKeySync(input.widgetKey) or a constant).
2. Fix P2 request_id fallback chain.
3. Decide on upstream P1: instrument the 7+ call sites in this PR (preferred — keeps the PR's stated value intact) or scope it to a follow-up and adjust docs/PR description.
4. Rotate WIDGET_AGENT_KEY after the widget leak fix lands, given the preview env had USAGE_TELEMETRY=1 + AXIOM_API_TOKEN set during the recent verification commits.