chore(registry): stabilize dataset contract foundation and bootstrap parity#10
Merged
Conversation
Add the dataset contract source, deterministic generator, and bootstrap-only generated artifacts needed to replace the hand-maintained bootstrap registries without touching api/health.js yet. Validation: npm run registry:check; node --test tests/bootstrap.test.mjs tests/market-breadth.test.mjs; node --test tests/edge-functions.test.mjs; npm run typecheck; npm run typecheck:api
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This was referenced Apr 11, 2026
Owner
Author
Greptile SummaryThis PR establishes
Confidence Score: 4/5Safe to merge; the regex bug disables a validation guard but does not affect the generated output or any runtime path. The architectural refactor is solid and the generated artifacts are verified correct and complete. The one concrete bug (\d regex in extractVersionTag) disables the version-mismatch validator without impacting the currently emitted registry or any runtime behavior — it's a latent defect that should be fixed before new datasets with version tags are added. Everything else is non-blocking style. registry/datasets.ts (extractVersionTag regex); scripts/generate-dataset-registry.ts (duplicate formatStringMap/formatTierMap) Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A["registry/datasets.ts\n(BOOTSTRAP_ALIASES + BOOTSTRAP_TIERS)"]
B["scripts/generate-dataset-registry.ts\n(validate + emit)"]
C["api/_generated/dataset-registry.js\n(BOOTSTRAP_CACHE_KEYS, BOOTSTRAP_TIERS)"]
D["server/_shared/_generated/bootstrap-registry.ts\n(BOOTSTRAP_CACHE_KEYS, BOOTSTRAP_TIERS)"]
E["api/bootstrap.js\n(edge function)"]
F["server/_shared/cache-keys.ts\n(re-export)"]
G["scripts/check-dataset-registry.mjs\n(git diff --exit-code)"]
A -->|"npm run registry:generate"| B
B -->|"writeFileSync"| C
B -->|"writeFileSync"| D
C -->|"import"| E
D -->|"re-export"| F
A -->|"npm run registry:check"| G
G -->|"re-runs generator then"| B
G -->|"verifies no diff"| C
G -->|"verifies no diff"| D
Prompt To Fix All With AIThis is a comment left during a code review.
Path: registry/datasets.ts
Line: 224
Comment:
**Regex uses `\\d` (literal backslash-d) instead of `\d` (digit)**
In a JavaScript/TypeScript regex literal, `\\d` means a literal backslash followed by the character `d`, not a digit. This means `extractVersionTag` will always return `undefined` for every Redis key (e.g. `market:sectors:v2`, `forecast:predictions:v2`), since none of them contain the literal string `\d`.
The practical consequence is that `redis.versionTag` is `undefined` for all datasets built by `buildDatasets()`, which silently bypasses the validation guard in `generate-dataset-registry.ts`:
```typescript
if (dataset.redis.versionTag && !hasMatchingVersionTag(dataset.redis.key, dataset.redis.versionTag)) {
fail(`Dataset ${dataset.id} has key/version mismatch`);
}
```
Since `versionTag` is never set, this check is never reached. The generated registry output is unaffected today (because `versionTag` isn't emitted), but the guard against key/version mismatches in future dataset additions is completely inert.
```suggestion
const match = key.match(/(?:^|[:\\-])(v\d+)(?:$|[:])/);
```
How can I resolve this? If you propose a fix, please make it concise.
---
This is a comment left during a code review.
Path: scripts/generate-dataset-registry.ts
Line: 85-103
Comment:
**`formatStringMap` and `formatTierMap` are identical**
Both functions have the exact same implementation. One of them (or both, renamed to `formatMap`) can be removed.
```suggestion
function formatMap(entries: Array<[string, string]>): string {
if (entries.length === 0) {
return '{}';
}
return `{\n${entries
.map(([name, value]) => ` ${JSON.stringify(name)}: ${JSON.stringify(value)},`)
.join('\n')}\n}`;
}
```
Then use `formatMap` in both `writeFileSync` calls.
How can I resolve this? If you propose a fix, please make it concise.
---
This is a comment left during a code review.
Path: tests/bootstrap.test.mjs
Line: 22-41
Comment:
**Test name and variable name reference the old `cache-keys.ts` location**
After this PR, `cacheKeysSrc` (line 12) is loaded from `server/_shared/_generated/bootstrap-registry.ts` — not from `server/_shared/cache-keys.ts`. The test description "generated keys match server/_shared/cache-keys.ts exports" is therefore misleading; it's actually comparing two generated files (`_generated/bootstrap-registry.ts` vs `api/_generated/dataset-registry.js`) to verify they're in sync.
While the test logic is correct, the test description and the `canonical` / `cacheKeysSrc` variable names will confuse future readers. Consider renaming to:
- `cacheKeysSrc` → `bootstrapRegistryTsSrc`
- Test name → "generated TS and JS registry files are in parity"
How can I resolve this? If you propose a fix, please make it concise.
---
This is a comment left during a code review.
Path: scripts/check-dataset-registry.mjs
Line: 5-12
Comment:
**`git diff --exit-code` output is piped away, making check failures harder to diagnose**
With `stdio: 'pipe'`, the diff between the stale and regenerated artifacts is silently swallowed. The developer only sees the custom error message and has to run `git diff` manually to find out what changed.
Switching to `stdio: 'inherit'` prints the diff before the process exits, making it immediately obvious what drifted:
```suggestion
execSync(
'git diff --exit-code -- api/_generated/dataset-registry.js server/_shared/_generated/bootstrap-registry.ts',
{ stdio: 'inherit' },
);
```
How can I resolve this? If you propose a fix, please make it concise.Reviews (1): Last reviewed commit: "chore(registry): introduce generated boo..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| } | ||
|
|
||
| function extractVersionTag(key: string): `v${number}` | undefined { | ||
| const match = key.match(/(?:^|[:\\-])(v\\d+)(?:$|[:])/); |
There was a problem hiding this comment.
\\d (literal backslash-d) instead of \d (digit)
In a JavaScript/TypeScript regex literal, \\d means a literal backslash followed by the character d, not a digit. This means extractVersionTag will always return undefined for every Redis key (e.g. market:sectors:v2, forecast:predictions:v2), since none of them contain the literal string \d.
The practical consequence is that redis.versionTag is undefined for all datasets built by buildDatasets(), which silently bypasses the validation guard in generate-dataset-registry.ts:
if (dataset.redis.versionTag && !hasMatchingVersionTag(dataset.redis.key, dataset.redis.versionTag)) {
fail(`Dataset ${dataset.id} has key/version mismatch`);
}Since versionTag is never set, this check is never reached. The generated registry output is unaffected today (because versionTag isn't emitted), but the guard against key/version mismatches in future dataset additions is completely inert.
Suggested change
| const match = key.match(/(?:^|[:\\-])(v\\d+)(?:$|[:])/); | |
| const match = key.match(/(?:^|[:\\-])(v\d+)(?:$|[:])/); |
Prompt To Fix With AI
This is a comment left during a code review.
Path: registry/datasets.ts
Line: 224
Comment:
**Regex uses `\\d` (literal backslash-d) instead of `\d` (digit)**
In a JavaScript/TypeScript regex literal, `\\d` means a literal backslash followed by the character `d`, not a digit. This means `extractVersionTag` will always return `undefined` for every Redis key (e.g. `market:sectors:v2`, `forecast:predictions:v2`), since none of them contain the literal string `\d`.
The practical consequence is that `redis.versionTag` is `undefined` for all datasets built by `buildDatasets()`, which silently bypasses the validation guard in `generate-dataset-registry.ts`:
```typescript
if (dataset.redis.versionTag && !hasMatchingVersionTag(dataset.redis.key, dataset.redis.versionTag)) {
fail(`Dataset ${dataset.id} has key/version mismatch`);
}
```
Since `versionTag` is never set, this check is never reached. The generated registry output is unaffected today (because `versionTag` isn't emitted), but the guard against key/version mismatches in future dataset additions is completely inert.
```suggestion
const match = key.match(/(?:^|[:\\-])(v\d+)(?:$|[:])/);
```
How can I resolve this? If you propose a fix, please make it concise.
Comment on lines
+85
to
+103
| function formatStringMap(entries: Array<[string, string]>): string { | ||
| if (entries.length === 0) { | ||
| return '{}'; | ||
| } | ||
|
|
||
| return `{\n${entries | ||
| .map(([name, value]) => ` ${JSON.stringify(name)}: ${JSON.stringify(value)},`) | ||
| .join('\n')}\n}`; | ||
| } | ||
|
|
||
| function formatTierMap(entries: Array<[string, string]>): string { | ||
| if (entries.length === 0) { | ||
| return '{}'; | ||
| } | ||
|
|
||
| return `{\n${entries | ||
| .map(([name, value]) => ` ${JSON.stringify(name)}: ${JSON.stringify(value)},`) | ||
| .join('\n')}\n}`; | ||
| } |
There was a problem hiding this comment.
formatStringMap and formatTierMap are identical
Both functions have the exact same implementation. One of them (or both, renamed to formatMap) can be removed.
Suggested change
| function formatStringMap(entries: Array<[string, string]>): string { | |
| if (entries.length === 0) { | |
| return '{}'; | |
| } | |
| return `{\n${entries | |
| .map(([name, value]) => ` ${JSON.stringify(name)}: ${JSON.stringify(value)},`) | |
| .join('\n')}\n}`; | |
| } | |
| function formatTierMap(entries: Array<[string, string]>): string { | |
| if (entries.length === 0) { | |
| return '{}'; | |
| } | |
| return `{\n${entries | |
| .map(([name, value]) => ` ${JSON.stringify(name)}: ${JSON.stringify(value)},`) | |
| .join('\n')}\n}`; | |
| } | |
| function formatMap(entries: Array<[string, string]>): string { | |
| if (entries.length === 0) { | |
| return '{}'; | |
| } | |
| return `{\n${entries | |
| .map(([name, value]) => ` ${JSON.stringify(name)}: ${JSON.stringify(value)},`) | |
| .join('\n')}\n}`; | |
| } |
Then use formatMap in both writeFileSync calls.
Prompt To Fix With AI
This is a comment left during a code review.
Path: scripts/generate-dataset-registry.ts
Line: 85-103
Comment:
**`formatStringMap` and `formatTierMap` are identical**
Both functions have the exact same implementation. One of them (or both, renamed to `formatMap`) can be removed.
```suggestion
function formatMap(entries: Array<[string, string]>): string {
if (entries.length === 0) {
return '{}';
}
return `{\n${entries
.map(([name, value]) => ` ${JSON.stringify(name)}: ${JSON.stringify(value)},`)
.join('\n')}\n}`;
}
```
Then use `formatMap` in both `writeFileSync` calls.
How can I resolve this? If you propose a fix, please make it concise.
Comment on lines
22
to
41
| it('generated keys match server/_shared/cache-keys.ts exports', () => { | ||
| const extractKeys = (src) => { | ||
| const block = src.match(/BOOTSTRAP_CACHE_KEYS[^=]*=\s*\{([^}]+)\}/); | ||
| if (!block) return {}; | ||
| const re = /(\w+):\s+'([a-z0-9_-]+(?::[a-z0-9_-]+)+:v\d+)'/g; | ||
| const re = /["']([^"']+)["']:\s*["']([^"']+)["']/g; | ||
| const keys = {}; | ||
| let m; | ||
| while ((m = re.exec(block[1])) !== null) keys[m[1]] = m[2]; | ||
| return keys; | ||
| }; | ||
| const canonical = extractKeys(cacheKeysSrc); | ||
| const inlined = extractKeys(bootstrapSrc); | ||
| const generated = extractKeys(generatedRegistrySrc); | ||
| assert.ok(Object.keys(canonical).length >= 10, 'Canonical registry too small'); | ||
| for (const [name, key] of Object.entries(canonical)) { | ||
| assert.equal(inlined[name], key, `Key '${name}' mismatch: canonical='${key}', inlined='${inlined[name]}'`); | ||
| assert.equal(generated[name], key, `Key '${name}' mismatch: canonical='${key}', generated='${generated[name]}'`); | ||
| } | ||
| for (const [name, key] of Object.entries(inlined)) { | ||
| for (const [name, key] of Object.entries(generated)) { | ||
| assert.equal(canonical[name], key, `Extra inlined key '${name}' not in canonical registry`); | ||
| } | ||
| }); |
There was a problem hiding this comment.
cache-keys.ts location
After this PR, cacheKeysSrc (line 12) is loaded from server/_shared/_generated/bootstrap-registry.ts — not from server/_shared/cache-keys.ts. The test description "generated keys match server/_shared/cache-keys.ts exports" is therefore misleading; it's actually comparing two generated files (_generated/bootstrap-registry.ts vs api/_generated/dataset-registry.js) to verify they're in sync.
While the test logic is correct, the test description and the canonical / cacheKeysSrc variable names will confuse future readers. Consider renaming to:
cacheKeysSrc→bootstrapRegistryTsSrc- Test name → "generated TS and JS registry files are in parity"
Prompt To Fix With AI
This is a comment left during a code review.
Path: tests/bootstrap.test.mjs
Line: 22-41
Comment:
**Test name and variable name reference the old `cache-keys.ts` location**
After this PR, `cacheKeysSrc` (line 12) is loaded from `server/_shared/_generated/bootstrap-registry.ts` — not from `server/_shared/cache-keys.ts`. The test description "generated keys match server/_shared/cache-keys.ts exports" is therefore misleading; it's actually comparing two generated files (`_generated/bootstrap-registry.ts` vs `api/_generated/dataset-registry.js`) to verify they're in sync.
While the test logic is correct, the test description and the `canonical` / `cacheKeysSrc` variable names will confuse future readers. Consider renaming to:
- `cacheKeysSrc` → `bootstrapRegistryTsSrc`
- Test name → "generated TS and JS registry files are in parity"
How can I resolve this? If you propose a fix, please make it concise.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Comment on lines
+5
to
+12
| try { | ||
| execSync( | ||
| 'git diff --exit-code -- api/_generated/dataset-registry.js server/_shared/_generated/bootstrap-registry.ts', | ||
| { stdio: 'pipe' }, | ||
| ); | ||
| } catch { | ||
| console.error('[dataset-registry] generated artifacts are out of date. Run: npm run registry:generate'); | ||
| process.exit(1); |
There was a problem hiding this comment.
git diff --exit-code output is piped away, making check failures harder to diagnose
With stdio: 'pipe', the diff between the stale and regenerated artifacts is silently swallowed. The developer only sees the custom error message and has to run git diff manually to find out what changed.
Switching to stdio: 'inherit' prints the diff before the process exits, making it immediately obvious what drifted:
Suggested change
| try { | |
| execSync( | |
| 'git diff --exit-code -- api/_generated/dataset-registry.js server/_shared/_generated/bootstrap-registry.ts', | |
| { stdio: 'pipe' }, | |
| ); | |
| } catch { | |
| console.error('[dataset-registry] generated artifacts are out of date. Run: npm run registry:generate'); | |
| process.exit(1); | |
| execSync( | |
| 'git diff --exit-code -- api/_generated/dataset-registry.js server/_shared/_generated/bootstrap-registry.ts', | |
| { stdio: 'inherit' }, | |
| ); |
Prompt To Fix With AI
This is a comment left during a code review.
Path: scripts/check-dataset-registry.mjs
Line: 5-12
Comment:
**`git diff --exit-code` output is piped away, making check failures harder to diagnose**
With `stdio: 'pipe'`, the diff between the stale and regenerated artifacts is silently swallowed. The developer only sees the custom error message and has to run `git diff` manually to find out what changed.
Switching to `stdio: 'inherit'` prints the diff before the process exits, making it immediately obvious what drifted:
```suggestion
execSync(
'git diff --exit-code -- api/_generated/dataset-registry.js server/_shared/_generated/bootstrap-registry.ts',
{ stdio: 'inherit' },
);
```
How can I resolve this? If you propose a fix, please make it concise.
Owner
Author
|
@codex review |
|
Codex Review: Didn't find any major issues. Hooray! ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Correct the dataset version-tag regex so generator validation is active again. Also collapse duplicate generator formatting helpers, surface registry diffs in registry:check, and clarify the bootstrap parity test wording.
|
PR author is not in the allowed authors list. |
lspassos1
marked this pull request as ready for review
April 11, 2026 21:31
lspassos1
pushed a commit
that referenced
this pull request
Apr 27, 2026
…oala73#3394) DNS (Cloudflare) and the Vercel domain are already provisioned by the operator; this lands the matching code-side wiring so the variant actually resolves and renders correctly. Changes: middleware.ts - Add `'energy.worldmonitor.app': 'energy'` to VARIANT_HOST_MAP. This also auto-includes the host in ALLOWED_HOSTS via the spread on line 87. - Add `energy` entry to VARIANT_OG with the Energy-Atlas-specific title + description from `src/config/variant-meta.ts:130-152`. OG image points at `https://energy.worldmonitor.app/favico/energy/og-image.png`, matching the per-variant convention used by tech / finance / commodity / happy. vercel.json - Add `https://energy.worldmonitor.app` to BOTH `frame-src` and `frame-ancestors` in the global Content-Security-Policy header. Without this, the variant subdomain would render but be blocked from being framed back into worldmonitor.app for any embedded flow (Slack/LinkedIn previews, future iframe widgets, etc.). This supersedes the CSP-only portion of PR koala73#3359 (which mixed CSP with unrelated relay/military changes). convex/payments/checkout.ts:108-117 - Add `https://energy.worldmonitor.app` to the checkout returnUrl allowlist. Without this, a PRO upgrade flow initiated from the energy subdomain would fail with "Invalid returnUrl" on Convex. src-tauri/tauri.conf.json:32 - Add `https://energy.worldmonitor.app` to the Tauri desktop CSP frame-src so the desktop app can embed the variant the same way it embeds the other 4. public/favico/energy/* (NEW, 7 files) - Stub the per-variant favicon directory by copying the root-level WorldMonitor brand assets (android-chrome 192/512, apple-touch, favicon 16/32/ico, og-image). This keeps the launch unblocked on design assets — every referenced URL resolves with valid bytes from day one. Replace with energy-themed designs in a follow-up PR; the file paths are stable. Other variant subdomains already on main (tech / finance / commodity / happy) are unchanged. APP_HOSTS in src/services/runtime.ts already admits any `*.worldmonitor.app` via `host.endsWith('.worldmonitor.app')` on line 226, so no edit needed there. Closes gaps §L #9, #10, #11 in docs/internal/energy-atlas-registry-expansion.md.
lspassos1
pushed a commit
that referenced
this pull request
May 20, 2026
* feat(convex): add mcpProTokens table + issue/validate/revoke + http routes (U1)
Non-key Pro MCP identity layer for the Pro-tier MCP access plan. Mirrors
apiKeys.ts shape but stores no key material — the row's _id IS the
bearer identifier (referenced from OAuth code/token records as
mcpTokenId).
- mcpProTokens table: {userId, clientId?, name?, createdAt, lastUsedAt?,
revokedAt?} indexed by_userId; userApiKeys untouched.
- Internal mutations: issueProMcpToken (tier ≥ 1 gate, 5-row cap with
silent oldest-revoke rotation, audit-preserving), validateProMcpToken,
internalRevokeProMcpToken (server-side rollback path for U5 — no Clerk
context needed), touchProMcpTokenLastUsed (debounced 5min).
- Public mutations: revokeProMcpToken + listProMcpTokens (Clerk-auth).
- HTTP internal routes: /api/internal-{issue,validate,revoke}-pro-mcp-token
with x-convex-shared-secret guard. Validate route schedules touch via
ctx.scheduler.runAfter, matching apiKeys pattern (http.ts:839).
28 new tests; full convex suite 245/245.
Plan unit U1: docs/plans/2026-05-10-001-feat-pro-mcp-clerk-auth-quota-plan.md
* feat(server): pro-mcp-token edge helper — issue/validate/revoke (U2)
Edge-runtime-safe wrappers around U1's Convex internal HTTP routes. No
positive cache on validate (revoke takes effect on next request);
negative-cache only at pro-mcp-token-neg:<tokenId> (60s TTL) for
known-bad bearers.
- issueProMcpTokenForUser: typed ProMcpIssueFailed (pro-required /
invalid-user-id / config / network), 3s timeout.
- validateProMcpToken: neg-cache short-circuit BEFORE Convex hit; fail-soft
on 5xx/timeout (returns null but does NOT write neg-cache sentinel —
prevents transient-blip poisoning of legitimate tokens). Convex's
validate route schedules touchProMcpTokenLastUsed internally; no
edge-side waitUntil needed.
- revokeProMcpToken: returns result object (no throw) so rollback callers
don't have their original error masked. Sets neg-cache sentinel after
successful revoke.
- invalidateProMcpTokenCache: public sentinel writer for U9.
25 tests including a load-bearing integration test for revoke → next
validate short-circuits without Convex round-trip.
Plan unit U2.
* feat(oauth): apex Clerk grant page + signed-grant mint API (U3)
Bridge between the apex Clerk session and the api-subdomain Pro MCP
consent flow. Apex /mcp-grant SPA reads the OAuth nonce + registered
client metadata so users see the real client_name + redirect host
(anti-phishing). On Authorize the page POSTs to mcp-grant-mint, gets
back a fixed redirect URL with a HMAC-signed grant token, and navigates.
- mcp-grant.html + src/mcp-grant-main.ts: Vite multi-page entry (matches
existing settings.html / live-channels.html convention).
- api/internal/mcp-grant-mint.ts: Clerk-auth POST. Re-checks tier ≥ 1,
re-validates redirect_uri allowlist (defense-in-depth), HMAC-signs
{userId, nonce, exp+5min}, SETEX mcp-grant:<nonce>, returns FIXED
redirect URL (https://api.worldmonitor.app/oauth/authorize-pro).
- api/internal/mcp-grant-context.ts: GET companion for the SPA to fetch
the real client_name + redirect_host. Same Pro-tier gate as mint.
- api/_mcp-grant-hmac.ts: shared sign/verify helper. Signature is over
exact post-base64url-decode bytes (key-order/whitespace independent).
U5 imports verifyGrant from this module.
- vercel.json: /mcp-grant → /mcp-grant.html rewrite + no-store header
rule. Catch-all SPA fallback adjusted to exclude the new page.
- api/oauth/register.js: export isAllowedRedirectUri so DCR allowlist is
reused (no parallel impl).
- 35 tests; full deploy-config + edge-functions + mcp + pro-mcp-token
suites green.
Plan unit U3.
* feat(oauth): consent page Pro-CTA-default + 'Use API key instead' (U4)
R1 of the plan: a logged-in Pro user authorising MCP from claude.ai
never sees the API-key input field by default. The consent page now
leads with a brand-green "Sign in with WorldMonitor Pro" CTA pointing
at the apex /mcp-grant page; the existing API-key form is hidden
behind a "Use API key instead" disclosure for Starter+ holders.
- Default state: form display:none; Pro CTA dominant.
- Error state: existing errorMsg path still works — when ke.textContent
is non-empty (or XHR-retry returns invalid_key), inline script auto-
reveals the form so the user doesn't lose context after a bad key.
- Deep-link: /oauth/authorize?...#api-key shows the form on initial
load, so Starter+ users can bookmark.
- Pro CTA href: https://worldmonitor.app/mcp-grant?nonce=<URL-encoded n>.
Apex page reads the OAuth nonce server-side (no client_name forwarded
via URL — already covered by U3's mcp-grant-context).
- nonce shared with U5: the same oauth:nonce:<n> row that U5's
/oauth/authorize-pro will GETDEL. Handler still mints exactly one
nonce per GET — no double-issue.
- XSS-escape preserved on client_name + redirect_host + nonce + errorMsg.
- 18 new tests; existing handler logic untouched.
Plan unit U4.
* feat(oauth): /oauth/authorize-pro bounce-back endpoint (U5)
Receives the apex grant bounce-back, validates the HMAC-signed grant,
atomically consumes both Redis nonces (mcp-grant + oauth:nonce), issues
a non-key Pro identity row in Convex via U2, and writes the OAuth code
with {kind:'pro', userId, mcpTokenId, ...} for U6 to read at exchange
time.
Security ordering (HMAC-first to avoid burning nonces on forged tokens):
1. verifyGrant (U3's _mcp-grant-hmac, no Redis)
2. URL-nonce vs payload-nonce match
3. GETDEL mcp-grant:<n>
4. strict {userId, exp} tuple match vs grant payload (forge defense)
5. GETDEL oauth:nonce:<n>
6. GET oauth:client:<client_id> + redirect_uri allowlist re-check
7. getEntitlements(userId) tier ≥ 1 re-check (lapse defense)
8. issueProMcpTokenForUser (U2)
9. SETEX oauth:code:<code> (kind:'pro', 600s)
10. 302 redirect_uri?code=<code>[&state=...] with Cache-Control:no-store
oauth:code shape (load-bearing for U6):
{kind:'pro', userId, mcpTokenId, client_id, redirect_uri,
code_challenge, scope:'mcp_pro'}
Best-effort revoke when SETEX fails after issueProMcpTokenForUser
succeeds — calls U2's no-throw revokeProMcpToken; logs orphan-row id
on revoke failure but never masks the original 500.
35 tests; vercel rewrite + api-route-exceptions registered as
external-protocol.
Plan unit U5.
* feat(oauth): McpAuthContext discriminated union + Pro token shape (U6)
Token endpoint and bearer resolver learn about Pro-shape tokens without
breaking Starter+ legacy paths.
api/_oauth-token.js:
- New resolveBearerToContext(token) returns a discriminated union:
{kind:'env_key', apiKey} -- legacy WORLDMONITOR_VALID_KEYS resolution
{kind:'pro', userId, mcpTokenId} -- new Pro identity
- resolveApiKeyFromBearer kept as backward-compat wrapper. Returns
apiKey for env_key kind, null for pro kind (so legacy callers can't
mis-handle a Pro bearer).
api/oauth/token.js → api/oauth/token.ts:
- Converted to TypeScript so it can import validateProMcpToken from
server/_shared/pro-mcp-token cleanly. Vercel routes by basename;
/oauth/token rewrite unaffected.
- Default handler wires injected deps via tokenHandler(req, deps) for
testability (mirrors U3/U5 pattern).
- authorization_code branch: dispatches on codeData.kind. Pro path
validates client_id + redirect_uri bind, PKCE-verifies, mints tokens
via storeProTokens (object shape). Legacy path unchanged.
- refresh_token branch: Pro refreshes call validateProMcpToken
(per-request Convex hit, no positive cache); revoked row → invalid_grant.
Cross-user defensive guard: Convex-returned userId must match bearer's.
family_id, mcpTokenId, scope ('mcp_pro') preserved across rotation.
- client_credentials grant untouched.
oauth:token:<uuid> shapes:
- Legacy (preserved): JSON.stringify("<sha256-hex-64>") or "<fingerprint-16>"
- Pro (new): JSON.stringify({kind:'pro', userId, mcpTokenId})
- oauth:refresh:<uuid> Pro shape includes client_id, scope, family_id
for rotation discipline (matches legacy semantics).
26 new tests; sibling suites (oauth-authorize, oauth-authorize-pro, mcp,
pro-mcp-token — 101 tests) regress green. tsc -p tsconfig.api.json clean.
Plan unit U6.
* feat(mcp): Pro identity, atomic INCR-first quota, internal HMAC fetch (U7)
Behavioral heart of the Pro MCP plan. api/mcp.ts now resolves bearers
as McpAuthContext (env_key | pro), runs Pro-specific pre-checks, and
enforces a hard 50/UTC-day cap via INCR-first reservation with
DECR-rollback on cap-exceed and on tool-dispatch failure.
Pre-dispatch chain for Pro context (synchronous, in order):
1. validateProMcpToken(mcpTokenId) — null = 401 revoked
2. defensive userId match — 401 cross-user
3. getEntitlements(userId) tier ≥ 1, mcpAccess: true, validUntil — fail-closed
4. per-minute slidingWindow(60, 60s) keyed pro-user:<userId> — fail-open
5. for tools/call ONLY: pipeline INCR + EXPIRE 172800
newCount > 50 → DECR rollback + -32029 + 429 + Retry-After
INCR Redis transient → -32603 + 503 + Retry-After:5 (hard-cap correctness)
6. dispatch tool; on throw → DECR rollback + -32603
Internal-HMAC tool fetches (replaces X-WorldMonitor-Key for Pro):
payload = ${ts}:${METHOD}:${pathname}:${queryHash}:${bodyHash}:${userId}
queryHash = SHA-256(canonicalQueryString) sorted keys, URL-encoded
bodyHash = SHA-256(bodyBytes) SHA-256("") for empty
sig = HMAC-SHA-256(MCP_INTERNAL_HMAC_SECRET, payload)
headers = X-WM-MCP-Internal: ${ts}.${b64url(sig)}, X-WM-MCP-User-Id
Replay defense: payload binds method+path+queryHash+bodyHash, so a
captured signature for /api/news/v1/list-feed-digest cannot be reused
on /api/intelligence/v1/deduct-situation. ±30s window.
server/_shared/mcp-internal-hmac.ts: single source of truth for sign
helpers. U8 verify imports the same canonicalisation primitives.
server/_shared/pro-mcp-token.ts: dailyCounterKey(userId), 50/172800s
constants exported for U9 to read the SAME Redis key the enforcement
writes.
_execute signatures changed (params, base, context) — option A from
the plan. Cache-only tools unchanged.
waitUntil unused: Convex internal-validate-pro-mcp-token schedules
touchProMcpTokenLastUsed itself (verified at convex/http.ts:1035-1040
from U1's commit 4530bdf).
22 new tests + 23 Starter+ regression tests in tests/mcp.test.mjs;
sibling chain (oauth-token, pro-mcp-token, oauth-authorize-pro,
mcp-grant-mint, mcp-proxy) all green. tsc + biome clean.
Plan unit U7.
* feat(gateway): HMAC-verify internal-MCP requests + sanitised propagation (U8)
Verifier counterpart to U7's signer. Gateway accepts X-WM-MCP-Internal
HMAC headers in lieu of an API key for Pro internal-MCP traffic, then
hands a freshly-constructed Request with sanitised trusted markers to
the downstream handler. isCallerPremium learns to honour those markers
so Pro framework/systemAppend semantics survive in summarize-article,
get-country-intel-brief, deduct-situation.
Gateway flow at the top of createDomainGateway's handler:
1. STRIP inbound x-wm-mcp-internal-verified + x-user-id (anti-injection)
2. If X-WM-MCP-Internal present:
verifyInternalMcpRequest re-canonicalises and HMAC-verifies the
request shape (method+pathname+queryHash+bodyHash+userId), ±30s
window, timing-safe compare. Failure → 401 invalid_internal_mcp_
signature (no fall-through to validateApiKey — present-but-bad is
a deliberate forge attempt).
getEntitlements(userId): tier ≥ 1 + mcpAccess === true (fail-closed).
Construct NEW Request via new Request(url, {method, body, headers})
with x-wm-mcp-internal-verified=<per-process nonce> +
x-user-id=<verified userId>. Skip validateApiKey + IP rate limit.
isCallerPremium adds a NEW first branch: timing-safe compare of
x-wm-mcp-internal-verified against the per-process nonce + non-empty
x-user-id + defensive getEntitlements re-fetch. Catches direct-edge-
function consumers (api/widget-agent.ts, chat-analyst.ts, me/entitlement.ts,
v2/shipping/webhooks/...) that don't run the gateway strip step.
DEFENSE-IN-DEPTH BEYOND PLAN: trusted marker is a per-process random
16-byte nonce, NOT the literal '1'. Sweep found direct edge functions
calling isCallerPremium without gateway-strip protection — a constant
marker would be spoofable from outside. The nonce is born once at
process startup, only the gateway knows it, comparison is timing-safe.
verifyInternalMcpRequest lives in server/_shared/mcp-internal-hmac.ts
next to U7's sign helpers — single source of truth for canonical form
+ payload + ±30s window. Drift between sign and verify is structurally
impossible.
26 new tests + 100 sibling regression tests (gateway-cdn-origin-policy,
premium-stock-gateway, premium-fetch, pro-mcp-token, mcp). tsc clean.
Plan unit U8.
* feat(catalog): mcpAccess feature flag + env vars + Pro-MCP docs (U10)
Catalog, schema, type, env, and docs scaffolding to make the Pro MCP
flow deployable end-to-end. U9 unblocks on this — settings UI gates on
hasFeature('mcpAccess') (distinct from existing apiAccess paywall).
- convex/config/productCatalog.ts: PlanFeatures.mcpAccess on every tier.
Free=false. Pro_monthly/Pro_annual=true. API_starter / API_starter_annual
/ API_business=true. Enterprise=true. Pro plan marketing copy mentions
"MCP access for Claude Desktop & other AI clients (50 calls/day)".
- convex/schema.ts + convex/payments/cacheActions.ts: mcpAccess optional
field on the entitlements validator; legacy rows pass.
- server/_shared/entitlement-check.ts: CachedEntitlements.features
mcpAccess?: boolean (consumed by U7 + U8).
- src/services/entitlements.ts: EntitlementState.features.mcpAccess?:
boolean — unblocks hasFeature('mcpAccess') for U9's settings tab gate.
- .env.example: new "Pro MCP" section with MCP_PRO_GRANT_HMAC_SECRET
(apex grant bridge, U3/U5) and MCP_INTERNAL_HMAC_SECRET (gateway
service-auth, U7/U8). Both 32-byte base64; both required.
- docs/mcp-server.mdx: "Pro sign-in flow" section + 50/day quota
callout + "Connected MCP clients" management pointer.
Bundled fixups for U7/U8 strict-mode TS errors:
- mcp-internal-hmac.ts: bufferToBase64Url uses for-of (avoids
noUncheckedIndexedAccess error on bytes[i]).
- gateway.ts: drop unused INTERNAL_MCP_USER_ID_HEADER import.
tsc -p tsconfig.api.json + tsc -p tsconfig.json clean.
9 entitlements tests + 71 gateway-internal-mcp + mcp tests pass.
Plan unit U10.
* feat(settings): Connected MCP clients tab + quota endpoint + revoke (U9)
User-facing surface that makes the Pro MCP plan visible. Settings UI
gets a new "Connected MCP clients" tab gated on hasFeature('mcpAccess')
(distinct from the existing apiAccess-gated 'API Keys' tab — Pro users
without apiAccess see only the new tab).
Endpoints:
- GET /api/user/mcp-quota → {used, limit:50, resetsAt}. Reads the
SAME Redis key (dailyCounterKey from U2) that U7 enforcement writes.
Single source of truth.
- POST /api/user/mcp-revoke {tokenId} → forwards Clerk-derived userId
(NOT client-supplied) to Convex internal-revoke-pro-mcp-token, then
calls invalidateProMcpTokenCache so any in-flight bearer with this
tokenId 401s within 60s. 404 on NOT_FOUND, 409 on ALREADY_REVOKED.
Tenancy enforced inside Convex (row.userId === userId).
UI (in src/components/UnifiedSettings.ts, no child component split —
matches existing API Keys tab pattern):
- Tab gated on hasFeature('mcpAccess') so Pro users see it without
needing apiAccess.
- Quota header auto-refreshes every 30s; interval cleared on tab-
switch / close / destroy.
- Each row: name, createdAt, lastUsedAt (relative), revokedAt
(struck-through + badge); active rows show Revoke button with
confirm() dialog (matches existing pattern).
- Empty state: click-to-copy https://api.worldmonitor.app/mcp.
Frontend service in src/services/mcp-clients.ts: listMcpClients (Convex
query), revokeMcpClient (POST /api/user/mcp-revoke), fetchMcpQuota.
11 + 14 = 25 new tests; api-route-exceptions registered as
internal-helper.
Plan unit U9.
---
PLAN COMPLETE — 10/10 units shipped. Follow-up polish noted in U9
report: CSS rules for .mcp-clients-* classes (suggest copy from
.api-keys-* rules); empty-state URL is hardcoded to
api.worldmonitor.app/mcp.
* style(settings): mcp-clients CSS rules mirroring api-keys (U9 polish)
* fix(pro-mcp): apply Tier-2 code-review residuals — security + correctness
Review pass after U10 (3 reviewers: code-reviewer, ce-security-reviewer,
ce-adversarial-reviewer). Findings BLOCKING + HIGH + MEDIUM + LOW
applied per user direction.
BLOCKING (gateway):
- Add validUntil check to gateway's HMAC-verify entitlement re-check.
Defense-in-depth against Convex-fallback paths returning a stale row
past its expiry.
HIGH — adv-001: cross-user OAuth nonce hijack:
- mcp-grant-mint now claims oauth nonces atomically via Upstash SET NX
semantics. If the nonce is already claimed by a DIFFERENT userId,
return 403 NONCE_CLAIMED_BY_OTHER_USER. Same userId multi-tab retry
is idempotent: re-sign the grant using the existing claim's exp so
authorize-pro's strict tuple equality still matches.
- mcp-grant-context performs the same cross-user check so the apex SPA
refuses to render consent context for a hijacked nonce.
MEDIUM — adv-008: refresh-token loss during Convex transient:
- validateProMcpToken returns a discriminated union {ok:'valid',userId}
| {ok:'revoked'} | {ok:'transient'}. Negative-cache writes only on
'revoked'. validateProMcpTokenOrNull wrapper preserves backward
compat for callers that don't need the distinction (per-request MCP
edge stays fail-closed).
- token.ts refresh path: on 'transient', best-effort restore the
consumed refresh token to Redis with the original TTL and return
503 server_error + Retry-After:5. Client retries; if Convex recovers,
refresh succeeds. No more permanent session loss on a Convex blip.
MEDIUM — adv-002: counter overshoot lockout:
- mcp.ts: on cap-exceeded after DECR rollback, if newCount is still
far above PRO_DAILY_QUOTA_LIMIT (overshoot from prior transient),
pipelined INCR+DECR probe + bounded DECR-sweep to converge the
counter back near the limit. Prevents one Redis hiccup from locking
out a paying Pro user for the rest of the UTC day.
MEDIUM — code-reviewer #4: 5-row cap race in Convex:
- mcpProTokens.issueProMcpToken now revokes ALL active rows beyond
MAX-1 (sorted by createdAt) on each issue, so concurrent inserts
that briefly produce 6 active rows converge back to 5 on the next
call.
LOW:
- adv-003: executeTool throws cache_all_null when every cache read
returns null AND there were keys configured — triggers DECR rollback
in dispatchToolsCall instead of silently burning quota on a degenerate
empty result.
- code-reviewer #10: gateway strips X-WM-MCP-Internal + X-WM-MCP-User-Id
from the trusted Request before forwarding to handler (no info leak).
- adv-004: 256 KB body cap (Content-Length + post-buffer count) on
both gateway strip and HMAC-verify paths — bounds memory amplification.
- adv-006: dailyCounterKey now env-prefixes (preview deploys can't
collide with production traffic on the same Upstash instance).
Reader (mcp-quota.ts) and writer (mcp.ts) share byte-identical keys
via the helper.
- adv-007 / code-reviewer #5: signInternalMcpRequest throws on
Blob/FormData/ReadableStream/plain-object bodies instead of silently
JSON.stringify'ing them (which would produce a hash that can't match
the wire bytes).
- code-reviewer #9: timestamp regex tightened to ^[0-9]{1,15}$ so
future ms-precision timestamps don't silently truncate through Number().
- code-reviewer #8: MCP_INTERNAL_HMAC_SECRET preflight in runProPreChecks
surfaces config errors at auth-resolution rather than mid-tool-fetch.
NITPICK:
- code-reviewer #6: rewritten readNegCache comment.
- code-reviewer #7: malformed-body error now reports reason
'malformed_request' instead of misleading 'auth_401'. New
RequestReason value added to usage.ts enum.
30 new tests + ~14 adapted; full convex suite 247/247, edge/server
suites 254/254. tsc clean across both tsconfig.json and tsconfig.api.json.
Plan: docs/plans/2026-05-10-001-feat-pro-mcp-clerk-auth-quota-plan.md
* chore(pro-marketing): regenerate /pro bundle for U10 catalog copy
U10's productCatalog.ts added an mcpAccess feature flag and updated the
Pro plan description copy to "MCP access for Claude Desktop & other AI
clients (50 calls/day)". scripts/generate-product-config.mjs cascades
that copy into pro-test/src/generated/tiers.json, which the /pro
marketing app builds into public/pro/assets/.
Vercel does NOT rebuild pro-test on deploy — public/pro/ ships whatever
is committed. Pre-push hook auto-ran the build and produced a new bundle
hash; this commit lands the regenerated artifacts so deploy picks them up.
Plan unit U10 (follow-on artifact).
* fix(pro-mcp): apply external review round-2 findings (P1+P2+P3)
Round-2 review on PR koala73#3646 surfaced 5 must-fix issues:
P1 — vercel.json `(?:...)` rejected by Vercel CI:
- Replace `mcp-grant(?:\\.html)?` with explicit `mcp-grant\\.html|mcp-grant`
inside the SPA catch-all negative-lookahead (lines 18 + 134).
- Split the headers `/mcp-grant(?:\\.html)?` source rule into two explicit
entries — Vercel's path-to-regexp `source` field doesn't accept `(?:...)`
with `?` quantifier as a standalone source.
- Update tests/deploy-config.test.mjs expectations to match.
P1 — api/api-route-exceptions.json points at deleted file:
- Update the entry path from `api/oauth/token.js` (deleted in U6) to
`api/oauth/token.ts`. `node scripts/enforce-sebuf-api-contract.mjs`
was failing pre-push CI.
P1 — mcpAccess migration gap on existing entitlement rows:
- convex/entitlements.ts now read-time merges `getFeaturesForPlan(planKey)`
defaults under stored features. Pre-U10 rows lacking `mcpAccess` see the
catalog default surfaced immediately, with NO wait for the next Dodo
webhook to rewrite the row. Stored features still win on conflict
(preserves per-user overrides). Mirrored explicitly in
convex/mcpProTokens.ts:55 (issueProMcpToken's direct ctx.db.query path
computes the same merge inline since it bypasses the query handler).
P2 — grant/issue path missing mcpAccess gate (let tier-1-without-mcpAccess
users complete OAuth then fail every tools/call at the gateway):
- mcp-grant-context.ts:95, mcp-grant-mint.ts:236, authorize-pro.ts:356,
convex/mcpProTokens.ts:55 now ALL gate on `tier ≥ 1 && mcpAccess === true`,
matching the downstream MCP-edge runProPreChecks check. Widens the
`getEntitlements` deps return type at all three handlers.
P3 — inline theme script blocked by global CSP:
- mcp-grant.html:34 inline `<script>` is removed (global CSP at vercel.json:92
is hash-allowlist-based and we don't allowlist this page's hashes).
- Theme application moved into the bundled module at src/mcp-grant-main.ts
(top of file, runs on module evaluation). Brief default-theme flash on
light-preference users is acceptable for a transient consent UI.
10 new tests covering: mcpAccess: false at each of the 4 gates,
mcpAccess: undefined (legacy row) at each of the 4 gates, and the
read-time catalog merge in both directions (legacy gets default,
override preserved).
Full gate post-fix:
- Convex 249/249 (+2 new merge tests).
- Edge/server 280/280 (+10 new gate tests).
- tsc both configs clean.
- sebuf contract clean (was failing before P1.2).
- Sentry coverage clean.
- deploy-config tests assert new explicit-source rules.
Plan: docs/plans/2026-05-10-001-feat-pro-mcp-clerk-auth-quota-plan.md
* fix(entitlement-cache): treat pre-U10 cache entries lacking mcpAccess as stale
Reviewer round-2 P2 (cache layer): _getEntitlementsImpl returned hot
Redis cache entries as-is, bypassing the read-time catalog merge that
convex/entitlements.ts:50 applies on the Convex read path. Paying users
with cache entries written before plan 2026-05-10-001 U10 deployed see
mcpAccess !== true at the grant/MCP gates and are blocked for up to
the 15-min cache TTL.
Cache predicate now also requires `typeof features.mcpAccess === 'boolean'`.
Legacy entries (mcpAccess: undefined) fall through to Convex, which
returns the merged shape and the cache is rewritten with the post-U10
layout. Self-healing, bounded to one extra Convex round-trip per
affected user during the migration window.
Targeted choice over alternatives:
- Importing convex/config/productCatalog into server/_shared to apply
the merge at the cache layer would pull non-edge-safe deps. Rejected.
- Treating ALL cached entries as stale would defeat the cache layer.
Rejected.
- typeof check on the new field is the minimum delta that fixes the
migration window without restructuring.
Test fixture in server/__tests__/entitlement-check.test.ts updated:
makeEntitlements now includes mcpAccess (tier ≥ 1 → true). Two new
focused tests:
1. Legacy cache entry without mcpAccess → cache rejected → Convex
round-trip → result returned (verifies fetch called once).
2. Post-U10 cache entry WITH mcpAccess → cache honored → no Convex
round-trip (verifies fetch called zero times).
Convex 251/251 (+2 new cache tests).
Plan: docs/plans/2026-05-10-001-feat-pro-mcp-clerk-auth-quota-plan.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This introduces the fork-first dataset contract foundation for bootstrap parity. The bootstrap alias map and tier map now come from generated artifacts derived from
registry/datasets.ts, whileapi/health.jsremains unchanged for the follow-up slice.Root cause
Bootstrap cache registration was duplicated across
api/bootstrap.js,server/_shared/cache-keys.ts, and tests. That duplication already drifted once when the generated registry rollout dropped the fourconsumerPrices*aliases from the pre-registry bootstrap set.Changes
registry/datasets.tsas the authored bootstrap contract sourceregistry:generateandregistry:checkapi/_generated/dataset-registry.jsandserver/_shared/_generated/bootstrap-registry.tsapi/bootstrap.jsandserver/_shared/cache-keys.tsto the generated bootstrap registryconsumerPricesOverview,consumerPricesCategories,consumerPricesMovers, andconsumerPricesSpreadbootstrap aliasesValidation
npm run registry:checknode --test tests/bootstrap.test.mjs tests/market-breadth.test.mjsnode --test tests/edge-functions.test.mjsnpm run typechecknpm run typecheck:apiRisk
Low. This is a compile-time and test-time refactor for bootstrap registration only;
api/health.jsstill uses the existing inline health registry in this PR.Refs #5