Skip to content

Commit 34d21b1

Browse files
authored
Merge pull request #1003 from Wibias/feat/ri-01-route-decision-traces
feat(routing): record durable route decision traces
2 parents 79fdaf9 + f6b4bce commit 34d21b1

13 files changed

Lines changed: 1899 additions & 16 deletions

File tree

‎devlog/_plan/260804_router_intelligence/000_master_plan.md‎

Lines changed: 490 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
# 001 - PR stack status ledger
2+
3+
Continuously updated during the programme. Every branch records: base SHA,
4+
head SHA, PR number/URL, verification result, and review state.
5+
6+
## Programme facts
7+
8+
- Stack base (dev): `e44d234f08e03dd4dbf0c4aa13af43046d86b0a6` (`upstream/dev`)
9+
- `origin/dev` (fork, stale ancestor): `be177ea501e5007f4a56d19d069ef5cd76ea24b9`
10+
- Bun: `1.3.14`; package version: `2.10.0`
11+
- Worktree: `D:\codex-worktrees\ocx-router-intelligence`
12+
- Push remote: `origin` (Wibias/opencodex); PR target: `lidge-jun/opencodex:dev`
13+
- All PRs opened as DRAFT; nothing merged by this programme.
14+
15+
## Related in-flight PRs (not superseded by this stack)
16+
17+
| PR | Branch | Note |
18+
|---|---|---|
19+
| #922 | `fix/914-account-neutral-network` | #914 alternative; consumed as health evidence input by RI-06 |
20+
| #966 | `codex/260804-issue914-transport-attribution` | #914 alternative; consumed as health evidence input by RI-06 |
21+
| #715 | `feat/priority-levels` | Pool selection order; out of scope |
22+
| #988 | `codex/providers-copy-doctor` | GUI providers/combos; conflict-checked at RI-10 |
23+
| #998 | `codex/260803-integration-switches` | Write substrate; rebase watch on request-log.ts |
24+
25+
No open PR found that implements the same vertical as any PR in this stack,
26+
so no stale PR is closed by this programme. Both #914 drafts overlap each
27+
other; closing one is a maintainer decision and neither is stale.
28+
29+
## Baseline
30+
31+
- Full-suite baseline on clean `upstream/dev` (worktree
32+
`D:\codex-worktrees\ocx-typecheck-base`, head `e44d234f0`): running in
33+
background; exact pass/fail counts appended here when done.
34+
- `bun x tsc --noEmit` on clean `upstream/dev`: **PASSED** (0 errors, verified
35+
in the pristine base worktree).
36+
- `bun run privacy:scan`: passed per-PR (see RI-01 below).
37+
38+
## Stack status
39+
40+
| RI | Branch | Base | Head SHA | PR | URL | Status |
41+
|---|---|---|---|---|---|---|
42+
| RI-01 | `feat/ri-01-route-decision-traces` | `e44d234f0` | pending | pending | pending | in progress |
43+
| RI-02 | `feat/ri-02-request-history-index` | `feat/ri-01` head | pending | pending | pending | queued |
44+
| RI-03 | `feat/ri-03-routing-analytics` | `feat/ri-02` head | pending | pending | pending | queued |
45+
| RI-04 | `feat/ri-04-policy-profile-core` | `feat/ri-03` head | pending | pending | pending | queued |
46+
| RI-05 | `feat/ri-05-capability-aware-routing` | `feat/ri-04` head | pending | pending | pending | queued |
47+
| RI-06 | `feat/ri-06-health-aware-routing` | `feat/ri-05` head | pending | pending | pending | queued |
48+
| RI-07 | `feat/ri-07-quota-aware-routing` | `feat/ri-06` head | pending | pending | pending | queued |
49+
| RI-08 | `feat/ri-08-cost-aware-routing` | `feat/ri-07` head | pending | pending | pending | queued |
50+
| RI-09 | `feat/ri-09-route-explainability-api` | `feat/ri-08` head | pending | pending | pending | queued |
51+
| RI-10 | `feat/ri-10-routing-intelligence-ui` | `feat/ri-09` head | pending | pending | pending | queued |
52+
53+
## Per-PR acceptance log
54+
55+
### RI-01 - feat/ri-01-route-decision-traces
56+
57+
- Base SHA: `e44d234f08e03dd4dbf0c4aa13af43046d86b0a6`
58+
- Reviewed commits:
59+
- `b5a8e7c4c` (implementation; author self-review + CodeRabbit review)
60+
- `2e0522b2` (privacy-scan fix after CI `gates` failure)
61+
- `pending` (CodeRabbit findings round; recorded after commit)
62+
- Findings (self-review): 3 test failures caught pre-push - (1) missing value
63+
import for `normalizeRouteDecisionTrace` in request-log hydration,
64+
(2) selected combo target marked ineligible because `ComboPick.attempted`
65+
includes the winner, (3) account-namespace fixture missing the canonical
66+
ChatGPT forward `baseUrl` (test-fixture bug, not product code).
67+
- Fixes: import fixed; `comboRouteCandidates` now excludes the selected target
68+
from `already-attempted`; fixture uses `https://chatgpt.com/backend-api/codex`.
69+
- Regression tests: all three cases are covered by the final
70+
`tests/route-decision-trace.test.ts` (14 tests, 75 assertions).
71+
- Findings (CodeRabbit, verified against code): 12 comments - 9 accepted
72+
(locale/plan docs, requestedModel bound doc, ledger SHA, combo tieBreak +
73+
duplicate getCombo, `truncated.requirements` flag, byte-accurate budget,
74+
parse-once evidence, hydration guard drops invalid traces, 2 regression
75+
tests, credential-test assertion hardening); 2 design-judgment comments
76+
(persist trace on every row - kept: bounded ~200 B single-candidate traces,
77+
plan mandates one trace per decision; docstring coverage - docstrings
78+
added to trace helpers); the privacy-scan finding was already fixed in
79+
`2e0522b2`.
80+
- Final commit: recorded after commit (round applies CodeRabbit + simplify
81+
fixes; new head pushes to #1003)
82+
- Verification:
83+
- `bun x tsc --noEmit`: PASSED (0 errors)
84+
- `bun run test tests/route-decision-trace.test.ts`: 14/14 pass
85+
- Focused regression suites: 253/253 pass across combos, codex-routing,
86+
usage-log, request-log, combo-management-api, codex-account-namespaces
87+
- `tests/server-combo-failover-e2e.test.ts`: 44/44 pass
88+
- `bun run privacy:scan`: passed
89+
- Remaining Low findings: none
90+
91+
### RI-02..RI-10
92+
93+
Appended as each PR is implemented.

‎src/router.ts‎

Lines changed: 139 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,13 @@
11
import type { CodexAccountMode, OcxConfig, OcxProviderConfig } from "./types";
2-
import { preservesPhysicalComboProvider, tryPickComboModel, type ComboPick } from "./combos";
2+
import {
3+
getCombo,
4+
isComboTargetInCooldown,
5+
preservesPhysicalComboProvider,
6+
targetKey,
7+
tryPickComboModel,
8+
type ComboPick,
9+
} from "./combos";
10+
import type { NormalizedComboConfig } from "./combos/types";
311
import { hasOwnProvider, resolveEnvValue } from "./config";
412
import { assertProviderDestinationAllowed } from "./lib/destination-policy";
513
import { redactSecretString, redactUrlForLog } from "./lib/redact";
@@ -14,17 +22,29 @@ import {
1422
import { decodeRoutedModelId, encodeRoutedModelId } from "./providers/slug-codec";
1523
import { getStaleCached } from "./codex/model-cache";
1624
import { codexAccountNamespaceEntries } from "./codex/account-namespaces";
25+
import {
26+
buildRouteDecisionTrace,
27+
type RouteDecisionKind,
28+
type RouteDecisionTraceV1,
29+
type TraceCandidateInput,
30+
} from "./routing/trace";
1731

1832
export interface RouteResult {
1933
providerName: string;
2034
provider: OcxProviderConfig;
2135
modelId: string;
36+
/** Which deterministic routing path produced this route (RI-01). */
37+
routeKind: RouteDecisionKind;
38+
/** Stable wire reason code for the selected route (RI-01). */
39+
routeReason: string;
2240
codexAccountMode?: CodexAccountMode;
2341
/** Exact account selected by an account-qualified native model. */
2442
codexAccountId?: string;
2543
/** Public namespace used by the account-qualified selector. */
2644
codexAccountNamespace?: string;
2745
combo?: ComboPick;
46+
/** Bounded route-decision trace (RI-01); never contains secrets. */
47+
routeDecision?: RouteDecisionTraceV1;
2848
}
2949

3050
const MODEL_PROVIDER_PATTERNS: Array<{ providerNames: string[]; prefixes: string[] }> = [
@@ -335,6 +355,34 @@ export class NoEnabledOpenAiProviderError extends Error {
335355
}
336356
}
337357

358+
/**
359+
* One immutable selection trace for a combo request: built once from the
360+
* initial pick, before any child dispatch. Fallback execution stays in the
361+
* usage entry's `attempts[]`; the trace never changes after selection.
362+
*/
363+
export function comboRouteDecisionTrace(
364+
config: OcxConfig,
365+
comboId: string,
366+
pick: ComboPick,
367+
requestedModel: string,
368+
): RouteDecisionTraceV1 {
369+
const combo = getCombo(config, comboId);
370+
return buildRouteDecisionTrace({
371+
requestedModel,
372+
routeKind: "combo",
373+
selected: {
374+
provider: pick.target.provider,
375+
model: pick.target.model,
376+
reason: "combo-pick",
377+
candidateIndex: pick.targetIndex,
378+
...(combo
379+
? { tieBreak: combo.strategy === "round-robin" ? "round-robin" : "failover" }
380+
: {}),
381+
},
382+
candidates: combo ? comboRouteCandidates(config, pick, combo) : undefined,
383+
});
384+
}
385+
338386
// Codex uses a small number of control-plane model ids that are not part of the public GPT/o
339387
// naming families. Keep this exact: a broad `codex-*` rule could capture a third-party model.
340388
const CODEX_INTERNAL_OPENAI_MODELS = new Set(["codex-auto-review"]);
@@ -344,16 +392,63 @@ function isBareOpenAiFamilyModel(modelId: string): boolean {
344392
&& (/^(?:gpt-|o1-|o3-|o4-)/.test(modelId) || CODEX_INTERNAL_OPENAI_MODELS.has(modelId));
345393
}
346394

347-
function routeResult(providerName: string, provider: OcxProviderConfig, modelId: string): RouteResult {
395+
function routeResult(
396+
providerName: string,
397+
provider: OcxProviderConfig,
398+
modelId: string,
399+
routeKind: RouteDecisionKind,
400+
routeReason: string,
401+
): RouteResult {
348402
const codexAccountMode = providerCodexAccountMode(providerName, provider);
349403
return {
350404
providerName,
351405
provider: routedProviderConfig(providerName, provider),
352406
modelId,
407+
routeKind,
408+
routeReason,
353409
...(codexAccountMode ? { codexAccountMode } : {}),
354410
};
355411
}
356412

413+
/**
414+
* Candidate evidence for a combo route: every configured target with its
415+
* selection-time eligibility and exclusion reasons. Purely observational; the
416+
* pick already happened and this never re-selects.
417+
*/
418+
function comboRouteCandidates(
419+
config: OcxConfig,
420+
pick: NonNullable<RouteResult["combo"]>,
421+
combo: NormalizedComboConfig,
422+
): TraceCandidateInput[] {
423+
const now = Date.now();
424+
return combo.targets.map((target, index) => {
425+
const key = targetKey(target);
426+
const provider = config.providers[target.provider];
427+
const configured = provider !== undefined;
428+
const enabled = configured && provider.disabled !== true;
429+
const inCooldown = isComboTargetInCooldown(pick.comboId, target, now);
430+
const isSelected = index === pick.targetIndex;
431+
// The pick's `attempted` list includes the winner itself; only non-selected
432+
// targets can be "already-attempted" (fallback picks exclude earlier tries).
433+
const alreadyAttempted = !isSelected && pick.attempted.includes(key);
434+
const exclusions: TraceCandidateInput["exclusions"] = [];
435+
if (!configured) exclusions.push({ code: "unconfigured" });
436+
if (configured && !enabled) exclusions.push({ code: "disabled" });
437+
if (inCooldown) exclusions.push({ code: "cooldown" });
438+
if (isSelected && inCooldown) exclusions.push({ code: "selected-despite-cooldown" });
439+
if (!isSelected && alreadyAttempted && exclusions.length === 0) {
440+
exclusions.push({ code: "already-attempted" });
441+
}
442+
if (!isSelected && exclusions.length === 0) exclusions.push({ code: "not-selected" });
443+
return {
444+
provider: target.provider,
445+
model: target.model,
446+
eligible: enabled && !inCooldown && !alreadyAttempted,
447+
exclusions,
448+
};
449+
});
450+
}
451+
357452
function routeModelInternal(config: OcxConfig, modelId: string, bypassCombos: boolean): RouteResult {
358453
const slash = modelId.indexOf("/");
359454
if (slash > 0) {
@@ -378,7 +473,7 @@ function routeModelInternal(config: OcxConfig, modelId: string, bypassCombos: bo
378473
throw new NoEnabledOpenAiProviderError(nativeModelId);
379474
}
380475
return {
381-
...routeResult(OPENAI_CODEX_PROVIDER_ID, provider, nativeModelId),
476+
...routeResult(OPENAI_CODEX_PROVIDER_ID, provider, nativeModelId, "explicit-account", "account-namespace"),
382477
// Exact account injection uses the pool credential machinery even when the canonical
383478
// provider is globally Direct. The fixed id bypasses pool selection entirely.
384479
codexAccountMode: "pool",
@@ -395,7 +490,7 @@ function routeModelInternal(config: OcxConfig, modelId: string, bypassCombos: bo
395490
// The selected target is already a concrete provider/model reference. Resolve it without
396491
// consulting combo aliases again, otherwise an alias that shadows the target can recurse.
397492
const routed = routeModelInternal(config, concrete, true);
398-
return { ...routed, combo };
493+
return { ...routed, combo, routeKind: "combo" as const, routeReason: "combo-pick" };
399494
}
400495
}
401496

@@ -415,23 +510,33 @@ function routeModelInternal(config: OcxConfig, modelId: string, bypassCombos: bo
415510
// Self-namespaced native id — the vendor segment equals the provider id, so the FULL ref is
416511
// itself a known model (e.g. orcarouter/auto). Route it whole instead of stripping to the
417512
// remainder, which would send a bare `auto` the upstream cannot resolve.
418-
if (known.includes(modelId)) return routeResult(provName, prov, modelId);
513+
if (known.includes(modelId)) {
514+
return routeResult(provName, prov, modelId, "explicit-provider", "explicit-provider-namespace");
515+
}
419516
// Codex-facing alias ids (`provider/vendor-model`) decode back to the native
420517
// slash id via an exact known-id lookup; raw full-slash selectors keep working.
421-
return routeResult(provName, prov, decodeRoutedModelId(modelId.slice(slash + 1), known));
518+
return routeResult(
519+
provName,
520+
prov,
521+
decodeRoutedModelId(modelId.slice(slash + 1), known),
522+
"explicit-provider",
523+
"explicit-provider-namespace",
524+
);
422525
}
423526
}
424527

425528
if (isBareOpenAiFamilyModel(modelId)) {
426529
const provider = config.providers[OPENAI_CODEX_PROVIDER_ID];
427-
if (provider && provider.disabled !== true) return routeResult(OPENAI_CODEX_PROVIDER_ID, provider, modelId);
530+
if (provider && provider.disabled !== true) {
531+
return routeResult(OPENAI_CODEX_PROVIDER_ID, provider, modelId, "native", "native-family");
532+
}
428533
throw new NoEnabledOpenAiProviderError(modelId);
429534
}
430535

431536
for (const [provName, prov] of activeProviderEntries(config)) {
432537
if (prov.defaultModel === modelId
433538
|| (typeof prov.defaultModel === "string" && encodeRoutedModelId(prov.defaultModel) === modelId)) {
434-
return routeResult(provName, prov, prov.defaultModel as string);
539+
return routeResult(provName, prov, prov.defaultModel as string, "explicit-provider", "configured-default-model");
435540
}
436541
}
437542

@@ -441,7 +546,9 @@ function routeModelInternal(config: OcxConfig, modelId: string, bypassCombos: bo
441546
for (const [provName, prov] of activeProviderEntries(config)) {
442547
if (prov.models && Array.isArray(prov.models)) {
443548
const hit = (prov.models as string[]).find(id => id === modelId || encodeRoutedModelId(id) === modelId);
444-
if (hit !== undefined) return routeResult(provName, prov, hit);
549+
if (hit !== undefined) {
550+
return routeResult(provName, prov, hit, "explicit-provider", "configured-model-list");
551+
}
445552
}
446553
}
447554

@@ -451,14 +558,34 @@ function routeModelInternal(config: OcxConfig, modelId: string, bypassCombos: bo
451558
if (hasOwnProvider(config.providers, config.defaultProvider)) {
452559
const defaultProv = config.providers[config.defaultProvider];
453560
if (defaultProv.disabled === true) throw new Error(`Default provider is disabled: ${config.defaultProvider}`);
454-
return routeResult(config.defaultProvider, defaultProv, modelId);
561+
return routeResult(config.defaultProvider, defaultProv, modelId, "default-provider", "default-provider");
455562
}
456563

457564
throw new Error(`No provider configured for model: ${modelId}`);
458565
}
459566

460567
export function routeModel(config: OcxConfig, modelId: string): RouteResult {
461-
return routeModelInternal(config, modelId, false);
568+
const route = routeModelInternal(config, modelId, false);
569+
const accountRef = route.codexAccountNamespace;
570+
const combo = route.combo ? getCombo(config, route.combo.comboId) : undefined;
571+
route.routeDecision = buildRouteDecisionTrace({
572+
requestedModel: modelId,
573+
routeKind: route.routeKind,
574+
selected: {
575+
provider: route.providerName,
576+
model: route.modelId,
577+
...(accountRef ? { accountRef } : {}),
578+
reason: route.routeReason,
579+
...(route.combo ? { candidateIndex: route.combo.targetIndex } : {}),
580+
...(combo
581+
? { tieBreak: combo.strategy === "round-robin" ? "round-robin" : "failover" }
582+
: {}),
583+
},
584+
candidates: route.routeKind === "combo" && route.combo && combo
585+
? comboRouteCandidates(config, route.combo, combo)
586+
: undefined,
587+
});
588+
return route;
462589
}
463590

464591
function routeByKnownModelPattern(config: OcxConfig, modelId: string): RouteResult | undefined {
@@ -469,7 +596,7 @@ function routeByKnownModelPattern(config: OcxConfig, modelId: string): RouteResu
469596
);
470597
if (matchingProvider) {
471598
const [provName, prov] = matchingProvider;
472-
return routeResult(provName, prov, modelId);
599+
return routeResult(provName, prov, modelId, "explicit-provider", "model-pattern");
473600
}
474601
}
475602
}

0 commit comments

Comments
 (0)