Skip to content

Commit 4f32280

Browse files
authored
Merge pull request #1404 from lidge-jun/fix/1345-cost-cap-contract
fix(routing): complete unknown cost-cap operator contract (#1181)
2 parents 2036ed7 + 01db57c commit 4f32280

17 files changed

Lines changed: 711 additions & 19 deletions

File tree

‎docs-site/src/content/docs/guides/routing-profile-editor.md‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ The **Models → Routing** tab in the OpenCodex dashboard can manage `config.rou
1111
2. Select **Create profile**.
1212
3. Enter an `id`. The canonical model id is `policy/<id>`.
1313
4. Add one or more explicit provider/model candidates.
14-
5. Configure optional requirements, scoring weights, cost limits, and unknown-evidence behavior.
14+
5. Configure optional requirements, scoring weights, cost limits (`maxEstimatedCostUsd`, optional `onUnknownCost`), and unknown-evidence behavior.
1515
6. Save the profile.
1616

1717
Profile ids are immutable after creation. To use a different id, create a new profile and remove the old one after updating callers.
@@ -28,6 +28,14 @@ The dashboard sends the same profile object used by `config.routingProfiles` to
2828

2929
A successful save persists the profile through the normal config writer, reconciles live state, and refreshes the model catalog. Validation failures leave the previous configuration unchanged and are shown in the editor.
3030

31+
When `limits.maxEstimatedCostUsd` is configured, `limits.onUnknownCost` defaults to `"allow"`: an unknown cost estimate does not get a
32+
cap-specific exclusion, and dry-run / live route-decision traces stamp
33+
`cost.capOutcome: "unknown-allowed"` so operators can tell the cap was not proven. Set `"exclude"`
34+
when the ceiling must fail closed (`cost-limit-unknown`, with
35+
`cost.capOutcome: "unknown-excluded"`). Configuring `onUnknownCost` alone is inert and does not emit a cap outcome. This is separate from
36+
`unknownEvidence.cost`, which can still exclude or penalize unknown prices independently of the
37+
cap outcome.
38+
3139
## Dry-run a saved profile
3240

3341
Select a saved profile and use **Dry-run evaluation** to add request evidence such as context-window size, tool use, image input, or structured output. Dry-run evaluates eligibility and scoring but never sends an upstream model request.
@@ -57,7 +65,7 @@ Example save payload:
5765
],
5866
"require": { "tools": true, "minContextWindow": 128000 },
5967
"optimize": { "latency": 0.55, "health": 0.25, "cost": 0.1, "quota": 0.1 },
60-
"limits": { "maxEstimatedCostUsd": 0.5 },
68+
"limits": { "maxEstimatedCostUsd": 0.5, "onUnknownCost": "allow" },
6169
"unknownEvidence": {
6270
"capability": "exclude",
6371
"health": "penalize",

‎docs-site/src/content/docs/reference/configuration/routing.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,7 @@ namespace, or reserved bare native families (`gpt-*`, `o1-*`, `o3-*`, `o4-*`, `c
114114
| `alias?` | `string` | — | Optional public model id in place of `policy/<id>`. |
115115
| `require?` | object | `{}` | Hard capability requirements evaluated before scoring (see below). |
116116
| `optimize?` | object | latency 0.55, health 0.25, cost 0.10, quota 0.10 | Scoring weights, normalized deterministically. `health`, `quota`, and `cost` have score dimensions; the configured-priority share is `1 - health - quota - cost` (default 0.55), and `latency` folds into that priority share rather than scoring independently. |
117-
| `limits?` | object | — | Hard limits, e.g. `maxEstimatedCostUsd` (enforced by the dry-run evaluator when candidate cost evidence is known). |
117+
| `limits?` | object | — | Hard limits. `maxEstimatedCostUsd` excludes a candidate when its estimated cost is known and above the cap. When that cap is set, `onUnknownCost` (`"allow"` default, or `"exclude"`) controls unknown estimates: allow prevents a cap-specific exclusion and records `cost.capOutcome: "unknown-allowed"`; exclude emits `cost-limit-unknown` and `capOutcome: "unknown-excluded"`. `onUnknownCost` alone (no cap) is inert. Separate from `unknownEvidence.cost`, which can still exclude or penalize unknown prices via `unknown-price` / scoring. |
118118
| `unknownEvidence?` | object | capability `exclude`, health/quota/cost `penalize` | How unknown evidence is treated per dimension: `allow`, `penalize`, or `exclude`. Unknown never becomes zero. |
119119

120120
`require` supports: `minContextWindow` (positive integer), `minQuotaHeadroom` (0..1 fraction),
@@ -145,7 +145,7 @@ candidate evidence is provided through the API (`POST /api/routing-profiles/dry-
145145
],
146146
"require": { "tools": true, "minContextWindow": 128000 },
147147
"optimize": { "latency": 0.55, "health": 0.25, "cost": 0.10, "quota": 0.10 },
148-
"limits": { "maxEstimatedCostUsd": 0.50 },
148+
"limits": { "maxEstimatedCostUsd": 0.50, "onUnknownCost": "allow" },
149149
"unknownEvidence": {
150150
"capability": "exclude",
151151
"health": "penalize",
@@ -185,8 +185,12 @@ Both are virtual namespaces with aliases and collision validation; they differ i
185185
is chosen. Profile scoring combines the configured-priority component with the health (RI-06),
186186
quota (RI-07), and cost (RI-08) score dimensions where evidence is present; the `latency` weight
187187
folds into the priority share rather than scoring independently. Cost is also enforced through the
188-
`limits.maxEstimatedCostUsd` cap (a candidate whose estimated cost exceeds the cap is excluded).
189-
Per-request route-decision traces are recorded when a policy profile executes.
188+
`limits.maxEstimatedCostUsd` cap: a candidate whose estimated cost is known and exceeds the cap is
189+
excluded (`cost-limit`). When a cap is configured and the estimate is unknown, the default `limits.onUnknownCost: "allow"`
190+
records `cost.capOutcome: "unknown-allowed"` on the route-decision trace without a cap exclusion;
191+
set `onUnknownCost: "exclude"` for a fail-closed ceiling (`cost-limit-unknown`). Cap outcome is not
192+
overall eligibility — `unknownEvidence.cost: "exclude"` can still add `unknown-price` and mark the
193+
candidate ineligible. Per-request route-decision traces are recorded when a policy profile executes.
190194

191195
### Catalog eligibility
192196

‎gui/src/i18n/de.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,20 @@ export const de: Record<TKey, string> = {
4646
"routing.candidate": "Kandidat",
4747
"routing.eligible": "Geeignet",
4848
"routing.exclusions": "Ausschlüsse",
49+
"routing.costCap": "Kostenobergrenze",
50+
"routing.capOutcome.satisfied": "innerhalb des Limits",
51+
"routing.capOutcome.exceeded": "über dem Limit",
52+
"routing.capOutcome.unknown-allowed": "unbekannt (erlaubt)",
53+
"routing.capOutcome.unknown-excluded": "unbekannt (ausgeschlossen)",
54+
"routing.exclusion.capability-unsatisfied": "Anforderung nicht erfüllt",
55+
"routing.exclusion.unknown-capability": "unbekannte Fähigkeit",
56+
"routing.exclusion.cost-limit": "über Kostenobergrenze",
57+
"routing.exclusion.cost-limit-unknown": "unbekannte Kosten unter Obergrenze",
58+
"routing.exclusion.cooldown": "Abklingzeit",
59+
"routing.exclusion.unknown-health": "unbekannter Zustand",
60+
"routing.exclusion.unknown-quota": "unbekanntes Kontingent",
61+
"routing.exclusion.unknown-price": "unbekannter Preis",
62+
"routing.exclusion.other": "Ausschluss: {code}",
4963
"routing.score": "Punktzahl",
5064
"routing.selected": "ausgewählt",
5165
"routing.yes": "ja",

‎gui/src/i18n/en.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,20 @@ export const en = {
8989
"routing.candidate": "Candidate",
9090
"routing.eligible": "Eligible",
9191
"routing.exclusions": "Exclusions",
92+
"routing.costCap": "Cost cap",
93+
"routing.capOutcome.satisfied": "within limit",
94+
"routing.capOutcome.exceeded": "over limit",
95+
"routing.capOutcome.unknown-allowed": "unknown (allowed)",
96+
"routing.capOutcome.unknown-excluded": "unknown (excluded)",
97+
"routing.exclusion.capability-unsatisfied": "capability not met",
98+
"routing.exclusion.unknown-capability": "unknown capability",
99+
"routing.exclusion.cost-limit": "over cost cap",
100+
"routing.exclusion.cost-limit-unknown": "unknown cost under cap",
101+
"routing.exclusion.cooldown": "cooldown",
102+
"routing.exclusion.unknown-health": "unknown health",
103+
"routing.exclusion.unknown-quota": "unknown quota",
104+
"routing.exclusion.unknown-price": "unknown price",
105+
"routing.exclusion.other": "exclusion: {code}",
92106
"routing.score": "Score",
93107
"routing.selected": "selected",
94108
"routing.yes": "yes",

‎gui/src/i18n/ja.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,20 @@ export const ja: Record<TKey, string> = {
4646
"routing.candidate": "候補",
4747
"routing.eligible": "対象",
4848
"routing.exclusions": "除外",
49+
"routing.costCap": "コスト上限",
50+
"routing.capOutcome.satisfied": "上限内",
51+
"routing.capOutcome.exceeded": "上限超過",
52+
"routing.capOutcome.unknown-allowed": "不明(許可)",
53+
"routing.capOutcome.unknown-excluded": "不明(除外)",
54+
"routing.exclusion.capability-unsatisfied": "能力要件未達",
55+
"routing.exclusion.unknown-capability": "能力不明",
56+
"routing.exclusion.cost-limit": "コスト上限超過",
57+
"routing.exclusion.cost-limit-unknown": "上限下でコスト不明",
58+
"routing.exclusion.cooldown": "クールダウン",
59+
"routing.exclusion.unknown-health": "健全性不明",
60+
"routing.exclusion.unknown-quota": "割当不明",
61+
"routing.exclusion.unknown-price": "価格不明",
62+
"routing.exclusion.other": "除外: {code}",
4963
"routing.score": "スコア",
5064
"routing.selected": "選択済み",
5165
"routing.yes": "はい",

‎gui/src/i18n/ko.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,20 @@ export const ko: Record<TKey, string> = {
4646
"routing.candidate": "후보",
4747
"routing.eligible": "적격",
4848
"routing.exclusions": "제외",
49+
"routing.costCap": "비용 상한",
50+
"routing.capOutcome.satisfied": "한도 이내",
51+
"routing.capOutcome.exceeded": "한도 초과",
52+
"routing.capOutcome.unknown-allowed": "알 수 없음(허용)",
53+
"routing.capOutcome.unknown-excluded": "알 수 없음(제외)",
54+
"routing.exclusion.capability-unsatisfied": "기능 미충족",
55+
"routing.exclusion.unknown-capability": "알 수 없는 기능",
56+
"routing.exclusion.cost-limit": "비용 상한 초과",
57+
"routing.exclusion.cost-limit-unknown": "상한 이하 비용 불명",
58+
"routing.exclusion.cooldown": "쿨다운",
59+
"routing.exclusion.unknown-health": "상태 불명",
60+
"routing.exclusion.unknown-quota": "할당량 불명",
61+
"routing.exclusion.unknown-price": "가격 불명",
62+
"routing.exclusion.other": "제외: {code}",
4963
"routing.score": "점수",
5064
"routing.selected": "선택됨",
5165
"routing.yes": "예",

‎gui/src/i18n/ru.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,20 @@ export const ru: Record<TKey, string> = {
4646
"routing.candidate": "Кандидат",
4747
"routing.eligible": "Допустим",
4848
"routing.exclusions": "Исключения",
49+
"routing.costCap": "Лимит стоимости",
50+
"routing.capOutcome.satisfied": "в пределах лимита",
51+
"routing.capOutcome.exceeded": "сверх лимита",
52+
"routing.capOutcome.unknown-allowed": "неизвестно (разрешено)",
53+
"routing.capOutcome.unknown-excluded": "неизвестно (исключено)",
54+
"routing.exclusion.capability-unsatisfied": "требование не выполнено",
55+
"routing.exclusion.unknown-capability": "неизвестная возможность",
56+
"routing.exclusion.cost-limit": "сверх лимита стоимости",
57+
"routing.exclusion.cost-limit-unknown": "неизвестная стоимость при лимите",
58+
"routing.exclusion.cooldown": "пауза",
59+
"routing.exclusion.unknown-health": "неизвестное состояние",
60+
"routing.exclusion.unknown-quota": "неизвестная квота",
61+
"routing.exclusion.unknown-price": "неизвестная цена",
62+
"routing.exclusion.other": "исключение: {code}",
4963
"routing.score": "Оценка",
5064
"routing.selected": "выбран",
5165
"routing.yes": "да",

‎gui/src/i18n/tr.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,20 @@ export const tr: Record<TKey, string> = {
8888
"routing.candidate": "Aday",
8989
"routing.eligible": "Uygun",
9090
"routing.exclusions": "Hariç Tutulanlar",
91+
"routing.costCap": "Maliyet tavanı",
92+
"routing.capOutcome.satisfied": "limit içinde",
93+
"routing.capOutcome.exceeded": "limit aşıldı",
94+
"routing.capOutcome.unknown-allowed": "bilinmiyor (izinli)",
95+
"routing.capOutcome.unknown-excluded": "bilinmiyor (hariç tutuldu)",
96+
"routing.exclusion.capability-unsatisfied": "yetenek karşılanmadı",
97+
"routing.exclusion.unknown-capability": "bilinmeyen yetenek",
98+
"routing.exclusion.cost-limit": "maliyet tavanı aşıldı",
99+
"routing.exclusion.cost-limit-unknown": "tavan altında bilinmeyen maliyet",
100+
"routing.exclusion.cooldown": "soğuma süresi",
101+
"routing.exclusion.unknown-health": "bilinmeyen sağlık",
102+
"routing.exclusion.unknown-quota": "bilinmeyen kota",
103+
"routing.exclusion.unknown-price": "bilinmeyen fiyat",
104+
"routing.exclusion.other": "hariç tutma: {code}",
91105
"routing.score": "Puan",
92106
"routing.selected": "seçildi",
93107
"routing.yes": "evet",

‎gui/src/i18n/zh.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,20 @@ export const zh: Record<TKey, string> = {
4646
"routing.candidate": "候选",
4747
"routing.eligible": "合格",
4848
"routing.exclusions": "排除项",
49+
"routing.costCap": "成本上限",
50+
"routing.capOutcome.satisfied": "未超限",
51+
"routing.capOutcome.exceeded": "已超限",
52+
"routing.capOutcome.unknown-allowed": "未知(允许)",
53+
"routing.capOutcome.unknown-excluded": "未知(排除)",
54+
"routing.exclusion.capability-unsatisfied": "能力未满足",
55+
"routing.exclusion.unknown-capability": "能力未知",
56+
"routing.exclusion.cost-limit": "超出成本上限",
57+
"routing.exclusion.cost-limit-unknown": "上限下成本未知",
58+
"routing.exclusion.cooldown": "冷却中",
59+
"routing.exclusion.unknown-health": "健康状态未知",
60+
"routing.exclusion.unknown-quota": "配额未知",
61+
"routing.exclusion.unknown-price": "价格未知",
62+
"routing.exclusion.other": "排除项:{code}",
4963
"routing.score": "分数",
5064
"routing.selected": "已选择",
5165
"routing.yes": "是",

‎gui/src/pages/RoutingProfiles.tsx‎

Lines changed: 67 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ import {
1111
type OptionalBoolean,
1212
type RoutingProfileDraft,
1313
type RoutingProfileDto,
14+
type UnknownCostCapMode,
1415
type UnknownEvidenceMode,
1516
} from "../routing-profile-editor-data";
1617
import { readJsonIfOk } from "../fetch-json";
@@ -23,6 +24,7 @@ type DryRunCandidate = {
2324
eligible: boolean;
2425
exclusions: Array<{ code: string; detail?: string }>;
2526
score?: { total: number; components: Record<string, number | undefined> };
27+
cost?: { capOutcome?: string; estimatedUsd?: number; incomplete?: boolean; limitUsd?: number };
2628
};
2729

2830
type Analytics = {
@@ -69,6 +71,7 @@ const NUMERIC_REQUIREMENTS = Object.keys(NUMERIC_REQUIREMENT_SPEC) as Array<keyo
6971
const OPTIMIZE_KEYS = ["latency", "health", "cost", "quota"] as const;
7072
const UNKNOWN_EVIDENCE_KEYS = ["capability", "health", "quota", "cost"] as const;
7173
const UNKNOWN_EVIDENCE_OPTIONS: UnknownEvidenceMode[] = ["allow", "penalize", "exclude"];
74+
const UNKNOWN_COST_CAP_OPTIONS: UnknownCostCapMode[] = ["allow", "exclude"];
7275

7376
function fmtMs(value: number | undefined, unavailable: string): string {
7477
return value === undefined ? unavailable : `${Math.round(value)}ms`;
@@ -78,6 +81,48 @@ function fmtRate(value: number | null | undefined, unavailable: string): string
7881
return value === null || value === undefined ? unavailable : `${Math.round(value * 100)}%`;
7982
}
8083

84+
function fmtCapOutcome(
85+
value: string | undefined,
86+
t: ReturnType<typeof useT>,
87+
unavailable: string,
88+
): string {
89+
switch (value) {
90+
case "satisfied":
91+
return t("routing.capOutcome.satisfied");
92+
case "exceeded":
93+
return t("routing.capOutcome.exceeded");
94+
case "unknown-allowed":
95+
return t("routing.capOutcome.unknown-allowed");
96+
case "unknown-excluded":
97+
return t("routing.capOutcome.unknown-excluded");
98+
default:
99+
return unavailable;
100+
}
101+
}
102+
103+
function fmtExclusion(code: string, t: ReturnType<typeof useT>): string {
104+
switch (code) {
105+
case "capability-unsatisfied":
106+
return t("routing.exclusion.capability-unsatisfied");
107+
case "unknown-capability":
108+
return t("routing.exclusion.unknown-capability");
109+
case "cost-limit":
110+
return t("routing.exclusion.cost-limit");
111+
case "cost-limit-unknown":
112+
return t("routing.exclusion.cost-limit-unknown");
113+
case "cooldown":
114+
return t("routing.exclusion.cooldown");
115+
case "unknown-health":
116+
return t("routing.exclusion.unknown-health");
117+
case "unknown-quota":
118+
return t("routing.exclusion.unknown-quota");
119+
case "unknown-price":
120+
return t("routing.exclusion.unknown-price");
121+
default:
122+
return t("routing.exclusion.other", { code });
123+
}
124+
}
125+
81126
function isPlainObject(value: unknown): value is Record<string, unknown> {
82127
return !!value && typeof value === "object" && !Array.isArray(value);
83128
}
@@ -709,10 +754,28 @@ export default function RoutingProfiles({
709754
value={draft.limits.maxEstimatedCostUsd}
710755
onChange={event => setDraft(current => current ? {
711756
...current,
712-
limits: { maxEstimatedCostUsd: event.target.value },
757+
limits: { ...current.limits, maxEstimatedCostUsd: event.target.value },
713758
} : current)}
714759
/>
715760
</label>
761+
<label className="field-label">
762+
<code>onUnknownCost</code>
763+
<select
764+
className="input"
765+
value={draft.limits.onUnknownCost}
766+
onChange={event => setDraft(current => current ? {
767+
...current,
768+
limits: {
769+
...current.limits,
770+
onUnknownCost: event.target.value as UnknownCostCapMode,
771+
},
772+
} : current)}
773+
>
774+
{UNKNOWN_COST_CAP_OPTIONS.map(mode => (
775+
<option key={mode} value={mode}>{t(`routing.unknownEvidence.${mode}` as const)}</option>
776+
))}
777+
</select>
778+
</label>
716779
</fieldset>
717780

718781
<fieldset style={{ border: 0, padding: 0, margin: 0 }}>
@@ -817,6 +880,7 @@ export default function RoutingProfiles({
817880
<th>{t("routing.candidate")}</th>
818881
<th>{t("routing.eligible")}</th>
819882
<th>{t("routing.exclusions")}</th>
883+
<th>{t("routing.costCap")}</th>
820884
<th>{t("routing.score")}</th>
821885
</tr>
822886
</thead>
@@ -828,7 +892,8 @@ export default function RoutingProfiles({
828892
{index === dryRunResult.selectedIndex ? ` ✓ (${t("routing.selected")})` : ""}
829893
</td>
830894
<td>{candidate.eligible ? t("routing.yes") : t("routing.no")}</td>
831-
<td>{candidate.exclusions.map(exclusion => exclusion.code).join(", ") || t("routing.none")}</td>
895+
<td>{candidate.exclusions.map(exclusion => fmtExclusion(exclusion.code, t)).join(", ") || t("routing.none")}</td>
896+
<td>{fmtCapOutcome(candidate.cost?.capOutcome, t, unavailable)}</td>
832897
<td>{candidate.score ? candidate.score.total.toFixed(3) : unavailable}</td>
833898
</tr>
834899
))}

0 commit comments

Comments
 (0)