Repository navigation
Description
Proposed Workgroup
MoM & Routing
Describe the bug
In src/semantic-router/pkg/selection/automix.go, AutoMixSelector.selectCostAware (lines 456–494) computes cost-adjusted selection values by multiplying candidate expected scores by costFactor = 1.0 / (1.0 + cap.Cost * selCtx.CostWeight) and scaling by 1.1 when VerificationProb >= VerificationThreshold.
When computeExpectedValue yields a negative expected value — which occurs naturally whenever a candidate's cost penalty exceeds its baseline catalog quality (such as unbenchmarked, uncataloged, or reasoning-differentiated models where quality = 0.0) — the multiplicative scaling inverts both the cost preference and the verification confidence bonus:
-
Cost Inversion: For an expensive model,
costFactoris much smaller than for a cheap model ($0.16$ vs$0.95$ ). Multiplying a negative score by a smaller positive factor brings it closer to zero. Thus, the expensive candidate achieves a higher score than the cheap candidate. -
Verification Bonus Inversion: When
value < 0, multiplying by1.1makes the value more negative (e.g.$-0.095 \times 1.1 = -0.105$ ). Candidates with verification confidence above threshold receive a 10% penalty instead of a 10% bonus, causing lower-confidence candidates to outrank higher-confidence ones. -
Unregistered Candidate Dropping: If a candidate model lacks a pre-populated
capabilitiesentry,selectCostAwaresilently discards it (if cap == nil { continue }), whereascomputeExpectedValuesafely assigns the neutral default$0.5$ .
To Reproduce
Configure two models in AutoMix with CostAwareRouting: true:
-
cheap-model: Cost = $0.10 / 1M, VerificationProb = 0.85 -
expensive-model: Cost = $10.00 / 1M, VerificationProb = 0.50 - Both models have baseline quality = 0.0 (or negative expected values: e.g. -0.10 vs -0.20).
Call selectCostAware.
Result: expensive-model is selected with score -0.033333, while cheap-model receives -0.104762. The 100x more expensive model with failing verification confidence is selected over the cheap model with passing verification confidence.
Expected behavior
- Cost-aware adjustments should penalize higher-cost models and reward lower-cost models monotonically, regardless of whether base expected values are positive or negative (e.g. through baseline utility shifting before discounting).
- The verification confidence bonus must always increase a candidate's ranking relative to candidates below the verification threshold.
- Candidates without pre-populated capabilities should receive fallback handling consistent with
computeExpectedValuerather than silent omission.
Impact
Under cost-aware AutoMix routing, when models have low initial benchmark scores or uncataloged quality, the router routes traffic to the most expensive candidate and penalizes models that pass verification checks.
Additional context
I have reproduced this with a unit test and have the fix and regression tests prepared locally. I will open a PR once the issue is accepted.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDone
Activity
Kaap10 commentedon Oct 26, 2026
I have reproduced this behavior with a unit test and already have the fix and regression tests prepared locally. Happy to claim with /assign and open the PR as soon as maintainers review and accept it!
ptimizeroracle commentedon Oct 28, 2026
reproduced on main @ f475d2e through
Select, with one nuance on the setupSelectpicks pricey. same pair with both values positive picks cheapnuance: when both models have quality 0 and only the cost penalty makes the value negative, the order is fine. raw is -0.03 * cost, and -0.03c / (1 + c w) still decreases with c, so with your two models (0.1 and 10, quality 0) cheap wins at cost_weight 0.2 and 1.0. the inversion needs different quality (or escalation reward) between candidates, like the -0.10 / -0.20 in your example. might be worth a regression test for both cases
(i used an ai assistant to help run the checks)
subin9 commentedon Nov 9, 2026
Thanks @Kaap10 and @ptimizeroracle for the careful reproduction. The decision-model redesign (#5885, in progress in #5886) removes the AutoMix selector and folds it into
multi_factor, so a fix here would not ship. Closing as not planned.