Skip to content

[Bug] AutoMix cost-aware selection inverts preference and verification bonus on negative expected values #5496

Closed as not planned

Description

@Kaap10

Proposed Workgroup

MoM & Routing

Describe the bug

In src/semantic-router/pkg/selection/automix.go, AutoMixSelector.selectCostAware (lines 456–494) computes cost-adjusted selection values by multiplying candidate expected scores by costFactor = 1.0 / (1.0 + cap.Cost * selCtx.CostWeight) and scaling by 1.1 when VerificationProb >= VerificationThreshold.

When computeExpectedValue yields a negative expected value — which occurs naturally whenever a candidate's cost penalty exceeds its baseline catalog quality (such as unbenchmarked, uncataloged, or reasoning-differentiated models where quality = 0.0) — the multiplicative scaling inverts both the cost preference and the verification confidence bonus:

  1. Cost Inversion: For an expensive model, costFactor is much smaller than for a cheap model ($0.16$ vs $0.95$). Multiplying a negative score by a smaller positive factor brings it closer to zero. Thus, the expensive candidate achieves a higher score than the cheap candidate.
  2. Verification Bonus Inversion: When value < 0, multiplying by 1.1 makes the value more negative (e.g. $-0.095 \times 1.1 = -0.105$). Candidates with verification confidence above threshold receive a 10% penalty instead of a 10% bonus, causing lower-confidence candidates to outrank higher-confidence ones.
  3. Unregistered Candidate Dropping: If a candidate model lacks a pre-populated capabilities entry, selectCostAware silently discards it (if cap == nil { continue }), whereas computeExpectedValue safely assigns the neutral default $0.5$.

To Reproduce

Configure two models in AutoMix with CostAwareRouting: true:

  • cheap-model: Cost = $0.10 / 1M, VerificationProb = 0.85
  • expensive-model: Cost = $10.00 / 1M, VerificationProb = 0.50
  • Both models have baseline quality = 0.0 (or negative expected values: e.g. -0.10 vs -0.20).

Call selectCostAware.
Result: expensive-model is selected with score -0.033333, while cheap-model receives -0.104762. The 100x more expensive model with failing verification confidence is selected over the cheap model with passing verification confidence.

Expected behavior

  1. Cost-aware adjustments should penalize higher-cost models and reward lower-cost models monotonically, regardless of whether base expected values are positive or negative (e.g. through baseline utility shifting before discounting).
  2. The verification confidence bonus must always increase a candidate's ranking relative to candidates below the verification threshold.
  3. Candidates without pre-populated capabilities should receive fallback handling consistent with computeExpectedValue rather than silent omission.

Impact

Under cost-aware AutoMix routing, when models have low initial benchmark scores or uncataloged quality, the router routes traffic to the most expensive candidate and penalizes models that pass verification checks.

Additional context

I have reproduced this with a unit test and have the fix and regression tests prepared locally. I will open a PR once the issue is accepted.

Activity

  1. Kaap10 commented on Oct 26, 2026

    @Kaap10
    ContributorAuthor

    I have reproduced this behavior with a unit test and already have the fix and regression tests prepared locally. Happy to claim with /assign and open the PR as soon as maintainers review and accept it!

  2. ptimizeroracle commented on Oct 28, 2026

    @ptimizeroracle

    reproduced on main @ f475d2e through Select, with one nuance on the setup

    • cost: cheap (cost 3, quality 0, raw -0.090) vs pricey (cost 10, quality 0.2, raw -0.100), cost_weight 0.2. after the cost factor pricey is -0.037 and cheap -0.062, so Select picks pricey. same pair with both values positive picks cheap
    • verification: same raw -0.060 for both, verif 0.90 vs 0.50. the 1.1 bonus takes the verified one to -0.047 and it loses to -0.043

    nuance: when both models have quality 0 and only the cost penalty makes the value negative, the order is fine. raw is -0.03 * cost, and -0.03c / (1 + c w) still decreases with c, so with your two models (0.1 and 10, quality 0) cheap wins at cost_weight 0.2 and 1.0. the inversion needs different quality (or escalation reward) between candidates, like the -0.10 / -0.20 in your example. might be worth a regression test for both cases

    (i used an ai assistant to help run the checks)

  3. added
    bugSomething isn't working
    needs-acceptanceAwaiting explicit Maintainer or Workgroup acceptance before implementation.
    wg/mom-routingOwned by the MoM and Routing Workgroup.
    on Oct 28, 2026
  4. subin9 commented on Nov 9, 2026

    @subin9
    Collaborator

    Thanks @Kaap10 and @ptimizeroracle for the careful reproduction. The decision-model redesign (#5885, in progress in #5886) removes the AutoMix selector and folds it into multi_factor, so a fix here would not ship. Closing as not planned.

  5. removed
    needs-acceptanceAwaiting explicit Maintainer or Workgroup acceptance before implementation.
    on Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingwg/mom-routingOwned by the MoM and Routing Workgroup.

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions