Skip to content

cost-optimized routing treats null-priced models as $0 (always picked as cheapest) #126

Closed

Description

@MitulShah1

Summary

The cost-optimized routing strategy treats any model that exists in the catalog as having a price, even when its catalog price is null (unknown / not per-token billed). Null-priced models compute to $0 and are therefore always selected as the "cheapest" route.

Root cause

models.Calculate collapses a nil price to 0 cost (models/calculator.go):

func perM(price *float64, n int) float64 {
    if price == nil || n == 0 { return 0 } // nil price → 0 cost
    return *price * float64(n) / 1_000_000
}

internal/strategies/costoptimized.go:58 then sets hasPrice from ModelFound, not from whether a real price exists:

candidates = append(candidates, priced{
    target:   t,
    costUSD:  result.InputUSD,   // 0 when price is null
    hasPrice: result.ModelFound, // true whenever the model is in the catalog
})

The cheapest-wins loop (lines 67-77) then picks the null-priced candidate because its costUSD is 0.

Impact

Any provider with null catalog pricing, when placed in a cost-optimized target group, is always chosen as cheapest regardless of actual cost.

This is a silent misroute: cost-optimizing operators would have all matching traffic pulled onto whichever provider happens to have unpriced catalog entries.

Expected behavior

A model with unknown (null) per-token price should be treated as unknown cost, not zero cost. Options:

  1. Change hasPrice to mean "has a non-null input/output price" (distinguish "in catalog" from "priced"). Exclude unknown-cost models from cheapest-selection, or rank them last, instead of treating them as $0.
  2. Optionally surface a config knob for how to treat unknown-cost candidates (skip vs. fallback vs. allow).

Repro sketch

  1. Configure a cost-optimized target group with two providers that both support model X: one with real catalog pricing, one with null pricing (e.g. a ZAI GLM model).
  2. Send a request for model X.
  3. Observe the null-priced provider is always selected as "cheapest."

Notes

  • Catalog side needs no change. null is the correct encoding for credit-based / unpriced models. The fix belongs in the router (and possibly the CostResult contract, e.g. a Priced bool separate from ModelFound).

Activity

  1. Rachit-Gandhi commented on May 24, 2026

    @Rachit-Gandhi
    Collaborator

    @MitulShah1 I would like to take this issue up, if you can assign it to me.

  2. MitulShah1 commented on May 24, 2026

    @MitulShah1
    ContributorAuthor

    Hey @Rachit-Gandhi thanks, assigned to you.

  3. Rachit-Gandhi commented on May 24, 2026

    @Rachit-Gandhi
    Collaborator

    @MitulShah1 There is no develop branch, should I raise a pr to main or a particular branch?

  4. MitulShah1 commented on May 24, 2026

    @MitulShah1
    ContributorAuthor

    @MitulShah1 There is no develop branch, should I raise a pr to main or a particular branch?

    Hey Yeah i will map to release branch later.

  5. added a commit that references this issue on May 29, 2026
    ac3e7c2
  6. added this to the v1.1.1 milestone on Jun 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions