Skip to content

[Bug]: cost-based-routing ignores cache_read price; on an exact input+output tie the cheaper provider is picked only if it's listed first (10x) #38064

Description

@xyzs996

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

cost-based-routing scores every candidate deployment on input + output price only. cache_read_input_token_cost is never read — the string cache_read does not appear anywhere in litellm/router_strategy/lowest_cost.py.

litellm/router_strategy/lowest_cost.py:267:

item_cost = item_input_cost + item_output_cost

and then, at the end of async_get_available_deployments (lines 301–304):

potential_deployments = sorted(potential_deployments, key=lambda x: x[1])
selected_deployment: Final = potential_deployments[0][0]

For traffic that mostly hits the cache — agents re-reading a codebase, RAG with a fixed prefix, anything with a long stable system prompt — the token type that dominates the bill is the one the ranking cannot see.

The sharpest case in your own price map is an exact tie. deepseek-v4-flash is listed under two providers with identical input and output prices and cache-read prices that differ 10x:

line key input output input + output cache_read as % of input
18698 fireworks_ai/deepseek-v4-flash 1.4e-07 2.8e-07 4.2e-07 2.8e-08 20%
49741 tencent/deepseek-v4-flash 1.4e-07 2.8e-07 4.2e-07 2.8e-09 2%

item_cost comes out to 4.2e-07 for both, so the sort sees a tie. Python's sort is stable and the code takes [0], so the winner is whichever deployment appears first in model_list — i.e. the order you happened to write your config in.

Impact

Mix-free version: 1M cache-read tokens cost $0.0280 on fireworks_ai and $0.0028 on tencent. The router scores the two at the same number and cannot tell them apart.

Priced at a coding agent's token mix (95.6% cache read / 4.1% cache miss / 0.3% output, measured over 8.04B tokens), the same 1M tokens cost $0.0333 vs $0.0092 — the config-order winner is 262% more expensive, on a strategy whose entire purpose is to be the cheap one.

I checked how wide this is rather than assuming: across the 49 model names in model_prices_and_context_window.json that appear under more than one provider and carry a real (non-zero) cache-read price, 1 ranks wrong at a cache-heavy mix — the one above. So today this is narrow. It gets wider every time a provider publishes a cache-read price, and it is already unbounded in the tie case, where the ranking carries no signal at all.

Suggested fix

Two options, smallest first.

1. Break the tie on cache-read price. Changes nothing for any deployment pair whose input + output already differ, so no existing routing decision moves except the ones that were arbitrary:

item_cache_cost = item_litellm_model_cost_map.get("cache_read_input_token_cost", item_input_cost)
...
potential_deployments.append((_deployment, item_cost, item_cache_cost))
...
potential_deployments = sorted(potential_deployments, key=lambda x: (x[1], x[2]))

2. Let the caller declare the cache-hit share. routing_args already exists on LowestCostLoggingHandler. Defaulting the share to 0 reproduces today's behaviour exactly, so nobody's routing changes until they opt in:

h = self.routing_args.get("cache_hit_ratio", 0.0)
item_cost = (item_cache_cost * h + item_input_cost * (1 - h)) + item_output_cost

Falling back to item_input_cost when a model has no cache-read price is deliberate: it is the conservative assumption (no discount) and it keeps models missing that field from being ranked as if cache reads were free.

I have not opened a PR because I am not able to sign the CLA. Line numbers and prices are from main as of today.

Relevant log output

$ pip install litellm
$ cat repro.py
import asyncio
from litellm import Router
from litellm.router_strategy.lowest_cost import LowestCostLoggingHandler

MODEL = "deepseek-v4-flash"

def build(order):
    return Router(
        model_list=[{"model_name": MODEL,
                     "litellm_params": {"model": f"{p}/deepseek-v4-flash"},
                     "model_info": {"id": p}} for p in order],
        routing_strategy="cost-based-routing")

async def pick(order):
    r = build(order)
    h = LowestCostLoggingHandler(router_cache=r.cache)
    d = await h.async_get_available_deployments(
        model_group=MODEL, healthy_deployments=r.model_list)
    return d["model_info"]["id"]

async def main():
    for order in (["fireworks_ai", "tencent"], ["tencent", "fireworks_ai"]):
        print(f"model_list order {order}  ->  routed to: {await pick(order)}")

asyncio.run(main())

$ python repro.py
config order ['fireworks_ai', 'tencent']  ->  routed to: fireworks_ai
config order ['tencent', 'fireworks_ai']  ->  routed to: tencent

Same two deployments, same prices, same request — the pick flips with the order of the list. The one on the left costs 10x per cache-read token.

(The run also prints two register_model warnings about fireworks_ai / tencent not being in the cost map; those come from the model_info.id labels I used for readable output and are unrelated to the routing decision.)

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

main, checked today

Activity

  1. added 3 commits that reference this issue on Sep 25, 2026
    888cb72
    97e9e97
    6febd44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions