Skip to content

fix(studio/pricing): long-context tier starts strictly above the threshold, not at it - #9527

Merged
danielhanchen merged 1 commit into
unslothai:mainfrom
xyzs996:long-context-boundary-exclusive
Aug 25, 2026
Merged

danielhanchen merged 1 commit into
unslothai:mainfrom
xyzs996:long-context-boundary-exclusive

Conversation

@xyzs996

@xyzs996 xyzs996 commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

The bug

calculate_cost flips a turn onto the long-context tier one token early:

# studio/backend/core/inference/pricing.py:246-254
lc_thresh = prices.get("long_context_threshold")
in_long_context_tier = (
    lc_thresh is not None
    and out["billable_input_tokens"] >= int(lc_thresh)   # <-- inclusive
    ...
)

OpenAI's own wording is strictly-above:

prompts with >272K input tokens are priced at 2x input and 1.5x output for the full session

— https://developers.openai.com/api/docs/models/gpt-5.4 (read 2026-08-22). The
tooltip on the pricing page this table cites in its header comment agrees:
≤272K input tokens for the standard column.

So a prompt of exactly 272,000 tokens is standard-rate at OpenAI, and
double-rate here. On gpt-5.5, 272,000 in / 1,000 out:

input output total
OpenAI bills $1.360 $0.030 $1.390
main bills $2.720 $0.045 $2.765

Two internal signals point the same way. The label this branch emits is
f"{model} (long-context >{lc_thresh})" — it already says >, on a request
that is not >. And test_openai_gpt55_short_context_under_272k_uses_base_rates
is named "under 272k" but only ever exercises 100k, so the naming convention in
the suite reads exclusive too.

Scope of the fix

All five long_context_threshold entries in pricing.py are OpenAI 272_000
(gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna, gpt-5.5, gpt-5.4). There is
no Google or Anthropic threshold in that table, so >= → > is the complete
fix here with no counter-vendor to worry about. (Worth knowing for later: this
is not a safe global convention. xAI documents the opposite — "requests whose
prompt reaches the listed token threshold are billed at the higher rate for
all tokens in the request", https://docs.x.ai/docs/models — so if a non-OpenAI
threshold is ever added to this table it will need a per-entry flag rather than
another blanket comparison.)

About the test I changed

test_openai_long_context_threshold_boundary_inclusive asserted the current
behaviour with the comment # Threshold is inclusive (>=)., so this is not a
case of an untested edge — the convention was pinned deliberately. I've renamed
it to ..._exclusive, flipped it to 272,000 → standard / 272,001 → long-context,
and put the vendor quote in the comment so the next reader sees where the
convention comes from. If that inclusive reading was deliberate for a reason not
in the repo, say so and I'll close this — but the vendor sentence and the
(long-context >{n}) label both read the other way.

The cache-inflation test (250k + 50k cache_creation → 300,000 billable) is
unaffected; it stays comfortably above the line.

Verification

Ran studio/backend/tests/test_pricing.py (37 passed) and
studio/backend/tests/test_pricing_edge.py (24 passed; the 25th,
test_build_usage_chunk_forwards_anthropic_cache_creation_breakdown, imports
core.inference.external_provider, which I didn't have wired up locally — it
doesn't touch pricing).

271999  gpt-5.5                          $1.360 / $0.030
272000  gpt-5.5                          $1.360 / $0.030   <- was long-context
272001  gpt-5.5 (long-context >272000)   $2.720 / $0.045

…shold

OpenAI prices "prompts with >272K input tokens" at 2x input / 1.5x output for
the full session, so a prompt of exactly 272,000 tokens is still on the standard
rate. calculate_cost used `>=`, which flipped the whole turn one token early --
gpt-5.5 at exactly 272,000 in was billed $2.72 instead of $1.36.

All five long_context_threshold entries in pricing.py are OpenAI 272_000, so
`>=` -> `>` is the complete fix with no other vendor affected. The boundary test
asserted the old behaviour and is updated to match the vendor wording.
@danielhanchen

Copy link
Copy Markdown
Member

Confirmed this hits studio/backend/core/inference/pricing.py where the long-context flip uses greater-or-equal while the emitted label and the vendor wording both say strictly above, so an exactly-272,000-token prompt is priced at the higher rate. Will get this reviewed.

@danielhanchen
danielhanchen merged commit c6b5166 into unslothai:main Aug 25, 2026
1 check passed
danielhanchen pushed a commit that referenced this pull request Aug 25, 2026
…shold (#9527)

OpenAI prices "prompts with >272K input tokens" at 2x input / 1.5x output for
the full session, so a prompt of exactly 272,000 tokens is still on the standard
rate. calculate_cost used `>=`, which flipped the whole turn one token early --
gpt-5.5 at exactly 272,000 in was billed $2.72 instead of $1.36.

All five long_context_threshold entries in pricing.py are OpenAI 272_000, so
`>=` -> `>` is the complete fix with no other vendor affected. The boundary test
asserted the old behaviour and is updated to match the vendor wording.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants