Skip to content

fix: count requests when providers omit usage - #4290

Merged
seratch merged 5 commits into
mainfrom
fix/pr-4281-typecheck
Aug 7, 2026
Merged

seratch merged 5 commits into
mainfrom
fix/pr-4281-typecheck

Conversation

@seratch

@seratch seratch commented Aug 7, 2026

Copy link
Copy Markdown
Member

This pull request takes over and completes the work started in #4281.

It counts completed model requests when providers omit usage across OpenAI Chat Completions, LiteLLM, and AnyLLM while preserving absent raw usage and Hosted Multi-Agent request accounting. It also aligns streamed tracing with non-streaming behavior, records span usage before terminal events are yielded, fixes the terminal-event test typing failure, and reuses the canonical usage-to-span serializer.

abhay-codes07 and others added 5 commits August 8, 2026 07:22
`Usage.requests` is documented as "Total requests made to the LLM API",
but a completed call was counted as zero requests whenever the provider
returned no usage block:

    usage = (Usage(requests=1, ...) if response.usage else Usage())

Some OpenAI-compatible providers and gateways omit `usage` entirely, and
LiteLLM already logs "No usage information returned from Litellm" for the
same situation. In those deployments `usage.requests` stays at zero for a
whole run even though every turn made a real HTTP call, so anything built
on it, such as request accounting or rate-limit tracking, silently
undercounts.

The retry path already takes the opposite position. `apply_retry_attempt_usage`
forces the successful attempt to count with `max(usage.requests, 1)`, so
the same single call is reported differently depending on whether a retry
happened to occur:

    failed_attempts=0 -> 1 HTTP call,  reported requests=0
    failed_attempts=1 -> 2 HTTP calls, reported requests=2

Count the request itself in the Chat Completions, LiteLLM and any_llm
models. Token counts stay at zero, since the provider genuinely did not
report them.

The Responses model is deliberately left alone. `OpenAIHostedMultiAgentModel`
subclasses it and merges several provider responses into a single
`ModelResponse`, tracking its own count through `_agents_sdk_request_count`.
There, a response without usage means the usage was already reported on an
earlier response rather than that a request went uncounted, so applying the
same change double-counts.

test_get_response_with_refusal asserted `requests == 0` for a completed
call with no usage. That expectation is the behaviour being corrected, so
it now asserts 1 while keeping the zeroed token assertions.
Addresses review feedback. The first revision only fixed the non-streaming
adapters, so OpenAI Chat Completions, LiteLLM and AnyLLM still reported
`requests == 0` when streamed: `ChatCmplStreamHandler` leaves the terminal
usage as `None` and the run loop turned that into `Usage()`. AnyLLM's
Responses path had the same non-streaming/streaming split.

Fixed at the owning adapter boundaries rather than in the run loop. An
adapter that reaches a terminal response without a provider usage payload
now marks that response with `mark_request_completed_without_usage`, and no
usage object is synthesized, so `raw_usage` stays absent and token counts
are not reported as real zeros.

`ChatCmplStreamHandler` is shared by the OpenAI Chat Completions, LiteLLM
and AnyLLM chat streams, so marking it there covers all three. AnyLLM's
Responses stream is marked where it captures the completed response.

The generic run-loop fallback is unchanged in effect. It now reads
`requests_for_response_without_usage`, which defaults to zero, so Hosted
Multi-Agent keeps working: it emits interim logical completions with
`usage=None` and reports accumulated request usage later, and would be
double-counted by an unconditional `Usage(requests=1)`.
Two follow-ups from review.

Rename `mark_request_completed_without_usage` and
`requests_for_response_without_usage` to underscore-prefixed helpers.
They are adapter and run-loop plumbing rather than supported
`agents.usage` API, so they should not create an accidental public
contract.

Also record the request in the generation span, not only in the run
result. Non-streaming already writes a span usage object with
`requests: 1` when the provider reports no usage, but the streaming
paths populated span usage only when the terminal response carried a
usage payload. A streamed run therefore reported one request while its
model span omitted the request entirely. The OpenAI Chat Completions,
LiteLLM and AnyLLM chat spans now fall back to a shared
`_span_usage_without_provider_totals()`, so streamed and non-streaming
tracing agree. Token totals stay at zero, since the provider reported
none.
Two more findings from automated review.

The generation span was populated after the stream loop finished. A
caller that consumes `stream_response()` directly and stops once it sees
`response.completed` closes the generator at that yield, so the loop
never resumes and the span was left without usage or output even though
the run counted the request. Both the OpenAI Chat Completions and
LiteLLM streams now populate the span at the point the terminal response
is captured, before it is handed out, matching what the AnyLLM chat path
already did.

The AnyLLM Responses path never set `span_response.span_data.usage` at
all, so with tracing enabled the run result and the turn and task spans
counted a request that the model response span did not show. It now
records usage the way `OpenAIResponsesModel` does, on both the streaming
and non-streaming paths, which also makes the usage-less request count
visible there.
Fix the terminal-event stream test typing so the full typecheck passes, and reuse the canonical usage-to-span serializer for usage-less responses.

Co-authored-by: abhay-codes07 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants