Repository navigation
fix: count requests when providers omit usage - #4290
Merged
Merged
Conversation
`Usage.requests` is documented as "Total requests made to the LLM API",
but a completed call was counted as zero requests whenever the provider
returned no usage block:
usage = (Usage(requests=1, ...) if response.usage else Usage())
Some OpenAI-compatible providers and gateways omit `usage` entirely, and
LiteLLM already logs "No usage information returned from Litellm" for the
same situation. In those deployments `usage.requests` stays at zero for a
whole run even though every turn made a real HTTP call, so anything built
on it, such as request accounting or rate-limit tracking, silently
undercounts.
The retry path already takes the opposite position. `apply_retry_attempt_usage`
forces the successful attempt to count with `max(usage.requests, 1)`, so
the same single call is reported differently depending on whether a retry
happened to occur:
failed_attempts=0 -> 1 HTTP call, reported requests=0
failed_attempts=1 -> 2 HTTP calls, reported requests=2
Count the request itself in the Chat Completions, LiteLLM and any_llm
models. Token counts stay at zero, since the provider genuinely did not
report them.
The Responses model is deliberately left alone. `OpenAIHostedMultiAgentModel`
subclasses it and merges several provider responses into a single
`ModelResponse`, tracking its own count through `_agents_sdk_request_count`.
There, a response without usage means the usage was already reported on an
earlier response rather than that a request went uncounted, so applying the
same change double-counts.
test_get_response_with_refusal asserted `requests == 0` for a completed
call with no usage. That expectation is the behaviour being corrected, so
it now asserts 1 while keeping the zeroed token assertions.
Addresses review feedback. The first revision only fixed the non-streaming adapters, so OpenAI Chat Completions, LiteLLM and AnyLLM still reported `requests == 0` when streamed: `ChatCmplStreamHandler` leaves the terminal usage as `None` and the run loop turned that into `Usage()`. AnyLLM's Responses path had the same non-streaming/streaming split. Fixed at the owning adapter boundaries rather than in the run loop. An adapter that reaches a terminal response without a provider usage payload now marks that response with `mark_request_completed_without_usage`, and no usage object is synthesized, so `raw_usage` stays absent and token counts are not reported as real zeros. `ChatCmplStreamHandler` is shared by the OpenAI Chat Completions, LiteLLM and AnyLLM chat streams, so marking it there covers all three. AnyLLM's Responses stream is marked where it captures the completed response. The generic run-loop fallback is unchanged in effect. It now reads `requests_for_response_without_usage`, which defaults to zero, so Hosted Multi-Agent keeps working: it emits interim logical completions with `usage=None` and reports accumulated request usage later, and would be double-counted by an unconditional `Usage(requests=1)`.
Two follow-ups from review. Rename `mark_request_completed_without_usage` and `requests_for_response_without_usage` to underscore-prefixed helpers. They are adapter and run-loop plumbing rather than supported `agents.usage` API, so they should not create an accidental public contract. Also record the request in the generation span, not only in the run result. Non-streaming already writes a span usage object with `requests: 1` when the provider reports no usage, but the streaming paths populated span usage only when the terminal response carried a usage payload. A streamed run therefore reported one request while its model span omitted the request entirely. The OpenAI Chat Completions, LiteLLM and AnyLLM chat spans now fall back to a shared `_span_usage_without_provider_totals()`, so streamed and non-streaming tracing agree. Token totals stay at zero, since the provider reported none.
Two more findings from automated review. The generation span was populated after the stream loop finished. A caller that consumes `stream_response()` directly and stops once it sees `response.completed` closes the generator at that yield, so the loop never resumes and the span was left without usage or output even though the run counted the request. Both the OpenAI Chat Completions and LiteLLM streams now populate the span at the point the terminal response is captured, before it is handed out, matching what the AnyLLM chat path already did. The AnyLLM Responses path never set `span_response.span_data.usage` at all, so with tracing enabled the run result and the turn and task spans counted a request that the model response span did not show. It now records usage the way `OpenAIResponsesModel` does, on both the streaming and non-streaming paths, which also makes the usage-less request count visible there.
Fix the terminal-event stream test typing so the full typecheck passes, and reuse the canonical usage-to-span serializer for usage-less responses. Co-authored-by: abhay-codes07 <[email protected]>
2 of 4 tasks
seratch
enabled auto-merge (squash)
August 7, 2026 22:42
seratch
disabled auto-merge
August 7, 2026 22:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request takes over and completes the work started in #4281.
It counts completed model requests when providers omit usage across OpenAI Chat Completions, LiteLLM, and AnyLLM while preserving absent raw usage and Hosted Multi-Agent request accounting. It also aligns streamed tracing with non-streaming behavior, records span usage before terminal events are yielded, fixes the terminal-event test typing failure, and reuses the canonical usage-to-span serializer.