Repository navigation
fix: avoid summing cumulative streaming usage on OpenAILike models - #11549
princejain756 wants to merge 1 commit into
Conversation
PR TriageIssue assigned to someone else: This PR references an issue that is already assigned:
Please confirm with the assignee or a maintainer in the issue comments before proceeding. This PR has been automatically closed because the linked issue is already assigned to someone else. Only a maintainer can reopen it. Once you have addressed the points above (for example, a maintainer has marked the linked issue |
|
Hey @harshsinha03, sorry about that, I saw the |
Summary
Fixes #11538
OpenAI-compatible servers like vLLM (with force-include-usage or continuous usage stats) emit running cumulative token counts on intermediate SSE chunks and a final usage summary chunk with an empty
choiceslist. BecauseOpenAILikeinheritedcollect_metrics_on_completion = False,_populate_stream_dataadded every chunk's cumulative usage together, and enablingcollect_metrics_on_completionstill dropped the finalchoices: []usage frame because_should_collect_metricsreturnedFalsewhenresponse.choiceswas empty.This sets
collect_metrics_on_completion: bool = Trueby default onOpenAILike, updatesOpenAIChat._should_collect_metricsto accept the final empty-choicesusage chunk, and replaces (while preserving timer, duration, and TTFT) rather than addsstream_data.response_metricswhencollect_metrics_on_completionis enabled so streams that attach usage to both thefinish_reasonchunk and the trailingchoices: []chunk do not double-count.Type of change
Checklist
./scripts/format.shand./scripts/validate.sh)Duplicate and AI-Generated PR Check
Additional Notes
Verified with
pytest libs/agno/tests/unit/models/openai/test_openai_metrics.py libs/agno/tests/unit/models/perplexity/test_perplexity_metrics.py(23 passed).