Skip to content

Commit fdbc2cc

Browse files
Python: fix(anthropic): report token usage as CompletionUsage (#14618)
### Motivation and Context The model diagnostics record token usage only when `metadata["usage"]` is a `CompletionUsage` (`utils/telemetry/model_diagnostics/decorators.py`). The other chat connectors build one, but `AnthropicChatCompletion` stores the raw `anthropic.types.Usage` (non-streaming) or a plain dict like `{"input_tokens": ..., "output_tokens": ...}` (streaming). As a result, spans for Anthropic calls never get `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens`. Prompt caching also isn't counted: Anthropic reports `cache_read_input_tokens` and `cache_creation_input_tokens` separately from `input_tokens`, and the streaming path drops them. ### Description - A new `_get_completion_usage` helper turns Anthropic usage into a `CompletionUsage`: - `prompt_tokens` = `input_tokens` + `cache_read_input_tokens` + `cache_creation_input_tokens`; - `prompt_tokens_details.cached_tokens` = `cache_read_input_tokens` (only when present); - `completion_tokens` = `output_tokens`. - Non-streaming: `metadata["usage"]` is now that `CompletionUsage`. - Streaming: `message_start` sets the prompt side. The final `message_delta` chunk carries the full usage: its own counts when it reports `input_tokens`, otherwise the `message_start` prompt counts plus the delta's `output_tokens`. Since the chunk metadata is merged with `|`, this is the usage of the combined message. - Fields are read with `getattr` because older `anthropic` releases (the requirement is `~= 0.32`) lack some of them. **Behavior change:** `metadata["usage"]` for Anthropic responses is now a `CompletionUsage`, the same type as the other connectors. Code that read `.input_tokens` / `.output_tokens` from it should use `.prompt_tokens` / `.completion_tokens`. The raw Anthropic usage is still available on `inner_content`. ### Contribution Checklist - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 (see the behavior change above) Tests, in `tests/unit/connectors/ai/anthropic/services/test_anthropic_chat_completion.py`, with model diagnostics enabled, 10 input + 100 cache-read + 20 cache-write + 7 output tokens: - `test_usage_is_reported_as_completion_usage` (non-streaming): `metadata["usage"]` is a `CompletionUsage` (130 prompt, 100 cached, 7 completion), and the span gets `gen_ai.usage.input_tokens=130`, `gen_ai.usage.output_tokens=7`; - `test_streaming_usage_is_reported_as_completion_usage`: the same checks on the combined streamed message. Both fail on `main`. `tests/unit/connectors/ai/anthropic` + `tests/unit/utils/model_diagnostics`: 65 passed (`test_connector_decorated.py` was not run here because it imports the Mistral connector). `ruff` 0.15.5 check/format is clean. `mypy` on the changed module reports the same 3 pre-existing errors as `main`, all in other modules. This change was drafted with Claude Code; the tests above were run before opening the PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 80161f5 commit fdbc2cc

2 files changed

Lines changed: 148 additions & 9 deletions

File tree

‎python/semantic_kernel/connectors/ai/anthropic/services/anthropic_chat_completion.py‎

Lines changed: 28 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -16,11 +16,14 @@
1616
from anthropic.types import (
1717
ContentBlockStopEvent,
1818
Message,
19+
MessageDeltaUsage,
1920
RawMessageDeltaEvent,
2021
RawMessageStartEvent,
2122
TextBlock,
2223
ToolUseBlock,
24+
Usage,
2325
)
26+
from openai.types.completion_usage import PromptTokensDetails
2427
from pydantic import ValidationError
2528

2629
from semantic_kernel.connectors.ai.anthropic.prompt_execution_settings.anthropic_prompt_execution_settings import (
@@ -32,6 +35,7 @@
3235
)
3336
from semantic_kernel.connectors.ai.anthropic.settings.anthropic_settings import AnthropicSettings
3437
from semantic_kernel.connectors.ai.chat_completion_client_base import ChatCompletionClientBase
38+
from semantic_kernel.connectors.ai.completion_usage import CompletionUsage
3539
from semantic_kernel.connectors.ai.function_choice_behavior import FunctionChoiceType
3640
from semantic_kernel.connectors.ai.prompt_execution_settings import PromptExecutionSettings
3741
from semantic_kernel.contents.chat_history import ChatHistory
@@ -309,11 +313,12 @@ def _create_streaming_chat_message_content(
309313
)
310314
elif isinstance(stream_event, RawMessageDeltaEvent):
311315
finish_reason = ANTHROPIC_TO_SEMANTIC_KERNEL_FINISH_REASON_MAP[str(stream_event.delta.stop_reason)]
312-
output_tokens = stream_event.usage.output_tokens
313-
if metadata is None:
314-
metadata = {"usage": {"output_tokens": output_tokens}}
315-
else:
316-
metadata = metadata | {"usage": metadata.get("usage", {}) | {"output_tokens": output_tokens}}
316+
usage = _get_completion_usage(stream_event.usage)
317+
start_usage = metadata.get("usage") if metadata else None
318+
if getattr(stream_event.usage, "input_tokens", None) is None and isinstance(start_usage, CompletionUsage):
319+
# The input token counts are only reported at message start.
320+
usage = start_usage.model_copy(update={"completion_tokens": usage.completion_tokens})
321+
metadata = (metadata or {}) | {"usage": usage}
317322

318323
return StreamingChatMessageContent(
319324
choice_index=0,
@@ -338,7 +343,7 @@ async def _send_chat_request(self, settings: AnthropicChatPromptExecutionSetting
338343

339344
response_metadata: dict[str, Any] = {"id": response.id}
340345
if hasattr(response, "usage") and response.usage is not None:
341-
response_metadata["usage"] = response.usage
346+
response_metadata["usage"] = _get_completion_usage(response.usage)
342347

343348
return [self._create_chat_message_content(response, response_metadata)]
344349

@@ -357,10 +362,10 @@ async def _send_chat_stream_request(
357362
"""
358363
try:
359364
async with self.async_client.messages.stream(**settings.prepare_settings_dict()) as stream:
360-
metadata: dict[str, Any] = {"usage": {}, "id": None}
365+
metadata: dict[str, Any] = {"usage": None, "id": None}
361366
async for stream_event in stream:
362367
if isinstance(stream_event, RawMessageStartEvent):
363-
metadata["usage"]["input_tokens"] = stream_event.message.usage.input_tokens
368+
metadata["usage"] = _get_completion_usage(stream_event.message.usage)
364369
metadata["id"] = stream_event.message.id
365370
elif isinstance(stream_event, (TextEvent, RawMessageDeltaEvent)) or (
366371
isinstance(stream_event, ContentBlockStopEvent)
@@ -391,3 +396,18 @@ def _get_tool_calls_from_message(self, message: Message) -> list[FunctionCallCon
391396
)
392397

393398
return tool_calls
399+
400+
401+
def _get_completion_usage(usage: Usage | MessageDeltaUsage) -> CompletionUsage:
402+
"""Convert Anthropic usage to CompletionUsage.
403+
404+
Anthropic reports the tokens read from and written to the prompt cache separately from `input_tokens`,
405+
so they are added back to get the full prompt token count. Older SDK versions lack some of these fields.
406+
"""
407+
cache_read_tokens = getattr(usage, "cache_read_input_tokens", None) or 0
408+
cache_creation_tokens = getattr(usage, "cache_creation_input_tokens", None) or 0
409+
return CompletionUsage(
410+
prompt_tokens=(getattr(usage, "input_tokens", None) or 0) + cache_read_tokens + cache_creation_tokens,
411+
prompt_tokens_details=PromptTokensDetails(cached_tokens=cache_read_tokens) if cache_read_tokens else None,
412+
completion_tokens=usage.output_tokens,
413+
)

‎python/tests/unit/connectors/ai/anthropic/services/test_anthropic_chat_completion.py‎

Lines changed: 120 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,19 @@
11
# Copyright (c) Microsoft. All rights reserved.
2+
from functools import reduce
23
from unittest.mock import AsyncMock, MagicMock, patch
34

45
import pytest
56
from anthropic import AsyncAnthropic
6-
from anthropic.types import Message
7+
from anthropic.lib.streaming import TextEvent
8+
from anthropic.types import Message, MessageDeltaUsage, RawMessageDeltaEvent, RawMessageStartEvent, TextBlock, Usage
9+
from anthropic.types.raw_message_delta_event import Delta
710

811
from semantic_kernel.connectors.ai.anthropic.prompt_execution_settings.anthropic_prompt_execution_settings import (
912
AnthropicChatPromptExecutionSettings,
1013
)
1114
from semantic_kernel.connectors.ai.anthropic.services.anthropic_chat_completion import AnthropicChatCompletion
1215
from semantic_kernel.connectors.ai.chat_completion_client_base import ChatCompletionClientBase
16+
from semantic_kernel.connectors.ai.completion_usage import CompletionUsage
1317
from semantic_kernel.connectors.ai.function_choice_behavior import FunctionChoiceBehavior
1418
from semantic_kernel.connectors.ai.open_ai.prompt_execution_settings.open_ai_prompt_execution_settings import (
1519
OpenAIChatPromptExecutionSettings,
@@ -27,6 +31,8 @@
2731
from semantic_kernel.functions.kernel_arguments import KernelArguments
2832
from semantic_kernel.functions.kernel_function_decorator import kernel_function
2933
from semantic_kernel.kernel import Kernel
34+
from semantic_kernel.utils.telemetry.model_diagnostics import gen_ai_attributes
35+
from semantic_kernel.utils.telemetry.model_diagnostics.model_diagnostics_settings import ModelDiagnosticSettings
3036

3137

3238
async def test_complete_chat_contents(
@@ -547,3 +553,116 @@ def test_chat_completion_reset_settings(
547553

548554
assert settings.tools is None
549555
assert settings.tool_choice is None
556+
557+
558+
@pytest.fixture
559+
def model_diagnostics_enabled(monkeypatch):
560+
monkeypatch.setattr(
561+
"semantic_kernel.utils.telemetry.model_diagnostics.decorators.MODEL_DIAGNOSTICS_SETTINGS",
562+
ModelDiagnosticSettings(enable_otel_diagnostics=True),
563+
)
564+
565+
566+
def _usage_span_attributes(mock_tracer: MagicMock) -> dict:
567+
mock_span = mock_tracer.start_span.return_value
568+
return {
569+
args[0]: args[1]
570+
for args, _ in mock_span.set_attribute.call_args_list
571+
if args[0] in (gen_ai_attributes.INPUT_TOKENS, gen_ai_attributes.OUTPUT_TOKENS)
572+
}
573+
574+
575+
@patch("semantic_kernel.utils.telemetry.model_diagnostics.decorators.tracer")
576+
async def test_usage_is_reported_as_completion_usage(
577+
mock_tracer: MagicMock,
578+
model_diagnostics_enabled,
579+
mock_settings: AnthropicChatPromptExecutionSettings,
580+
):
581+
response = Message(
582+
id="test_message_id",
583+
content=[TextBlock(text="Hello", type="text")],
584+
model="claude-sonnet-4-5",
585+
role="assistant",
586+
stop_reason="end_turn",
587+
stop_sequence=None,
588+
type="message",
589+
usage=Usage(input_tokens=10, output_tokens=7, cache_read_input_tokens=100, cache_creation_input_tokens=20),
590+
)
591+
client = MagicMock(spec=AsyncAnthropic)
592+
client.messages = MagicMock()
593+
client.messages.create = AsyncMock(return_value=response)
594+
chat_completion = AnthropicChatCompletion(ai_model_id="test_model_id", api_key="", async_client=client)
595+
chat_history = ChatHistory()
596+
chat_history.add_user_message("test_user_message")
597+
598+
result = await chat_completion.get_chat_message_content(chat_history, mock_settings)
599+
600+
usage = result.metadata["usage"]
601+
assert isinstance(usage, CompletionUsage)
602+
assert usage.prompt_tokens == 130
603+
assert usage.prompt_tokens_details.cached_tokens == 100
604+
assert usage.completion_tokens == 7
605+
assert _usage_span_attributes(mock_tracer) == {
606+
gen_ai_attributes.INPUT_TOKENS: 130,
607+
gen_ai_attributes.OUTPUT_TOKENS: 7,
608+
}
609+
610+
611+
@patch("semantic_kernel.utils.telemetry.model_diagnostics.decorators.tracer")
612+
async def test_streaming_usage_is_reported_as_completion_usage(
613+
mock_tracer: MagicMock,
614+
model_diagnostics_enabled,
615+
mock_settings: AnthropicChatPromptExecutionSettings,
616+
):
617+
stream_events = [
618+
RawMessageStartEvent(
619+
message=Message(
620+
id="test_message_id",
621+
content=[],
622+
model="claude-sonnet-4-5",
623+
role="assistant",
624+
stop_reason=None,
625+
stop_sequence=None,
626+
type="message",
627+
usage=Usage(
628+
input_tokens=10, output_tokens=1, cache_read_input_tokens=100, cache_creation_input_tokens=20
629+
),
630+
),
631+
type="message_start",
632+
),
633+
TextEvent(type="text", text="Hello", snapshot="Hello"),
634+
RawMessageDeltaEvent(
635+
delta=Delta(stop_reason="end_turn", stop_sequence=None),
636+
type="message_delta",
637+
usage=MessageDeltaUsage(output_tokens=7),
638+
),
639+
]
640+
641+
async def async_generator():
642+
for event in stream_events:
643+
yield event
644+
645+
stream_mock = AsyncMock()
646+
stream_mock.__aenter__.return_value = async_generator()
647+
client = MagicMock(spec=AsyncAnthropic)
648+
client.messages = MagicMock()
649+
client.messages.stream.return_value = stream_mock
650+
chat_completion = AnthropicChatCompletion(ai_model_id="test_model_id", api_key="", async_client=client)
651+
chat_history = ChatHistory()
652+
chat_history.add_user_message("test_user_message")
653+
654+
chunks = [
655+
chunk
656+
async for chunks in chat_completion.get_streaming_chat_message_contents(chat_history, mock_settings)
657+
for chunk in chunks
658+
]
659+
660+
usage = reduce(lambda x, y: x + y, chunks).metadata["usage"]
661+
assert isinstance(usage, CompletionUsage)
662+
assert usage.prompt_tokens == 130
663+
assert usage.prompt_tokens_details.cached_tokens == 100
664+
assert usage.completion_tokens == 7
665+
assert _usage_span_attributes(mock_tracer) == {
666+
gen_ai_attributes.INPUT_TOKENS: 130,
667+
gen_ai_attributes.OUTPUT_TOKENS: 7,
668+
}

0 commit comments

Comments
 (0)