Skip to content

Commit 381d7b1

Browse files
nik13claude
andcommitted
fix(ag2): count Anthropic cached input; agent and chat replies in Observe
Anthropic input tokens: AG2's Anthropic mapper reports prompt_tokens without cache reads and writes (its OpenAI mapper includes them), and AnthropicConfig caches by default, so a chat span showed 4 input tokens for a 9,750-token prompt. On anthropic spans whose tokens count, cache_read_input_tokens and cache_creation_input_tokens are now added to gen_ai.usage.input_tokens, and total_tokens is recomputed when present. The trace total can then exceed AG2's UsageReport by the cached tokens (README, CHANGELOG). Duplicate record_usage spans keep AG2's own numbers under ag2.usage.*. Chat reply: a single completion goes on output.value as its message object. Observe's chat view shows that as the assistant reply and shows nothing for a one-item list. Agent I/O: invoke_agent carries no content in AG2. With capture_content=True it now gets the first chat input and the last chat output of its turn, so the trace list shows the question and the answer. Hidden content is not copied, and the agent's own values win. Checked live on CLIProxy: Anthropic chat spans store 9751/94 and 9753/88 tokens; the capture run's trace list shows Input/Output, and the chat span shows the assistant reply bubble. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 75637f5 commit 381d7b1

4 files changed

Lines changed: 305 additions & 7 deletions

File tree

‎python/frameworks/ag2/CHANGELOG.md‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,22 @@
2424
rollups whose sub-agent (`ag2.usage.label`) is itself instrumented in the
2525
same trace. `aggregation` and `compaction` usage keeps its promoted tokens:
2626
AG2 calls the model outside the middleware for those, so the usage span is
27-
the only copy. The trace total now equals AG2's `UsageReport` total.
27+
the only copy. The trace total now equals AG2's `UsageReport` total, except
28+
for Anthropic with prompt caching (next entry).
29+
- Anthropic input tokens include cached tokens. AG2's Anthropic mapper reports
30+
`prompt_tokens` without cache reads and writes (its OpenAI mapper includes
31+
them), so with `AnthropicConfig`'s default prompt caching a chat span showed
32+
4 input tokens for a 9,750-token prompt. On `anthropic` spans whose tokens
33+
count, `cache_read_input_tokens` + `cache_creation_input_tokens` are added
34+
to `gen_ai.usage.input_tokens` (and `total_tokens` recomputed when present).
35+
The trace total then exceeds AG2's `UsageReport` by the cached tokens, which
36+
that report leaves out for Anthropic; a duplicate `record_usage` span keeps
37+
AG2's numbers under `ag2.usage.*`.
38+
- With `capture_content=True`, a single chat completion goes on `output.value`
39+
as its message object (Observe's chat view shows it as the assistant reply;
40+
it shows nothing for a one-item list), and `invoke_agent` spans get the first
41+
chat input and last chat output of their turn, so the trace list shows the
42+
question and the answer. Content a `TraceConfig` flag hides is not copied.
2843
- Requires `fi-instrumentation-otel>=1.1.0` (was `>=1.0.0`): published 1.0.0
2944
crashes in `register()` with `opentelemetry-exporter-otlp-proto-http` 1.45.0
3045
(`AttributeError: 'HTTPSpanExporter' object has no attribute '_headers'`).

‎python/frameworks/ag2/README.md‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -165,11 +165,11 @@ not copied to `input.value` / `output.value` either.
165165
| Span kind | `AGENT`, `LLM`, `TOOL` from `gen_ai.operation.name` | `await_human_input` and `record_usage` spans get no kind (see below) |
166166
| Model | `gen_ai.request.model`, `gen_ai.response.model` pass through | When AG2 did not set them |
167167
| Provider | `gen_ai.provider.name` from `provider_name` or the model response | When neither is known |
168-
| Tokens | `gen_ai.usage.input_tokens` / `output_tokens` pass through; cache tokens are aliased to `gen_ai.usage.cache_read_tokens` / `cache_write_tokens` and thinking tokens to `gen_ai.usage.reasoning.output_tokens`, originals kept | When the usage field is zero/absent. Cost is not emitted |
168+
| Tokens | `gen_ai.usage.input_tokens` / `output_tokens` pass through, except on `anthropic` spans, where cache read and write tokens are added to `input_tokens` (and `total_tokens` recomputed when present; see the note below). Cache tokens are aliased to `gen_ai.usage.cache_read_tokens` / `cache_write_tokens` and thinking tokens to `gen_ai.usage.reasoning.output_tokens`, originals kept | When the usage field is zero/absent. Cost is not emitted |
169169
| Session | Not emitted by AG2. `session.id` from traceAI's `using_session` / `using_attributes` context, which wins over a static value; or set it yourself: `setup(..., span_attributes={"session.id": "..."})` | When neither is set |
170170
| User, metadata, tags | `user.id`, `metadata`, `tag.tags` (and the other traceAI context keys) from `using_user` / `using_metadata` / `using_tags` / `using_attributes`; a key AG2 or `span_attributes` already set is not overridden | When the context does not set them |
171171
| Tools | `gen_ai.tool.name`, `gen_ai.tool.call.id`, `gen_ai.tool.type`; arguments/result only with `capture_content=True` | |
172-
| Input / output | `input.value` / `output.value` (what Observe shows as a span's input and output), copied from the captured chat messages, tool arguments and result, or human input | Without `capture_content=True`, when a `TraceConfig` flag hides the source, and always on `invoke_agent` and `record_usage` spans (AG2 records no content there) |
172+
| Input / output | `input.value` / `output.value` (what Observe shows as a span's input and output), copied from the captured chat messages, tool arguments and result, or human input. A single chat completion goes on `output.value` as its message object, which Observe's chat view shows as the assistant reply. `invoke_agent` gets the first chat input and the last chat output of its turn, so the trace list shows the question and the answer | Without `capture_content=True`, when a `TraceConfig` flag hides the source, and always on `record_usage` spans (AG2 records no content there) |
173173
| Errors | OTel status `ERROR` on failing tool / model / agent spans | |
174174

175175
Notes:
@@ -193,7 +193,15 @@ Notes:
193193
| `subtask` | Only if the sub-agent named in `ag2.usage.label` is itself instrumented and its `invoke_agent` span ended earlier in the same trace | Moved when it is, kept otherwise |
194194

195195
With that, the trace total equals AG2's own `UsageReport` total; the tests
196-
check this with real AG2 for every kind above. For the `subtask` match, the
196+
check this with real AG2 for every kind above. Exception: with an
197+
Anthropic model and prompt caching (`AnthropicConfig` caches by default),
198+
AG2's Anthropic mapper reports `prompt_tokens` without the cached tokens,
199+
while its OpenAI mapper and the GenAI conventions count them as input. The
200+
processor adds `cache_read_input_tokens` and `cache_creation_input_tokens`
201+
to `gen_ai.usage.input_tokens` on `anthropic` spans whose tokens count, so
202+
the trace total is the real prompt size and exceeds `UsageReport` by the
203+
cached tokens. A `record_usage` span that repeats a chat span keeps AG2's
204+
own numbers under `ag2.usage.*`. For the `subtask` match, the
197205
sub-agent's `TelemetryMiddleware` must use the agent's own name (`setup`
198206
does; `create_telemetry_middleware(agent_name=...)` must pass `agent.name`).
199207
On ag2 1.0.0 to 1.0.2 there are no `record_usage` spans, so aggregation,

‎python/frameworks/ag2/tests/test_processor.py‎

Lines changed: 157 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -484,7 +484,9 @@ def test_trace_config_reads_env(monkeypatch):
484484
{
485485
"input.value": '[{"content": "hi", "role": "user"}]',
486486
"input.mime_type": "application/json",
487-
"output.value": '[{"content": "yo", "role": "assistant"}]',
487+
# One completion: the message object, which Observe's chat view
488+
# shows as the assistant reply (it shows nothing for a list).
489+
"output.value": '{"content": "yo", "role": "assistant"}',
488490
"output.mime_type": "application/json",
489491
},
490492
),
@@ -571,6 +573,160 @@ def test_existing_input_and_output_values_are_kept(pipeline):
571573
assert (got["input.value"], got["output.value"]) == ("mine", "also mine")
572574

573575

576+
def test_chat_with_several_completions_keeps_the_list():
577+
messages = '[{"content": "a", "role": "assistant"}, {"content": "b", "role": "assistant"}]'
578+
got = normalize_attributes({"gen_ai.operation.name": "chat", "gen_ai.output.messages": messages})
579+
assert got["output.value"] == messages
580+
581+
582+
# --- invoke_agent input / output -------------------------------------------
583+
# AG2 records no content on invoke_agent; the first chat input and the last
584+
# chat output of the agent's turn are copied there so Observe's trace list
585+
# shows the question and the answer.
586+
587+
588+
def _agent_turn(provider, chats, agent_attrs=None):
589+
tracer = provider.get_tracer(AG2_INSTRUMENTATION_SCOPE)
590+
with tracer.start_as_current_span(
591+
"invoke_agent a", attributes={"gen_ai.operation.name": "invoke_agent", **(agent_attrs or {})}
592+
):
593+
for chat in chats:
594+
with tracer.start_as_current_span("chat m", attributes={"gen_ai.operation.name": "chat", **chat}):
595+
pass
596+
597+
598+
_TURN = [
599+
{
600+
"gen_ai.input.messages": '[{"content": "Q?", "role": "user"}]',
601+
"gen_ai.output.messages": '[{"content": null, "role": "assistant", "tool_calls": []}]',
602+
},
603+
{
604+
"gen_ai.input.messages": '[{"content": "Q?", "role": "user"}, {"content": "tool says", "role": "tool"}]',
605+
"gen_ai.output.messages": '[{"content": "A.", "role": "assistant"}]',
606+
},
607+
]
608+
609+
610+
def _agent(exporter):
611+
(agent,) = [s for s in exporter.get_finished_spans() if s.name == "invoke_agent a"]
612+
return attrs(agent)
613+
614+
615+
def test_agent_span_gets_first_chat_input_and_last_chat_output(pipeline):
616+
provider, exporter = pipeline
617+
_agent_turn(provider, _TURN)
618+
got = _agent(exporter)
619+
assert got["input.value"] == '[{"content": "Q?", "role": "user"}]'
620+
assert got["output.value"] == '{"content": "A.", "role": "assistant"}'
621+
assert got["input.mime_type"] == got["output.mime_type"] == "application/json"
622+
623+
624+
def test_agent_span_has_no_io_without_captured_content(pipeline):
625+
provider, exporter = pipeline
626+
_agent_turn(provider, [{"gen_ai.request.model": "m"}, {"gen_ai.request.model": "m"}])
627+
got = _agent(exporter)
628+
assert not {"input.value", "output.value", "input.mime_type", "output.mime_type"} & set(got)
629+
630+
631+
def test_agent_span_io_follows_hide_flags():
632+
provider, exporter = memory_provider()
633+
install_span_processor(provider, config=TraceConfig(hide_inputs=True))
634+
_agent_turn(provider, _TURN)
635+
got = _agent(exporter)
636+
assert "input.value" not in got and "Q?" not in str(got)
637+
assert got["output.value"] == '{"content": "A.", "role": "assistant"}'
638+
639+
640+
def test_agent_span_keeps_its_own_io(pipeline):
641+
provider, exporter = pipeline
642+
_agent_turn(provider, _TURN, agent_attrs={"input.value": "mine", "output.value": "also mine"})
643+
got = _agent(exporter)
644+
assert (got["input.value"], got["output.value"]) == ("mine", "also mine")
645+
646+
647+
def test_agent_io_state_is_released(pipeline):
648+
provider, exporter = pipeline
649+
_agent_turn(provider, _TURN)
650+
(processor,) = [p for p in provider._active_span_processor._span_processors if isinstance(p, AG2SpanProcessor)]
651+
assert not processor._agent_io
652+
653+
654+
# --- Anthropic input tokens --------------------------------------------------
655+
# AG2's Anthropic mapper reports prompt_tokens without cached tokens (its OpenAI
656+
# mapper includes them). gen_ai.usage.input_tokens means total input, so on an
657+
# anthropic span that counts, the cache read and write tokens are added.
658+
659+
660+
def test_anthropic_input_tokens_include_cached_tokens():
661+
got = normalize_attributes(
662+
{
663+
"gen_ai.operation.name": "chat",
664+
"gen_ai.provider.name": "anthropic",
665+
"gen_ai.usage.input_tokens": 4,
666+
"gen_ai.usage.output_tokens": 97,
667+
"gen_ai.usage.total_tokens": 101,
668+
"gen_ai.usage.cache_read_input_tokens": 9747,
669+
"gen_ai.usage.cache_creation_input_tokens": 2,
670+
}
671+
)
672+
assert got["gen_ai.usage.input_tokens"] == 9753
673+
assert got["gen_ai.usage.total_tokens"] == 9850
674+
assert (got["gen_ai.usage.cache_read_tokens"], got["gen_ai.usage.cache_write_tokens"]) == (9747, 2)
675+
676+
677+
def test_anthropic_without_total_gets_no_total_invented():
678+
got = normalize_attributes(
679+
{
680+
"gen_ai.operation.name": "chat",
681+
"gen_ai.provider.name": "anthropic",
682+
"gen_ai.usage.input_tokens": 4,
683+
"gen_ai.usage.output_tokens": 97,
684+
"gen_ai.usage.cache_read_input_tokens": 9747,
685+
}
686+
)
687+
assert got["gen_ai.usage.input_tokens"] == 9751
688+
assert "gen_ai.usage.total_tokens" not in got
689+
690+
691+
@pytest.mark.parametrize("provider_name", ["openai", None])
692+
def test_other_providers_input_tokens_unchanged(provider_name):
693+
source = {
694+
"gen_ai.operation.name": "chat",
695+
"gen_ai.usage.input_tokens": 11,
696+
"gen_ai.usage.cache_read_input_tokens": 3,
697+
}
698+
if provider_name:
699+
source["gen_ai.provider.name"] = provider_name
700+
assert normalize_attributes(source)["gen_ai.usage.input_tokens"] == 11
701+
702+
703+
def test_anthropic_duplicate_usage_span_stays_off_the_counted_keys():
704+
got = normalize_attributes(
705+
{
706+
"ag2.span.type": "usage",
707+
"ag2.usage.kind": "model_call",
708+
"gen_ai.provider.name": "anthropic",
709+
"gen_ai.usage.input_tokens": 4,
710+
"gen_ai.usage.cache_read_input_tokens": 9747,
711+
}
712+
)
713+
assert "gen_ai.usage.input_tokens" not in got
714+
assert got["ag2.usage.input_tokens"] == 4 # AG2's own number, kept as reported
715+
716+
717+
def test_anthropic_counted_usage_span_includes_cached_tokens():
718+
got = normalize_attributes(
719+
{
720+
"ag2.span.type": "usage",
721+
"ag2.usage.kind": "aggregation",
722+
"gen_ai.provider.name": "anthropic",
723+
"gen_ai.usage.input_tokens": 4,
724+
"gen_ai.usage.cache_read_input_tokens": 100,
725+
}
726+
)
727+
assert got["gen_ai.usage.input_tokens"] == 104
728+
729+
574730
# --- installation ----------------------------------------------------------
575731

576732

0 commit comments

Comments
 (0)