Skip to content

Commit 00802f3

Browse files
committed
fix(ag2): count aggregation and subtask usage exactly once
Review of #210 (reproduced with real ag2 1.1.2) found two token-total errors against AG2's own UsageReport: - aggregation: ag2 aggregate.py 97-99 / 193-195 call the model client on a throwaway Context(MemoryStream()) that never reaches on_llm_call, so the record_usage aggregation span is the only copy of that spend. Demoting it dropped it from the trace (11 promoted vs 111 actual input tokens). 'aggregation' is removed from the demoted kinds; compaction stays kept. - subtask: the rollup (run_task.py 63-93, label = sub-agent name) repeats the sub-agent's chat spans when that sub-agent is itself instrumented (91 vs 51). AG2SpanProcessor now records agent names from ended invoke_agent spans per trace in a bounded LRU map (1024 traces) and moves a subtask rollup's promoted tokens to ag2.usage.* only when its ag2.usage.label names an agent seen in the same trace. An uninstrumented worker's rollup keeps its tokens (the only copy). Comment, module docstring, README usage table and CHANGELOG describe the moves per usage kind. Tests first, red on ec9fc07 (7 failed): new tests/test_usage_totals.py runs real AG2 with the scripted TestConfig model and asserts summed promoted input and output tokens equal UsageReport for aggregation, instrumented and uninstrumented subagent_tool workers, a namesake in another trace, and one run covering main loop + compaction + aggregation + subtask; synthetic processor tests for aggregation/compaction kept, subtask demoted/kept and the bounded map. Mutants "demote every subtask" and "agent names not scoped per trace" each turn guard tests red. 63 passed on ag2 1.1.2, 61 passed + 2 skipped on 1.0.3. approved: Nikhil 2026-10-03 blanket Refs: TH-8236
1 parent ec9fc07 commit 00802f3

5 files changed

Lines changed: 416 additions & 22 deletions

File tree

‎python/frameworks/ag2/CHANGELOG.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,3 +8,11 @@
88
`gen_ai.operation.name` and aliases AG2's cache/thinking usage keys onto the
99
GenAI semconv names (originals kept). Does not wrap `Agent` and never
1010
imports `autogen`. Refs: TH-8236.
11+
- Token totals: Future AGI sums the promoted `gen_ai.usage.input_tokens` /
12+
`output_tokens` / `total_tokens` over every span of a trace. The processor
13+
moves those keys to `ag2.usage.*` (values kept) on `record_usage model_call`
14+
spans, which repeat the chat span's tokens, and on `record_usage subtask`
15+
rollups whose sub-agent (`ag2.usage.label`) is itself instrumented in the
16+
same trace. `aggregation` and `compaction` usage keeps its promoted tokens:
17+
AG2 calls the model outside the middleware for those, so the usage span is
18+
the only copy. The trace total now equals AG2's `UsageReport` total.

‎python/frameworks/ag2/README.md‎

Lines changed: 21 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -125,13 +125,28 @@ Notes:
125125

126126
- AG2 also emits `await_human_input {agent}` spans. That operation is not in
127127
the Microsoft Agent Framework kind table, so no kind is guessed for it.
128-
- From ag2 1.0.3, AG2 also emits a `record_usage {kind}` span per usage event,
129-
which carries the same token counts as the `chat` span it accounts for.
128+
- From ag2 1.0.3, AG2 also emits a `record_usage {kind}` span per usage event
129+
(`ag2.usage.kind` = `model_call`, `aggregation`, `compaction` or `subtask`).
130130
It has no operation name and gets no span kind; the aliases are applied.
131-
Future AGI totals tokens by summing every span in a trace, so on
132-
`model_call` (and `aggregation`) usage spans the processor moves
133-
`gen_ai.usage.input_tokens` / `output_tokens` / `total_tokens` to
134-
`ag2.usage.*`. The values stay on the span; each model call counts once.
131+
Future AGI totals tokens by summing the promoted
132+
`gen_ai.usage.input_tokens` / `output_tokens` / `total_tokens` over every
133+
span in a trace, so the processor moves those three keys to
134+
`ag2.usage.input_tokens` / `output_tokens` / `total_tokens` (values kept on
135+
the span) exactly where a chat span already counts the same spend:
136+
137+
| `ag2.usage.kind` | Same tokens already on a chat span? | Promoted tokens |
138+
|---|---|---|
139+
| `model_call` | Yes: the chat span of that LLM call | Moved to `ag2.usage.*` |
140+
| `aggregation` | No: memory aggregation calls the model client directly, outside the middleware | Kept |
141+
| `compaction` | No: history compaction calls the model client directly | Kept |
142+
| `subtask` | Only if the sub-agent named in `ag2.usage.label` is itself instrumented and its `invoke_agent` span ended earlier in the same trace | Moved when it is, kept otherwise |
143+
144+
With that, the trace total equals AG2's own `UsageReport` total; the tests
145+
check this with real AG2 for every kind above. For the `subtask` match, the
146+
sub-agent's `TelemetryMiddleware` must use the agent's own name (`setup`
147+
does; `create_telemetry_middleware(agent_name=...)` must pass `agent.name`).
148+
On ag2 1.0.0 to 1.0.2 there are no `record_usage` spans, so aggregation,
149+
compaction and uninstrumented sub-agent spend does not reach the trace.
135150
- Network trace propagation (`ag2.otel.traceparent`) is upstream. Apart from
136151
that usage move, the processor never removes attributes it did not hide by
137152
request.

‎python/frameworks/ag2/tests/test_processor.py‎

Lines changed: 71 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -183,13 +183,14 @@ def test_model_call_usage_span_does_not_double_count_trace_tokens(pipeline):
183183

184184

185185
def test_subtask_usage_span_keeps_promoted_tokens(pipeline):
186-
"""Only kinds that duplicate a chat span are demoted; others keep their tokens."""
186+
"""A subtask rollup with no instrumented worker in the trace is the only copy."""
187187
provider, exporter = pipeline
188188
_emit(
189189
provider,
190190
{
191191
"ag2.span.type": "usage",
192192
"ag2.usage.kind": "subtask",
193+
"ag2.usage.label": "worker",
193194
"gen_ai.usage.input_tokens": 4,
194195
},
195196
name="record_usage subtask",
@@ -198,6 +199,75 @@ def test_subtask_usage_span_keeps_promoted_tokens(pipeline):
198199
assert attrs(span)["gen_ai.usage.input_tokens"] == 4
199200

200201

202+
@pytest.mark.parametrize("kind", ["aggregation", "compaction"])
203+
def test_out_of_band_usage_spans_keep_promoted_tokens(pipeline, kind):
204+
"""aggregate.py / compact.py call the model client outside on_llm_call, so
205+
no chat span carries these tokens."""
206+
provider, exporter = pipeline
207+
_emit(
208+
provider,
209+
{
210+
"ag2.span.type": "usage",
211+
"ag2.usage.kind": kind,
212+
"gen_ai.usage.input_tokens": 100,
213+
"gen_ai.usage.output_tokens": 50,
214+
},
215+
name=f"record_usage {kind}",
216+
)
217+
(span,) = exporter.get_finished_spans()
218+
got = attrs(span)
219+
assert got["gen_ai.usage.input_tokens"] == 100
220+
assert got["gen_ai.usage.output_tokens"] == 50
221+
assert "ag2.usage.input_tokens" not in got
222+
223+
224+
def _subtask_after_worker(provider, worker_name: str, label: str) -> None:
225+
tracer = provider.get_tracer(AG2_INSTRUMENTATION_SCOPE)
226+
with tracer.start_as_current_span("invoke_agent planner", attributes={"gen_ai.operation.name": "invoke_agent", "gen_ai.agent.name": "planner"}):
227+
with tracer.start_as_current_span(
228+
f"invoke_agent {worker_name}",
229+
attributes={"gen_ai.operation.name": "invoke_agent", "gen_ai.agent.name": worker_name},
230+
):
231+
pass
232+
with tracer.start_as_current_span(
233+
"record_usage subtask",
234+
attributes={
235+
"ag2.span.type": "usage",
236+
"ag2.usage.kind": "subtask",
237+
"ag2.usage.label": label,
238+
"gen_ai.usage.input_tokens": 40,
239+
"gen_ai.usage.output_tokens": 4,
240+
},
241+
):
242+
pass
243+
244+
245+
def test_subtask_usage_span_demoted_when_its_agent_ran_instrumented_in_the_trace(pipeline):
246+
provider, exporter = pipeline
247+
_subtask_after_worker(provider, worker_name="worker", label="worker")
248+
(rollup,) = [s for s in exporter.get_finished_spans() if s.name == "record_usage subtask"]
249+
got = attrs(rollup)
250+
assert "gen_ai.usage.input_tokens" not in got and "gen_ai.usage.output_tokens" not in got
251+
assert (got["ag2.usage.input_tokens"], got["ag2.usage.output_tokens"]) == (40, 4)
252+
253+
254+
def test_subtask_usage_span_kept_when_label_is_another_agent(pipeline):
255+
provider, exporter = pipeline
256+
_subtask_after_worker(provider, worker_name="other_worker", label="worker")
257+
(rollup,) = [s for s in exporter.get_finished_spans() if s.name == "record_usage subtask"]
258+
assert attrs(rollup)["gen_ai.usage.input_tokens"] == 40
259+
260+
261+
def test_agent_name_map_is_bounded():
262+
processor = AG2SpanProcessor(config=TraceConfig(), max_tracked_traces=2)
263+
provider = TracerProvider()
264+
provider.add_span_processor(processor)
265+
for name in ("a", "b", "c"):
266+
_emit(provider, {"gen_ai.operation.name": "invoke_agent", "gen_ai.agent.name": name})
267+
assert len(processor._agents_by_trace) == 2
268+
assert [set(v) for v in processor._agents_by_trace.values()] == [{"b"}, {"c"}]
269+
270+
201271
def test_processor_only_adds_keys_by_default(pipeline):
202272
"""Nothing AG2 set is stripped, including propagation-related keys."""
203273
provider, exporter = pipeline
Lines changed: 236 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,236 @@
1+
"""Trace token totals against AG2's own ``UsageReport``, with real AG2.
2+
3+
fi-collector promotes ``gen_ai.usage.input_tokens`` / ``output_tokens`` /
4+
``total_tokens`` into its token columns on any span, and Observe sums them over
5+
every span of a trace. So the sum of those keys over a trace must equal the
6+
total AG2 itself reports for the run. The model is AG2's scripted
7+
``ag2.testing.TestConfig``: no network, no API key.
8+
"""
9+
10+
from __future__ import annotations
11+
12+
import asyncio
13+
from typing import Any, Iterable, Tuple
14+
15+
import pytest
16+
17+
from traceai_ag2 import setup
18+
19+
from ._support import attrs, emits_record_usage, memory_provider
20+
21+
PROMOTED_INPUT = "gen_ai.usage.input_tokens"
22+
PROMOTED_OUTPUT = "gen_ai.usage.output_tokens"
23+
PROMOTED_TOTAL = "gen_ai.usage.total_tokens"
24+
25+
needs_record_usage = pytest.mark.skipif(
26+
not emits_record_usage(),
27+
reason="record_usage spans (the only carrier of aggregation/compaction/sub-task rollup spend) start at ag2 1.0.3",
28+
)
29+
30+
31+
def _ag2_version() -> Tuple[int, ...]:
32+
from importlib.metadata import version
33+
34+
parts = []
35+
for piece in version("ag2").split(".")[:3]:
36+
digits = "".join(ch for ch in piece if ch.isdigit())
37+
parts.append(int(digits or 0))
38+
return tuple(parts)
39+
40+
41+
def _response(text: str, prompt_tokens: int, completion_tokens: int) -> Any:
42+
from ag2.events import ModelMessage, ModelResponse
43+
from ag2.usage import Usage
44+
45+
return ModelResponse(
46+
ModelMessage(text),
47+
usage=Usage(
48+
prompt_tokens=prompt_tokens,
49+
completion_tokens=completion_tokens,
50+
total_tokens=prompt_tokens + completion_tokens,
51+
),
52+
model="fake-model",
53+
)
54+
55+
56+
def _trace_sums(spans: Iterable[Any]) -> Tuple[int, int, int]:
57+
spans = list(spans)
58+
assert len({s.context.trace_id for s in spans}) == 1, "expected one trace"
59+
total_in = sum(attrs(s).get(PROMOTED_INPUT, 0) for s in spans)
60+
total_out = sum(attrs(s).get(PROMOTED_OUTPUT, 0) for s in spans)
61+
total = sum(attrs(s).get(PROMOTED_TOTAL, 0) for s in spans)
62+
return total_in, total_out, total
63+
64+
65+
def _run(agent: Any, prompt: str) -> Any:
66+
async def _go() -> Any:
67+
reply = await agent.ask(prompt)
68+
return await reply.usage()
69+
70+
return asyncio.run(_go())
71+
72+
73+
def _assert_trace_matches_report(spans: Any, report: Any) -> None:
74+
total_in, total_out, total = _trace_sums(spans)
75+
assert (total_in, total_out) == (report.total.prompt_tokens, report.total.completion_tokens)
76+
# AG2 never sets gen_ai.usage.total_tokens; if a later release does, it must
77+
# not be counted twice either.
78+
assert total in (0, report.total.prompt_tokens + report.total.completion_tokens)
79+
80+
81+
def _knowledge(*, aggregate: bool = False, compact: bool = False) -> Any:
82+
from ag2.knowledge import MemoryKnowledgeStore
83+
from ag2.knowledge.config import KnowledgeConfig
84+
from ag2.testing import TestConfig
85+
86+
kwargs: dict = {"store": MemoryKnowledgeStore(), "expose_tool": False, "write_event_log": False}
87+
if aggregate:
88+
from ag2.aggregate import AggregateTrigger, ConversationSummaryAggregate
89+
90+
kwargs["aggregate"] = ConversationSummaryAggregate(TestConfig(_response("summary", 100, 50)))
91+
kwargs["aggregate_trigger"] = AggregateTrigger(on_end=True)
92+
if compact:
93+
from ag2.compact import CompactTrigger, SummarizeCompact
94+
95+
kwargs["compact"] = SummarizeCompact(target=2, config=TestConfig(_response("compacted", 300, 30)))
96+
kwargs["compact_trigger"] = CompactTrigger(max_events=2)
97+
return KnowledgeConfig(**kwargs)
98+
99+
100+
def _delegating_planner(worker: Any, **planner_kwargs: Any) -> Any:
101+
from ag2 import Agent
102+
from ag2.events import ToolCallEvent
103+
from ag2.testing import TestConfig
104+
from ag2.tools.subagents import subagent_tool
105+
106+
return Agent(
107+
"planner",
108+
config=TestConfig(
109+
ToolCallEvent("task_worker", arguments='{"objective": "x"}', id="call_task_1"),
110+
_response("ok", 11, 7),
111+
),
112+
tools=[subagent_tool(worker, description="delegate")],
113+
**planner_kwargs,
114+
)
115+
116+
117+
def _worker() -> Any:
118+
from ag2 import Agent
119+
from ag2.testing import TestConfig
120+
121+
return Agent("worker", config=TestConfig(_response("done", 40, 4)))
122+
123+
124+
# --- R1: memory aggregation calls the model outside on_llm_call ---------------
125+
126+
127+
@needs_record_usage
128+
def test_aggregation_usage_counts_once_in_trace_total():
129+
"""aggregate.py calls the client on a throwaway stream: no chat span repeats it."""
130+
from ag2 import Agent
131+
from ag2.testing import TestConfig
132+
133+
provider, exporter = memory_provider()
134+
agent = Agent("mem_bot", config=TestConfig(_response("hello", 11, 7)), knowledge=_knowledge(aggregate=True))
135+
setup(agent, tracer_provider=provider)
136+
report = _run(agent, "hi")
137+
138+
assert (report.total.prompt_tokens, report.total.completion_tokens) == (111, 57)
139+
spans = exporter.get_finished_spans()
140+
(aggregation,) = [s for s in spans if s.name == "record_usage aggregation"]
141+
assert attrs(aggregation)[PROMOTED_INPUT] == 100
142+
assert attrs(aggregation)[PROMOTED_OUTPUT] == 50
143+
_assert_trace_matches_report(spans, report)
144+
145+
146+
# --- R2: a sub-task rollup repeats an instrumented worker's chat spans --------
147+
148+
149+
@needs_record_usage
150+
def test_subtask_rollup_keeps_tokens_when_worker_is_not_instrumented():
151+
"""The rollup is the only copy of the worker's spend: it must stay promoted."""
152+
provider, exporter = memory_provider()
153+
worker = _worker()
154+
planner = _delegating_planner(worker)
155+
setup(planner, tracer_provider=provider)
156+
report = _run(planner, "plan")
157+
158+
spans = exporter.get_finished_spans()
159+
(rollup,) = [s for s in spans if s.name == "record_usage subtask"]
160+
assert attrs(rollup)["ag2.usage.label"] == "worker"
161+
assert attrs(rollup)[PROMOTED_INPUT] == 40
162+
assert attrs(rollup)[PROMOTED_OUTPUT] == 4
163+
_assert_trace_matches_report(spans, report)
164+
165+
166+
def test_subtask_rollup_counts_once_when_worker_is_instrumented():
167+
"""setup(planner, worker): the worker's chat spans already carry its tokens."""
168+
provider, exporter = memory_provider()
169+
worker = _worker()
170+
planner = _delegating_planner(worker)
171+
setup(planner, worker, tracer_provider=provider)
172+
report = _run(planner, "plan")
173+
174+
spans = exporter.get_finished_spans()
175+
assert [s.name for s in spans if s.name.startswith("invoke_agent")] == ["invoke_agent worker", "invoke_agent planner"]
176+
for rollup in [s for s in spans if s.name == "record_usage subtask"]:
177+
a = attrs(rollup)
178+
assert PROMOTED_INPUT not in a and PROMOTED_OUTPUT not in a
179+
assert a["ag2.usage.input_tokens"] == 40
180+
assert a["ag2.usage.output_tokens"] == 4
181+
_assert_trace_matches_report(spans, report)
182+
183+
184+
@needs_record_usage
185+
def test_subtask_rollup_for_an_agent_seen_in_another_trace_keeps_tokens():
186+
"""Agent names are tracked per trace: an instrumented worker's earlier,
187+
separate run must not demote a later rollup from an uninstrumented copy."""
188+
from ag2 import Agent
189+
from ag2.testing import TestConfig
190+
191+
provider, exporter = memory_provider()
192+
instrumented_namesake = Agent("worker", config=TestConfig(_response("solo", 1, 1)))
193+
setup(instrumented_namesake, tracer_provider=provider)
194+
_run(instrumented_namesake, "warm up")
195+
exporter.clear()
196+
197+
planner = _delegating_planner(_worker())
198+
setup(planner, tracer_provider=provider)
199+
report = _run(planner, "plan")
200+
_assert_trace_matches_report(exporter.get_finished_spans(), report)
201+
202+
203+
# --- every usage route in one run ---------------------------------------------
204+
205+
206+
@needs_record_usage
207+
@pytest.mark.parametrize("instrument_worker", [False, True], ids=["worker-uninstrumented", "worker-instrumented"])
208+
def test_trace_tokens_equal_usage_report_across_every_usage_route(instrument_worker):
209+
"""Main loop (chat + record_usage model_call), compaction, aggregation and a
210+
sub-task rollup in one run: summed promoted input and output tokens equal
211+
AG2's UsageReport.
212+
213+
Compaction is left out on ag2 < 1.0.4: there ``UsageReport`` reads the
214+
compacted history, so it drops usage events compaction removed (measured on
215+
1.0.3: report 111 input tokens, trace and actual spend 451). The trace is
216+
right in that case and the report is not, so it is no ground truth.
217+
"""
218+
compact = _ag2_version() >= (1, 0, 4)
219+
provider, exporter = memory_provider()
220+
worker = _worker()
221+
planner = _delegating_planner(worker, knowledge=_knowledge(aggregate=True, compact=compact))
222+
agents = (planner, worker) if instrument_worker else (planner,)
223+
setup(*agents, tracer_provider=provider)
224+
report = _run(planner, "plan")
225+
226+
spans = exporter.get_finished_spans()
227+
names = {s.name for s in spans}
228+
expected_routes = ["record_usage model_call", "record_usage aggregation", "record_usage subtask"]
229+
if compact:
230+
expected_routes.append("record_usage compaction")
231+
for expected in expected_routes:
232+
assert expected in names, f"{expected} not emitted; the scenario no longer covers it"
233+
spent_in = 11 + 40 + 100 + (300 if compact else 0)
234+
spent_out = 7 + 4 + 50 + (30 if compact else 0)
235+
assert (report.total.prompt_tokens, report.total.completion_tokens) == (spent_in, spent_out)
236+
_assert_trace_matches_report(spans, report)

0 commit comments

Comments
 (0)