feat(ag2): traceAI-ag2 for AG2 1.x via native TelemetryMiddleware (TH-8236) - #210

The Node OTLP exporter streams with Transfer-Encoding: chunked and sends
no Content-Length. The Receiver read Content-Length only, so every Node
export came back 400 with zero spans; two TH-8103 children had to put a
de-chunking relay in front of it. Receiver now accepts both framings.

It also records one entry per accepted export (path, lower-cased
headers, flattened resource attributes) via requests(), so a contract
test can assert X-Api-Key/X-Secret-Key and project_name through the
shared harness instead of a private recorder.

Verified: 6 harness tests pass, and the TanStack example's real Node
exporter delivered 4 chunked exports (4 spans, collector path, both
auth headers, project_name/project_type) with no relay.

Refs: TH-8339, TH-8103

New package python/frameworks/ag2 (traceAI-ag2, import traceai_ag2) for
AG2 1.x (PyPI ag2, import ag2). Native-first per D-8236:

- setup(*agents, tracer_provider=None, capture_content=False, ...) registers
  the Future AGI exporter (register(project_type=OBSERVE) when no provider is
  passed) and attaches AG2's own TelemetryMiddleware to each agent through the
  public Agent.add_middleware, overriding the upstream capture_content=True
  default. Idempotent per agent. Agent is not wrapped or patched.
- create_telemetry_middleware() for agents built later or per-call middleware.
- AG2SpanProcessor, prepended ahead of the exporter, on the
  opentelemetry.instrumentation.ag2 scope only: sets gen_ai.span.kind from
  gen_ai.operation.name (chat/execute_tool/invoke_agent), copies
  cache_creation_input_tokens, cache_read_input_tokens and thinking_tokens to
  the dotted GenAI semconv keys (originals kept), and applies TraceConfig as a
  second content gate.
- ag2>=1.0.0 (first stable 1.x with TelemetryMiddleware), no dependency on
  autogen or autogen-agentchat; fi-instrumentation-otel>=1.0.0.

Tests: processor unit tests, setup tests against real ag2 with its scripted
TestConfig model, import isolation (no autogen), wire-level resource/header
test, and shared-harness Receiver contract tests over real OTLP/HTTP.

Refs: TH-8236

record_usage model_call repeats the chat span's input/output tokens
(ag2 1.1.2 telemetry.py 437-440 vs 501-504). fi-collector promotes
gen_ai.usage.* into the token columns on any span (adapter.go
inputTokenKeys/outputTokenKeys/totalTokenKeys) and Observe sums
total_tokens over every span of a trace (span_reader v2, tasks/session.py),
so the trace total doubled.

On model_call and aggregation usage spans the processor now moves the
promoted keys to ag2.usage.*. Values are kept. subtask and compaction
usage spans keep their tokens: no chat span repeats them.

Tests first: the contract test now asserts the summed promoted input
tokens equal one model call; 2 new processor tests. 52 passed on ag2
1.1.2, 50 passed + 2 skipped on 1.0.3.

approved: Nikhil 2026-10-03 blanket
Refs: TH-8236

Published fi-instrumentation-otel 1.0.0 crashes in register() with
opentelemetry-exporter-otlp-proto-http 1.45.0 (otel.py:289
AttributeError: 'HTTPSpanExporter' object has no attribute '_headers').

Evidence, this branch's suite with PYTHONPATH=python/frameworks/ag2:
python/tests (no in-repo fi_instrumentation) and ag2 1.1.2:
- --with fi-instrumentation-otel==1.0.0: 5 failed, 58 passed
  (3x the _headers AttributeError, 1x register() got an unexpected
  keyword argument 'timeout')
- --with fi-instrumentation-otel==1.1.0: 63 passed

Floor raised in pyproject.toml with the reason; README and CHANGELOG note it.

approved: Nikhil 2026-10-03 blanket
Refs: TH-8236

R4: a second setup(..., config=TraceConfig(hide_inputs=True,
hide_outputs=True)) on a provider that already had an AG2SpanProcessor
was silently ignored, so content was exported despite the hide request
(reviewer probe: gen_ai.input/output.messages still on the chat span).
install_span_processor now applies an explicit config to the installed
processor (AG2SpanProcessor.update_config; most recent call wins) and
logs a warning when it changes. config=None keeps the installed config,
so a later setup() never re-exposes content.

R5: on_start did nothing, so using_session / using_user / using_metadata /
using_attributes were ignored on AG2 spans. on_start now copies
fi_instrumentation get_attributes_from_context() onto AG2-scope spans
without overriding keys AG2 already set (later AG2 set_attribute calls
still win), except session.id: the context session is re-applied in
on_end so it beats a static span_attributes value. Pending sessions are
kept in a bounded map keyed by (trace_id, span_id) and released on end.
Foreign-scope spans are untouched.

README (setup, sessions, conformance table), setup()/processor docstrings
and CHANGELOG updated.

Tests first, red on ec9fc07: test_later_setup_config_replaces_installed_
processor_config, test_using_session_sets_session_on_every_ag2_span,
test_using_attributes_context_reaches_ag2_spans,
test_context_session_wins_but_other_ag2_keys_are_not_overridden,
test_pending_context_sessions_are_bounded_and_released_on_end (also red
against a no-eviction mutant: 40 == 16). Guards that stay green:
test_later_setup_without_config_keeps_installed_config,
test_context_attributes_skip_foreign_scope_spans. 70 passed on ag2 1.1.2.

approved: Nikhil 2026-10-03 blanket
Refs: TH-8236

R6: AG2 records message content as one JSON string per span
(gen_ai.input.messages / gen_ai.output.messages, telemetry.py 477/517 at
1.1.2), never as the flattened gen_ai.input.messages.{i}.message.content
keys that TraceConfig.mask's hide_input_text / hide_output_text /
hide_input_images rules match. With capture_content=True, those flags
left every prompt and completion in place, while the module docstring
and README claimed "the other TraceConfig.mask rules" applied.

The processor now drops whole attributes (privacy-safe; the text inside
the JSON cannot be redacted on its own):
- hide_inputs / hide_input_text: gen_ai.input.messages,
  gen_ai.system_instructions, gen_ai.tool.call.arguments,
  ag2.human_input.prompt
- hide_outputs / hide_output_text: gen_ai.output.messages,
  gen_ai.tool.call.result, ag2.human_input.response
- hide_input_messages: gen_ai.input.messages, gen_ai.system_instructions
- hide_output_messages: gen_ai.output.messages
hide_input_images and hide_embedding_vectors have nothing to act on (AG2
serialises only text parts and records no embeddings); a real-AG2 test
pins that image bytes never reach a span. Module docstring, README
(privacy table of which flags apply) and CHANGELOG say so.

Tests first, red on fbaac8e (7 failed, 75 passed; final test files copied
into a detached fbaac8e worktree): test_trace_config_flags_on_ag2_json_
content[hide_input_text, hide_output_text, hide_inputs,
hide_input_messages, hide_input_text+hide_output_text],
test_text_hide_flags_remove_ag2_message_json and
test_hide_input_text_keeps_outputs (real AG2). Passing on fbaac8e by
design: the hide_outputs / hide_output_messages params,
test_flags_with_nothing_to_act_on_leave_ag2_content,
test_ag2_never_records_image_bytes. Green: 82 passed on ag2 1.1.2;
75 passed, 7 skipped on ag2 1.0.0.

approved: Nikhil 2026-10-03 blanket
Refs: TH-8236

R7: the provider from fi_instrumentation.register() keeps
_default_processor=True, so the application's first
provider.add_span_processor() call after setup() shut down and cleared
the whole chain (fi_instrumentation/otel.py add_span_processor,
lines 329-340), AG2SpanProcessor included. The new exporter then got
un-normalized AG2 spans (no gen_ai.span.kind, no usage moves, no
TraceConfig content gate).

install_span_processor now wraps the provider instance's
add_span_processor once (marker attribute): the AG2 processor is lifted
out of the chain for the call, so fi's reset neither shuts it down nor
drops it, and is put back first afterwards. provider.shutdown() still
shuts it down. Installing on a provider whose active processor is a
ConcurrentMultiSpanProcessor logs a warning, since parallel on_end calls
do not guarantee normalization before export. Docstring, README (set-up
note, troubleshooting) and CHANGELOG updated.

Tests first, red on 4a9922e (2 failed, 2 passed):
test_later_add_span_processor_on_fi_provider_keeps_ag2_processor_first
(chain was just the new SimpleSpanProcessor) and
test_install_warns_when_processors_run_concurrently. Passing on
4a9922e by design: test_later_add_span_processor_on_sdk_provider_keeps_
ag2_processor_first and test_provider_shutdown_still_shuts_the_processor_
down; mutant "do not put the processor back" turns both later_add tests
red. Green: 86 passed on ag2 1.1.2; 79 passed, 7 skipped on ag2 1.0.0.

approved: Nikhil 2026-10-03 blanket
Refs: TH-8236

…spans

R7: AG2SpanProcessor.on_end replaced span._attributes with a plain dict.
ReadableSpan.dropped_attributes (exported as dropped_attributes_count)
reads BoundedAttributes.dropped and returns 0 for anything else, so
every normalized AG2 span reported 0 dropped attributes, and the span's
max_span_attributes / max_attribute_length limits and its ended-span
immutability were lost.

on_end now rebuilds a BoundedAttributes with the previous maxlen,
max_value_len and immutability, and carries the previous dropped count
over. Keys the processor adds count against the same limit: on a span
already full, the oldest key is evicted and counted, as the SDK does
(only reachable at the limit, 128 by default). Docstring, README note
and CHANGELOG updated.

Tests first, red on c1196a4 (2 failed):
test_normalizing_keeps_the_sdk_dropped_attribute_count (assert 0 == 1
on span.dropped_attributes) and
test_keys_the_processor_adds_count_against_span_limits (plain dict kept
3 keys over a limit of 2). Green: 88 passed on ag2 1.1.2.

approved: Nikhil 2026-10-03 blanket
Refs: TH-8236

…g again

setup() without tracer_provider called fi register() on every call, so
setup(planner) then setup(worker) put each agent on its own provider with
its own AG2SpanProcessor. The planner's processor never saw the worker's
invoke_agent span, kept the subtask rollup's promoted tokens, and the trace
counted the worker's spend twice (real AG2: 91/15 input/output tokens
against a UsageReport of 51/11).

setup() now remembers the provider it registered and reuses it on later
calls without tracer_provider until that provider is shut down. A later
config= still replaces the installed TraceConfig; a different project_name
is ignored with a warning. README and CHANGELOG updated.

approved: Nikhil 2026-10-03 blanket
Refs: TH-8236

…bute limits

on_start writes the traceAI context keys (session.id, user.id, metadata, ...)
before AG2 sets any attribute, so they are the oldest keys on every AG2 span.
When the keys the processor adds (span kind, usage aliases) overflow
max_span_attributes, BoundedAttributes evicts oldest-first, so session.id
and user.id went first and the span dropped out of its session's totals.

on_end now moves the context keys behind AG2's keys before re-bounding, so
AG2's oldest keys are evicted instead. The eviction count is unchanged.
Test: real AG2 chat span with using_attributes at a lowered limit.

approved: Nikhil 2026-10-03 blanket
Refs: TH-8236

…tured

setup() adds TelemetryMiddleware as the innermost middleware. Observed with
real AG2 (1.1.2 and 1.0.0) and now pinned by tests:
- a middleware registered before setup() that answers on_llm_call without
  calling call_next leaves no chat span (invoke_agent ends UNSET);
- one that answers on_turn leaves no span at all;
- AG2's halt (HaltEvent from AlertPolicy) with Agent(assembly=...) policies
  leaves no chat span, because the halt check is added before telemetry;
- with per-call Plugin policies (ag2 1.1.2 only) the halted call gets a chat
  span with status UNSET and no attribute carrying the halt reason.

No capture is added. README gains a Limits section; CHANGELOG notes it.

approved: Nikhil 2026-10-03 blanket
Refs: TH-8236

- Tested-version line names only what the suite ran: ag2 1.0.0, 1.0.3 and
  1.1.2 on Python 3.11, and ag2 1.1.2 on Python 3.10, 3.12 and 3.13.
- hide_llm_invocation_parameters has nothing to act on: AG2 1.0.0 and 1.1.2
  set no gen_ai.request.parameters. A drift guard now checks AG2's
  telemetry source for that key; the processor docstring says the same.
- Limits: the subtask dedupe matches by agent name within a trace (M1);
  create_telemetry_middleware() defaults agent_name=None ("unknown") and
  tracer_provider=None (OTel global provider) (N2); concurrent first
  add_span_processor calls can race with fi's unlocked reset (N4).

approved: Nikhil 2026-10-03 blanket
Refs: TH-8236

AG2's cache_read_input_tokens / cache_creation_input_tokens were aliased to
the dotted semconv names gen_ai.usage.cache_read.input_tokens /
cache_creation.input_tokens. Future AGI's attribute registry reads
gen_ai.usage.cache_read_tokens / cache_write_tokens and never the dotted
names, so cache counts never reached it. Alias to those keys instead (the
originals stay). A record_usage span that repeats a chat span's usage moves
them to ag2.usage.* with the token counts, so each call carries them once.

Checked live: an AnthropicConfig agent on CLIProxy (AG2's default prompt
caching) stored cache_read_tokens 9747 and cache_write_tokens 9749 on its
chat spans in ClickHouse, and ag2.usage.* on the record_usage spans.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

…erve

Anthropic input tokens: AG2's Anthropic mapper reports prompt_tokens without
cache reads and writes (its OpenAI mapper includes them), and AnthropicConfig
caches by default, so a chat span showed 4 input tokens for a 9,750-token
prompt. On anthropic spans whose tokens count, cache_read_input_tokens and
cache_creation_input_tokens are now added to gen_ai.usage.input_tokens, and
total_tokens is recomputed when present. The trace total can then exceed
AG2's UsageReport by the cached tokens (README, CHANGELOG). Duplicate
record_usage spans keep AG2's own numbers under ag2.usage.*.

Chat reply: a single completion goes on output.value as its message object.
Observe's chat view shows that as the assistant reply and shows nothing for
a one-item list.

Agent I/O: invoke_agent carries no content in AG2. With capture_content=True
it now gets the first chat input and the last chat output of its turn, so
the trace list shows the question and the answer. Hidden content is not
copied, and the agent's own values win.

Checked live on CLIProxy: Anthropic chat spans store 9751/94 and 9753/88
tokens; the capture run's trace list shows Input/Output, and the chat span
shows the assistant reply bubble.

Co-Authored-By: Claude Opus 5.5 <[email protected]>