Skip to content

Commit c1196a4

Browse files
committed
fix(ag2): keep AG2SpanProcessor first after a later add_span_processor
R7: the provider from fi_instrumentation.register() keeps _default_processor=True, so the application's first provider.add_span_processor() call after setup() shut down and cleared the whole chain (fi_instrumentation/otel.py add_span_processor, lines 329-340), AG2SpanProcessor included. The new exporter then got un-normalized AG2 spans (no gen_ai.span.kind, no usage moves, no TraceConfig content gate). install_span_processor now wraps the provider instance's add_span_processor once (marker attribute): the AG2 processor is lifted out of the chain for the call, so fi's reset neither shuts it down nor drops it, and is put back first afterwards. provider.shutdown() still shuts it down. Installing on a provider whose active processor is a ConcurrentMultiSpanProcessor logs a warning, since parallel on_end calls do not guarantee normalization before export. Docstring, README (set-up note, troubleshooting) and CHANGELOG updated. Tests first, red on 4a9922e (2 failed, 2 passed): test_later_add_span_processor_on_fi_provider_keeps_ag2_processor_first (chain was just the new SimpleSpanProcessor) and test_install_warns_when_processors_run_concurrently. Passing on 4a9922e by design: test_later_add_span_processor_on_sdk_provider_keeps_ ag2_processor_first and test_provider_shutdown_still_shuts_the_processor_ down; mutant "do not put the processor back" turns both later_add tests red. Green: 86 passed on ag2 1.1.2; 79 passed, 7 skipped on ag2 1.0.0. approved: Nikhil 2026-10-03 blanket Refs: TH-8236
1 parent 4a9922e commit c1196a4

4 files changed

Lines changed: 164 additions & 9 deletions

File tree

‎python/frameworks/ag2/CHANGELOG.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,3 +36,10 @@
3636
`hide_input_messages` also drops `gen_ai.system_instructions`. The README
3737
lists which flags apply; `hide_input_images` and `hide_embedding_vectors`
3838
have nothing to act on.
39+
- A later `trace_provider.add_span_processor(...)` no longer removes
40+
`AG2SpanProcessor`. fi's provider shuts down and clears every processor on
41+
the first such call after `register()`; `install_span_processor` now wraps
42+
the provider instance's `add_span_processor` so the AG2 processor sits out
43+
that reset and is put back first. Installing on a provider that uses
44+
`ConcurrentMultiSpanProcessor` logs a warning (normalization before export
45+
is not guaranteed there).

‎python/frameworks/ag2/README.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,11 @@ a later `setup(..., config=TraceConfig(...))` replaces its `TraceConfig` (the
9292
most recent call wins and a warning is logged when it changes), while
9393
`config=None` keeps the current one.
9494

95+
Span processors you add after `setup()` with
96+
`trace_provider.add_span_processor(...)` are fine: `setup` wraps that method on
97+
the provider instance so `AG2SpanProcessor` stays first and is not shut down
98+
when fi's provider replaces its default exporter.
99+
95100
Sessions and users come from traceAI's context helpers, as with the other
96101
traceAI integrations:
97102

@@ -190,6 +195,10 @@ Notes:
190195
- Duplicate spans: do not attach a second `TelemetryMiddleware`, and do not
191196
also enable `traceai-autogen` in an `ag2` process.
192197
- Export failures are logged and never fail the agent.
198+
- Spans arrive without `gen_ai.span.kind`: the provider runs its processors
199+
concurrently (`TracerProvider(active_span_processor=ConcurrentMultiSpanProcessor())`;
200+
`setup` logs a warning), so the exporter can read a span before
201+
`AG2SpanProcessor` normalizes it. Use the default synchronous processor.
193202
- Content appeared without opting in: you constructed `TelemetryMiddleware`
194203
yourself. Pass `capture_content=False`, or use `create_telemetry_middleware`.
195204
- This package never imports `autogen`. If your code imports `autogen`, you

‎python/frameworks/ag2/tests/test_processor.py‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22

33
from __future__ import annotations
44

5+
import logging
6+
57
import pytest
68
from fi_instrumentation.instrumentation.config import TraceConfig
79
from opentelemetry.sdk.trace import TracerProvider
@@ -424,6 +426,83 @@ def test_install_keeps_fi_register_exporter():
424426
provider.shutdown()
425427

426428

429+
def _fi_provider(exporter):
430+
from fi_instrumentation import register
431+
from fi_instrumentation.fi_types import ProjectType
432+
433+
return register(
434+
project_type=ProjectType.OBSERVE,
435+
project_name="ag2-unit",
436+
batch=False,
437+
span_exporter=exporter,
438+
verbose=False,
439+
)
440+
441+
442+
def test_later_add_span_processor_on_fi_provider_keeps_ag2_processor_first():
443+
"""fi's provider shuts down and clears every processor on the first
444+
add_span_processor() after register() (otel.py 329-340); the user's new
445+
exporter must still receive normalized AG2 spans."""
446+
provider = _fi_provider(InMemorySpanExporter())
447+
try:
448+
assert install_span_processor(provider) is True
449+
replacement = InMemorySpanExporter()
450+
provider.add_span_processor(SimpleSpanProcessor(replacement))
451+
452+
chain = provider._active_span_processor._span_processors
453+
assert isinstance(chain[0], AG2SpanProcessor), chain
454+
assert sum(isinstance(p, AG2SpanProcessor) for p in chain) == 1
455+
assert [type(p) for p in chain[1:]] == [SimpleSpanProcessor]
456+
457+
_emit(provider, {"gen_ai.operation.name": "chat", "gen_ai.usage.thinking_tokens": 2})
458+
(span,) = replacement.get_finished_spans()
459+
got = attrs(span)
460+
assert got["gen_ai.span.kind"] == "LLM"
461+
assert got["gen_ai.usage.reasoning.output_tokens"] == 2
462+
finally:
463+
provider.shutdown()
464+
465+
466+
def test_later_add_span_processor_on_sdk_provider_keeps_ag2_processor_first():
467+
provider, first = memory_provider()
468+
assert install_span_processor(provider) is True
469+
assert install_span_processor(provider) is False # no second wrapper either
470+
second = InMemorySpanExporter()
471+
provider.add_span_processor(SimpleSpanProcessor(second))
472+
provider.add_span_processor(SimpleSpanProcessor(InMemorySpanExporter()))
473+
474+
chain = provider._active_span_processor._span_processors
475+
assert isinstance(chain[0], AG2SpanProcessor), chain
476+
assert sum(isinstance(p, AG2SpanProcessor) for p in chain) == 1
477+
assert len(chain) == 4
478+
_emit(provider, {"gen_ai.operation.name": "chat"})
479+
for exporter in (first, second):
480+
(span,) = exporter.get_finished_spans()
481+
assert attrs(span)["gen_ai.span.kind"] == "LLM"
482+
483+
484+
def test_provider_shutdown_still_shuts_the_processor_down():
485+
provider, _ = memory_provider()
486+
install_span_processor(provider)
487+
(processor,) = [p for p in provider._active_span_processor._span_processors if isinstance(p, AG2SpanProcessor)]
488+
provider.shutdown()
489+
assert processor._shutdown is True
490+
491+
492+
def test_install_warns_when_processors_run_concurrently(caplog):
493+
"""A ConcurrentMultiSpanProcessor runs on_end of every processor in
494+
parallel, so the exporter may read a span before it is normalized."""
495+
from opentelemetry.sdk.trace import ConcurrentMultiSpanProcessor
496+
497+
provider = TracerProvider(active_span_processor=ConcurrentMultiSpanProcessor(), shutdown_on_exit=False)
498+
try:
499+
with caplog.at_level(logging.WARNING, logger="traceai_ag2"):
500+
assert install_span_processor(provider) is True
501+
assert any("ConcurrentMultiSpanProcessor" in r.getMessage() for r in caplog.records), caplog.text
502+
finally:
503+
provider.shutdown()
504+
505+
427506
def test_install_without_sdk_provider_returns_false():
428507
class _NoChain:
429508
pass

‎python/frameworks/ag2/traceai_ag2/_setup.py‎

Lines changed: 69 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -9,17 +9,22 @@
99

1010
from __future__ import annotations
1111

12+
import contextlib
13+
import functools
1214
import logging
1315
from typing import Any, Dict, Iterable, Optional
1416

1517
from fi_instrumentation.instrumentation.config import TraceConfig
1618
from opentelemetry import trace as trace_api
17-
from opentelemetry.sdk.trace import TracerProvider
19+
from opentelemetry.sdk.trace import ConcurrentMultiSpanProcessor, TracerProvider
1820

1921
from ._processor import AG2SpanProcessor
2022

2123
logger = logging.getLogger(__name__)
2224

25+
# Set on the wrapper ``_keep_processor_first`` installs, so it wraps once.
26+
_KEEP_FIRST_MARKER = "_traceai_ag2_keeps_processor_first"
27+
2328

2429
class _Unset:
2530
def __repr__(self) -> str: # pragma: no cover - cosmetic
@@ -89,6 +94,16 @@ def install_span_processor(
8994
``fi_instrumentation.register()`` treats its exporter as a replaceable
9095
default and shuts it down on the first ``add_span_processor`` call, and the
9196
processor must run before the exporter reads the span.
97+
98+
That same ``add_span_processor`` call, made later by the application,
99+
would also shut down and drop this processor. So the provider instance's
100+
``add_span_processor`` is wrapped once: the processor is taken out of the
101+
chain for the call (fi's reset never shuts it down) and put back first
102+
afterwards. Spans ending during that call are not normalized.
103+
104+
A provider whose active processor is a ``ConcurrentMultiSpanProcessor``
105+
runs every processor's ``on_end`` in parallel, so an exporter can read a
106+
span before it is normalized; a warning is logged for it.
92107
"""
93108
provider = tracer_provider if tracer_provider is not None else trace_api.get_tracer_provider()
94109
active = getattr(provider, "_active_span_processor", None)
@@ -101,19 +116,19 @@ def install_span_processor(
101116
return False
102117

103118
processor = AG2SpanProcessor(config=config)
104-
lock = getattr(active, "_lock", None)
105-
if lock is not None:
106-
with lock:
107-
existing = tuple(active._span_processors)
108-
installed = next((p for p in existing if isinstance(p, AG2SpanProcessor)), None)
109-
if installed is None:
110-
active._span_processors = (processor,) + existing
111-
else: # pragma: no cover - SDK multi-processors always carry a lock
119+
with _chain_lock(active):
112120
existing = tuple(active._span_processors)
113121
installed = next((p for p in existing if isinstance(p, AG2SpanProcessor)), None)
114122
if installed is None:
115123
active._span_processors = (processor,) + existing
124+
_keep_processor_first(provider, active)
116125
if installed is None:
126+
if isinstance(active, ConcurrentMultiSpanProcessor):
127+
logger.warning(
128+
"traceai-ag2: the tracer provider runs span processors concurrently "
129+
"(ConcurrentMultiSpanProcessor), so exporters may read AG2 spans before "
130+
"AG2SpanProcessor normalizes them. Use the default synchronous processor."
131+
)
117132
return True
118133
if config is not None and installed.update_config(config):
119134
logger.warning(
@@ -124,6 +139,51 @@ def install_span_processor(
124139
return False
125140

126141

142+
def _chain_lock(active: Any) -> Any:
143+
lock = getattr(active, "_lock", None)
144+
# SDK multi-processors always carry a lock.
145+
return lock if lock is not None else contextlib.nullcontext() # pragma: no branch
146+
147+
148+
def _keep_processor_first(provider: Any, active: Any) -> None:
149+
"""Wrap ``provider.add_span_processor`` so :class:`AG2SpanProcessor` stays first.
150+
151+
fi's ``TracerProvider.add_span_processor`` shuts down and clears the whole
152+
chain on its first call after ``register()`` (``fi_instrumentation/otel.py``
153+
``add_span_processor``: ``_default_processor``). The wrapper lifts the AG2
154+
processor out of the chain for the call, so it is neither shut down nor
155+
dropped, then puts it back in front of whatever the call left.
156+
"""
157+
original = getattr(provider, "add_span_processor", None)
158+
if original is None or getattr(original, _KEEP_FIRST_MARKER, False):
159+
return
160+
161+
@functools.wraps(original)
162+
def add_span_processor(*args: Any, **kwargs: Any) -> Any:
163+
with _chain_lock(active):
164+
processors = tuple(active._span_processors)
165+
ours = tuple(p for p in processors if isinstance(p, AG2SpanProcessor))
166+
if ours:
167+
active._span_processors = tuple(p for p in processors if not isinstance(p, AG2SpanProcessor))
168+
try:
169+
return original(*args, **kwargs)
170+
finally:
171+
if ours:
172+
with _chain_lock(active):
173+
rest = tuple(p for p in active._span_processors if not isinstance(p, AG2SpanProcessor))
174+
active._span_processors = ours[:1] + rest
175+
176+
setattr(add_span_processor, _KEEP_FIRST_MARKER, True)
177+
try:
178+
provider.add_span_processor = add_span_processor
179+
except (AttributeError, TypeError): # pragma: no cover - providers with __slots__
180+
logger.warning(
181+
"traceai-ag2: cannot guard %s.add_span_processor; add span processors "
182+
"before calling setup(), or AG2 spans may reach exporters un-normalized.",
183+
type(provider).__name__,
184+
)
185+
186+
127187
def _has_telemetry_middleware(agent: Any) -> bool:
128188
from ag2.middleware.builtin.telemetry import TelemetryMiddleware
129189

0 commit comments

Comments
 (0)