fix(fi_instrumentation): keep register()'s exporter when another span processor is added (TH-8727) - #281

What does this PR do?

The provider register() returns now keeps its Future AGI exporter when another span processor is added. Before, the first add_span_processor() call shut the exporter down and removed it. Traces then stopped reaching Future AGI, and nothing reported an error.

• The fix is one line. register() no longer runs tracer_provider._default_processor = True (python/fi_instrumentation/otel.py:244 on main). A processor added later runs next to the Future AGI exporter, as add_span_processor() does in upstream OpenTelemetry. The other otel.py changes are docstrings and a comment.
• traceai-strands needed a matching change. With the core fix, README Option 1 can send duplicate requests when both exporters target Future AGI with matching headers. configure_strands_tracing() now skips Strands' own OTLP exporter only when the provider already sends to the same traces URL with exactly the same headers (same names and values, none missing on either side), and keeps its exporter when another processor is added. On fi-instrumentation-otel 1.1.0, traceai-strands 0.1.1 adds the exporter exactly as 0.1.0 does.
• Releases, neither published: fi-instrumentation-otel 1.1.1 and traceai-strands 0.1.1. Publishing waits for the owner. traceai-strands 0.1.1 is safe on 1.1.0, so it can go out first, and should go out no later than fi-instrumentation-otel 1.1.1 (see "Behaviour change and compatibility").

Why?

Linear: TH-8727. #273 called this the underlying register() trap and tracked it as TH-8672.

Root cause.

• register() added its processor, then marked it as a default: otel.py:244 on main.
• When that mark is set, the TracerProvider.add_span_processor override (otel.py:330-341 on main) shuts down every processor and removes it.
• So any later add_span_processor() call silently dropped the exporter register() built: the Future AGI exporter, or the custom one passed as register(span_exporter=...).

Who it hit.

• TH-8665, LiveKit: livekit-agents 1.8 adds its privacy filter through add_span_processor, so traceAI-livekit sent no traces at all. fix(livekit): keep the Future AGI exporter when LiveKit adds its privacy filter (TH-8665) #273 fixed that inside traceAI-livekit, by clearing the mark in enable_*_attribute_mapping().
• traceai-strands, README Option 1 (register(), then configure_strands_tracing(tracer_provider=...)): Strands adds its OTLP exporter through add_span_processor, which dropped the Future AGI exporter. Spans then went only where OTEL_EXPORTER_OTLP_* pointed, or to localhost:4318 when it was unset.
• TH-8320: the repro below comes from that investigation.
• Any customer who adds a processor to the provider register() returns, including one who passed span_exporter=.

Repro. One span, sent to a local OTLP/HTTP sink on 127.0.0.1. register(project_type=OBSERVE, verbose=False), then optionally add_span_processor(SimpleSpanProcessor(InMemorySpanExporter())):

traceai-strands, README Option 1, 3 spans, with real strands-agents 1.61.1. Both collectors are on 127.0.0.1 and record the X-Api-Key each request carried. Each cell is the number of spans Future AGI got with the keys (the copies it accepts); a span counted twice is a duplicate. "The fix" is this repo's fi-instrumentation-otel; "1.1.0" is PyPI.

The 43e25de columns are the previous head of this PR. Its host-and-port check skipped Strands' exporter on 1.1.0 too, where register()'s exporter is still marked, so the next processor replaced it and Future AGI got nothing. It also skipped the only exporter that carried the keys. The last row on 1.1.0 is the core bug itself; traceai-strands cannot fix it there.

The change

• fi_instrumentation (otel.py):
  • The _default_processor = True line in register() is deleted. register()'s own add_span_processor() call still replaces the default the provider adds on construction; now the mark just stays cleared afterwards.
  • Docstrings for register(), TracerProvider and add_span_processor say what happens now. A comment at otel.py:246-247 explains the call.
  • register(verbose=True) no longer prints "add_span_processor will overwrite this default", which is now untrue. Only the mark controlled that line.
  • A bare TracerProvider() behaves exactly as on main.
• traceai-strands (traceai_strands/_instrumentor.py):
  • Before setup_otlp_exporter(), _strands_otlp_exporter_target() (_instrumentor.py:95-121) builds the upstream OTLP/HTTP span exporter the way Strands' setup_otlp_exporter() does, with no arguments. It reads back the traces URL and headers that exporter resolved from OTEL_EXPORTER_OTLP_*, then shuts it down. The implementation checks _headers on the exporter (the location in opentelemetry-exporter-otlp-proto-http 1.29-1.44, verified against 1.29 and 1.39) and _client._headers from 1.45. In 1.29-1.44, headers passed to the exporter replace the environment's, which are used only when none are passed, and an explicitly empty OTEL_EXPORTER_OTLP_TRACES_ENDPOINT resolves to an empty URL; from 1.45, environment and passed headers are merged, and an empty traces endpoint falls back to the base endpoint.
  • _strands_otlp_exporter_duplicates() (_instrumentor.py:141-173) skips the exporter only when all of these hold. The skip is logged at INFO (_instrumentor.py:276-285).
    1. The provider has no _default_processor mark, so it keeps its exporter on the next add_span_processor() call. This is the flag the core's override reads, checked on the provider being configured. fi-instrumentation-otel 1.1.0 leaves it set on the provider register() returns; a bare fi_instrumentation TracerProvider() has it until its first add_span_processor() call.
    2. A processor on the provider exports to the same traces URL. URLs are compared by scheme, userinfo, host, port, path and query, so a different path, scheme, query or URL credentials count as another destination. Scheme and host case, a written-out default port, and repeated or trailing slashes are normalised. The fragment is ignored.
    3. That exporter sends exactly the same header set as Strands' exporter would: same names (case-insensitively) and values, with none missing on either side. A header present on only one exporter keeps Strands' exporter, including an extra routing header on the provider, keys only one exporter carries, or the x-project-name header that project_name= adds. Client certificate, client key or credential-provider environment variables set for Strands' exporter keep it too. OpenTelemetry's SDK constant names for the two credential-provider variables start with _; the actual environment variable names checked here do not.
  • In every other case, Strands' exporter is added as on main. The console exporter and the meter are unchanged.
  • README Option 1 says exactly when Strands' exporter is added: always, for the URL the variables resolve to (localhost:4318 when unset), except for that one duplicate case. It also covers what happens on 1.1.0. The environment section says the variables only set where Strands' own exporter sends. The Option 2 example now gives otlp_endpoint a base URL: passing a URL already ending in /v1/traces used to create /v1/traces/v1/traces because Strands appends the signal path.
  • Version 0.1.1, __version__, and a new CHANGELOG.
• traceAI-livekit: its enable_* workaround stays, because the package allows fi-instrumentation-otel 1.1.0, which still sets the mark. It does nothing on the fixed release. Only its comment changes, plus one test, described below. No livekit release is needed.

Why a behaviour check in traceai-strands, not a fi-instrumentation-otel>=1.1.1 floor:

• The check reads the one thing that decides whether skipping is safe, on the provider actually being configured. It holds however the environment was resolved: pip --no-deps, a lock pinned to 1.1.0, or a vendored core.
• A floor would make traceai-strands 0.1.1 uninstallable until fi-instrumentation-otel 1.1.1 is published. That blocks the safe release order, and this repo's CI installs the package's dependencies from PyPI, so the install would fail too.
• The fix's version number is not settled: it becomes 1.2.1 if feat(python): hide error details with the content hide flags #272 ships first (see the notes below). A floor would have to follow it.

Behaviour change and compatibility

Changed

• add_span_processor() on the provider register() returns adds the processor next to register()'s exporter instead of replacing it. Code that relied on it to swap out register()'s exporter now exports to both. To export somewhere else instead, pass register(span_exporter=...).
• register(span_exporter=...) followed by add_span_processor() keeps the custom exporter, which main dropped, so spans go to both.
• traceai-strands 0.1.1 does not add Strands' OTLP exporter when it would only duplicate what the provider already sends (the three conditions above).

The CHANGELOG now has these under ### Changed (python/CHANGELOG.md, 1.1.1). The ### Fixed entry keeps only the bug.

Upgrade note

A user on traceai-strands 0.1.0 whose OTEL_EXPORTER_OTLP_* points at Future AGI can get duplicate requests after upgrading only fi-instrumentation-otel. traceai-strands 0.1.1 skips its exporter only when the provider and Strands exporters have exactly the same URL and header set; if the headers differ, it keeps Strands' exporter. The fi-instrumentation-otel 1.1.1 CHANGELOG names the upgrade case. With extra headers on Strands' exporter (for example Authorization=Bearer ..., as in the strands README's environment example, or project_name=), 0.1.1 keeps it, so a backend accepting both copies still gets two.

Unchanged

• A TracerProvider() constructed directly: its docstrings document this, and tests/test_batch_span_processor_tuning.py:335-336 and :356-357 rely on it (a bare TracerProvider(), then its first add_span_processor() call).
• register() on its own, and every register() argument.
• traceai-strands with any other destination, including the default localhost:4318 with nothing listening, which logs export warnings as on main.
• traceai-strands with fi-instrumentation-otel 1.1.0: Strands' exporter is added exactly as in 0.1.0 and replaces register()'s. Option 1 therefore still loses Future AGI spans there unless Strands' exporter points at Future AGI with the keys. That is the core bug, and the strands README says to upgrade fi-instrumentation-otel.

Known limits

• register(transport=Transport.GRPC) sends to grpc.futureagi.com, while Strands' exporter is HTTP. The URLs differ, so a duplicate there is not detected.
• OTEL_EXPORTER_OTLP_ENDPOINT=https://api.futureagi.com resolves to https://api.futureagi.com/v1/traces, not register()'s /tracer/v1/traces, so Strands' exporter is kept for it. The README shows the URL that is recognised.
• A client certificate or credential provider set for Strands' exporter always keeps it, even when register()'s exporter uses the same one: the probe cannot compare them.

How was it tested?

Python 3.11, as in CI, on fresh environments built from each package's [dev] setup, plus this repo's core where noted. The Poetry lock used by CI pins OpenTelemetry 1.34.1. Collector URLs point at 127.0.0.1.

• python/tests/test_register_keeps_exporter.py, 9 tests. They send through the real Future AGI exporters to an OTLP/HTTP and an OTLP/gRPC collector on 127.0.0.1, then check the span names each one received.
  • register() + add_span_processor(in-memory): both the collector and the in-memory exporter get the span. Run over HTTP and gRPC, each with batch=True and batch=False.
  • Two more processors after register(): all three get the span.
  • register(span_exporter=custom) + another processor: the custom exporter and the other processor get it, and the collector gets nothing. Run with batch=True and batch=False.
  • register(verbose=True) prints the tracing details without "will overwrite this default".
  • A bare TracerProvider(): the first call replaces the default, the next one adds, and nothing reaches the collector.
  • On main's otel.py, 8 of the 9 fail. Only the bare-provider test passes, which pins that this behaviour is unchanged.
• fi_instrumentation suite (the CI job's command, on this head): 208 passed on the locked OpenTelemetry 1.34.1, the 199 already on main plus 9 new. The same 208 pass on 1.29.0. On 1.45.1, 205 pass and three existing timeout-introspection tests fail because upstream HTTPSpanExporter no longer has _timeout; these same three failures reproduce with PyPI fi-instrumentation-otel 1.1.0.
• traceai-strands: 102 tests, 41 of them new.
  • 34 unit cases for the decision: traces URL and base-URL resolution, otlp_endpoint=, URL normalization, different URL components, default endpoint, no processors, identical headers, a header extra on either exporter, keys only on either exporter, the same header names with other values (other keys; restored in cb66f71), x-project-name, unreadable headers, client certificate and both actual credential-provider environment variables, both OTel header-storage layouts, probe shutdown, and a marked provider with and without the console exporter.
  • 7 real-export cases run README Option 1 in a fresh interpreter, because conftest.py mocks opentelemetry for the session. They use register() and real exporters: exact same Future AGI URL and keys (no duplicate), same URL but keys only on the provider (Strands exporter stays), a base endpoint, the console exporter, a processor added afterwards, keys only in otlp_headers=, and a second collector. CI has no strands-agents (it is not a dev dependency), so there the scenario uses a copy of StrandsTelemetry.setup_otlp_exporter() and setup_console_exporter(). With strands-agents installed, it uses the real one.
  • Results at cb66f71: 102 passed on this repo's core (Python 3.11). At b1bbf5a, before the one added test: 101 passed on this repo's core plus real strands-agents 1.61.1, and with opentelemetry 1.29.0, 1.44.0 and 1.45.1; 99 passed and 2 skipped on PyPI fi-instrumentation-otel 1.1.0, with and without real strands-agents. CI re-runs the strands package on cb66f71. The two skips are the real-export cases requiring the core fix (the second collector and the provider-only-auth case).
  • Fail-first checks with main's _instrumentor.py and 43e25de's implementation reproduced the original duplicate, missing-key and dropped-exporter cases; the new tests exercise those regressions, including extra headers on either side.
• traceAI-livekit, 58 tests:
  • all pass on this repo's core and on fi-instrumentation-otel 1.1.0 from PyPI;
  • with the enable_* workaround deleted, all 58 still pass on this repo's core, so the core fix alone covers TH-8665. 8 fail on 1.1.0, which is why the workaround stays;
  • test_add_span_processor_after_register_alone_matches_the_installed_core replaces ..._follows_the_mark_enable_clears. The expected mark is version-based: 1.1.0 and earlier set it, later releases do not. The test asserts that expected mark, then checks which processors remain. With the deleted line put back in this repo's otel.py, it fails.
• Both repro tables above were re-run on this head.
• Earlier round, on e1b803d, whose register() behaviour is the same: openai (7), anthropic (8) and bedrock (21 passed, plus the known test_bedrock_image_attributes failure) gave the same results on 1.1.0 and on the fixed core. CI runs all 49 packages against this repo's core, because python/fi_instrumentation/ changed.
• ruff check --select F,E9 finds nothing new on the changed files: the 7 findings it reports are in the same files on 43e25de. The repo's wider ruff config flags the same typing-style rules (List, Optional) the surrounding code already uses.

Checklist

• Branch is off main
• Commit messages follow Conventional Commits
• No TODOs or commented-out code left in
• No real API keys or secrets (tests use placeholders and local collectors)
• VOCABULARY check (N/A: docs/VOCABULARY.md is not in this repo)

Notes for reviewers

• Commits:
  • e1b803d is the original change. Its message is superseded by 79be724. It describes a private _remove_default_processor() helper and an INFO log on the bare provider, neither of which remain.
  • 79be724 cuts the core change back to the one-line deletion. Its commit description also says the livekit test reads the provider mark; that test-description is superseded by 36d302f, which made the expected outcome version-based again.
  • 43e25de is the first traceai-strands change (host-and-port check).
  • ff8e7ba replaces that check with the three conditions above, with a subset header check; b1bbf5a made the header check exact (same names and values both ways). ff8e7ba also updates the tests, README and CHANGELOG.
  • cb66f71 restores the same-names-other-values test (a names-only check passed every test after b1bbf5a) and fixes the exact-match wording in the core CHANGELOG and the test docstring.
  • 36d302f makes the livekit test version-based again: it expects the installed release's behavior instead of reading the provider mark, superseding 79be724's description.
  • 908179f moves the behaviour changes under ### Changed in the core CHANGELOG.
  • b1bbf5a uses the actual OTLP credential-provider environment variable names, requires exact header-set equality, adds both-direction header and provider-only-auth tests, and corrects the OTel storage and Option 2 URL documentation.
• Version and merge order with feat(python): hide error details with the content hide flags #272 (draft, bumps the same package to 1.2.0):
  • feat(python): hide error details with the content hide flags #272 changes three version files: python/pyproject.toml, python/CHANGELOG.md and python/fi_instrumentation/instrumentation/version.py (0.1.19 to 1.2.0). This PR changes the first two and leaves version.py at 0.1.19. That value was already stale at 1.1.0, and nothing in fi_instrumentation reads it.
  • Whichever PR merges second sets all three consistently. If feat(python): hide error details with the content hide flags #272 merges first, this entry becomes 1.2.1, and version.py should read 1.2.1 too. If this PR merges first, feat(python): hide error details with the content hide flags #272's 1.2.0 stands in all three.
  • Elsewhere, only published versions are named: "traceai-strands 0.1.1" in the CHANGELOG note, "1.1.0 and earlier" in the READMEs and livekit comments, and _LAST_RELEASE_WITH_THE_MARK = (1, 1, 0) in the livekit test. That constant changes only if a release with the mark is published after 1.1.0, for example feat(python): hide error details with the content hide flags #272 shipping 1.2.0 before this fix. The livekit lock pins 1.1.0 today.
• Other SDKs, checked for the same pattern and not changed here:
  • TypeScript (@traceai/fi-core) is written to have the same bug, but a second bug hides it today.
    • The provider register() returns keeps defaultProcessorAttached = true (typescript/packages/fi-core/src/otel.ts:387), and addSpanProcessor (otel.ts:403-411) is written to throw the default processor away.
    • On @opentelemetry/sdk-trace-base 2.x, addSpanProcessor does nothing: it pushes onto _registeredSpanProcessors, a field 2.x no longer reads. The SDK builds its processor list once, in the constructor (otel.ts:386). So an added processor never receives spans, and the Future AGI exporter is kept.
    • register({ batch: true }) (otel.ts:599-601) goes through the same call, so it silently keeps the default SimpleSpanProcessor. It also clears _defaultProcessorAttached, a typo for defaultProcessorAttached.
    • A probe of the built fi-core 1.0.0 against a local sink gave 1 Future AGI export and 0 in-memory spans for both batch settings, with and without an added processor.
    • A fix that only makes addSpanProcessor work would bring TH-8727 to TypeScript. It must also stop register() from leaving its exporter marked as a replaceable default (clear defaultProcessorAttached). That needs its own ticket and fi-core release.
  • Java (java/traceai-java-core/src/main/java/ai/traceai/TraceAI.java:111,117): processors go into upstream SdkTracerProviderBuilder, which only adds. Not affected.
  • C# (csharp/src/FIInstrumentation/TraceAI.cs:110 and the AddOtlpExporter calls after it): uses upstream TracerProviderBuilder, which only adds. Not affected.
  • Go: there is no Go code on main, but open PRs add it.
    • feat(go): Go SDK with OpenAI instrumentor #166 (feat/go-sdk, head 529dc6e) adds a Go SDK core in go/traceai plus an OpenAI instrumentor.
    • feat(go/anthropic): Anthropic Go instrumentor (TH-8272) #259-feat(go/genai): Gemini Go instrumentor (TH-8273) #262 and feat(go/eino): CloudWeGo Eino instrumentor (TH-8275) #265 add Go instrumentors (go/anthropic, go/openai, go/a2a, go/genai, go/eino) with examples.
    • As of their current heads, all of them build providers with upstream sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(...)) or WithBatcher(...), and upstream only adds processors. None keeps a replaceable default, so none is affected. Recheck if a Go register() with its own default processor lands.
• Follow-ups outside this repo, once fi-instrumentation-otel 1.1.1 is published:
  • In future-agi/docs (main 042a3fe1) and future-agi/unified-docs (main 84895232), these pages describe the old trap as current behaviour. They should say the workaround is only needed on fi-instrumentation-otel 1.1.0 and earlier. The workaround itself does no harm on 1.1.1.
    • public/fi_verify.py:117-129
    • src/pages/docs/cookbook/quickstart/instrument-and-verify.mdx:401-403 and :691
  • src/pages/docs/observe/troubleshooting/noisy-or-incomplete-traces.mdx:122 adds a processor after register(). On 1.1.0 and earlier, that silently drops the Future AGI exporter.
  • Once traceAI-livekit raises its floor past 1.1.0, its enable_* workaround and the mark-dependent test can go.

🤖 Generated with Claude Code