Skip to content

fix(ai): replace Object.create(span) with delegating wrapper to preserve OTel span private-field access#9152

Merged
heyitsgrace996 merged 7 commits into
DataDog:masterfrom
aperezfals:fix/ai-instrumentation-otel-bridge-private-field-setStatus
Jul 16, 2026
Merged

fix(ai): replace Object.create(span) with delegating wrapper to preserve OTel span private-field access#9152
heyitsgrace996 merged 7 commits into
DataDog:masterfrom
aperezfals:fix/ai-instrumentation-otel-bridge-private-field-setStatus

Conversation

@aperezfals

Copy link
Copy Markdown
Contributor

What does this PR do?

Replaces the Object.create(span) prototype-clone pattern in wrapTracer (in packages/datadog-instrumentations/src/ai.js) with an explicit delegating wrapper object. The wrapper implements the full OTel Span interface and forwards every method call directly to the real span instance, overriding only end, setAttributes, and recordException to publish to diagnostic channels before delegating.

Motivation

When dd-trace is registered as the global OTel TracerProvider, the spans passed to startActiveSpan callbacks are BridgeSpanBase instances, which store span status in the private class field #statusCode. Object.create(span) produces a prototype-chain clone that fails JavaScript's private-field brand check — so any call to setStatus() on the clone throws:

TypeError: Cannot read private member #statusCode from an object whose class did not declare it
The AI SDK calls setStatus({ code: ERROR }) only on its error path, so this TypeError is thrown while handling another error, masking the original failure entirely. All the user sees is the cryptic #statusCode message.

new Proxy(span) has the same limitation — there is no prototype-chain-based approach that can cross a private-field brand check. The fix delegates every method call to span directly, so this === span inside every implementation and private-field access always resolves on the real instance. This works for any span implementation regardless of what private fields it uses now or in the future.

Side benefit: resolves the existing // TODO: does this cause memory leaks? comment — the new wrapper is a plain object with no extra references and is eligible for GC as soon as the callback returns.

Fixes #9151.

Additional Notes

wrapTracer is now exported alongside wrapModelWithLifecycle so the regression test can call it directly without going through the full orchestrion channel setup.
The regression test (describe('wrapTracer') in packages/datadog-instrumentations/test/ai.spec.js) creates a PrivateFieldSpan class with a private #statusCode field — the same pattern as BridgeSpanBase — and asserts that setStatus() on the wrapped span does not throw.
new Proxy(span) was considered and ruled out: it has the same brand-check limitation as Object.create.

@aperezfals
aperezfals requested review from a team as code owners June 30, 2026 09:51
@aperezfals
aperezfals requested review from tlhunter and removed request for a team June 30, 2026 09:51

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d190a9a806

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".


shimmer.wrap(freshSpan, 'recordException', function (recordException) {
return function (exception) {
return span.setAttributes(attributes)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve chainable span methods on wrapper

When a wrapped tracer callback uses the OTel chainable API, e.g. span.setAttributes(attrs).end(), this returns the underlying span instead of freshSpan, so the subsequent end() runs on the unwrapped span and skips the dd-trace:vercel-ai asyncEnd publication. The old prototype clone kept chains on the wrapper for span implementations that return this; the delegating wrapper should return freshSpan after forwarding chainable calls such as setAttribute, setAttributes, addEvent, addLink(s), setStatus, and updateName.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed on af71d1f

@sabrenner sabrenner left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi @aperezfals, thanks for the contribution & resolution to the issue you identified! i left one comment related to testing this.

but additionally, another thought - i think you should not need to rely on both our integration and the underlying OTel interface/integration when using the Datadog OTel trace provider, but rather one or the other. so, from your issue, using either

import tracer from 'dd-trace'
tracer.init(...)

import { generateText } from 'ai'

await generateText({ ... }) // leave out experimental telemetry and tracer as the DD instrumentation will handle it properly

or

export DD_TRACE_DISABLED_INSTRUMENTATIONS=ai
import tracer from 'dd-trace'
tracer.init(...)
new tracer.TracerProvider().register()

import { generateText } from 'ai'

await generateText({ ...,  experimental_telemetry: { isEnabled: true, tracer: trace.getTracer('ai') } }) // leave out experimental telemetry and tracer as the DD instrumentation will handle it properly

but not both. let me know if migrating to one of these scenarios (either relying on the DD instrumentation but not OTel trace provider, or not relying on the DD instrumentation but on the OTel trace provider) helps in your case without this fix.

}

module.exports = { wrapModelWithLifecycle }
module.exports = { wrapModelWithLifecycle, wrapTracer }

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i prefer we do not export the wrapper function here (it's a practice in this repo we generally try not to follow, wrapModelWithLifecycle was an exception). as an alternative, to test it, let's instead add a test in this test file that properly reproduces the issue as you were experiencing it, for a more true end-to-end regression test, is possible.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed here: f7b1af9

@aperezfals

Copy link
Copy Markdown
Contributor Author

hi @aperezfals, thanks for the contribution & resolution to the issue you identified! i left one comment related to testing this.

but additionally, another thought - i think you should not need to rely on both our integration and the underlying OTel interface/integration when using the Datadog OTel trace provider, but rather one or the other. so, from your issue, using either

import tracer from 'dd-trace'
tracer.init(...)

import { generateText } from 'ai'

await generateText({ ... }) // leave out experimental telemetry and tracer as the DD instrumentation will handle it properly

or

export DD_TRACE_DISABLED_INSTRUMENTATIONS=ai
import tracer from 'dd-trace'
tracer.init(...)
new tracer.TracerProvider().register()

import { generateText } from 'ai'

await generateText({ ...,  experimental_telemetry: { isEnabled: true, tracer: trace.getTracer('ai') } }) // leave out experimental telemetry and tracer as the DD instrumentation will handle it properly

but not both. let me know if migrating to one of these scenarios (either relying on the DD instrumentation but not OTel trace provider, or not relying on the DD instrumentation but on the OTel trace provider) helps in your case without this fix.

Hi @sabrenner, thanks for looking at this and for the thoughtful suggestion! I want to clarify why I think the fix is still needed even setting aside the "dual path" scenario.

The bug actually reproduces with a single path through dd-trace — no explicit experimental_telemetry.tracer required. The setup is:

const tracer = require('dd-trace').init({ ... })
tracer.use('ai', { enabled: true })      // DD ai instrumentation active
new tracer.TracerProvider().register()    // DD registered as global OTel provider
await generateText({ model, messages })  // no experimental_telemetry at all

Here's the sequence that leads to the crash:

  1. selectTelemetryAttributesChannel (in packages/datadog-instrumentations/src/ai.js) forces telemetry.isEnabled: true on the ai SDK options, so telemetry is enabled even when the user didn't configure it.
  2. The ai SDK calls its internal getTracer(), which (with no explicit tracer: passed) fetches the global OTel tracer via trace.getTracer('ai').
  3. Since DD is the global OTel provider, that returns a DD bridge tracer.
  4. getTracerChannel fires and wrapTracer(bridgeTracer) is called.
  5. When the ai SDK calls startActiveSpan, it gets a BridgeSpanBase instance.
  6. Object.create(bridgeSpan) creates a prototype clone that fails the private-field brand check.
  7. Any ai SDK error path calls span.setStatus({ code: SpanStatusCode.ERROR }) on the clone → TypeError: Cannot read private member #statusCode.

So the issue is not about using two overlapping export paths — it's about two dd-trace subsystems (the ai instrumentation and the OTel bridge) being incompatible with each other when used together. The ai instrumentation forces OTel telemetry on even when the user doesn't configure it explicitly, and if DD is the global OTel provider, that's enough to hit the crash.

Your Option A avoids the crash because it skips registering the OTel provider (so trace.getTracer() returns a noop tracer with no private fields). Your Option B avoids it because it disables the ai instrumentation entirely (so wrapTracer is never called). But both workarounds require giving up a legitimate feature: either OTel bridge compatibility or DD ai instrumentation. The intended configuration — DD as the global OTel provider for ecosystem compatibility, with DD ai instrumentation active for AI-specific span enrichment — is reasonable and should work.

The Object.create(span) pattern is also fundamentally unsafe as a generic span wrapper regardless of this specific scenario: it breaks for any span implementation that uses private fields, not just BridgeSpanBase. The delegating wrapper fixes the root incompatibility for all current and future span implementations.

@aperezfals
aperezfals force-pushed the fix/ai-instrumentation-otel-bridge-private-field-setStatus branch from f7b1af9 to 99b5672 Compare July 2, 2026 09:26

@sabrenner sabrenner left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's about two dd-trace subsystems (the ai instrumentation and the OTel bridge) being incompatible with each other when used together

i think this is a valid concern. i would say it might make sense to still disable the dd-trace AI instrumentation in this case and use the OTel instrumentation from the vercel AI sdk, but for the cases where you explicitly want to use the DD instrumentation while still registering it as the global OTel provider, this change makes sense.

i'll get CI run on this PR, and if it all looks good, we'll get this landed!

@sabrenner
sabrenner force-pushed the fix/ai-instrumentation-otel-bridge-private-field-setStatus branch from 4779546 to 407f413 Compare July 7, 2026 19:07
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Jul 7, 2026

Copy link
Copy Markdown

Tests

🔄 Datadog auto-retried 1 job - 1 passed on retry View in Datadog

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: fcb83e9 | Docs | Datadog PR Page | Give us feedback!

// "Cannot read private member from an object whose class did not declare it".
// The delegating wrapper must call through to the real span instance to avoid this.
class PrivateFieldSpan {
#statusCode = 0

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

feel free to add an eslint ignore to this line, the test here is that it's able to be accessed to set, which is done below, it's just never actually used, which is fine for the point of this test

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@aperezfals

Copy link
Copy Markdown
Contributor Author

@sabrenner can you run CI again?

@tlhunter
tlhunter removed their request for review July 14, 2026 17:08
@heyitsgrace996
heyitsgrace996 force-pushed the fix/ai-instrumentation-otel-bridge-private-field-setStatus branch from 6f5d0b2 to fcb83e9 Compare July 15, 2026 21:38
@heyitsgrace996
heyitsgrace996 merged commit 76d4d45 into DataDog:master Jul 16, 2026
818 of 821 checks passed
@heyitsgrace996

Copy link
Copy Markdown
Contributor

Hey @aperezfals, I've gone ahead and re-run the CI + merged since everything looks good :)
This fix will be included in an upcoming release of dd-trace-js, you can keep an eye on the release notes to confirm it's included.

Thanks again for the contribution!

dd-octo-sts Bot pushed a commit that referenced this pull request Jul 16, 2026
…rve OTel span private-field access (#9152)

* fix(ai): replace Object.create(span) with delegating wrapper to preserve OTel span private-field access

* fix(ai): replace Object.create(span) with delegating wrapper to preserve OTel span private-field access and chaining

* test(ai): replace exported wrapTracer unit tests with E2E regression in index.spec.js

* fix(ai): replace Object.create(span) with delegating wrapper to preserve OTel span private-field access

* fix(ai): replace Object.create(span) with delegating wrapper to preserve OTel span private-field access and chaining

* test(ai): replace exported wrapTracer unit tests with E2E regression in index.spec.js

* test(ai): eslint ignore non used statusCode var
@dd-octo-sts dd-octo-sts Bot mentioned this pull request Jul 16, 2026
dd-octo-sts Bot pushed a commit that referenced this pull request Jul 16, 2026
…rve OTel span private-field access (#9152)

* fix(ai): replace Object.create(span) with delegating wrapper to preserve OTel span private-field access

* fix(ai): replace Object.create(span) with delegating wrapper to preserve OTel span private-field access and chaining

* test(ai): replace exported wrapTracer unit tests with E2E regression in index.spec.js

* fix(ai): replace Object.create(span) with delegating wrapper to preserve OTel span private-field access

* fix(ai): replace Object.create(span) with delegating wrapper to preserve OTel span private-field access and chaining

* test(ai): replace exported wrapTracer unit tests with E2E regression in index.spec.js

* test(ai): eslint ignore non used statusCode var
@dd-octo-sts dd-octo-sts Bot mentioned this pull request Jul 16, 2026
juan-fernandez pushed a commit that referenced this pull request Jul 16, 2026
…rve OTel span private-field access (#9152)

* fix(ai): replace Object.create(span) with delegating wrapper to preserve OTel span private-field access

* fix(ai): replace Object.create(span) with delegating wrapper to preserve OTel span private-field access and chaining

* test(ai): replace exported wrapTracer unit tests with E2E regression in index.spec.js

* fix(ai): replace Object.create(span) with delegating wrapper to preserve OTel span private-field access

* fix(ai): replace Object.create(span) with delegating wrapper to preserve OTel span private-field access and chaining

* test(ai): replace exported wrapTracer unit tests with E2E regression in index.spec.js

* test(ai): eslint ignore non used statusCode var
juan-fernandez pushed a commit that referenced this pull request Jul 16, 2026
…rve OTel span private-field access (#9152)

* fix(ai): replace Object.create(span) with delegating wrapper to preserve OTel span private-field access

* fix(ai): replace Object.create(span) with delegating wrapper to preserve OTel span private-field access and chaining

* test(ai): replace exported wrapTracer unit tests with E2E regression in index.spec.js

* fix(ai): replace Object.create(span) with delegating wrapper to preserve OTel span private-field access

* fix(ai): replace Object.create(span) with delegating wrapper to preserve OTel span private-field access and chaining

* test(ai): replace exported wrapTracer unit tests with E2E regression in index.spec.js

* test(ai): eslint ignore non used statusCode var
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

3 participants