feat(otel-thread-ctx): Node.js OTEP-4947 thread-context writer#9210
feat(otel-thread-ctx): Node.js OTEP-4947 thread-context writer#9210szegedi wants to merge 7 commits into
Conversation
Overall package sizeSelf size: 7.5 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.2 | 124.41 kB | 440.65 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #9210 +/- ##
==========================================
- Coverage 98.42% 98.34% -0.09%
==========================================
Files 939 942 +3
Lines 126531 126945 +414
Branches 10760 10781 +21
==========================================
+ Hits 124541 124843 +302
- Misses 1990 2102 +112 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
BenchmarksBenchmark execution time: 2026-07-13 11:54:01 Comparing candidate commit 275b832 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2315 metrics, 43 unstable metrics.
|
797fe55 to
d608888
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d608888464
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // the result on the span. | ||
| function getCachedWebTags (span) { | ||
| const cached = getCache(span) | ||
| if (cached.resolved) return cached.webTags |
There was a problem hiding this comment.
When a child span is entered before its HTTP parent receives route/resource tags, this cache stores resolved=true with webTags === undefined for the child. A later dd-trace:span:tags:update on the parent only publishes resolvedCh for the parent span, so re-entering the already-cached child hits this fast path and never re-walks to pick up the now-resolved endpoint; the OTel thread-context record (and wall-profiler context through the shared cache) will keep missing endpoints for common flows where routing tags are set after downstream spans start.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
This is a valid issue, although it's been pre-existing for very long time (it existed before in wall.js code before we extracted it into web-tags-cache.js. This is rather non-trivial to fix, we need to either not-cache-negative-answers (extra walks each activation), or track a parent-children reverse map to invalidate on parent transition. Given the complexity, I find it acceptable as a corner case, but I can file a JIRA issue to keep track of it.
| // wins; subsequent calls are no-ops regardless of their argument. In | ||
| // practice all callers observe the same global ACF state. | ||
| function ensureChannelsActivated (asyncContextFrameEnabled) { | ||
| if (channelsActivated) return |
There was a problem hiding this comment.
If DD_TRACE_OTEL_CTX_ENABLED initializes these channels first on an ACF-capable runtime, this flag is set after only the enterWith wrapper is installed. A later wall-profiler start in auto mode with DD_PROFILING_ASYNC_CONTEXT_FRAME_ENABLED=false calls ensureChannelsActivated(false), but returns here before installing the async_hooks.before publisher and run() wrapper that non-ACF profiling relies on, so code-hotspot/endpoint contexts stop updating for that configuration.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
This is true, but the scenario is quite unrealistic as it requires the combination of DD_TRACE_OTEL_CTX_ENABLED=1 DD_PROFILING_ASYNC_CONTEXT_FRAME_ENABLED=0 DD_PROFILING_ENABLED=auto. I'll file a JIRA as a low-priority follow-up.
…times Addresses Codex review point #4 on #9210: getThreadLocalMetadata would happily return a payload on macOS/Windows or on Node without an active AsyncContextFrame, which would cause libdatadog to advertise a threadlocal block that no writer is actually producing. Gate the function on the same platform + ACF conditions start() already checks, so callers see 'no threadlocal block'.
Addresses Codex review point #2 on #9210: an OTEP-4947 record can only carry the first-written value for each attribute key, and HTTP plugins routinely set 'http.method' up front and add 'http.route' (plus 'resource.name') later once routing has resolved. The writer previously committed the endpoint on first activation, so a request to '/users/:id' was recorded as 'GET' forever. Introduce an isEndpointFinal(tags) heuristic and only write the endpoint when the tag bag looks stable — either 'resource.name' is set, or both 'http.method' and 'http.route' are. Otherwise mark the context as needsEndpoint and re-check on every 'dd-trace:span:tags:update' fire (switched from webTagsCache.resolvedCh, which only fires on presence transitions and would miss content-only updates).
50b56d7 to
eb0ec07
Compare
Addresses Codex review point #2 on #9210: HTTP plugins routinely set 'http.method' up front and add 'http.route' (plus 'resource.name') later once routing has resolved. The writer previously committed the endpoint on first activation, so an out-of-process reader sampling mid-request saw a bare 'GET' as the endpoint for a call to '/users/:id'. OTEP-4947 duplicates are last-wins, so a later appendAttributes would overwrite the interim value for readers that decode the record in full — but a sampling reader can still observe the incomplete value. Introduce an isEndpointFinal(tags) heuristic and only write the endpoint when the tag bag looks stable — either 'resource.name' is set, or both 'http.method' and 'http.route' are. Otherwise mark the context as needsEndpoint and re-check on every 'dd-trace:span:tags:update' fire (switched from webTagsCache.resolvedCh, which only fires on presence transitions and would miss content-only updates).
7a1f597 to
1a19d66
Compare
pprof 5.16.0 is the first release containing the OTEP-4947 thread-context writer (ThreadContext class, getContext/clearContext, and the getProcessContextAttributes helper this branch depends on). libdatadog 0.12.1 is a bug-fix bump over 0.12.0.
Move the dd-trace storage diagnostics channels (storage:enter, storage:before, span:finish, span:tags:update), the legacy-storage enterWith/run shimmer, and the getActiveSpan helper out of profiling/profilers/wall.js into a new packages/dd-trace/src/storage-channels.js. No behavior change for the wall profiler. The extraction is to let a forthcoming OTEP-4947 thread-context writer reuse the same channel infrastructure without duplicating the shimmer or pulling in the profiler module. Adjust wall.spec.js's proxyquire stubs to replace the new module instead of mocking datadog-core directly.
The wall profiler and the OTEP-4947 thread-context will both walk each span's started-spans chain to find the nearest web-server ancestor, each caching the answer under its own Symbol. Move the walk and cache into packages/dd-trace/src/web-tags-cache.js. - getCachedWebTags(span): lazy parent-chain walk, cached on a shared Symbol. - onSpanTagsUpdated(span): call from a tagsUpdate subscriber; if the walk previously came up empty and the span is now a web-server span, promote its tags into the cache. Returns true iff the cache transitioned from undefined to a real value — a signal to consumers that they should snapshot the new value into whatever they built while the answer was undefined. wall.js still snapshots webTags into its per-sample ProfilingContext because label generation reads it from the sample-context ref (not from the span); its tagsUpdate handler now refreshes that snapshot only when the shared cache signals a transition. No functional change. CODEOWNERS gets the new file scoped to @DataDog/profiling-js.
New module packages/dd-trace/src/otel-thread-ctx.js that mirrors the
active trace ID, span ID and current endpoint into a thread-local
OTEP-4947 record. An out-of-process eBPF reader discovers the record
via the otel_thread_ctx_nodejs_v1 TLS symbol exported by the
@datadog/pprof addon.
Highlights:
- Gates on Linux + AsyncContextFrame (Node 24+ default, or Node 22/23
with --experimental-async-context-frame). isACFActive from
datadog-core/src/storage is the single source of truth for that
check.
- One ThreadContext is allocated per span the first time it's
activated and cached on the span via a Symbol slot. Re-entries in
any async-context frame re-install the same wrap via setContext;
`getContext() === cachedContext` is the JS-reference identity
check that replaces any byte-level comparison (same allocation-
churn fix as the wall profiler in dd-trace-js#8638).
- The record always carries the local root span ID (16-char hex,
index 0 by libdatadog convention), thread name and thread id
(stable per-thread, computed once at module load). The endpoint
attribute is appended in place when a web-server ancestor is
discovered via the span:tags:update channel.
- On span:finish, if the writer's record currently belongs to the
finishing span, setContext(undefined) is called so an out-of-
process reader doesn't keep seeing a finished span as the active
thread context (matters in enterWith-style sticky activation).
Activation gate:
- DD_TRACE_OTEL_CTX_ENABLED (boolean, default false) is registered
in supported-configurations.json as implementation A, exposed as
config.DD_TRACE_OTEL_CTX_ENABLED. proxy.js calls
otel-thread-ctx.start() iff the flag is set; the module itself
starts in a no-op state otherwise.
Tests + docker rig:
- packages/dd-trace/test/otel-thread-ctx.spec.js covers start()
gating, enter/skip behavior, span-drift round-trip, span-finish,
and the late-tags append path with the wire-record attribute
layout (15 cases).
- scripts/docker/{Dockerfile,run-otel-thread-ctx-spec.sh} build a
node:24-bookworm image and run the spec against the locally built
sibling pprof-nodejs; driven by `npm run test:docker:otel-thread-ctx`.
Forthcoming follow-up: publishing the corresponding
threadlocal.attribute_key_map via process discovery, which requires
bumping @DataDog/libdatadog.
Sets up the OTEP-4719 process context so an out-of-process reader can decode
the on-the-wire records the thread-context writer emits. The metadata is
published through libdatadog-nodejs's process-discovery napi crate: bumped
here to 0.12.0, which exposes the ThreadLocalMetadata substruct with the full
'threadlocal.*' block (attribute key map, schema-version string, and extra
KeyValues for reader-side layout constants).
The pieces:
- Add getThreadLocalMetadata() in otel-thread-ctx.js. Pulls the process-context
snapshot from @datadog/pprof (its otelThreadCtx.getProcessContextAttributes
is the source of truth for the schema-version string and V8 layout constants
the reader needs) and reshapes it into the napi ThreadLocalMetadata form:
{ attributeKeys, schemaVersion, extraAttributes: [{ key, intValue|stringValue }] }.
Returns undefined when @datadog/pprof isn't installed or doesn't expose the
helper.
- Wire tracer_metadata.js to pass the substruct (or undefined) as the last
positional arg to processDiscovery.TracerMetadata(...), replacing the flat
threadlocalAttributeKeys array. Gated on the same DD_TRACE_OTEL_CTX_ENABLED
flag that activates the writer.
- Bump the @DataDog/libdatadog optionalDependency from 0.10.0 to 0.12.0.
The spec is technically already picked up by the top-level 'packages/dd-trace/test/*.spec.js' glob in test:trace:core:ci (apm-capabilities.yml), but the file is profiling-team-owned per CODEOWNERS. Adding a dedicated step in profiling.yml gives the owning team direct failure visibility in their own workflow.
1a19d66 to
275b832
Compare
| "DD_TRACE_OTEL_CTX_ENABLED": [ | ||
| { | ||
| "implementation": "A", | ||
| "type": "boolean", | ||
| "default": "false" | ||
| } | ||
| ], |
There was a problem hiding this comment.
Fingers crossed we can enable by default (on supported node.js versions) soon ;)
| // Positional attribute layout. The local root span ID stays at index 0 by | ||
| // convention (mirrors libdatadog's libdd-otel-thread-ctx, where | ||
| // `local_root_span_id` is always the first entry in | ||
| // `threadlocal.attribute_key_map`), encoded as a 16-character lowercase | ||
| // hex string. Endpoint, thread name, and thread id follow. Adding more | ||
| // means assigning the next index and updating ATTRIBUTE_KEYS | ||
| // accordingly. |
There was a problem hiding this comment.
Minor: Since we're supplying a threadlocal.attribute_key_map I'm not sure if libdatadog will still prepend the local root span id, might be worth checking
There was a problem hiding this comment.
It does prepend it. At least, the version we built libdatadog-nodejs process_discovery crate still did, if this changes in a later libdatadog release, we should update accordingly. I know there's some talk of retiring local root span id?
There was a problem hiding this comment.
If it worked when you did it, that's fine!
I know there's some talk of retiring local root span id
Yes, it's on my todo list to experiment a bit, but I'll share it loudly if it looks like we're going in that direction so yeah for now let's keep it.
2 participants
Summary
Adds a Node.js writer for the OpenTelemetry Thread Local Context Record (OTEP-4947), letting out-of-process readers (typically eBPF profilers) sample the active trace/span ID and a small attribute payload with no cooperation from the tracer at read time. Gated behind
DD_TRACE_OTEL_CTX_ENABLED(default off).The writer itself lives in
@datadog/pprof(5.16.0+); this branch wires it into the tracer and publishes the accompanying OTEP-4719 process context via libdatadog-nodejs (0.12.1+).Relevant PRs in other repos that this PR builds upon:
A note on future work: unifying CPU profiler context and OTel thread context
There's lots of similarities in span-related context management between the new writer in
otel-thread-ctx.jsand the in-process CPU profiler inprofiler/wall.js. Two of the commits in this PR extract common functionality (storage-channels.jsandweb-tags-cache.js) for both, these are elaborated more on below. It would be possible to implement this so thatwall.jsno longer maintains its own context data, but always uses the OTel context instead. Java and PHP profilers already do this. For us, the biggest blocker is that OTel context record relies on Async Context Frame, and the CPU profiler still needs to support Node.js 22-23 where it's off by default, so we can't unify before our lowest supported version is 24 where ACF is on by default.It also has some extra runtime cost, but that's only a minor aspect.
wall.jscurrently establishes its context very cheaply by only retaining references to span-related objects, and deferring string conversion until profile serialization, so it only happens for those contexts (~6k of them/minute) that were captured with samples. In contrast, utf-encoded string data need to be written into the OTel thread context immediately for each created span, since we don't know when it will be captured by an external eBPF reader.We will likely still do the unification, especially if both are enabled by default so the OTel context record is generated anyhow. We can either do the unification during dd-trace-js 6.x cycle but then
wall.jswill be more complex as it'll have to handle both kinds of contexts, or defer until 7.x next year when the minimum supported Node.js version will be 24 so we can dropwall.jsown context and just use OTel.What's in this branch (commit by commit):
Bump @datadog/pprof to 5.16.0 and @datadog/libdatadog to 0.12.1pprof 5.16.0 is the first release containing the writer bits this branch depends on; libdatadog 0.12.1 is a bug-fix bump over 0.12.0.
Extract storage-channels module from wall profilerPull the
dd-trace:storage:enter/:before/dd-trace:span:finish/:tags:updatediagnostic-channel wiring out of the wall profiler into a sharedpackages/dd-trace/src/storage-channels.js, so both the wall profiler and the new thread-context writer can subscribe to the same normalized activation stream. No functional change to the wall profiler.Extract shared web-tags cache from wall profilerThe OTel thread context writer will need to walk the started-spans chain per span to find the nearest web-server ancestor, just like wall profiler does. For thi reason, we extract the functionality into
packages/dd-trace/src/web-tags-cache.js: a single Symbol on the span, one lazy walk per span, and add-trace:web-tags:resolveddiagnostics channel that fires once per span at the moment a previously-empty answer transitions to a real value viadd-trace:span:tags:update.Add OTEP-4947 thread context writerAfter the first three preparatory commits, this is the actual new functionality.
packages/dd-trace/src/otel-thread-ctx.js: the writer. Subscribes to storage-channels; on eachstorage:enter, builds (or reuses) aThreadContextfrom@datadog/pprof.otelThreadCtxfor the active span, populates trace/span IDs plus a positional attribute array (index 0 =datadog.local_root_span_id, thendatadog.trace_endpointfor web-server spans,datadog.thread_name,datadog.thread_id), and installs it viacontext.enter(). Handles span-drift (re-installs the same cached context when we switch spans and back) and span-finish (clears the writer if the record is still ours to avoid leaking stale state past enterWith-style activation). Late endpoint discovery is handled via the shared web-tags cache (see below): the writer subscribes towebTagsCache.resolvedChand appends the endpoint attribute in place when the shared cache signals a transition.packages/dd-trace/src/proxy.js: gatedrequire('./otel-thread-ctx').start()after profiler init.DD_TRACE_OTEL_CTX_ENABLEDadded tosupported-configurations.json; the generated.d.tspicks it up.scripts/docker/: atest:docker:otel-thread-ctxharness that builds insidenode:24-bookworm(the writer is Linux+AsyncContextFrame-only; macOS dev machines fall through to the harness).packages/dd-trace/test/otel-thread-ctx.spec.js, 20 cases):start()gate matrix, on-enter build/skip/drift, span-finish clear-vs-leave, tags-update endpoint append, and the process-context helper.Publish OTEP-4947 process-context metadata via process discoveryFor OTel thread context record to work correctly, information also needs to be published in the process context.
Sets up the OTEP-4719 process context so an out-of-process reader can decode the on-the-wire records. Adds
getThreadLocalMetadata()inotel-thread-ctx.js— pulls the snapshot from@datadog/pprof.otelThreadCtx.getProcessContextAttributes(schema-version string, attribute key map, V8 layout constants) and reshapes it into the napiThreadLocalMetadataform.tracer_metadata.jspasses it as the last positional arg toprocessDiscovery.TracerMetadata(...). Returnsundefinedif pprof is missing → publishes without a threadlocal block.Platform / runtime scope
--experimental-async-context-frameon 22/23). On any other environment,start()logs and returnsfalse— no runtime cost.@datadog/pprofis an optionalDependency; if it isn't installed, the writer stays inert and no threadlocal block is published in the process context.Test plan
yarn test:otel-thread-ctx— 20/20 passyarn mocha --timeout 60000 packages/dd-trace/test/tracer_metadata.spec.js— 11/11 passyarn mocha --timeout 60000 packages/dd-trace/test/profiling/profilers/wall.spec.js— 32/32 pass (unchanged from master after the shared-cache extraction)Jira: PROF-15220