feat(llmobs): propagate session_id across distributed traces [MLOB-7756]#18952
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 8 commits intoJul 10, 2026
Conversation
session_id was inherited in-process but dropped when a trace crossed a service boundary, so downstream LLM spans lost their session. Add session_id to the LLMObs propagating tag set (_dd.p.llmobs_session_id), mirroring how ml_app propagates: inject on the active span/context, extract onto the rebuilt distributed context, and apply to the child span at activation. An explicitly-set session_id still wins over a propagated one. MLOB-7756 Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Codeowners resolved as |
Circular import analysis
|
🎉 All green!🧪 All tests passed 🔗 Commit SHA: 2140fa2 | Docs | Datadog PR Page | Give us feedback! |
Extend session propagation so the first session_id set in a trace becomes the trace-level default. Spans whose parent chain carries no session (e.g. siblings under a session-less parent) now fall back to that default, and it propagates across service boundaries via the existing _dd.p.llmobs_session_id tag. An explicit session_id on a span still overrides locally, so multiple sessions can still coexist in one trace. MLOB-7756 Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f8137ad116
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Yun-Kim
reviewed
Jul 9, 2026
Yun-Kim
reviewed
Jul 9, 2026
…7.yaml Co-authored-by: Yun Kim <[email protected]>
…lmobs_span_data _annotate_llmobs_span_data only writes the span meta_struct; establishing the trace-level default on the shared propagating context now happens in _start_span, after the effective session (explicit / annotation-context / inherited) is applied. Placing it here rather than _activate_llmobs_span is deliberate: _activate_llmobs_span runs before the explicit session_id is applied and never sees it, so it would only ever observe inherited sessions (already covered by the parent chain). Addresses review feedback on #18952. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
_inject_llmobs_context no longer writes the active span's session into the shared propagating context. The trace-level default is already on context._meta (set in _start_span) and is carried on x-datadog-tags by the standard propagator, so the explicit write was redundant and could overwrite the trace default with an active override, causing later session-less siblings to inherit the wrong session. Addresses review feedback on #18952. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…override Adds an in-process test (inspecting the injected x-datadog-tags) and an end-to-end distributed test asserting that when an override child is active at injection time, the trace-level default session is what propagates - and that the override does not corrupt the default for later spans. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Shortens the propagating tag key to conserve x-datadog-tags budget. Coordinated with dd-trace-js and dd-trace-go to keep the wire contract identical across SDKs. Addresses review feedback on #18952. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Yun-Kim
approved these changes
Jul 10, 2026
Keeps the llmobs_ namespace (consistent with _dd.p.llmobs_parent_id / _ml_app / _trace_id) while shortening session_id -> sid to conserve x-datadog-tags budget. Addresses review feedback on #18952. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
gh-worker-dd-mergequeue-cf854d
Bot
deleted the
nicole-cybul/llmobs-session-id-distributed-propagation
branch
July 10, 2026 21:18
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
to DataDog/dd-trace-go
that referenced
this pull request
Jul 15, 2026
…56] (#5011) ### What Does This Do Adds LLMObs `session_id` to the cross-service propagation set (`_dd.p.llmobs_sid`), so a downstream service's LLM spans inherit the `session_id` set on an upstream service's span. Mirrors the existing `ml_app` propagation and the dd-trace-py change in [dd-trace-py#18952](DataDog/dd-trace-py#18952). ### Changes - `internal/llmobs/context.go` — add `SessionID` to `PropagatedLLMSpan`. - `internal/llmobs/span.go` — add `SessionID()` accessor; `propagatedSessionID` falls back to `s.propagated.SessionID` (mirrors `propagatedMLApp`). - `ddtrace/tracer/span.go` — add `keyPropagatedLLMObsSessionID` (`_dd.p.llmobs_sid`); inject the active span's resolved session in `setLLMObsPropagatingTags` (only when non-empty, to conserve `x-datadog-tags` budget). - `ddtrace/tracer/context.go` — extract `_dd.p.llmobs_sid` in `propagatedLLMSpanFromTags`. The tag key `_dd.p.llmobs_sid` matches the wire contract agreed across SDKs (dd-trace-py, dd-trace-js): the `llmobs_` namespace is kept (consistent with `_dd.p.llmobs_parent_id` / `_ml_app` / `_trace_id`) while `session_id` is shortened to `sid` to conserve the `x-datadog-tags` budget. An explicit `session_id` on a span still takes precedence over a propagated one. ### Scope note This PR is **cross-service propagation only**. It does **not** add the in-process "trace-level default" (a sibling under a session-less parent inheriting the trace's session), which py/js get via their trace-shared propagating bag. In go, `internal/llmobs` can't reach the tracer's trace-level propagating tags (one-way import boundary), so a trace default would need a separate trace-scoped store with lifecycle management — deferred as a follow-up. Go already covers the common case (session on the root span → all descendants inherit via the parent-chain walk). ### Testing [Simulating a distributed session](https://github.com/ncybul/sdk-manual-testing/blob/master/go/cross_service.go) ([trace](https://app.datadoghq.com/llm/traces?query=%40ml_app%3Asdk-manual-testing%20%40event_type%3Aspan%20%40parent_id%3Aundefined&agg_m=count&agg_m_source=base&agg_t=count&fromUser=false&is_llm_session=false&refresh_mode=sliding&selectedTab=overview&sp=%5B%7B%22p%22%3A%7B%22eventId%22%3A%22AwAAAZ9b5TBDrCwCXwAAABhBWjliNVRCREFBQVZqSm10cEJZZUFBQUEAAAAkZjE5ZjViZTgtNWVmNy00YThlLTk5ZTEtNGUzNzllYTJjYzVkAAForA%22%7D%2C%22i%22%3A%22llm-obs-panel%22%7D%5D&spanId=7720399594986875361&start=1783952628955&end=1783953528955&paused=false)) - `session_id:my-session` set in service A which calls service B and inherits the same session ID [Simulating session inheritance in a trace unless explicitly annotated](https://github.com/ncybul/sdk-manual-testing/blob/master/go/in_process.go) ([trace](https://app.datadoghq.com/llm/traces?query=%40ml_app%3Asdk-manual-testing%20%40event_type%3Aspan%20%40parent_id%3Aundefined&agg_m=count&agg_m_source=base&agg_t=count&fromUser=false&is_llm_session=false&refresh_mode=sliding&selectedTab=overview&sp=%5B%7B%22p%22%3A%7B%22eventId%22%3A%22AwAAAZ9b6iswjqdBewAAABhBWjliNmlzd0FBQlMyTmJ0U0pfSEFBQUEAAAAkZjE5ZjViZWEtN2MzOS00MThkLTlkYjgtYmE3ZWIxZTMxNGY2AAAm-w%22%7D%2C%22i%22%3A%22llm-obs-panel%22%7D%5D&spanId=6028805347540461692&start=1783952718901&end=1783953618901&paused=false)) - root span sets `session_id:my-session` which is inherited by its children spans except those explicitly annotated with `session_id:other-session` MLOB-7756 Co-authored-by: kakkoyun <[email protected]> Co-authored-by: nicole.cybul <[email protected]>
vlad-scherbich
pushed a commit
that referenced
this pull request
Jul 15, 2026
…56] (#18952) This PR adds propagation support for the LLMObs `session_id` across distributed contexts. This PR also establishes the first seen `session_id` as the default for all subsequent spans in the trace. Annotation contexts / explicit session tagging are still respected with the consequence that one trace can have multiple session IDs (this is the current accepted behavior). ## Description <!-- Provide an overview of the change and motivation for the change --> ## Testing [Simulating a distributed session](https://github.com/ncybul/sdk-manual-testing/blob/master/cross_service.py) ([trace](https://app.datadoghq.com/llm/traces?query=%40ml_app%3Asdk-manual-testing%20%40event_type%3Aspan%20%40parent_id%3Aundefined&agg_m=count&agg_m_source=base&agg_t=count&fromUser=false&is_llm_session=false&refresh_mode=sliding&selectedTab=overview&sp=%5B%7B%22p%22%3A%7B%22eventId%22%3A%22AwAAAZ9Her5LhIWFvQAAABhBWjlIZXI1TEFBQUhjUkhSTTROd0FBQUEAAAAkMDE5ZjQ3N2EtZjJmYy00ZThhLWI1OTYtN2U0M2U3NDI0YmFhAAAF4w%22%7D%2C%22i%22%3A%22llm-obs-panel%22%7D%5D&spanId=1706347288156013108&start=1783607073040&end=1783610673040&paused=false)) - `session_id:my-session` set in service A which calls service B and inherits the same session ID [Simulating default session being applied to subsequent spans in a trace unless explicitly annotated](https://github.com/ncybul/sdk-manual-testing/blob/master/in_process.py) ([trace](https://app.datadoghq.com/llm/traces?query=%40ml_app%3Asdk-manual-testing%20%40event_type%3Aspan%20%40parent_id%3Aundefined&agg_m=count&agg_m_source=base&agg_t=count&fromUser=false&is_llm_session=false&refresh_mode=sliding&selectedTab=overview&sp=%5B%7B%22p%22%3A%7B%22eventId%22%3A%22AwAAAZ9Hf--jl2WBRQAAABhBWjlIZi0takFBQ19vQ0UyalFQX0FBQUEAAAAkMDE5ZjQ3ODAtMDRiMC00NGVlLTk0MTgtN2VlNDdkMDc5ZDcwAAADRw%22%7D%2C%22i%22%3A%22llm-obs-panel%22%7D%5D&spanId=9624447665336394777&start=1783610133070&end=1783611033070&paused=false)) - first span sets `session_id:my-session` which is inherited by subsequent spans except those explicitly annotated with `session_id:other-session` ## Risks <!-- Note any risks associated with this change, or "None" if no risks --> ## Additional Notes <!-- Any other information that would be helpful for reviewers --> Co-authored-by: nicole.cybul <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds propagation support for the LLMObs
session_idacross distributed contexts. This PR also establishes the first seensession_idas the default for all subsequent spans in the trace.Annotation contexts / explicit session tagging are still respected with the consequence that one trace can have multiple session IDs (this is the current accepted behavior).
Description
Testing
Simulating a distributed session (trace)
session_id:my-sessionset in service A which calls service B and inherits the same session IDSimulating default session being applied to subsequent spans in a trace unless explicitly annotated (trace)
session_id:my-sessionwhich is inherited by subsequent spans except those explicitly annotated withsession_id:other-sessionRisks
Additional Notes