Skip to content

feat(llmobs): propagate session_id across distributed traces [MLOB-7756]#18952

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 8 commits into
mainfrom
nicole-cybul/llmobs-session-id-distributed-propagation
Jul 10, 2026
Merged

feat(llmobs): propagate session_id across distributed traces [MLOB-7756]#18952
gh-worker-dd-mergequeue-cf854d[bot] merged 8 commits into
mainfrom
nicole-cybul/llmobs-session-id-distributed-propagation

Conversation

@ncybul

@ncybul ncybul commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

This PR adds propagation support for the LLMObs session_id across distributed contexts. This PR also establishes the first seen session_id as the default for all subsequent spans in the trace.

Annotation contexts / explicit session tagging are still respected with the consequence that one trace can have multiple session IDs (this is the current accepted behavior).

Description

Testing

Simulating a distributed session (trace)

  • session_id:my-session set in service A which calls service B and inherits the same session ID

Simulating default session being applied to subsequent spans in a trace unless explicitly annotated (trace)

  • first span sets session_id:my-session which is inherited by subsequent spans except those explicitly annotated with session_id:other-session

Risks

Additional Notes

session_id was inherited in-process but dropped when a trace crossed a
service boundary, so downstream LLM spans lost their session. Add
session_id to the LLMObs propagating tag set (_dd.p.llmobs_session_id),
mirroring how ml_app propagates: inject on the active span/context,
extract onto the rebuilt distributed context, and apply to the child
span at activation. An explicitly-set session_id still wins over a
propagated one.

MLOB-7756

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@ncybul ncybul changed the title feat(llmobs): propagate session_id across distributed traces feat(llmobs): propagate session_id across distributed traces [MLOB-7756] Jul 8, 2026
@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 8, 2026

Copy link
Copy Markdown

Codeowners resolved as

ddtrace/llmobs/_constants.py                                            @DataDog/ml-observability
ddtrace/llmobs/_llmobs.py                                               @DataDog/ml-observability
releasenotes/notes/llmobs-propagate-session-id-4a06a4bc6ecb4ef7.yaml    @DataDog/apm-python
tests/llmobs/test_propagation.py                                        @DataDog/ml-observability

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 8, 2026

Copy link
Copy Markdown

Circular import analysis

⚠️ Existing circular imports

There are 4974 circular imports that already exist on the base branch and have not been changed by this PR.

Show existing cycles (showing 5 of 4974 shortest)
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.botocore -> ddtrace.internal.datastreams
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.kombu -> ddtrace.internal.datastreams
ddtrace.internal.core -> ddtrace._trace.span -> ddtrace.internal.core
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.aiokafka -> ddtrace.internal.datastreams
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.kafka -> ddtrace.internal.datastreams

To see all cycles, download the cycles-base.json and cycles-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/cycles.py compare cycles-base.json cycles-pr.json

@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Jul 8, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 2140fa2 | Docs | Datadog PR Page | Give us feedback!

Extend session propagation so the first session_id set in a trace
becomes the trace-level default. Spans whose parent chain carries no
session (e.g. siblings under a session-less parent) now fall back to
that default, and it propagates across service boundaries via the
existing _dd.p.llmobs_session_id tag. An explicit session_id on a span
still overrides locally, so multiple sessions can still coexist in one
trace.

MLOB-7756

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@ncybul
ncybul marked this pull request as ready for review July 9, 2026 15:32
@ncybul
ncybul requested review from a team as code owners July 9, 2026 15:33
@ncybul
ncybul requested review from Kyle-Verhoog and duncanista July 9, 2026 15:33

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f8137ad116

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread ddtrace/llmobs/_llmobs.py
Comment thread ddtrace/llmobs/_llmobs.py Outdated
Comment thread ddtrace/llmobs/_utils.py Outdated
Comment thread ddtrace/llmobs/_llmobs.py
Comment thread ddtrace/llmobs/_utils.py Outdated
Comment thread ddtrace/llmobs/_constants.py Outdated
ncybul and others added 5 commits July 9, 2026 16:17
…lmobs_span_data

_annotate_llmobs_span_data only writes the span meta_struct; establishing
the trace-level default on the shared propagating context now happens in
_start_span, after the effective session (explicit / annotation-context /
inherited) is applied. Placing it here rather than _activate_llmobs_span
is deliberate: _activate_llmobs_span runs before the explicit session_id
is applied and never sees it, so it would only ever observe inherited
sessions (already covered by the parent chain).

Addresses review feedback on #18952.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
_inject_llmobs_context no longer writes the active span's session into
the shared propagating context. The trace-level default is already on
context._meta (set in _start_span) and is carried on x-datadog-tags by
the standard propagator, so the explicit write was redundant and could
overwrite the trace default with an active override, causing later
session-less siblings to inherit the wrong session.

Addresses review feedback on #18952.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…override

Adds an in-process test (inspecting the injected x-datadog-tags) and an
end-to-end distributed test asserting that when an override child is
active at injection time, the trace-level default session is what
propagates - and that the override does not corrupt the default for
later spans.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Shortens the propagating tag key to conserve x-datadog-tags budget.
Coordinated with dd-trace-js and dd-trace-go to keep the wire contract
identical across SDKs.

Addresses review feedback on #18952.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Keeps the llmobs_ namespace (consistent with _dd.p.llmobs_parent_id /
_ml_app / _trace_id) while shortening session_id -> sid to conserve
x-datadog-tags budget.

Addresses review feedback on #18952.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 8e44831 into main Jul 10, 2026
590 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the nicole-cybul/llmobs-session-id-distributed-propagation branch July 10, 2026 21:18
gh-worker-dd-mergequeue-cf854d Bot pushed a commit to DataDog/dd-trace-go that referenced this pull request Jul 15, 2026
…56] (#5011)

### What Does This Do

Adds LLMObs `session_id` to the cross-service propagation set (`_dd.p.llmobs_sid`), so a downstream service's LLM spans inherit the `session_id` set on an upstream service's span. Mirrors the existing `ml_app` propagation and the dd-trace-py change in [dd-trace-py#18952](DataDog/dd-trace-py#18952).

### Changes

- `internal/llmobs/context.go` — add `SessionID` to `PropagatedLLMSpan`.
- `internal/llmobs/span.go` — add `SessionID()` accessor; `propagatedSessionID` falls back to `s.propagated.SessionID` (mirrors `propagatedMLApp`).
- `ddtrace/tracer/span.go` — add `keyPropagatedLLMObsSessionID` (`_dd.p.llmobs_sid`); inject the active span's resolved session in `setLLMObsPropagatingTags` (only when non-empty, to conserve `x-datadog-tags` budget).
- `ddtrace/tracer/context.go` — extract `_dd.p.llmobs_sid` in `propagatedLLMSpanFromTags`.

The tag key `_dd.p.llmobs_sid` matches the wire contract agreed across SDKs (dd-trace-py, dd-trace-js): the `llmobs_` namespace is kept (consistent with `_dd.p.llmobs_parent_id` / `_ml_app` / `_trace_id`) while `session_id` is shortened to `sid` to conserve the `x-datadog-tags` budget.

An explicit `session_id` on a span still takes precedence over a propagated one.

### Scope note

This PR is **cross-service propagation only**. It does **not** add the in-process "trace-level default" (a sibling under a session-less parent inheriting the trace's session), which py/js get via their trace-shared propagating bag. In go, `internal/llmobs` can't reach the tracer's trace-level propagating tags (one-way import boundary), so a trace default would need a separate trace-scoped store with lifecycle management — deferred as a follow-up. Go already covers the common case (session on the root span → all descendants inherit via the parent-chain walk).

### Testing

[Simulating a distributed session](https://github.com/ncybul/sdk-manual-testing/blob/master/go/cross_service.go) ([trace](https://app.datadoghq.com/llm/traces?query=%40ml_app%3Asdk-manual-testing%20%40event_type%3Aspan%20%40parent_id%3Aundefined&agg_m=count&agg_m_source=base&agg_t=count&fromUser=false&is_llm_session=false&refresh_mode=sliding&selectedTab=overview&sp=%5B%7B%22p%22%3A%7B%22eventId%22%3A%22AwAAAZ9b5TBDrCwCXwAAABhBWjliNVRCREFBQVZqSm10cEJZZUFBQUEAAAAkZjE5ZjViZTgtNWVmNy00YThlLTk5ZTEtNGUzNzllYTJjYzVkAAForA%22%7D%2C%22i%22%3A%22llm-obs-panel%22%7D%5D&spanId=7720399594986875361&start=1783952628955&end=1783953528955&paused=false))

- `session_id:my-session` set in service A which calls service B and inherits the same session ID

[Simulating session inheritance in a trace unless explicitly annotated](https://github.com/ncybul/sdk-manual-testing/blob/master/go/in_process.go) ([trace](https://app.datadoghq.com/llm/traces?query=%40ml_app%3Asdk-manual-testing%20%40event_type%3Aspan%20%40parent_id%3Aundefined&agg_m=count&agg_m_source=base&agg_t=count&fromUser=false&is_llm_session=false&refresh_mode=sliding&selectedTab=overview&sp=%5B%7B%22p%22%3A%7B%22eventId%22%3A%22AwAAAZ9b6iswjqdBewAAABhBWjliNmlzd0FBQlMyTmJ0U0pfSEFBQUEAAAAkZjE5ZjViZWEtN2MzOS00MThkLTlkYjgtYmE3ZWIxZTMxNGY2AAAm-w%22%7D%2C%22i%22%3A%22llm-obs-panel%22%7D%5D&spanId=6028805347540461692&start=1783952718901&end=1783953618901&paused=false))

- root span sets `session_id:my-session` which is inherited by its children spans except those explicitly annotated with `session_id:other-session`

MLOB-7756

Co-authored-by: kakkoyun <[email protected]>
Co-authored-by: nicole.cybul <[email protected]>
vlad-scherbich pushed a commit that referenced this pull request Jul 15, 2026
…56] (#18952)

This PR adds propagation support for the LLMObs `session_id` across distributed contexts. This PR also establishes the first seen `session_id` as the default for all subsequent spans in the trace.

Annotation contexts / explicit session tagging are still respected with the consequence that one trace can have multiple session IDs (this is the current accepted behavior).

## Description

<!-- Provide an overview of the change and motivation for the change -->

## Testing

[Simulating a distributed session](https://github.com/ncybul/sdk-manual-testing/blob/master/cross_service.py) ([trace](https://app.datadoghq.com/llm/traces?query=%40ml_app%3Asdk-manual-testing%20%40event_type%3Aspan%20%40parent_id%3Aundefined&agg_m=count&agg_m_source=base&agg_t=count&fromUser=false&is_llm_session=false&refresh_mode=sliding&selectedTab=overview&sp=%5B%7B%22p%22%3A%7B%22eventId%22%3A%22AwAAAZ9Her5LhIWFvQAAABhBWjlIZXI1TEFBQUhjUkhSTTROd0FBQUEAAAAkMDE5ZjQ3N2EtZjJmYy00ZThhLWI1OTYtN2U0M2U3NDI0YmFhAAAF4w%22%7D%2C%22i%22%3A%22llm-obs-panel%22%7D%5D&spanId=1706347288156013108&start=1783607073040&end=1783610673040&paused=false))
- `session_id:my-session` set in service A which calls service B and inherits the same session ID

[Simulating default session being applied to subsequent spans in a trace unless explicitly annotated](https://github.com/ncybul/sdk-manual-testing/blob/master/in_process.py) ([trace](https://app.datadoghq.com/llm/traces?query=%40ml_app%3Asdk-manual-testing%20%40event_type%3Aspan%20%40parent_id%3Aundefined&agg_m=count&agg_m_source=base&agg_t=count&fromUser=false&is_llm_session=false&refresh_mode=sliding&selectedTab=overview&sp=%5B%7B%22p%22%3A%7B%22eventId%22%3A%22AwAAAZ9Hf--jl2WBRQAAABhBWjlIZi0takFBQ19vQ0UyalFQX0FBQUEAAAAkMDE5ZjQ3ODAtMDRiMC00NGVlLTk0MTgtN2VlNDdkMDc5ZDcwAAADRw%22%7D%2C%22i%22%3A%22llm-obs-panel%22%7D%5D&spanId=9624447665336394777&start=1783610133070&end=1783611033070&paused=false))
- first span sets `session_id:my-session` which is inherited by subsequent spans except those explicitly annotated with `session_id:other-session`

## Risks

<!-- Note any risks associated with this change, or "None" if no risks -->

## Additional Notes

<!-- Any other information that would be helpful for reviewers -->


Co-authored-by: nicole.cybul <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants