Skip to content

feat(llmobs): propagate session_id to following spans and across services [MLOB-7756]#9279

Merged
ncybul merged 3 commits into
masterfrom
nicole-cybul/llmobs-session-id-propagation
Jul 14, 2026
Merged

feat(llmobs): propagate session_id to following spans and across services [MLOB-7756]#9279
ncybul merged 3 commits into
masterfrom
nicole-cybul/llmobs-session-id-propagation

Conversation

@ncybul

@ncybul ncybul commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

What

Propagates LLMObs session_id so it reaches the spans that follow it in a trace — including spans under a session-less parent and spans in downstream services.

The first session_id set in a trace is stored on the trace-shared propagating tags as _dd.p.llmobs_sid. Later spans inherit it (via the tagger's resolution fallback), and it rides x-datadog-tags across service boundaries via the standard propagator's _dd.p.* inject/extract. An explicit session_id on a span still overrides locally, so multiple sessions can still coexist in one trace.

Mirrors the dd-trace-py change in dd-trace-py#18952 and the same _dd.p.llmobs_* propagation pattern already used for ml_app.

Changes

  • packages/dd-trace/src/llmobs/constants/tags.js — add PROPAGATED_SESSION_ID_KEY (_dd.p.llmobs_sid).
  • packages/dd-trace/src/llmobs/tagger.js — resolve session_id from explicit arg → parent → trace default; establish the trace default on _trace.tags (first-writer). No index.js inject change needed: _injectTags already propagates any _dd.p.* trace tag.
  • packages/dd-trace/test/llmobs/tagger.spec.js — tests for establishing the default, sibling-gap inheritance, and explicit override not clobbering the default.

Test plan

Simulating a distributed session (trace)

  • session_id:my-session set in service A which calls service B and inherits the same session ID

Simulating default session being applied to subsequent spans in a trace unless explicitly annotated (trace)

  • first span sets session_id:my-session which is inherited by subsequent spans except those explicitly annotated with session_id:other-session

MLOB-7756

…ices

The first session_id set in a trace is stored on the trace-shared
propagating tags (_dd.p.llmobs_session_id). Later spans - including
those under a session-less parent and spans in downstream services -
inherit it, and it rides x-datadog-tags across service boundaries via
the standard propagator. An explicit session_id still overrides locally.

MLOB-7756

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@dd-octo-sts

dd-octo-sts Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Overall package size

Self size: 6.71 MB
Deduped: 7.37 MB
No deduping: 7.37 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.1 | 122.62 kB | 437.94 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@pr-commenter

pr-commenter Bot commented Jul 9, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-07-10 21:07:26

Comparing candidate commit ac05921 in PR branch nicole-cybul/llmobs-session-id-propagation with baseline commit 4398f17 in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2324 metrics, 34 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:appsec-appsec-enabled-24

  • unstable execution_time [-203.236ms; +211.279ms] or [-7.682%; +7.986%]

scenario:appsec-appsec-enabled-26

  • unstable execution_time [-235.422ms; +239.827ms] or [-9.226%; +9.399%]

scenario:appsec-appsec-enabled-with-attacks-24

  • unstable execution_time [-157561.220µs; +159254.920µs] or [-5.113%; +5.168%]

scenario:appsec-appsec-enabled-with-attacks-26

  • unstable execution_time [-197455.819µs; +197316.253µs] or [-6.764%; +6.760%]

scenario:appsec-control-20

  • unstable execution_time [-125.168ms; +121.229ms] or [-7.627%; +7.387%]

scenario:appsec-control-24

  • unstable execution_time [-114.010ms; +111.535ms] or [-9.218%; +9.018%]

scenario:appsec-control-26

  • unstable execution_time [-126.872ms; +121.500ms] or [-10.234%; +9.801%]

scenario:appsec-iast-with-vulnerability-control-20

  • unstable execution_time [-28.167ms; +30.653ms] or [-5.131%; +5.584%]

scenario:debugger-line-probe-with-snapshot-default-24

  • unstable cpu_user_time [-1771.033ms; +2603.223ms] or [-21.385%; +31.434%]
  • unstable execution_time [-1753.112ms; +2623.642ms] or [-19.454%; +29.114%]
  • unstable instructions [-15.3G instructions; +22.6G instructions] or [-22.625%; +33.503%]
  • unstable max_rss_usage [-7.113MB; +10.879MB] or [-4.539%; +6.943%]
  • unstable throughput [-739.152op/s; +508.152op/s] or [-20.356%; +13.994%]

scenario:debugger-line-probe-with-snapshot-default-26

  • unstable cpu_user_time [-2.091s; +4.561s] or [-20.946%; +45.694%]
  • unstable execution_time [-2.201s; +4.625s] or [-20.565%; +43.223%]
  • unstable instructions [-18.8G instructions; +40.6G instructions] or [-22.462%; +48.503%]
  • unstable max_rss_usage [-6.682MB; +14.724MB] or [-4.156%; +9.158%]
  • unstable throughput [-1002.076op/s; +484.234op/s] or [-32.097%; +15.510%]

scenario:debugger-line-probe-without-snapshot-24

  • unstable cpu_user_time [-2.485s; +0.341s] or [-27.930%; +3.830%]
  • unstable execution_time [-2.506s; +0.331s] or [-26.011%; +3.434%]
  • unstable instructions [-21.3G instructions; +2.8G instructions] or [-29.180%; +3.889%]
  • unstable throughput [-97.108op/s; +694.230op/s] or [-2.807%; +20.070%]

scenario:debugger-line-probe-without-snapshot-26

  • unstable cpu_user_time [-2857.002ms; +2884.952ms] or [-28.637%; +28.917%]
  • unstable execution_time [-2847.518ms; +2882.972ms] or [-26.605%; +26.936%]
  • unstable instructions [-25436.8M instructions; +25679.3M instructions] or [-30.428%; +30.718%]
  • unstable max_rss_usage [-9518.632KB; +9149.032KB] or [-5.925%; +5.694%]
  • unstable throughput [-630.959op/s; +618.307op/s] or [-20.238%; +19.832%]

scenario:dogstatsd-with-tags-20

  • unstable cpu_user_time [-315.241ms; +382.475ms] or [-6.532%; +7.925%]
  • unstable execution_time [-308.101ms; +380.133ms] or [-6.280%; +7.748%]
  • unstable throughput [-130362.760op/s; +108271.958op/s] or [-7.623%; +6.331%]

scenario:plugin-claude-agent-sdk-compact-hook-indexed-26

  • unstable cpu_usage_percentage [-6.999%; +4.081%]

scenario:plugin-graphql-long-with-depth-on-max-20

  • unstable cpu_user_time [-751.582ms; +704.152ms] or [-5.869%; +5.499%]
  • unstable execution_time [-748.682ms; +700.661ms] or [-5.722%; +5.355%]
  • unstable throughput [-3.248op/s; +3.462op/s] or [-5.280%; +5.628%]

@datadog-official

datadog-official Bot commented Jul 9, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 96.53% (+0.00%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ac05921 | Docs | Datadog PR Page | Give us feedback!

@codecov

codecov Bot commented Jul 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.57%. Comparing base (4398f17) to head (ac05921).
⚠️ Report is 7 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #9279      +/-   ##
==========================================
- Coverage   96.57%   96.57%   -0.01%     
==========================================
  Files         918      918              
  Lines      121521   121579      +58     
  Branches    21089    20993      -96     
==========================================
+ Hits       117355   117411      +56     
- Misses       4166     4168       +2     
Flag Coverage Δ
aiguard 53.60% <12.50%> (-0.04%) ⬇️
aiguard-integration 56.98% <12.50%> (-0.05%) ⬇️
apm-bucket-0 53.85% <12.50%> (-0.02%) ⬇️
apm-bucket-1 58.05% <12.50%> (-0.11%) ⬇️
apm-bucket-2 56.09% <12.50%> (-0.10%) ⬇️
apm-capabilities-tracing 60.31% <12.50%> (-0.03%) ⬇️
apm-integrations-aerospike 52.91% <12.50%> (-0.10%) ⬇️
apm-integrations-confluentinc-kafka-javascript 57.03% <12.50%> (-0.11%) ⬇️
apm-integrations-couchbase 53.25% <12.50%> (-0.10%) ⬇️
apm-integrations-http 57.94% <12.50%> (-0.10%) ⬇️
apm-integrations-kafkajs 57.69% <12.50%> (-0.13%) ⬇️
apm-integrations-next 54.22% <12.50%> (-0.10%) ⬇️
apm-integrations-prisma 54.15% <12.50%> (-0.10%) ⬇️
apm-integrations-tedious 53.02% <12.50%> (-0.10%) ⬇️
appsec 68.94% <12.50%> (-0.11%) ⬇️
appsec-express_fastify_graphql 65.58% <12.50%> (-0.09%) ⬇️
appsec-integration 45.58% <12.50%> (-0.02%) ⬇️
appsec-kafka_ldapjs_lodash 59.07% <12.50%> (-0.10%) ⬇️
appsec-mongodb-core_mongoose_mysql 62.22% <12.50%> (-0.09%) ⬇️
appsec-next 52.83% <12.50%> (-0.09%) ⬇️
appsec-node-serialize_passport_postgres 61.86% <12.50%> (-0.09%) ⬇️
appsec-sourcing_stripe_template 60.21% <12.50%> (-0.09%) ⬇️
debugger 65.71% <12.50%> (-0.08%) ⬇️
instrumentations-bucket-0 48.73% <12.50%> (-0.10%) ⬇️
instrumentations-bucket-1 54.93% <12.50%> (-0.10%) ⬇️
instrumentations-bucket-10 56.60% <12.50%> (-0.10%) ⬇️
instrumentations-bucket-11 48.72% <12.50%> (-0.10%) ⬇️
instrumentations-bucket-12 48.82% <12.50%> (-0.09%) ⬇️
instrumentations-bucket-13 48.66% <12.50%> (-0.10%) ⬇️
instrumentations-bucket-2 49.71% <12.50%> (-0.09%) ⬇️
instrumentations-bucket-3 53.81% <12.50%> (-0.10%) ⬇️
instrumentations-bucket-4 49.14% <12.50%> (-0.10%) ⬇️
instrumentations-bucket-5 52.99% <12.50%> (-0.09%) ⬇️
instrumentations-bucket-6 55.58% <12.50%> (-0.11%) ⬇️
instrumentations-bucket-7 53.58% <12.50%> (-0.10%) ⬇️
instrumentations-bucket-8 54.61% <12.50%> (-0.10%) ⬇️
instrumentations-bucket-9 56.07% <12.50%> (-0.09%) ⬇️
instrumentations-instrumentation-couchbase 48.16% <12.50%> (-0.09%) ⬇️
instrumentations-integration-esbuild 33.96% <ø> (-0.01%) ⬇️
llmobs-ai_anthropic_bedrock 57.73% <68.75%> (-0.07%) ⬇️
llmobs-bucket-1 56.99% <100.00%> (-0.06%) ⬇️
llmobs-openai 57.56% <68.75%> (-0.08%) ⬇️
llmobs-sdk 59.96% <100.00%> (-0.08%) ⬇️
llmobs-vertex-ai 54.21% <68.75%> (-0.08%) ⬇️
master-coverage 96.57% <100.00%> (?)
openfeature 54.59% <12.50%> (-0.05%) ⬇️
openfeature-unit 49.81% <12.50%> (-0.10%) ⬇️
platform-core_esbuild_instrumentations-misc 38.44% <12.50%> (-0.03%) ⬇️
platform-integration 62.21% <12.50%> (+<0.01%) ⬆️
platform-shimmer_unit-guardrails_webpack 37.27% <12.50%> (+<0.01%) ⬆️
plugins-bucket-0 53.16% <12.50%> (-0.09%) ⬇️
plugins-bucket-1 55.06% <12.50%> (-0.06%) ⬇️
plugins-bucket-11 55.58% <12.50%> (-0.07%) ⬇️
plugins-bucket-18 57.05% <12.50%> (-0.10%) ⬇️
plugins-bucket-19 55.33% <12.50%> (-0.13%) ⬇️
plugins-bucket-20 57.42% <12.50%> (-0.10%) ⬇️
plugins-bucket-4 53.83% <12.50%> (-0.10%) ⬇️
plugins-bullmq_cassandra_cookie 57.37% <12.50%> (-0.10%) ⬇️
plugins-cookie-parser_crypto_dd-trace-api 52.30% <12.50%> (-0.10%) ⬇️
plugins-fetch_fs_generic-pool 54.44% <12.50%> (-0.10%) ⬇️
plugins-google-cloud-pubsub_grpc_handlebars 59.85% <12.50%> (-0.10%) ⬇️
plugins-hapi_hono_ioredis 55.84% <12.50%> (-0.10%) ⬇️
plugins-jest_knex_langgraph 51.64% <12.50%> (-0.10%) ⬇️
plugins-ldapjs_light-my-request_limitd-client 53.39% <12.50%> (-0.10%) ⬇️
plugins-lodash_mariadb_memcached 54.19% <12.50%> (-0.10%) ⬇️
plugins-moleculer_mongodb_mongodb-core 57.26% <12.50%> (-0.10%) ⬇️
plugins-mongoose_multer_mysql 54.69% <12.50%> (-0.10%) ⬇️
plugins-mysql2_nats_node-serialize 56.18% <12.50%> (-0.11%) ⬇️
plugins-opensearch_passport-http_pino 55.03% <12.50%> (-0.10%) ⬇️
plugins-postgres_process_pug 54.27% <12.50%> (-0.10%) ⬇️
plugins-redis_router_sequelize 57.01% <12.50%> (-0.10%) ⬇️
plugins-test-and-upstream-rhea_undici_url 57.01% <12.50%> (-0.08%) ⬇️
plugins-valkey_vm_winston 53.82% <12.50%> (-0.10%) ⬇️
plugins-ws 54.78% <12.50%> (-0.10%) ⬇️
profiling 58.18% <12.50%> (-0.09%) ⬇️
serverless-aws-sdk-aws-sdk 50.65% <12.50%> (-0.08%) ⬇️
serverless-aws-sdk-bedrockruntime 50.67% <12.50%> (-0.09%) ⬇️
serverless-aws-sdk-client 51.95% <12.50%> (-0.09%) ⬇️
serverless-aws-sdk-dynamodb 51.73% <12.50%> (-0.09%) ⬇️
serverless-aws-sdk-eventbridge 46.23% <12.50%> (-0.08%) ⬇️
serverless-aws-sdk-kinesis 54.69% <12.50%> (-0.09%) ⬇️
serverless-aws-sdk-lambda 52.77% <12.50%> (-0.09%) ⬇️
serverless-aws-sdk-s3 51.51% <12.50%> (-0.09%) ⬇️
serverless-aws-sdk-serverless-peer-service 54.74% <12.50%> (-0.06%) ⬇️
serverless-aws-sdk-sns 55.46% <12.50%> (-0.10%) ⬇️
serverless-aws-sdk-sqs 55.95% <12.50%> (-0.04%) ⬇️
serverless-aws-sdk-stepfunctions 51.23% <12.50%> (-0.09%) ⬇️
serverless-aws-sdk-util 48.45% <12.50%> (-0.10%) ⬇️
serverless-bucket-0 55.20% <12.50%> (-0.06%) ⬇️
serverless-bucket-1 55.98% <12.50%> (-0.10%) ⬇️
test-optimization-cucumber 72.91% <12.50%> (-0.07%) ⬇️
test-optimization-cypress 66.37% <12.50%> (+0.06%) ⬆️
test-optimization-jest 73.87% <12.50%> (-0.09%) ⬇️
test-optimization-mocha 74.27% <12.50%> (-0.01%) ⬇️
test-optimization-playwright-playwright-atr 61.55% <12.50%> (-0.02%) ⬇️
test-optimization-playwright-playwright-efd 61.74% <12.50%> (-0.02%) ⬇️
test-optimization-playwright-playwright-final-status 61.69% <12.50%> (-0.03%) ⬇️
test-optimization-playwright-playwright-impacted-tests 61.42% <12.50%> (+0.14%) ⬆️
test-optimization-playwright-playwright-reporting 61.32% <12.50%> (-0.03%) ⬇️
test-optimization-playwright-playwright-test-management 62.23% <12.50%> (-0.07%) ⬇️
test-optimization-playwright-playwright-test-span 61.46% <12.50%> (-0.08%) ⬇️
test-optimization-selenium 60.65% <12.50%> (-0.18%) ⬇️
test-optimization-testopt 59.14% <12.50%> (+0.04%) ⬆️
test-optimization-vitest 71.55% <12.50%> (-0.92%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Matches the wire contract agreed in dd-trace-py#18952: keep the llmobs_
namespace, shorten session_id -> sid to conserve x-datadog-tags budget.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@ncybul
ncybul marked this pull request as ready for review July 10, 2026 18:55
@ncybul
ncybul requested a review from a team as a code owner July 10, 2026 18:55

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: afae342a84

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread packages/dd-trace/src/llmobs/tagger.js Outdated
Comment thread packages/dd-trace/src/llmobs/tagger.js Outdated
…registration

Move the trace-level default establishment into _setTag, the single choke
point for session writes, so sessions post-populated by integrations after
span start (e.g. claude-agent-sdk's _setTag(span, SESSION_ID, ...)) also seed
the trace default. First-writer wins, so an explicit session still overrides.

Addresses review feedback on #9279.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Comment thread packages/dd-trace/src/llmobs/tagger.js
@ncybul
ncybul merged commit 4fc6e58 into master Jul 14, 2026
657 checks passed
@ncybul
ncybul deleted the nicole-cybul/llmobs-session-id-propagation branch July 14, 2026 13:36
This was referenced Jul 14, 2026
juan-fernandez pushed a commit that referenced this pull request Jul 14, 2026
…ices [MLOB-7756] (#9279)

* feat(llmobs): propagate session_id to following spans and across services

The first session_id set in a trace is stored on the trace-shared
propagating tags (_dd.p.llmobs_session_id). Later spans - including
those under a session-less parent and spans in downstream services -
inherit it, and it rides x-datadog-tags across service boundaries via
the standard propagator. An explicit session_id still overrides locally.

MLOB-7756

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* refactor(llmobs): shorten propagated session key to _dd.p.llmobs_sid

Matches the wire contract agreed in dd-trace-py#18952: keep the llmobs_
namespace, shorten session_id -> sid to conserve x-datadog-tags budget.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* fix(llmobs): establish trace-default session in _setTag, not just at registration

Move the trace-level default establishment into _setTag, the single choke
point for session writes, so sessions post-populated by integrations after
span start (e.g. claude-agent-sdk's _setTag(span, SESSION_ID, ...)) also seed
the trace default. First-writer wins, so an explicit session still overrides.

Addresses review feedback on #9279.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
juan-fernandez pushed a commit that referenced this pull request Jul 14, 2026
…ices [MLOB-7756] (#9279)

* feat(llmobs): propagate session_id to following spans and across services

The first session_id set in a trace is stored on the trace-shared
propagating tags (_dd.p.llmobs_session_id). Later spans - including
those under a session-less parent and spans in downstream services -
inherit it, and it rides x-datadog-tags across service boundaries via
the standard propagator. An explicit session_id still overrides locally.

MLOB-7756

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* refactor(llmobs): shorten propagated session key to _dd.p.llmobs_sid

Matches the wire contract agreed in dd-trace-py#18952: keep the llmobs_
namespace, shorten session_id -> sid to conserve x-datadog-tags budget.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* fix(llmobs): establish trace-default session in _setTag, not just at registration

Move the trace-level default establishment into _setTag, the single choke
point for session writes, so sessions post-populated by integrations after
span start (e.g. claude-agent-sdk's _setTag(span, SESSION_ID, ...)) also seed
the trace default. First-writer wins, so an explicit session still overrides.

Addresses review feedback on #9279.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants