feat(traces): [SVLS-8734] respect Datadog-Client-Computed-Stats header by lym953 · Pull Request #1118 · DataDog/datadog-lambda-extension

lym953 · 2026-03-20T12:53:08Z

Background

traces/stats that go through Lambda extension can generate stats in three ways:

tracer generates stats and sends them to extension at /v0.6/stats. Only works if the env var DD_TRACE_STATS_COMPUTATION_ENABLED is true.
extension generates stats from traces. It's off by default, and works only if env var DD_COMPUTE_TRACE_STATS_ON_EXTENSION is true.
Datadog backend generates stats from traces. Works only if the trace tag _dd.compute_stats is 1.

A separate thing: the Go tracer sends a dummy aws.lambda span to the extension, just to send metadata. The extension extracts metadata from this span received, and sets it on the real aws.lambda span generated on the extension side. The latter is what's finally sent to the trace intake.

Summary

If a trace has the header Datadog-Client-Computed-Stats with a truthy value (which means 1 is on), then turn off 2 and 3 to avoid duplicate counts, i.e.

make extension not generate stats
set trace tag _dd.compute_stats to 0

As a special case, if the dummy aws.lambda span sent by the Go tracer has a Datadog-Client-Computed-Stats header with truthy value, then extension propagates this value to the aws.lambda span generated by the extension.

The _dd.compute_stats tag is determined by the combination of compute_trace_stats_on_extension and client_computed_stats:

Input: `compute_trace_stats_on_extension`	Input: `client_computed_stats`	Expected: `_dd.compute_stats`	Expected: Extension generates stats?
`false`	`false`	`"1"`	No
`false`	`true`	`"0"`	No
`true`	`false`	`"0"`	Yes
`true`	`true`	`"0"`	No

Details

_dd.compute_stats is now always set in process_traces() (centralized from tags_from_env() in tags.rs), with the value determined by both compute_trace_stats_on_extension and client_computed_stats as shown in the table above
When client_computed_stats is true or compute_trace_stats_on_extension is true, the extension skips stats generation in send_processed_traces() to avoid double-counting
COMPUTE_STATS_KEY is pub(crate) in tags.rs so it can be referenced from trace_processor
Propagates client_computed_stats to the extension-generated aws.lambda invocation span: when the Go tracer sends its placeholder span (dd-tracer-serverless-span) with Datadog-Client-Computed-Stats: t, that flag is stored in the invocation Context and passed through to send_spans, so the extension also skips stats generation for the aws.lambda span it creates

Test plan

Automated tests

Passed the new unit tests

Manual tests

Steps

Deploy a Lambda function that uses github.com/DataDog/dd-trace-go/v2 v2.6.0
Set env var:
DD_TRACE_STATS_COMPUTATION_ENABLED: true, which turns on 1
DD_COMPUTE_TRACE_STATS_ON_EXTENSION: true, which tries to turn on 2

Result

Before: 2 hits (duplicate):

one with resource_name=dd-tracer-serverless-span, generated by Go tracer
one with resource_name=yiming2-xxx

After: 1 hit. Only the one with resource_name=dd-tracer-serverless-span

🤖 Generated with Claude Code

Copilot

Pull request overview

Updates the Lambda extension trace pipeline to respect the Datadog-Client-Computed-Stats header so stats are not double-computed (by the extension and/or backend) when the tracer already computed them.

Changes:

When client_computed_stats is true, forces _dd.compute_stats to "0" in trace payload tags so backend stats computation is disabled.
Skips extension-side stats generation when client_computed_stats is true, even if DD_COMPUTE_TRACE_STATS_ON_EXTENSION is enabled.
Makes COMPUTE_STATS_KEY pub(crate) and adds unit tests covering the new behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
`bottlecap/src/traces/trace_processor.rs`	Overrides `_dd.compute_stats` when the tracer computed stats; skips extension stats generation; adds tests.
`bottlecap/src/tags/lambda/tags.rs`	Exposes `COMPUTE_STATS_KEY` within the crate for reuse.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-20T16:48:03Z

+    /// Verifies that when `client_computed_stats` is true, `process_traces` sets
+    /// `_dd.compute_stats` to "0" in the payload tags, overriding the value set by
+    /// `get_function_tags_map`, and that `send_processed_traces` does not generate stats.
+    #[tokio::test]


This test’s doc comment says it verifies send_processed_traces skips stats generation when client_computed_stats is true, but the config in this test has compute_trace_stats_on_extension: false, so stats generation is skipped regardless of the header. Consider removing the send_processed_traces portion here (the following test already covers the header-based skip), or adjust the setup so this test actually exercises the new condition.

When the Datadog-Client-Computed-Stats header is set (truthy), the tracer has already computed stats, so the extension should not double-count them. This change: - Skips extension-side stats generation when client_computed_stats is true - Sets _dd.compute_stats to 0 in the trace payload tags so the backend also skips stats computation Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

… header is set - Extended existing test to also assert that send_processed_traces does not generate stats when client_computed_stats is true - Added new test covering the case where compute_trace_stats_on_extension is true but client_computed_stats is also true — no stats should be generated Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

…aws.lambda span When the Go tracer sends its placeholder span (dd-tracer-serverless-span) with Datadog-Client-Computed-Stats: t, store that flag in the invocation Context and pass it through to send_spans so the extension does not double-count stats for the aws.lambda span it generates. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

litianningdatadog · 2026-03-23T20:50:14Z

+                if header_tags.client_computed_stats {
+                    tracer_payload
+                        .tags
+                        .insert(COMPUTE_STATS_KEY.to_string(), "0".to_string());


if client_computed_stats = true, COMPUTE_STATS_KEY is set to 0. Does it also imply if client_computed_stats = false, COMPUTE_STATS_KEY should be set to 1 to avoid implicit inconsistency?

Good question. COMPUTE_STATS_KEY should be set to 1 only if both header_tags.client_computed_stats and config.compute_trace_stats_on_extension are false. Let me centralize the logic for clarity.

litianningdatadog

left a comment

…_processor Moved COMPUTE_STATS_KEY insertion from tags_from_env() to process_traces(). The tag is now always set with the correct value: - "1" (backend computes stats) when neither the extension nor the tracer computes them - "0" (skip backend stats) when either compute_trace_stats_on_extension is true or the Datadog-Client-Computed-Stats header is set by the tracer Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

…ogic - test_process_traces_client_computed_stats_overrides_compute_stats_tag: add assertion that _dd.compute_stats is "1" when both flags are false - test_send_processed_traces_skips_stats_when_client_computed_stats: add assertion that _dd.compute_stats is "0" when compute_trace_stats_on_extension is true - test_client_computed_stats_propagated_to_aws_lambda_span: add assertion that _dd.compute_stats is "0" in the built payload when client_computed_stats is true Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

…nation tests Replace the two previous tests with a shared helper and four focused tests covering every combination of compute_trace_stats_on_extension and client_computed_stats, verifying both the _dd.compute_stats tag value and whether the extension generates stats. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

…ace.aws.lambda.hits PR #1118 moved `_dd.compute_stats` from each span's `meta` (applied via `get_tags_map()` / `ChunkProcessor`) to the payload-level `tracer_payload.tags` only. The Datadog APM backend reads this signal from the root span's `meta` field, not from the payload-level tags, so it stopped computing stats and `trace.aws.lambda.hits` disappeared from the trace explorer. Fix: propagate `_dd.compute_stats` to every span's `meta` in the same loop that writes it to `tracer_payload.tags`, restoring the pre-#1118 behavior while keeping the payload-level tag as well. Also add debug logging to `handle_traces` to surface the header tags on every incoming trace request, which aids diagnosing similar issues. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

…om payload tags - Add `compute_stats_value` field to `ChunkProcessor` so the tag is set on each span's meta alongside the other function tags, instead of in a separate post-processing loop. - Remove `_dd.compute_stats` from `tracer_payload.tags`: it was never set there before PR #1118 and is not needed — stats work via the span meta path alone. - Compute and log the value once at the top of `process_traces`, before constructing `ChunkProcessor`. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

…ts header (#1118)" This reverts commit 55659d9.

…1176) ## Summary - Reverts #1118 (`feat(traces): [SVLS-8734] respect Datadog-Client-Computed-Stats header`) which introduced logic to skip backend stats when the tracer claims client-computed-stats, causing stats to vanish entirely in some scenarios. - Reverts #1136 (`fix(tests): remove stale _dd.compute_stats:1 assertion from logs integration test`) which was a follow-up test adjustment for #1118. Restores the pre-v95 behavior where `_dd.compute_stats` is set solely by `DD_COMPUTE_TRACE_STATS_ON_EXTENSION`, keeping the backend as a safety net. ## Test plan - [x] `cargo fmt --check` passes - [x] `cargo clippy --all-targets -- -D warnings` passes - [x] `cargo test` — all 513 tests pass [SVLS-8734]: https://datadoghq.atlassian.net/browse/SVLS-8734?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

…1176) - Reverts #1118 (`feat(traces): [SVLS-8734] respect Datadog-Client-Computed-Stats header`) which introduced logic to skip backend stats when the tracer claims client-computed-stats, causing stats to vanish entirely in some scenarios. - Reverts #1136 (`fix(tests): remove stale _dd.compute_stats:1 assertion from logs integration test`) which was a follow-up test adjustment for Restores the pre-v95 behavior where `_dd.compute_stats` is set solely by `DD_COMPUTE_TRACE_STATS_ON_EXTENSION`, keeping the backend as a safety net. - [x] `cargo fmt --check` passes - [x] `cargo clippy --all-targets -- -D warnings` passes - [x] `cargo test` — all 513 tests pass [SVLS-8734]: https://datadoghq.atlassian.net/browse/SVLS-8734?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

…tead of baking it Move _dd.compute_stats from a baked-in function tag to a per-span backend directive stamped by the trace processor, and respect the tracer's Datadog-Client-Computed-Stats signal. - tags.rs: stop inserting _dd.compute_stats in tags_from_env (it no longer leaks into _dd.tags.function); make COMPUTE_STATS_KEY pub(crate). - trace_processor.rs: add client_computed_stats to ChunkProcessor; stamp _dd.compute_stats="1" on each span's meta only when neither the extension (compute_trace_stats_on_extension) nor the tracer (client_computed_stats) computes stats, matching the Go agent (set "1" or leave absent). Add && !client_computed_stats to the extension-side stats-generation guard. - Tests: truth-table on Span.meta (#1118 guard), stats-skip guard via the real StatsConcentratorService, updated tags.rs unit tests, and removed the stale _dd.compute_stats assertions from the logs/metrics integration tests. All 548 workspace tests pass; fmt + clippy clean.

## Overview Makes the extension respect the tracer's `Datadog-Client-Computed-Stats` header and moves `_dd.compute_stats` from a baked-in function tag to a per-span backend directive. We tried supporting the `Datadog-Client-Computed-Stats` header before in #1118, but that was reverted in #1176. ### Background Span attribute `_dd.compute_stats` asks the backend to compute trace stats. It must be set to `"1"` only when nobody else computed them — neither the extension (agent) nor the tracer. Previously the extension: 1. **Baked `_dd.compute_stats` into the function tags** unconditionally (`tags_from_env`), which also leaked the key into `_dd.tags.function`. 2. **Ignored `Datadog-Client-Computed-Stats`** entirely, so when a tracer computed stats client-side, the backend was still asked to compute them. ### Canonical semantics (validated against the Go agent) The Go agent (`pkg/serverless/tags/tags.go`) only ever sets `_dd.compute_stats = "1"`, never `"0"`, and leaves the key absent otherwise. This PR matches that: > Set `_dd.compute_stats="1"` iff `!compute_trace_stats_on_extension && !client_computed_stats`; otherwise leave it absent. | `client_computed_stats` (header from tracer) | `compute_on_extension` (extension config) | who computes | stamp `_dd.compute_stats="1"`? | -------------------- | --------------------- | --------------- | --- | true | (ignored) | tracer | ❌ no | false | true | extension | ❌ no | false | false | backend | ✅ yes ### Changes - **`tags/lambda/tags.rs`** — stop baking `_dd.compute_stats` in `tags_from_env` (no longer leaks into `_dd.tags.function`); `COMPUTE_STATS_KEY` is now `pub` so the integration test can reuse it instead of re-declaring the literal. - **Path A: `traces/trace_processor.rs`** — `ChunkProcessor` gains `client_computed_stats` and stamps `_dd.compute_stats="1"` per-span only when neither side computes stats; the extension-side stats-generation guard in `send_processed_traces` now also skips when `client_computed_stats` is set. - **Path B (extension-generated `aws.lambda` span)** — `client_computed_stats` is propagated from the tracer's placeholder span through `context.rs` → `processor.rs` → `processor_service.rs` → `trace_agent.rs`, so Path B reuses the same `ChunkProcessor` stamping (single source of truth). - **OTLP: `otlp/agent.rs`** — the OTLP stats-generation guard previously checked only `compute_trace_stats_on_extension`, so an OTLP request carrying `Datadog-Client-Computed-Stats` would still generate extension-side stats and double-count against the tracer. It now also skips when `client_computed_stats` is set, mirroring the `send_processed_traces` guard. - **Single source of truth: `traces/trace_processor.rs`** — the three decisions over the same two inputs (the per-span `_dd.compute_stats` stamp, plus the extension-side stats-generation guards in `send_processed_traces` and `otlp/agent.rs`) are now derived from one `StatsComputedBy::resolve(compute_on_extension, client_computed_stats)` helper, so the stamp and the guards can't silently drift apart. ### Note on the header value (cross-runtime) `Datadog-Client-Computed-Stats` is not standardized (`"true"` .NET/Java/PHP/Python, `"yes"` JS/Ruby/C++, `"t"` Go). bottlecap consumes the already-parsed `client_computed_stats` bool from `libdd_trace_utils`, where any non-empty value → `true`, so the fix triggers on every runtime. A separate libdatadog PR ([DataDog/libdatadog#2071](DataDog/libdatadog#2071)) aligns the header parsing with the Go agent's `isHeaderTrue`/`ParseBool` rules; this PR only consumes the bool and does not depend on that change. ## Testing - ~**Tier 0** (header-parsing contract)~ — moved to the libdatadog bump PR #1244, since those tests assert libdatadog parsing behavior that the `db05e1f → 48da0d8` bump changes. This branch keeps Tiers 1–3 and rebases onto [#1244](//pull/1244) after it merges. - **Tier 1** (`trace_processor.rs`) — truth-table on `Span.meta`, stats-skip guard via the real `StatsConcentratorService`, and updated `tags.rs` unit tests asserting the key no longer appears in the tags map. Fixed the logs/metrics integration tests that asserted the old leak. - **Tier 2** (`context.rs`, `processor.rs`) — context-level flag recording and an end-to-end Path B test driving `send_ctx_spans` through the `trace_tx` channel, asserting `_dd.compute_stats` on the `aws.lambda` span across the truth table. - **Tier 3** (`apm_integration_test.rs`) — full fake-intake E2E routing a trace through `SendingTraceProcessor::send_processed_traces`: asserts on the captured `AgentPayload` span meta and on `stats_payloads()` (stats suppressed unless the extension computes and the tracer did not). ### ⚠️ TODO before merging - [x] Rebase onto the libdatadog bump PR ([#1244](#1244)) after it merges. - [ ] E2E tests. --- > *"Computing stats twice doesn't make them twice as true, it just makes the backend twice as grumpy."* — Claude 🤖 [APMSVLS-487](https://datadoghq.atlassian.net/browse/APMSVLS-487) [APMSVLS-487]: https://datadoghq.atlassian.net/browse/APMSVLS-487?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

lym953 changed the title ~~feat(traces): respect Datadog-Client-Computed-Stats header~~ feat(traces): [SVLS-8734] respect Datadog-Client-Computed-Stats header Mar 20, 2026

lym953 marked this pull request as ready for review March 20, 2026 16:44

lym953 requested a review from a team as a code owner March 20, 2026 16:44

lym953 requested review from Copilot and shreyamalpani March 20, 2026 16:44

Copilot started reviewing on behalf of lym953 March 20, 2026 16:45 View session

Copilot AI reviewed Mar 20, 2026

View reviewed changes

purple4reina approved these changes Mar 20, 2026

View reviewed changes

lym953 marked this pull request as draft March 20, 2026 17:01

lym953 marked this pull request as ready for review March 23, 2026 18:47

lym953 and others added 6 commits March 23, 2026 16:15

fmt

2ca26df

fix(lint): add clippy::unwrap_used allow to new async tests

e7682fa

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

Add a test

b1abd8a

lym953 force-pushed the yiming.luo/ignore-tracer-stats branch from 5a248ff to b1abd8a Compare March 23, 2026 20:15

litianningdatadog reviewed Mar 23, 2026

View reviewed changes

lym953 marked this pull request as draft March 24, 2026 15:59

lym953 and others added 4 commits March 24, 2026 12:02

Modify comments

79ab0bf

docs(traces): add combination table to check_compute_stats_behavior doc

1e20726

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

lym953 marked this pull request as ready for review March 24, 2026 16:25

lym953 mentioned this pull request Mar 24, 2026

feat(otlp): add gRPC protocol support for trace ingestion #1105

Merged

4 tasks

lucaspimentel reviewed Mar 25, 2026

View reviewed changes

Comment thread bottlecap/src/lifecycle/invocation/context.rs Outdated

lym953 added 2 commits March 25, 2026 13:58

Modify comment for client_computed_stats in Context

f64fd5c

Fix clippy and fmt

c52d188

duncanista approved these changes Mar 26, 2026

View reviewed changes

lym953 merged commit 55659d9 into main Mar 26, 2026
48 of 50 checks passed

lym953 deleted the yiming.luo/ignore-tracer-stats branch March 26, 2026 20:12

duncanista mentioned this pull request Apr 7, 2026

fix(traces): keep backend as safety net for trace stats #1171

Closed

3 tasks

lym953 mentioned this pull request Apr 7, 2026

[SVLS-8734] fix(traces): restore _dd.compute_stats to span meta to fix missing trace stats #1172

Closed

duncanista added a commit that referenced this pull request Apr 8, 2026

Revert "feat(traces): [SVLS-8734] respect Datadog-Client-Computed-Sta…

328cc90

…ts header (#1118)" This reverts commit 55659d9.

duncanista mentioned this pull request Apr 8, 2026

revert(traces): revert Datadog-Client-Computed-Stats header support #1176

Merged

3 tasks

lucaspimentel mentioned this pull request Jun 4, 2026

[APMSVLS-487] respect Datadog-Client-Computed-Stats header #1245

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(traces): [SVLS-8734] respect Datadog-Client-Computed-Stats header#1118

feat(traces): [SVLS-8734] respect Datadog-Client-Computed-Stats header#1118
lym953 merged 13 commits into
mainfrom
yiming.luo/ignore-tracer-stats

lym953 commented Mar 20, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 20, 2026

Uh oh!

litianningdatadog Mar 23, 2026 •

edited

Loading

Uh oh!

lym953 Mar 24, 2026

Uh oh!

litianningdatadog left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

lym953 commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

Summary

Details

Test plan

Automated tests

Manual tests

Steps

Result

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 20, 2026

Choose a reason for hiding this comment

Uh oh!

litianningdatadog Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lym953 Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

litianningdatadog left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

lym953 commented Mar 20, 2026 •

edited

Loading

litianningdatadog Mar 23, 2026 •

edited

Loading