SLES 2889 | Fixing payload size#1257
Merged
Merged
Conversation
process_traces captured body_size from the raw msgpack request at ingress and passed it through unchanged. But every span is then enriched with the tags-provider map (function_arn, region, env, ...), so the protobuf payload actually sent is substantially larger than that ingress figure. Both size caps gate on this value: the TraceAggregator's per-batch limit and libdatadog's coalesce limit. Sizing by the smaller ingress number lets the extension bundle and send payloads that exceed the backend's limit, which rejects them with 413 Payload Too Large (and retries that can never succeed). Recompute body_size from the protobuf encoded_len of the payload that will actually be sent. This previously happened only when compute_trace_stats_on_extension was enabled; it now runs unconditionally so the default configuration is correct too.
The TraceAggregator's per-batch cap was 3.2 MB. Combined with the enriched-protobuf sizing fix, a single trace whose enriched payload sits between 3.2 MB and the trace intake's ~15 MB limit could neither be split within one SendData nor flushed: get_batch() put it back on the queue indefinitely, stalling every trace behind it (silently dropped traces and apparent timeouts). Raise MAX_CONTENT_SIZE_BYTES to 12 MB so such a payload flushes in a single batch while staying under the ~15 MB intake limit. Add debug logging of the per-batch total and the post-enrichment payload size for diagnosability.
Deploys a Node Lambda that produces a single ~10 MB trace from real instrumented spans (a payload-logging service capturing sizeable order documents on a few hundred spans) and asserts the trace reaches Datadog with its root span and >100 child spans. Reads the extension's post-enrichment payload-size debug line from CloudWatch and asserts it lands in (3.2 MB, 12 MB) -- above the old cap (so it would have stalled before the fix) and below the new cap and the backend limit. Registered in bin/app.ts and the test-suites datasource.
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses 413 “Payload Too Large” failures for large single-invocation traces by ensuring outbound trace batching uses the post-enrichment, protobuf-encoded payload size (the size actually sent), and by increasing the trace batch size cap to better align with trace intake limits. It also adds unit + integration coverage to prevent regressions and to validate end-to-end behavior for ~10MB traces.
Changes:
- Compute
body_sizefrom the enriched protobuf payload size for batching accuracy (not just when on-extension stats are enabled). - Increase trace aggregator max batch size to 12 MB and add debug logs for enriched and batched sizes.
- Add a new integration test suite that generates and validates delivery of a large single-invocation trace.
Reviewed changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| integration-tests/tests/utils/cloudwatch.ts | Adds CloudWatch Logs helper for filtering extension/Lambda log lines (region/pagination handling needs fixes). |
| integration-tests/tests/payload-size.test.ts | New integration test asserting large enriched + batched payloads flush without 413 and are delivered intact. |
| integration-tests/lib/stacks/payload-size.ts | New CDK stack deploying a Node Lambda that emits a large trace with debug logging enabled. |
| integration-tests/lambda/large-trace-node/index.js | Node Lambda code to generate a large single-invocation trace by attaching large tag payloads across many spans. |
| integration-tests/bin/app.ts | Registers the new PayloadSize stack in the integration test app. |
| bottlecap/src/traces/trace_processor.rs | Fixes body_size accounting to reflect enriched protobuf payload size; adds debug log; adds unit test. |
| bottlecap/src/traces/trace_aggregator.rs | Raises max trace batch cap to 12MB; adds debug log for per-batch coalesced size. |
| .gitlab/datasources/test-suites.yaml | Adds payload-size suite to CI test suite list. |
Comment on lines
+1
to
+6
| import { | ||
| CloudWatchLogsClient, | ||
| FilterLogEventsCommand, | ||
| } from '@aws-sdk/client-cloudwatch-logs'; | ||
|
|
||
| const logsClient = new CloudWatchLogsClient({ region: 'us-east-1' }); |
Comment on lines
+21
to
+40
| const messages: string[] = []; | ||
| let nextToken: string | undefined; | ||
|
|
||
| do { | ||
| const response = await logsClient.send( | ||
| new FilterLogEventsCommand({ | ||
| logGroupName, | ||
| filterPattern, | ||
| startTime, | ||
| endTime, | ||
| nextToken, | ||
| }), | ||
| ); | ||
| for (const event of response.events ?? []) { | ||
| if (event.message) { | ||
| messages.push(event.message); | ||
| } | ||
| } | ||
| nextToken = response.nextToken; | ||
| } while (nextToken); |
duncanista
reviewed
Jun 15, 2026
Comment on lines
+134
to
+141
| if !self.buffer.is_empty() { | ||
| debug!( | ||
| "TRACES | batched {} payload(s) totaling {batch_size} bytes (cap {} bytes)", | ||
| self.buffer.len(), | ||
| self.max_content_size_bytes | ||
| ); | ||
| } | ||
|
|
Contributor
There was a problem hiding this comment.
Wonder if libdatadog has something similar?
Contributor
Author
There was a problem hiding this comment.
I don't think they do. If they did, would still prefer for it to be here as well since we use this log message for our integration tests to ensure we are actually testing against a large payload.
duncanista
approved these changes
Jun 17, 2026
lucaspimentel
approved these changes
Jun 17, 2026
lucaspimentel
left a comment
lucaspimentel
left a comment
Member
There was a problem hiding this comment.
LGTM aside from Copilot's comment about the hard-coded us-east-1 which seems legit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://datadoghq.atlassian.net/browse/SLES-2889
Overview
Large single-invocation traces could be rejected by the trace intake with a 413, because the extension undercounted how big a payload would be once it left the extension.
The extension enriches every span after the tracer posts it (adding the tags-provider map), so the protobuf payload it sends is larger than the raw msgpack it received. The trace aggregator's batch cap therefore undercounted the true outbound bytes and packed too much into a single request, whose combined size exceeded the intake limit → 413.
The batch cap was below realistic enriched payload sizes. The cap was set to the old 3.2 MB documented limit. However, from trace intake config, the limit is actually 15.
This PR:
body_sizefrom the enriched, protobuf-encoded payload unconditionally (not only when on-extension stats are enabled), so the aggregator's batch cap reflects exactly what will be sent.Testing
body_sizeequals the protobufencoded_lenof the payload actually sent, and exceeds the (smaller) ingress size — on the default config (no on-extension stats).payload-sizesuite deploys a Node Lambda that emits a ~10 MB single-invocation trace (400 spans carrying large captured payloads) and verifies it: enriches to >10 MB, flushes in a single batch >10 MB, arrives with no 413, and is delivered intact (one trace, root span present, all 400 spans). Run against a locally published extension layer in the serverless sandbox.