Comparing changes

…_ENABLED (#928) https://datadoghq.atlassian.net/browse/SVLS-7818 ## Overview Add DD_LOGS_ENABLED environment variable and YAML config option as an alias for DD_SERVERLESS_LOGS_ENABLED. Both variables now use OR logic, meaning logs are enabled if either variable is set to true. Changes: - Add logs_enabled field to EnvConfig and YamlConfig structs - Implement OR logic in merge_config functions: logs are enabled if either DD_LOGS_ENABLED or DD_SERVERLESS_LOGS_ENABLED is true - Add comprehensive test coverage with 9 test cases covering all combinations of the two variables - Maintain backward compatibility with existing configurations - Default value remains true when neither variable is set ## Testing Set DD_LOGS_ENABLED and DD_SERVERLESS_LOGS_ENABLED to false and expect: - [Log can be found in AWS console](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/$252Faws$252Flambda$252Fltn-fullinstrument-bn-cold-node22-lambda/log-events/2025$252F11$252F13$252F$255B$2524LATEST$255D455478dcbc944055b5be933e2e099f6a$3FfilterPattern$3DREPORT+RequestId) - [Log could NOT be found in DD console](https://ddserverless.datadoghq.com/logs?query=source%3Alambda%20%40lambda.arn%3A%22arn%3Aaws%3Alambda%3Aus-east-1%3A425362996713%3Afunction%3Altn-fullinstrument-bn-cold-node22-lambda%22%20AND%20%22REPORT%20RequestId%22&agg_m=count&agg_m_source=base&agg_t=count&clustering_pattern_field_path=message&cols=host%2Cservice%2C%40lambda.request_id&fromUser=true&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=desc&viz=stream&from_ts=1763063694206&to_ts=1763065424700&live=false) Otherwise the log should be available in DD console.

@nhulston

Now we support proxying requests from `/debugger/v1/input`. This PR adds `/debugger/v2/input` and `/debugger/v1/diagnostics`. #925 ## Test ### Steps 1. Set up Exception Replay following https://docs.datadoghq.com/error_tracking/backend/exception_replay/ 2. Build a test layer and install it on the Lambda 3. Set `DD_TRACE_DEBUG` to `true` 4. Change Lambda timeout from 3s to 30s. Seems Exception Replay dramatically increases the duration of the first invocation. It took 8–9s for my tests. ### Result Before: - In CloudWatch logs, see the error multiple times: `debugger::unsupported_agentUnsupported Datadog agent detected. Snapshots from Dynamic Instrumentation/Exception Replay/Code Origin for Spans will not be uploaded. Please upgrade to version 7.49.0 or later` After: - No such error. - See tracer debug log: `Detected /debugger/v2/input endpoint` - See the error on Error Tracking page <img width="486" height="166" alt="image" src="https://github.com/user-attachments/assets/3d48cb2b-0d5a-4806-97e1-e8146642ce64" /> ## Notes Thanks @nhulston @joeyzhao2018 @purple4reina for discussion and helping debug.

Upgrade libdatadog. Including: - Rename a few creates: - `ddcommon` -> `libdd-common` - `datadog-trace-protobuf` -> `libdd-trace-protobuf` - `datadog-trace-utils` -> `libdd-trace-utils` - `datadog-trace-normalization` -> `libdd-trace-normalization` - `datadog-trace-stats` -> `libdd-trace-stats` - Use the new API to send traces, which takes in an http_client object instead of proxy url string GitHub issue: #860 Jira: https://datadoghq.atlassian.net/browse/SLES-2499 Slack discussion: https://dd.slack.com/archives/C01TCF143GB/p1762526199549409

## Overview Support sending trace stats to Datadog when the Lambda is in a VPC with a proxy. ## Testing ### Steps 1. Set up a Lambda in a VPC with a proxy. See [Using an agent proxy](https://datadoghq.atlassian.net/wiki/spaces/SLS/pages/4168745841/Using+an+Agent+Proxy) (internal) 2. Build an extension layer and install it on the Lambda 3. Invoke the Lambda. After 20 seconds, invoke it again. This should force trace stats for the first invocation to be flushed. ### Result **Before:** stats flush failed with error: ``` Error sending stats: Failed to send trace stats: client error (Connect) ``` <img width="671" height="55" alt="image" src="https://github.com/user-attachments/assets/e815d296-304c-4832-90fe-44a5a3981d73" /> <img width="905" height="55" alt="image" src="https://github.com/user-attachments/assets/32fe9cc2-43ed-41b7-b53d-89750c2122be" /> **After:** stats flush was successful. <img width="656" height="56" alt="image" src="https://github.com/user-attachments/assets/e2b54b8b-3b0b-41e6-9c11-673bd7f2f894" /> <img width="885" height="52" alt="image" src="https://github.com/user-attachments/assets/3dc4c0b5-6681-4f2c-926c-edb278359943" /> ## Dependency DataDog/libdatadog#1345 ## Notes GitHub issue: #860 Jira: https://datadoghq.atlassian.net/browse/SLES-2499

@astuyve

## TL;DR - Support deduping spans by span_id - which fixes trace stats overcount issue for Node.js ## Problem Node.js tracer sometimes sends duplicate traces to the extension, causing over-count of trace stats. In our tests, the over-count is usually 1–4 for 50, 500 or 5000 invocations. This only happens when the "default" flushing strategy (which uses the "continuous" strategy) is used, and doesn't happen when the "end" strategy is used. ## Cause of problem I think it's similar to a known problem of continuous flushing. - The known problem is, when Lambda runtime is frozen, the connection between **extension** and **DD endpoint** can time out, causing data flush failure. - In the trace stats case, the problem is, when Lambda runtime is frozen, the connection between **tracer** and **extension** can time out. Extension receives the request from tracer, then freezes before sending a response, causing the tracer's request to time out, which makes the tracer resend the trace. This doesn't happen with the END flush strategy because in that case, after the Lambda handler finishes, extension still needs to flush the data and doesn't freeze so fast, and it has enough time to respond to tracer. ## Testing ### Steps - Build a test extension layer - Run e2e tests, including: - Install it on various Lambda functions - Invoke these functions with various traffic pattern - Check the trace stats result ### Result **Before:** - Over-count happens for (1) Node.js runtime + (2) "Sampling" test, which uses the default flush strategy - (vs expected 50) <img width="1336" height="277" alt="Screenshot 2025-11-20 at 9 30 00 PM" src="https://github.com/user-attachments/assets/3bf33c26-8996-43e4-8df5-7313247926b3" /> **After:** - No over-count (vs expected 5000) <img width="1447" height="421" alt="Screenshot 2025-11-20 at 9 25 32 PM" src="https://github.com/user-attachments/assets/21576483-dbdc-499f-acc8-1801d3927b0b" /> ## Options considered 1. **At most once**: Disable retry in tracer, at least for Lambda. 2. **At least once** - 2.1 Do nothing. Call out this as a known limitation. - 2.2 Treat tracer as VIP. Before calling /next, make sure tracer's requests have been responded. This may cause regression on invocation duration, especially when volume is high. 3. **Exactly once**: Implement dedup in extension, by trace_id. This PR chooses 3 because it's the easiest. ## Notes Thanks @astuyve @rochdev @purple4reina for discussion. Thanks Cursor for writing most of the code. The under-count issue will be addressed separately. Related issue: #688

## Problem The `bottlecap` GitLab job usually takes < 10 minutes, but it sometimes gets stuck and times out at 1 hour. This failure signal causes confusion in PRs and blocks PR merge. As a result, engineers often need to rerun this job. ## This PR For this job, add: 1. timeout of 10 minutes 2. a retry in case of `stuck_or_timeout_failure` or `runner_system_failure` ## Testing Will merge and see if these errors become less frequent.

https://datadoghq.atlassian.net/browse/SVLS-8080 ## Overview Merge Lambda Managed Instance feature branch ## Testing Covered by individual commits Co-authored-by: shreyamalpani <[email protected]> Co-authored-by: duncanista <[email protected]> Co-authored-by: astuyve <[email protected]> Co-authored-by: jchrostek-dd <[email protected]> Co-authored-by: tianning.li <[email protected]>

https://datadoghq.atlassian.net/browse/SVLS-8080 ## Overview v90 release prep ## Testing n/a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Comparing changes

Open a pull request

Commits on Nov 24, 2025

Commits on Nov 25, 2025

Commits on Dec 1, 2025

This comparison is taking too long to generate.

Uh oh!