chore: merge main by jaronoff97 · Pull Request #18 · usetero/datadog-lambda-extension

jaronoff97 · 2026-06-30T21:39:31Z

Please include Jira ticket in title.

Overview

Testing

…n logger (DataDog#1042) ## Summary Root cause: commit 3b533a0 introduced `serde_json::json!()` for JSON formatting, adding per-call heap pressure and a new crate dependency that contributes to Lambda init duration regression JIRA: https://datadoghq.atlassian.net/browse/SVLS-8619 - Replaces `serde_json::json!()` in `logger.rs` with direct `write!`/`writeln!` calls to the tracing `Writer`, eliminating two heap allocations (a `Map` + `Value`) per log call - Adds a `write_json_escaped` helper that handles all 6 mandatory JSON escape sequences inline (`"`, `\`, `\n`, `\r`, `\t`, U+0000–U+001F) - Removes the `"json"` feature from `tracing-subscriber`, dropping `tracing-serde` as a transitive dependency and reducing binary size - The JSON output format is **identical** — no observable behavior change ## Test plan - unit tests - self-monitoring tests

DataDog#1043) https://datadoghq.atlassian.net/browse/SVLS-8619 ## Summary - Bumps `opentelemetry-proto` from `0.29` to `0.30.0` - Bumps `tonic-types` from `0.12` to `0.13` - Removes stale `LICENSE-3rdparty.csv` entries for `glob`, `indexmap` (bluss repo), and the now-absent duplicate `indexmap` entry ## Why `opentelemetry-proto 0.29` required `tonic ^0.12`, which forced a second copy of the full tonic gRPC stack into the Lambda binary alongside the `tonic 0.13` already pulled in by `dogstatsd`/`datadog-protos` (saluki, introduced in `674cd358`). This duplicate adds binary size and contributes to Lambda init duration regression (SVLS-8619). `opentelemetry-proto 0.30.0` requires `tonic ^0.13` and `prost ^0.13`, aligning with what saluki already brings in. No external dependency changes (saluki, libdatadog) are required for this fix. ## Result - `tonic 0.12` removed from binary — single `tonic 0.13` copy remaining - `Cargo.lock` shrinks by ~58 lines (removed crates: `glob`, one `indexmap` duplicate, `tonic 0.12` stack) - No API breakage — `opentelemetry 0.29 → 0.30` is a single minor bump with no changes to the APIs used in `bottlecap/src/otlp/` ## Test plan - unit tests passed - Self-monitoring tests

## Overview Regression on performance because `[email protected]` checks for multiple folders as opposed to only one ## Testing Manual testing in AWS Lambda and Self Monitoring Data

…ds (DataDog#1044) https://datadoghq.atlassian.net/browse/SVLS-8609 Issue reported: DataDog#1041 ## Summary - axum 0.7+ applies a 2 MB default body limit globally; the `/lambda/start-invocation` endpoint was rejecting payloads larger than 2 MB with `length limit exceeded` - Raises the limit to 6 MB (`DefaultBodyLimit::max(6 * 1024 * 1024)`) to match [Lambda's maximum synchronous invocation payload size](https://docs.aws.amazon.com/lambda/latest/api/API_Invoke.html) - Adds three unit tests covering: accept above the old 2 MB default, accept at boundary (6 MB − 1 byte), reject above 6 MB (413) ## Test plan - Unit tests - Run `bash local_tests/repro-large-payload.sh` which sets up the local test env and sends a 3 MB payload to `/lambda/start-invocation` and verifies HTTP 200 with no `length limit exceeded` error | Scenario|Log| |--------|--------| | Without the fix|```{"level":"ERROR","message":"DD_EXTENSION \| ERROR \| Failed to extract request body: Failed to buffer the request body: length limit exceeded"}``` | | With the fix| ```{"level":"DEBUG","message":"DD_EXTENSION \| DEBUG \| Received start invocation request from headers:{\"datadog-meta-lang\": \"java\", \"user-agent\": \"curl/8.3.0\", \"host\": \"localhost:8124\", \"lambda-runtime-aws-request-id\": \"test-large-payload-request\", \"expect\": \"100-continue\", \"accept\": \"*/*\", \"content-length\": \"3200074\", \"content-type\": \"application/json\"}"}``` |

…ARN (DataDog#1046) JIRA: https://datadoghq.atlassian.net/browse/SLES-2729 Issue: DataDog#1000 **Overview** When AWS Lambda freezes or terminates a function instance, the OS immediately closes all TCP sockets. The extension's proxy HTTP handler (handle_proxy) was mid-read on the request body — this read fails with error reading a body from connection and was logged at ERROR level, generating noise with no actionable signal. This is expected Lambda lifecycle behaviour: the Lambda function completed successfully and nothing in the extension is broken. The ERROR level implies an extension fault, which misleads operators. **Change** Introduce a warn_response helper (mirroring the existing error_response / success_response trio) and use it for body-read failures in handle_proxy. The log message content is unchanged — only the severity is downgraded from ERROR to WARN. Testing - unit tests

…ataDog#1048) ## Overview When the Lambda function's runtime version contains "RuntimeVersion", add the tag `durable_function:true` to enhanced metrics generated by the extension, so the facet `durable_function` appears on Serverless View. Sample value of runtime version: - Python: `python:3.14.DurableFunction.v9` - Node.js: `nodejs:24.DurableFunction.v10` ## Testing ### Steps 1. Build a layer and install it on a durable function 2. Invoke it 3. Check Serverless view ### Result 1. When I search by `durable_function:true`, the function appears in the result. <img width="446" height="210" alt="image" src="https://github.com/user-attachments/assets/2fa14641-fd37-4b27-90bb-85cfce2570e3" /> 2. A facet `durable_function` appears on the sidebar. <img width="437" height="151" alt="image" src="https://github.com/user-attachments/assets/2dcc4113-5caa-4376-a92e-ed54b29dd453" /> 3. When I search by `-durable_function:true`, other functions appear in the result. <img width="500" height="240" alt="image" src="https://github.com/user-attachments/assets/5b1308ff-5a7d-4c9f-9593-93dfb26d625d" /> --------- Co-authored-by: Claude Sonnet 4.6 <[email protected]>

…1055) ## Overview Fixes an issue where AWS filters out by logs with "level" in a structured JSON. So any DD_LOG_LEVEL logs won't show up in CWL, but they will be sent to Datadog correctly. This changes it to "status" so they can be displayed in CWL and they can also arrive to Datadog. ## Testing Tested manually in AWS Lambda.

…ataDog#1056)

## Summary - Delete `compile_go_agent.sh` and Go agent Dockerfiles (`Dockerfile.go_agent.compile`, `Dockerfile.go_agent.alpine.compile`) which are no longer needed since the extension was rewritten in Rust (bottlecap) - Remove the now-orphaned `AGENT_BRANCH` and `AGENT_VERSION` pipeline input variables from `.gitlab-ci.yml` ## Jira [SVLS-8636](https://datadoghq.atlassian.net/browse/SVLS-8636) [SVLS-8636]: https://datadoghq.atlassian.net/browse/SVLS-8636?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

## Overview Upgrades dogstatsd which allows us to use our own reqwest client instead of expecting the server to create one This gives us control over how we are building a specialized client which might need a proxy, certificates, and more. ## Testing Unit tests, integration tests, and self monitoring

fixing qemu flakiness.

## Overview The extension was constructing up to 6 separate HTTP clients across different subsystems (metrics, logs, trace proxy, trace flusher, stats flusher, and Lambda API). Three of the reqwest::Client instances (metrics, logs, trace proxy) were identically configured via get_client(config), and two HttpClient (hyper) instances (trace flusher, stats flusher) were identically configured via http_client::create_client(). Each independent client maintains its own connection pool and TLS sessions, which means: - Duplicate TLS handshakes to the same Datadog intake hosts - No connection reuse across subsystems hitting the same endpoints - Unnecessary memory overhead from redundant connection pools This change consolidates to 3 clients total: 1. reqwest::Client (no-proxy) — Lambda Extensions API (intentionally isolated, localhost only) 2. reqwest::Client (shared) — Metrics, Logs, Trace proxy 3. HttpClient (shared) — Trace flusher, Stats flusher Both reqwest::Client and HttpClient are Arc-based internally, so cloning is a refcount increment that shares the underlying connection pool. This also removes the lazy OnceCell initialization from TraceFlusher and StatsFlusher in favor of eager construction at startup. The lazy init was retrying on failure, but the only failure modes (bad proxy URL, unreadable cert file) are configuration errors that won't self-resolve between retries, so failing fast is preferable. ## Testing Unit tests, integration tests, and self monitoring

## Overview Add DD_SKIP_SSL_VALIDATION support, parsed from both env and YAML, matching the datadog-agent's behavior — applied to all outgoing HTTP clients (reqwest via danger_accept_invalid_certs, hyper via custom ServerCertVerifier). ## Motivation Customers in environments with corporate proxies or custom CA setups need the ability to disable TLS certificate validation, matching the existing datadog-agent config option. The Go agent applies tls.Config{InsecureSkipVerify: true} to all HTTP transports via a central CreateHTTPTransport() — we mirror this by wiring the config through to both client builders. And [SLES-2710](https://datadoghq.atlassian.net/browse/SLES-2710) ## Changes Config (config/mod.rs, config/env.rs, config/yaml.rs): - Add skip_ssl_validation: bool to Config, EnvConfig, and YamlConfig with default false reqwest client (http.rs): - .danger_accept_invalid_certs(config.skip_ssl_validation) on the shared client builder hyper client (traces/http_client.rs): - Custom NoVerifier implementing rustls::client::danger::ServerCertVerifier that accepts all certificates - Uses CryptoProvider::get_default() (not hardcoded aws_lc_rs) for FIPS-safe signature scheme reporting - New skip_ssl_validation parameter on create_client() ## Testing Unit tests and self monitoring [SLES-2710]: https://datadoghq.atlassian.net/browse/SLES-2710?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

…taDog#1024) ## Overview 1. The first thing [handle_end_invocation](https://github.com/DataDog/datadog-lambda-extension/blob/2c61aca453a3ab41df58d407e1659705cdcee273/bottlecap/src/lifecycle/listener.rs#L163) does is spawn a task, let's call it `anonymousTask` 2. `handle_end_invocation` then [immediately returns 200](https://github.com/DataDog/datadog-lambda-extension/blob/2c61aca453a3ab41df58d407e1659705cdcee273/bottlecap/src/lifecycle/listener.rs#L181) so the tracer can continue 3. `anonymousTask` is busy with a complex body in `extract_request_body` for the time being 4. Because the tracer has continued, eventually `PlatformRuntimeDone` [is processed](https://github.com/DataDog/datadog-lambda-extension/blob/2c61aca453a3ab41df58d407e1659705cdcee273/bottlecap/src/lifecycle/invocation/processor_service.rs#L481) 5. Given our customer is not managed (`initialization_type: SnapStart`) then `PlatformRuntimeDone` [tries to pair_platform_runtime_done_event](https://github.com/DataDog/datadog-lambda-extension/blob/2c61aca453a3ab41df58d407e1659705cdcee273/bottlecap/src/lifecycle/invocation/processor.rs#L496) which is `None` because `anonymousTask` is still busy with the body 6. We then [jump to process_on_platform_runtime_done](https://github.com/DataDog/datadog-lambda-extension/blob/2c61aca453a3ab41df58d407e1659705cdcee273/bottlecap/src/lifecycle/invocation/processor.rs#L506) 7. Span and trace ids [are not there yet](https://github.com/DataDog/datadog-lambda-extension/blob/2c61aca453a3ab41df58d407e1659705cdcee273/bottlecap/src/lifecycle/invocation/processor.rs#L517), and they are never checked again after this 8. `anonymousTask` finally completes, but that's irrelevant because `send_ctx_spans` is only run on `PlatformRuntimeDone` which assumes [universal_instrumentation_end has already been sent](https://github.com/DataDog/datadog-lambda-extension/blob/2c61aca453a3ab41df58d407e1659705cdcee273/bottlecap/src/lifecycle/listener.rs#L177) ## Why this looks likely In the customer's logs we can see * `05:11:48.463` `datadog.trace.agent.core.DDSpan - Finished span (WRITTEN): DDSpan [ t_id=2742542901019652192` * `05:11:48.489` PlatformRuntimeDone received * `05:11:48.630` REPORT RequestId `1db22159-7200-43c8-bec1-11b89df4f099` (last log emitted in an execution) * `05:11:53.784` START RequestId: `8c801767-e21b-43f7-bd11-078bb64bc430` (new request id, 5s later) * `05:11:53.789` [Received end invocation request from headers:{""x-datadog-trace-id"": ""2742542901019652192"...](https://github.com/DataDog/datadog-lambda-extension/blob/2c61aca453a3ab41df58d407e1659705cdcee273/bottlecap/src/lifecycle/listener.rs#L258) -> we are now trying to finish the span after the request is long gone 🙃 In this specific run, the lambda even had time to stop before continuin with the anonymous task from `handle_end_invocation`. ## Motivation [SLES-2666](https://datadoghq.atlassian.net/browse/SLES-2666) ## Performance This PR makes the reading of the body synchronous with the response. This will delay handing over execution to outside the extension until the body is read. But that is irrelevant because it is a requirement to read the body and send `universal_instrumentation_end` before relinquishing control. ## Testing Suggestions very welcome [SLES-2666]: https://datadoghq.atlassian.net/browse/SLES-2666?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: Jordan Gonzalez <[email protected]>

## Summary - Sign the image for ci image jobs per [instruction](https://datadoghq.atlassian.net/wiki/spaces/SECENG/pages/2744681107/Image+Integrity+User+Guide) Fixes: https://datadoghq.atlassian.net/browse/SVLS-8644 Co-authored-by: tianning.li <[email protected]>

…revent memory leak (DataDog#1050) JIRA: https://datadoghq.atlassian.net/browse/SVLS-8625 Git issue reported: DataDog#1049 ## What does this PR do? Fixes a memory leak in the extension where `ContextBuffer` accumulated `Context` objects indefinitely across warm Lambda invocations. **Root cause:** `on_platform_report()` reads the `Context` for runtime duration and CPU/network enhanced metrics, but never called `context_buffer.remove()` afterwards. Each warm invocation appended a new `Context` to the buffer; none were ever freed. **Fix:** Call `self.context_buffer.remove(request_id)` at the end of `on_platform_report()`, after all processing for the invocation is complete. **Impact:** Most visible when `DD_CAPTURE_LAMBDA_PAYLOAD=true` — each retained `Context` holds the full captured request/response payload as span metadata (~500 KB per invocation). Without the fix, a Lambda function invoked thousands of times would accumulate hundreds of MB of leaked memory in the extension process. ## Testing - Unit tests - A test with 50+ invocations shows | without the fix | with the fix | |--------|--------| | `buffer size after remove: 52`|`buffer size after remove: 1`| Co-authored-by: Claude Sonnet 4.6 <[email protected]>

…taDog#1090) Bumps [aquasecurity/trivy-action](https://github.com/aquasecurity/trivy-action) from 0.34.2 to 0.35.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/aquasecurity/trivy-action/releases">aquasecurity/trivy-action's releases</a>.</em></p> <blockquote> <h2>v0.35.0</h2> <h2>What's Changed</h2> <ul> <li>chore(deps): Update trivy to v0.69.3 by <a href="https://github.com/aqua-bot"><code>@aqua-bot</code></a> in <a href="https://redirect.github.com/aquasecurity/trivy-action/pull/519">aquasecurity/trivy-action#519</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/aquasecurity/trivy-action/compare/0.34.2...0.35.0">https://github.com/aquasecurity/trivy-action/compare/0.34.2...0.35.0</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/aquasecurity/trivy-action/commit/57a97c7e7821a5776cebc9bb87c984fa69cba8f1"><code>57a97c7</code></a> chore(deps): Update trivy to v0.69.3 (<a href="https://redirect.github.com/aquasecurity/trivy-action/issues/519">#519</a>)</li> <li>See full diff in <a href="https://github.com/aquasecurity/trivy-action/compare/97e0b3872f55f89b95b2f65b3dbab56962816478...57a97c7e7821a5776cebc9bb87c984fa69cba8f1">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=aquasecurity/trivy-action&package-manager=github_actions&previous-version=0.34.2&new-version=0.35.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…race_stats_on_extension is enabled (DataDog#1060) ## Summary - When `compute_trace_stats_on_extension` is `true`, traces with sampling priority `<= 0` (`AUTO_DROP` / `USER_DROP`) are now dropped and **not** flushed to the Datadog backend. - When `compute_trace_stats_on_extension` is `false` (the current default), traces are not dropped because all traces need to be sent to Datadog so trace stats can be computed. - Trace stats computation is unaffected: dropped traces are still processed through `process_traces` and their stats are forwarded to the stats concentrator. ## Test plan - Passed the added unit test ### Manual test #### Steps - Add logging for trace payload flushed, and for traces being dropped - Build a layer and install it on a Lambda function - Invoke the function 6 times #### Result Logs show that - some traces were dropped. - the `aws.lambda` span for some of the 6 invocations were not flushed to Datadog. 🤖 Partially generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <[email protected]>

…aDog#1067) Instead of passing the whole Request into the spawned task and calling extract_request_body (which consumes the request and early-returns on failure), the fix: 1. Splits the request upfront with request.into_parts() — headers are preserved regardless of body extraction outcome 2. Attempts body buffering via Bytes::from_request 3. Falls back to Bytes::new() on failure (e.g. oversized payload) with a warn! log 4. Always calls universal_instrumentation_end — trace context, span finalization, and status code extraction proceed with degraded (empty) payload rather than being silently dropped The FromRequest import was added to the module-level axum imports to support calling Bytes::from_request directly. --------- Co-authored-by: Claude Opus 4.6 <[email protected]>

https://datadoghq.atlassian.net/browse/SVLS-8626 ## Overview OTLP currently expects protobuf payload. Update to allow for protobuf and json. ## Testing Add integration tests which sends json payload. --------- Co-authored-by: Claude Opus 4.5 <[email protected]>

## This PR After dropping sampled-out traces, recompute `body_size`. This ensures the `TraceAggregator`'s 3.2 MB batch limit is evaluated against only the data that will actually be sent to the backend. Upgrades the version of some external crates because those are the versions used in libdatadog. ## Testing - Unit tests ## Note The recomputed size uses the protobuf encoded payload size, different from the original size, computed from: (from Claude) > - Datadog tracer path: src/traces/trace_agent.rs:506 — body_size is the first element of the tuple returned by get_traces_from_request_body() / get_v05_traces_from_request_body(), which is the raw HTTP request body byte count (msgpack-encoded bytes). > - OTLP path: src/otlp/agent.rs:167 — let body_size = size_of_val(&traces) where traces is a TracerPayloadCollection, giving only the stack size of the Rust enum (~32 bytes). The new approach be more accurate because it's computed at a later point for the size of protobuf data, which is what's finally sent to Datadog. By the way, the size doesn't need to be accurate. An approximate size is good enough. See: https://github.com/DataDog/libdatadog/blob/3f3efefb2ff45d7a5491b770480396d001b87631/libdd-trace-utils/src/send_data/mod.rs#L148 Also here are some sample logs (logging node not included in the PR) showing the different sizes computed in different ways: > body_size (original): 582, body_size (encoded_len): 1355 > body_size (original): 373, body_size (encoded_len): 1269 > body_size (original): 336, body_size (encoded_len): 899 --------- Co-authored-by: Claude Sonnet 4.6 <[email protected]>

… Lambda (DataDog#1094) ## Summary Fixes a regression introduced in v93 where customers see a sharp increase in "Max retries exceeded, returning request error" errors after upgrading from v92. - Disables HTTP connection pooling for the trace/stats flusher by setting `pool_max_idle_per_host(0)` - Prevents stale connections from being reused after Lambda freeze/resume cycles ## Problem PR DataDog#1018 introduced HTTP client caching for performance improvements. However, the cached client maintains a connection pool that doesn't work well with Lambda's freeze/resume execution model: 1. Lambda executes, HTTP client created with connection pool 2. Extension flushes traces, connections remain open in pool 3. Lambda **freezes** (paused between invocations - can be seconds to minutes) 4. Lambda **resumes**, cached client reuses stale connections 5. TCP errors → "Max retries exceeded" In v92, a new HTTP client was created per-flush, so there were never stale connections to reuse. ## Solution Disable connection pooling by setting `pool_max_idle_per_host(0)`. This ensures each request gets a fresh connection, avoiding stale connection issues while still benefiting from client caching (TLS session reuse, configuration reuse, etc.). This matches the pattern used in libdatadog's `new_client_periodic()` which explicitly disables pooling with the comment: > "This client does not keep connections because otherwise we would get a pipe closed every second connection because of low keep alive in the agent." ## Related - Fixes [SVLS-8672](https://datadoghq.atlassian.net/browse/SVLS-8672) - Fixes DataDog#1092 - Regression introduced in DataDog#1018 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.5 <[email protected]> Co-authored-by: jordan gonzález <[email protected]>

…vars (DataDog#1097) Allow passing custom tracer library versions via environment variables (`NODE_TRACER_LAYER_ARN`, `PYTHON_TRACER_LAYER_ARN`, `JAVA_TRACER_LAYER_ARN`, `DOTNET_TRACER_LAYER_ARN`). Current hardcoded values remain as defaults. No updates were made to actual extension code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.5 <[email protected]>

…DataDog#1095) Bumps [styfle/cancel-workflow-action](https://github.com/styfle/cancel-workflow-action) from 0.13.0 to 0.13.1. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/styfle/cancel-workflow-action/releases">styfle/cancel-workflow-action's releases</a>.</em></p> <blockquote> <h2>0.13.1</h2> <h3>Patches</h3> <ul> <li>Fix: update Node.js runtime from 20 to 24: <a href="https://redirect.github.com/styfle/cancel-workflow-action/issues/217">#217</a></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/styfle/cancel-workflow-action/commit/d07a454dad7609a92316b57b23c9ccfd4f59af66"><code>d07a454</code></a> 0.13.1</li> <li><a href="https://github.com/styfle/cancel-workflow-action/commit/2838e032676d58d98ce3e3161f323f8164018142"><code>2838e03</code></a> fix: update Node.js runtime from 20 to 24 (<a href="https://redirect.github.com/styfle/cancel-workflow-action/issues/217">#217</a>)</li> <li>See full diff in <a href="https://github.com/styfle/cancel-workflow-action/compare/0.13.0...0.13.1">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=styfle/cancel-workflow-action&package-manager=github_actions&previous-version=0.13.0&new-version=0.13.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…g#1240) ## What Log the `handle_traces` request-body-read failure at **DEBUG** instead of **ERROR**. Fixes DataDog#1232. ## Why This failure happens probably because the execution environment is frozen when tracer sends traces to extension, causing timeout for TCP connections, which is not actionable on extension side. Lowering the log level to reduce noise. ## Notes If any customer sees other issues such as missing traces, let us know and we will treat it as a separate issue. --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

DataDog#1224) ## Overview Introduce a team-shared coding-agent configuration so any agent (Claude Code, Codex, etc.) gets the same project-level guidance, while still allowing personal overrides. - `AGENTS.md` — agent-agnostic project guide (architecture, data flow, source-tree map, PR conventions). Reviewed against the codebase: corrected the `proxy/` and `metrics/` descriptions and replaced the simplistic data-flow diagram with one that lists the real event sources (Lambda Extension API polling, telemetry listener, lifecycle listener, DogStatsD, trace/OTLP agents). - `CLAUDE.md` — single-line entrypoint that loads both `@AGENTS.md` (shared) and `@CLAUDE_PERSONAL.md` (personal, untracked). - `.gitignore` — now ignores `/CLAUDE_PERSONAL.md` instead of the shared files. - `.claude/settings.json` — `PreToolUse` hook that runs `cd bottlecap && cargo clippy --workspace --all-targets --features default && cargo fmt --all -- --check` before any `git commit` Bash call. Failures exit 2 and block the commit. Timeout 60s; matches via `if: "Bash(git commit*)"` so chained commands won't bypass it. The settings split mirrors the CLAUDE.md/CLAUDE_PERSONAL.md pattern via Claude Code's built-in convention: `.claude/settings.json` is committed (team-wide), `.claude/settings.local.json` is gitignored (personal overrides). ## Testing The team will test this in daily work and iterate as needed.

…#1234) ## Overview Adds support for excluding specific auto-injected tags from customer DogStatsD metrics enrichment via a new `DD_CUSTOMER_METRICS_EXCLUDE_TAGS` environment variable. This lets users drop high-cardinality enrichment tags (e.g. `function_arn`, `region`) from custom metrics to reduce billing. ## Changes - New `customer_metrics_exclude_tags` config (comma-separated list of tag keys) parsed from `DD_CUSTOMER_METRICS_EXCLUDE_TAGS`, with default empty - `start_dogstatsd()` filters the enrichment tag list against the exclusion set when initializing the metrics aggregator - Config parsing + merging support in `bottlecap/src/config/env.rs` and `mod.rs` - Unit tests cover env-var parsing and default behavior - New integration suite (`tests/custom-metrics.test.ts`) deploys two Lambdas — one with no exclusions, one with `function_arn,region` excluded — invokes them, and asserts via the Datadog metrics API that the enrichment tags are present/absent as expected ## Testing - `cargo fmt`, `cargo check`, `cargo clippy --features default`, `cargo test --workspace` all pass locally - Integration test deployed to sandbox and validated against the live Datadog API --------- Co-authored-by: datadog-bits <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

remove mention of serverless-init

… dedup (DataDog#1241) ## Background From our knowledge (before this PR), here's the behavior when each runtime OOMs: - emits runtime-specific error message. This can happen on **Java**, **Node** (case 1 in the table below) and **.NET** - In `PlatformRuntimeDone` event, `error_type` is `Runtime.OutOfMemory`. This can happen on **Python** and **Ruby**. - In `PlatformReport` event, `max_memory_used == memory_size`. This can happen on **Python**, **Ruby**, **Node** and **Go**. To capture OOM for all these scenarios (except Node case 2, which was just called out in DataDog#1237) without double counting, right now the extension emits `aws.lambda.enhanced.out_of_memory` metric in these scenarios: - when we see runtime-specific error messages for Java, Node and .NET - when we see `Runtime.OutOfMemory` - when we see `max_memory_used == memory_size` for Go, i.e. only when runtime is `provided.al2`. We don't do this for other runtimes (Python, Ruby, Node) to avoid double counting. <img width="768" height="323" alt="image" src="https://github.com/user-attachments/assets/549a8820-6b86-462d-a857-0269d2990a02" /> ## Problem In issue DataDog#1237, a customer called out a new scenario: "Node (case 2)" in the table. The only evidence of OOM is `max_memory_used == memory_size`, and there is no runtime-specific log message. As a result, OOMs like this are not captured by the OOM enhanced metric. ## This PR - Regardless of runtime, use all the three ways to capture OOM. - In addition, dedup by request_id to avoid double counting. - Add one integration test per runtime (except for Node, which has 2 tests) ## Test plan Passed the added unit tests and integration tests. ## To reviewers Most of the code changes are for integration tests. ## Details (generated by Claude Code) Closes the gap surfaced in DataDog#1237: a Node.js Lambda that hit its memory limit (`Memory Size 192 MB / Max Memory Used 192 MB`, `Status: timeout`) did not emit `aws.lambda.enhanced.out_of_memory` because none of the three existing detection paths matched. - **Why the existing paths missed it.** V8 spent its budget in GC rather than declaring `JavaScript heap out of memory`, so the runtime log-line match never fired. The runtime crashed on a wall-clock timeout, so `PlatformRuntimeDone` reported no `error_type`. And the `max_memory_used_mb == memory_size_mb` check in `PlatformReport` was gated on `runtime.starts_with("provided.al")` to avoid double-counting against the log path, so Node was excluded. - **What changes.** Drop the `provided.al*` restriction so the equality check applies to every runtime. To avoid double-counting against the two pre-existing paths (some invocations satisfy both equality and `Runtime.OutOfMemory` simultaneously), add a per-`Context` `oom_emitted` flag. All three detection paths funnel through a new `Processor::try_increment_oom_metric`, which checks/sets the flag and is a no-op on subsequent calls for the same `request_id`. - **Plumbing.** `Event::OutOfMemory` now carries an `Option<String> request_id`. The log-path detector reads it from `LambdaProcessor::invocation_context.request_id` (set on `PlatformStart`, cleared on `PlatformRuntimeDone`/`PlatformReport`). `None` is only realistic in Managed Instance mode (extensions can't subscribe to INVOKE there); the helper falls back to a best-effort emit without dedup in that case. --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

…r fix (DataDog#1244) ## Overview Bump all `libdd-*` dependencies from `db05e1f` to `48da0d8` to pick up [libdatadog PR #2071](DataDog/libdatadog#2071), which fixes how the `datadog-client-computed-stats` and `datadog-client-computed-top-level` headers are parsed so the behavior matches the Go trace-agent. Bottlecap consumes this exact path in `bottlecap/src/traces/trace_agent.rs` via `(&parts.headers).into()`, so it directly benefits from the fix. ### Adapting to breaking API changes between the two revisions - `HttpClientTrait` was renamed to `HttpClientCapability` - `SendData::send` now requires `C: HttpClientCapability + SleepCapability`, so implemented `SleepCapability` for `HttpClient` ### Testing Added Tier 0 coverage in `trace_agent.rs` pinning the new header-parsing behavior at the `(&headers).into()` boundary `handle_traces` uses: - `true`, `yes`, `t`, `1` -> flag set - `false`, `0`, `f`, `F`, `FALSE`, `False` -> flag clear - absent/empty -> clear - `client_computed_top_level` no longer set by presence alone (pre-bump regression) These test would fail on the pre-bump rev (`db05e1f`), so they lock in the fix. > *"Turns out `yes` means yes and `false` means false. Took a whole PR to teach the parser what every toddler already knows."* — Claude 🤖 [APMSVLS-487](https://datadoghq.atlassian.net/browse/APMSVLS-487) [APMSVLS-487]: https://datadoghq.atlassian.net/browse/APMSVLS-487?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

…ataDog#1248) ## Overview Fixes the remaining cases of [DataDog#1092](DataDog#1092) — customers continuing to see `Max retries exceeded, returning request error` and `TRACES | Request failed: No requests sent` after the v94 patch in DataDog#1094. PR DataDog#1094 disabled connection pooling on the **hyper** `HttpClient` used by the trace flusher and stats flusher. It did not touch the **reqwest** `Client` in `bottlecap/src/http.rs`, which is used by: - the trace proxy (`proxy_flusher`) — the path AppSec and many tracer configurations route through - the metrics flusher - the logs flusher That client was still configured with `pool_idle_timeout(270s)` and `tcp_keepalive(120s)`, leaving idle connections in the pool across Lambda freeze/resume cycles. Connections go stale during a freeze (the OS tears down the remote side while the local kernel still considers them established), and the next request after resume fails with broken pipe / RST → retries exhaust → the user-visible errors. ## Change `bottlecap/src/http.rs`: replace `pool_idle_timeout(270s) + tcp_keepalive(120s)` with `pool_max_idle_per_host(0)`. Mirrors the fix in `bottlecap/src/traces/http_client.rs` and the pattern in libdatadog's `new_client_periodic`. Each request opens a fresh connection; TLS session resumption inside the client still amortizes the handshake. Trade-off: one extra TLS handshake per flush. Matches the pre-v93 behavior that customers had no complaints about. ## Testing - New unit test `shared_client_does_not_pool_connections` in `bottlecap/src/http.rs`: - Spawns a minimal HTTP/1.1 keep-alive server that counts accepted TCP connections. - Issues two sequential GETs through `get_client(...)`. - Asserts the server saw **2** connections. - Verified the test **fails** against the unpatched code (`opened=1`, connection reused from pool) — proves it catches the regression. - Verified the test **passes** against the patched code (`opened=2`, fresh connection each time). - `cargo test --lib` — 547 tests pass. - `cargo clippy --lib --tests -- -D warnings` — clean. - `cargo fmt --check` — clean. ## Other clients (unchanged) | Client | File | Pooling | Notes | |---|---|---|---| | hyper `HttpClient` | `traces/http_client.rs` | already disabled (DataDog#1094) | trace + stats flushers | | reqwest `Client` (shared) | `bottlecap/src/http.rs` | **disabled here** | trace proxy, metrics, logs | | reqwest `Client` (Lambda Extensions API) | `bin/bottlecap/main.rs` | unchanged | localhost-only; pooling irrelevant |

…#1245) ## Overview Makes the extension respect the tracer's `Datadog-Client-Computed-Stats` header and moves `_dd.compute_stats` from a baked-in function tag to a per-span backend directive. We tried supporting the `Datadog-Client-Computed-Stats` header before in DataDog#1118, but that was reverted in DataDog#1176. ### Background Span attribute `_dd.compute_stats` asks the backend to compute trace stats. It must be set to `"1"` only when nobody else computed them — neither the extension (agent) nor the tracer. Previously the extension: 1. **Baked `_dd.compute_stats` into the function tags** unconditionally (`tags_from_env`), which also leaked the key into `_dd.tags.function`. 2. **Ignored `Datadog-Client-Computed-Stats`** entirely, so when a tracer computed stats client-side, the backend was still asked to compute them. ### Canonical semantics (validated against the Go agent) The Go agent (`pkg/serverless/tags/tags.go`) only ever sets `_dd.compute_stats = "1"`, never `"0"`, and leaves the key absent otherwise. This PR matches that: > Set `_dd.compute_stats="1"` iff `!compute_trace_stats_on_extension && !client_computed_stats`; otherwise leave it absent. | `client_computed_stats` (header from tracer) | `compute_on_extension` (extension config) | who computes | stamp `_dd.compute_stats="1"`? | -------------------- | --------------------- | --------------- | --- | true | (ignored) | tracer | ❌ no | false | true | extension | ❌ no | false | false | backend | ✅ yes ### Changes - **`tags/lambda/tags.rs`** — stop baking `_dd.compute_stats` in `tags_from_env` (no longer leaks into `_dd.tags.function`); `COMPUTE_STATS_KEY` is now `pub` so the integration test can reuse it instead of re-declaring the literal. - **Path A: `traces/trace_processor.rs`** — `ChunkProcessor` gains `client_computed_stats` and stamps `_dd.compute_stats="1"` per-span only when neither side computes stats; the extension-side stats-generation guard in `send_processed_traces` now also skips when `client_computed_stats` is set. - **Path B (extension-generated `aws.lambda` span)** — `client_computed_stats` is propagated from the tracer's placeholder span through `context.rs` → `processor.rs` → `processor_service.rs` → `trace_agent.rs`, so Path B reuses the same `ChunkProcessor` stamping (single source of truth). - **OTLP: `otlp/agent.rs`** — the OTLP stats-generation guard previously checked only `compute_trace_stats_on_extension`, so an OTLP request carrying `Datadog-Client-Computed-Stats` would still generate extension-side stats and double-count against the tracer. It now also skips when `client_computed_stats` is set, mirroring the `send_processed_traces` guard. - **Single source of truth: `traces/trace_processor.rs`** — the three decisions over the same two inputs (the per-span `_dd.compute_stats` stamp, plus the extension-side stats-generation guards in `send_processed_traces` and `otlp/agent.rs`) are now derived from one `StatsComputedBy::resolve(compute_on_extension, client_computed_stats)` helper, so the stamp and the guards can't silently drift apart. ### Note on the header value (cross-runtime) `Datadog-Client-Computed-Stats` is not standardized (`"true"` .NET/Java/PHP/Python, `"yes"` JS/Ruby/C++, `"t"` Go). bottlecap consumes the already-parsed `client_computed_stats` bool from `libdd_trace_utils`, where any non-empty value → `true`, so the fix triggers on every runtime. A separate libdatadog PR ([DataDog/libdatadog#2071](DataDog/libdatadog#2071)) aligns the header parsing with the Go agent's `isHeaderTrue`/`ParseBool` rules; this PR only consumes the bool and does not depend on that change. ## Testing - ~**Tier 0** (header-parsing contract)~ — moved to the libdatadog bump PR DataDog#1244, since those tests assert libdatadog parsing behavior that the `db05e1f → 48da0d8` bump changes. This branch keeps Tiers 1–3 and rebases onto [DataDog#1244](/DataDog/pull/1244) after it merges. - **Tier 1** (`trace_processor.rs`) — truth-table on `Span.meta`, stats-skip guard via the real `StatsConcentratorService`, and updated `tags.rs` unit tests asserting the key no longer appears in the tags map. Fixed the logs/metrics integration tests that asserted the old leak. - **Tier 2** (`context.rs`, `processor.rs`) — context-level flag recording and an end-to-end Path B test driving `send_ctx_spans` through the `trace_tx` channel, asserting `_dd.compute_stats` on the `aws.lambda` span across the truth table. - **Tier 3** (`apm_integration_test.rs`) — full fake-intake E2E routing a trace through `SendingTraceProcessor::send_processed_traces`: asserts on the captured `AgentPayload` span meta and on `stats_payloads()` (stats suppressed unless the extension computes and the tracer did not). ### ⚠️ TODO before merging - [x] Rebase onto the libdatadog bump PR ([DataDog#1244](DataDog#1244)) after it merges. - [ ] E2E tests. --- > *"Computing stats twice doesn't make them twice as true, it just makes the backend twice as grumpy."* — Claude 🤖 [APMSVLS-487](https://datadoghq.atlassian.net/browse/APMSVLS-487) [APMSVLS-487]: https://datadoghq.atlassian.net/browse/APMSVLS-487?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

…ataDog#1252) The `clippy` + `fmt` pre-commit hook in `.claude/settings.json` used a bare `cd bottlecap`, which only resolved from the repo root. Switched to `cd "$CLAUDE_PROJECT_DIR/bottlecap"` so it works from subdirectories too.

## Overview Motivation: Customers without an application tracer can hit Lambda timeouts where the extension synthesizes an `aws.lambda` trace ID for timeout handling. That generated trace ID was also copied onto the cold start span, making the extension send a cold start span even though no tracer had established an APM trace. Changes: - Only copy the invocation trace ID to the cold start span when tracer instrumentation was detected. - Preserve the Node/Python behavior where `aws.lambda.load` sets the cold start trace ID directly before the cold start span is sent. - Added regression coverage for timeout handling without a tracer and for preserving tracer-provided cold start trace IDs. ## Testing - `RUSTUP_TOOLCHAIN=stable cargo fmt --check` - Attempted `RUSTUP_TOOLCHAIN=stable cargo test --lib cold_start --no-default-features`; blocked before compilation because the sandbox allowlist returned 403 fetching `DataDog/serverless-components` for the `datadog-fips` git dependency. --- PR by Bits - [View session in Datadog](https://app.datadoghq.com/code/9495ef61-5bdd-4785-95d4-5b9ecf23d513) Comment @DataDog to request changes Co-authored-by: datadog-bits <[email protected]>

…uest id for on-demand (DataDog#1250) ## Overview When the tracer sends the `lambda-runtime-aws-request-id` header (in the start-invocation and end-invocation requests), use request ID based pairing instead of FIFO for both `UniversalInstrumentationStart` and `UniversalInstrumentationEnd` in On Demand mode in addition to Managed Instance mode. FIFO pairing is preserved as a fallback for tracers that do not send the header. Motivation: [SLES-2877](https://datadoghq.atlassian.net/browse/SLES-2877) [SLES-2877]: https://datadoghq.atlassian.net/browse/SLES-2877?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

…OG_BUFFER_SIZE (DataDog#1239) ## Background For durable functions, the extension holds logs and waits to get the invocation context from the tracer so the extension can enrich the logs. ## Problem When tracing is off (which is not recommended for durable functions), logs are flushed when: 1. the buffer is full (buffer size is all logs for 50 invocations). This increases delay. Or 2. when extension shuts down. However, this is not guaranteed to happen due to race conditions, causing missing logs. ## This PR - Adds `lambda_durable_function_log_buffer_size` config field (env var `DD_LAMBDA_DURABLE_FUNCTION_LOG_BUFFER_SIZE`, default: `0`) controlling the max number of request ID keys held in `held_logs`. - When set to `0`, logs bypass the hold mechanism and are sent immediately without durable execution context enrichment. - Setting the default to 0 for now because the tracer changes for durable functions are not released. Later, after tracer changes are released, we can change the default back to 50 and ask customers to install the tracer for durable functions. ## Test plan ### Steps - Build an extension layer - install it on a durable function - execute the function - wait for a few minutes until the environment shuts down ### Result Before: - No log appears in Datadog - and the invocation doesn't appear either After: - The invocation appears <img width="407" height="106" alt="image" src="https://github.com/user-attachments/assets/f2679f37-a80e-48ef-b84c-049beeb0f7b3" /> - Logs also appear <img width="578" height="562" alt="image" src="https://github.com/user-attachments/assets/55ac197e-22fa-455e-a8eb-40a46ad99dce" /> --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

…1255) ## Overview Two tests added in DataDog#1254 were failing on `main`. The Rust "Test Suite" CI job is not a required check (only `devflow/mergegate` is), so the red suite did not block DataDog#1254 from merging. ### 1. `enrich_ctx_keeps_tracer_set_cold_start_trace_id_without_tracer_detected` Annotated `#[test]` but calls `setup()`, which spawns a Tokio task via `tokio::spawn(service.run())`. Running it panicked with `there is no reactor running, must be called from the context of a Tokio 1.x runtime`. Changed to `#[tokio::test] async fn` to match its sibling tests. ### 2. `timeout_without_tracer_does_not_send_cold_start_span` Passed its `trace_sender` by value into `on_platform_runtime_done`, dropping the only `Arc<SendingTraceProcessor>` before `try_recv()`. The receiver then reported `Disconnected`, but the test asserted `Empty`, so it failed even though the production code correctly never sent the cold start span. Fixed by passing a clone and holding the sender until after the assertion so the channel stays connected. No production code changes; both fixes are test-only. > *"A test that drops its own sender and then complains the line went dead: the unit-testing equivalent of cutting the phone cord and filing a complaint about the silence."* — Claude 🤖

…ake (DataDog#1256) - [x] ~Note: stacked on DataDog#1255. Merge that PR first.~ ## Overview Extends the in-process fake-intake APM tests (added in DataDog#1194) with **value-level** coverage of trace stats aggregation. The existing stats tests assert serialization fidelity (a hand-built `ClientStatsPayload` survives the roundtrip) and stats presence/suppression (`_dd.compute_stats` truth table). None assert that the *computed* aggregates reaching the intake are correct. This PR routes concrete spans through the real pipeline (`SendingTraceProcessor` → `StatsConcentratorService` → `StatsFlusher` → fake-intake) and asserts on the decoded `ClientGroupedStats`: - `e2e_stats_count_aggregates_identical_spans` — 3 identical top-level spans collapse into one group with `hits == 3`, `top_level_hits == 3`. - `e2e_stats_counts_errors_separately_from_hits` — `errors` counts only `error != 0`; `hits` counts all. - `e2e_stats_sums_span_durations` — `duration` is the nanosecond sum of aggregated spans. - `e2e_stats_groups_by_resource` — distinct resources stay in separate groups. - `e2e_stats_groups_by_span_kind` — distinct `span.kind` stay in separate groups. - `e2e_stats_groups_by_http_status_code` — distinct `http.status_code` stay in separate groups. These verify bottlecap's wiring and serialization end-to-end (a span entering the trace path comes out the intake correctly aggregated), which libdatadog's concentrator unit tests cannot see. The aggregation *math* itself is owned and exhaustively tested upstream in `libdd-trace-stats`. To support multiple spans, `run_processor_pipeline` was generalized into `run_processor_pipeline_with_traces(.., traces)`, with a thin single-span wrapper preserving the existing `_dd.compute_stats` tests unchanged. ### Out of scope: time-bucketing Deliberately not added. Bucket assignment lives entirely inside `SpanConcentrator` and is thoroughly tested in `libdd-trace-stats` (`test_concentrator_stats_counts` over multiple buckets, `test_concentrator_oldest_timestamp_cold/hot`, `test_force_flush`). bottlecap's wiring does not touch bucketing, so a test here would duplicate upstream coverage and require a production clock seam (the concentrator seeds `oldest_timestamp` from wall-clock) for no unique coverage. ## Testing This PR is only adding more tests. If they pass, we're good. Ran locally: - `cargo test --test apm_integration_test` - `cargo fmt --all -- --check` - `cargo clippy --workspace --all-targets --features default` > *"Six tests walk into a bucket. They all collapse into one group — turns out they were identical the whole time."* — Claude 🤖

…ts (DataDog#1189) ## Overview Configure `SpanConcentrator` in bottlecap with the Go agent's default `ComputeStatsBySpanKind` span kinds and `basePeerTags` peer tag keys, fixing two gaps in agent-side trace stats computation: - [`APMSVLS-464`](https://datadoghq.atlassian.net/browse/APMSVLS-464): `span_kinds_stats_computed` was empty, so non-top-level, non-measured spans with `span.kind` = `server`/`client`/`producer`/`consumer`) were silently excluded from stats. - [`APMSVLS-463`](https://datadoghq.atlassian.net/browse/APMSVLS-463): `peer_tag_keys` was empty, so client/producer/consumer spans had no per-dependency granularity in stats output (e.g., all S3 buckets, DynamoDB tables, Kafka topics lumped together). ### Changes - Add `STATS_ELIGIBLE_SPAN_KINDS` constant (4 span kinds matching the Go agent's `KindsComputed`) - Add `DEFAULT_PEER_TAG_KEYS` constant (43 peer tag keys matching the Go agent's `basePeerTags` from `mappings.json`) - Pass both to `SpanConcentrator::new()` instead of empty vecs - Add unit tests that exercise the `StatsConcentratorService` pipeline for both features - Add fake-intake integration tests that route real spans through the concentrator → flusher → intake path, verifying span kind and peer tags survive msgpack/gzip serialization end to end ### Follow-up Both constants are hand-copied from the Go agent (`KindsComputed` and `basePeerTags`), which is the source of truth, and the same lists are also duplicated in `serverless-components`. A future refactor should keep the Rust agents in sync with the Go agent rather than each maintaining its own copy. ## Testing - `test_span_kind_stats_computed`: sends a non-root, non-measured client span through the service, verifies stats are produced with `span_kind="client"` - `test_peer_tags_populated`: sends a client span with `db.instance` and `db.system` meta, verifies `peer_tags` contains both in the stats output - `stats_span_kind_through_fake_intake`: feeds a non-root, non-measured `server` span through the full concentrator → flusher → fake-intake path, asserts the decoded `StatsPayload` carries `span_kind="server"` - `stats_peer_tags_through_fake_intake`: feeds a `client` span with `db.instance`/`db.system` meta through the same path, asserts the decoded `peer_tags` contain both keys after serialization - Tests were written first and confirmed failing before the fix was applied - Full test suite still passes > *"Turns out an empty vec is a great way to compute stats on absolutely nothing. Efficient, but not very useful."* — Claude 🤖

DataDog#1253) ## Overview The init duration metric is sent on the `PlatformInitReport` event. In the case of this event not being received / arriving late, this PR adds a fallback to use the init duration from the `PlatformReport` event on cold start invocations in On Demand mode. ## Motivation Saw failures on emitting the init duration metric in E2E tests. ## Testing Added unit tests and E2E tests no longer fail on sending the init duration metric on this PR.

https://datadoghq.atlassian.net/browse/SLES-2889 ## Overview Large single-invocation traces could be rejected by the trace intake with a 413, because the extension undercounted how big a payload would be once it left the extension. The extension enriches every span after the tracer posts it (adding the tags-provider map), so the protobuf payload it sends is larger than the raw msgpack it received. The trace aggregator's batch cap therefore undercounted the true outbound bytes and packed too much into a single request, whose combined size exceeded the intake limit → 413. The batch cap was below realistic enriched payload sizes. The cap was set to the old 3.2 MB documented limit. However, from [trace intake config](https://github.com/DataDog/logs-backend/blob/6b17c3834760efe43f0fc2ae09f8c03f893f32fb/domains/intake/apps/intake/src/main/resources/track-types.conf#L269), the limit is actually 15. This PR: - Computes `body_size` from the enriched, protobuf-encoded payload **unconditionally** (not only when on-extension stats are enabled), so the aggregator's batch cap reflects exactly what will be sent. - Raises the batch cap from 3.2 MB to 12 MB — kept below the trace intake's ~15 MB limit so a large enriched payload flushes in a single batch without a 413. - Adds debug logs for the post-enrichment payload size and per-batch coalesced size. ## Testing - **Unit:** new test asserting `body_size` equals the protobuf `encoded_len` of the payload actually sent, and exceeds the (smaller) ingress size — on the default config (no on-extension stats). - **Integration:** new `payload-size` suite deploys a Node Lambda that emits a ~10 MB single-invocation trace (400 spans carrying large captured payloads) and verifies it: enriches to >10 MB, flushes in a single batch >10 MB, arrives with no 413, and is delivered intact (one trace, root span present, all 400 spans). Run against a locally published extension layer in the serverless sandbox.

…g#1263) (DataDog#1264) ## Overview Consolidates the 6 open Dependabot/ADMS vulnerability PRs into a single PR. | Source PR | Manifest | Notes | |---|---|---| | DataDog#1258 | `integration-tests/lambda/otlp-java/pom.xml` | `io.opentelemetry:opentelemetry-api` 1.58.0 → 1.63.0 | | DataDog#1259 | `local_tests/golang/go.{mod,sum}` | `aws-sdk-go`, `golang.org/x/net`, `golang.org/x/sys` minor bumps | | DataDog#1260 | `integration-tests/lambda/otlp-node/package*.json` | Superseded by DataDog#1263 (lock already has `@grpc/[email protected]`, `@protobufjs/utf8` override carried over) | | DataDog#1261 | `integration-tests/package*.json` | `uuid` 9 → 14 (ported manually onto DataDog#1262 to avoid lock conflict) | | DataDog#1262 | `integration-tests/package*.json` | `axios` 1.15 → 1.18, plus overrides for `@opentelemetry/core`, `fast-xml-parser`, `fast-xml-builder`, `fast-uri`, `brace-expansion` | | DataDog#1263 | `integration-tests/lambda/otlp-node/package*.json` | Major OTel SDK bump (`@opentelemetry/sdk-node` 0.54 → 0.219, `protobufjs` 7.5.4 → 7.6.4) | Closes the following Dependabot alerts: DataDog#191, DataDog#192 (`@grpc/grpc-js`); DataDog#183–DataDog#190 + older axios alerts; protobufjs alerts via the major OTel SDK bump; uuid DataDog#180; OpenTelemetry Java DataDog#176. ## Testing GitHub Actions checks (Build All, Test Suite, Clippy, Audit, CodeQL) passed on each individual source PR. `dd-gitlab/*` checks failed identically on all 6 — appears to be an unrelated infra/permissions issue with the bot-authored branches. Re-running CI on this branch (created from a maintainer account) should clear them. The vendored deps live entirely under `integration-tests/` and `local_tests/`, so they have no impact on the shipped extension binary. --------- Co-authored-by: gh-worker-campaigns-3e9aa4[bot] <244854796+gh-worker-campaigns-3e9aa4[bot]@users.noreply.github.com>

…nt-config (DataDog#1249) ## Overview First step of migrating bottlecap's in-tree configuration module onto the shared `datadog-agent-config` crate (lives in [DataDog/serverless-components](https://github.com/DataDog/serverless-components/tree/main/crates/datadog-agent-config)). This PR adds the foundation only — it introduces a `LambdaConfig` extension struct (in `bottlecap/src/config/mod.rs`, alongside the existing legacy code) that implements the upstream `ConfigExtension` trait, plus a `LambdaConfigSource` deserialization shape that figment uses for both env-var and YAML loading (dual extraction). Nothing in bottlecap consumes the extension yet — the existing `bottlecap::config::Config` struct is untouched. The follow-up DataDog#1251 (stacked on this) replaces it with `Config<LambdaConfig>`, deletes the duplicated env.rs / yaml.rs / deserializer modules, and removes the legacy `#[macro_export]` `merge_*` macros at the top of `mod.rs` (`LambdaConfig::merge_from` deliberately uses fully-qualified upstream macro paths today to coexist with them). ### Why this shape The upstream crate is purpose-built for this migration — see PR DataDog/serverless-components#111 ("ConfigExtension trait") which landed exactly the extensibility hook we need. Each consumer (bottlecap here, future serverless agents elsewhere) supplies its own extension type for fields that don't generalize, while sharing one canonical implementation for all the core agent fields (site, api_key, logs/APM/OTLP/proxy/trace propagation, etc.). ### Fields in the extension The 19 Lambda-specific fields that have no upstream equivalent: - `api_key_secret_arn`, `kms_api_key`, `api_key_ssm_arn`, `api_key_secret_reload_interval` - `serverless_logs_enabled` (OR-merged with the `DD_LOGS_ENABLED` alias) - `serverless_flush_strategy` (custom `FlushStrategy` deserializer for `"end" | "end,N" | "periodically,N" | "continuously,N"`) - `enhanced_metrics`, `lambda_proc_enhanced_metrics` - `capture_lambda_payload`, `capture_lambda_payload_max_depth` - `compute_trace_stats_on_extension`, `span_dedup_timeout` - `dd_org_uuid` (sourced from `DD_ORG_UUID`, source field `org_uuid` → config field `dd_org_uuid`) - `serverless_appsec_enabled`, `appsec_rules`, `appsec_waf_timeout` - `api_security_enabled`, `api_security_sample_delay` - `custom_metrics_exclude_tags` (sourced from `DD_LAMBDA_CUSTOMER_METRICS_EXCLUDE_TAGS` / `lambda_customer_metrics_exclude_tags:` in YAML) Kept in the extension rather than upstreamed because they're Lambda-runtime concerns. `custom_metrics_exclude_tags` is arguably generalizable to other serverless targets but stays here for now; we can migrate it upstream when another consumer needs it. ### Pre-requisites that already merged - DataDog/serverless-components#135 — bumps libdatadog rev across all serverless-components crates from `0a3304c` to `48da0d8` to match bottlecap. - DataDog/serverless-components#136 — switches `datadog-agent-config`'s libdd transitive deps to `default-features = false` and exposes `https` / `fips` opt-in features. Without this, the FIPS dep tree was poisoned by an implicit `https` (ring) path. Bottlecap pins all three serverless-components crates (`dogstatsd`, `datadog-fips`, `datadog-agent-config`) to the merge SHA `bb4dedee` so cargo deduplicates `dogstatsd`. ## Testing 37 new tests in `bottlecap/src/config/mod.rs::lambda_config_tests` cover each extension field from both env vars and YAML where applicable, plus: - Per-field env round-trip (`api_key_secret_arn`, `kms_api_key`, `api_key_ssm_arn`, …) - YAML coverage for the two source-to-config renames: `org_uuid` → `dd_org_uuid` and `lambda_customer_metrics_exclude_tags` → `custom_metrics_exclude_tags` - `DD_LOGS_ENABLED` ↔ `DD_SERVERLESS_LOGS_ENABLED` OR-merge semantics - `FlushStrategy` — `"end"`, `"end,N"` (`EndPeriodically`), `"periodically,N"`, `"continuously,N"`, invalid → `Default` - Duration parsing — seconds, microseconds, `_ignore_zero` variant - env precedence over YAML - Forgiving fallback when a single field is malformed (default preserved instead of failing the whole extraction) ``` $ cargo test --lib config::lambda_config_tests test result: ok. 37 passed; 0 failed; 0 ignored ``` Existing bottlecap config behavior is unchanged (this PR adds; it doesn't replace yet). ## Follow-up - DataDog#1251 — stacked on this branch — does the actual wire-in: replaces `bottlecap::config::Config` with `Config<LambdaConfig>`, deletes the duplicated config files, and removes the legacy `#[macro_export]` macros at the top of `mod.rs`.

…aConfig> (DataDog#1251) ## Overview Stacked on top of DataDog#1249. Completes the migration to the shared `datadog-agent-config` crate: `bottlecap::config::Config` is now a type alias for `datadog_agent_config::Config<LambdaConfig>`, Lambda-specific fields live under `.ext`, and the 10 in-tree config submodules that duplicated upstream implementations are removed. ## Structural changes - `bottlecap/src/config/mod.rs` shrinks from **2243 → 602 lines**. The legacy `Config` struct, `ConfigBuilder`, `ConfigSource`, `ConfigError`, the `#[macro_export]` `merge_*` macros, all per-field deserializer helpers, and the test module that mirrored upstream's behavior — all gone. What remains: a `type Config` alias, a `get_config(path)` wrapper, the `LambdaConfig` extension itself (from the parent PR), and re-exports of upstream modules under the same `crate::config::*` paths so existing imports keep working. - New module **`bottlecap/src/config/propagation_wrapper.rs`** holds a newtype `PropConfig(Arc<Config>)` so we can implement the foreign `PropagationConfig` trait on the foreign `Config<E>` type without tripping Rust's orphan rule. The wrapper is scoped to the dd-trace-rs propagator boundary — nothing else uses it. - `bottlecap/src/traces/propagation/mod.rs` wraps the inner propagator in `PropConfig` instead of passing `Config` directly. All call sites that previously handed `Arc<Config>` to the propagator are unchanged — the wrapping happens inside `DatadogCompositePropagator::new`. - **Deleted files** (each redundant with upstream): - `additional_endpoints.rs`, `apm_replace_rule.rs`, `env.rs`, `flush_strategy.rs`, `log_level.rs`, `logs_additional_endpoints.rs`, `processing_rule.rs`, `service_mapping.rs`, `trace_propagation_style.rs`, `yaml.rs` - **~70 field-access sites** referencing Lambda-specific fields (`config.api_key_secret_arn`, `config.serverless_logs_enabled`, `config.enhanced_metrics`, etc.) updated to read through `config.ext.X`. Test struct literals that constructed `Config { ... }` with Lambda fields now nest them under `ext: LambdaConfig { ..., ..Default::default() }`. ## What stays the same - `LambdaConfig` itself (and its 33 tests) — already shipped in the parent PR; no behavior change in this commit. - All other tests pass: **501 lib + 5 integration tests green**. - The `.ext` indirection is invisible to callers that hold an `Arc<Config>` thanks to Rust's field-access auto-deref — they just go from `config.foo` to `config.ext.foo` for the 19 Lambda fields. ## Testing ``` $ cargo test --lib test result: ok. 501 passed; 0 failed; 0 ignored $ cargo test --tests test result: ok. (all integration suites green) ``` ## Follow-ups - Re-target this PR's base from `feat/use-datadog-agent-config-crate` to `main` once DataDog#1249 merges. - Optional: contribute `custom_metrics_exclude_tags` upstream so it becomes part of core `Config` rather than living in the Lambda extension (discussed in DataDog#1249).

…g#1246) ## What does this PR do? The usage-metrics monitoring loop polls `/proc` every 10ms and reads `/proc/{pid}/fd` for every PID each tick. When `read_dir` fails (e.g. `/proc/1/fd` is not readable by the extension), the existing code logged a `trace!` message per unreadable PID per tick. At ~40 ticks per invocation (tested with a Hello World golang function), a single persistently-unreadable PID produced ~40 trace messages per invocation (~4000 per 100 invocations). These failures are benign — a PID's `fd` directory may be unreadable while the process is otherwise alive (typically a permission issue). The other `/proc/{pid}` readers (`fd_max`, `threads_max`, `threads_use`) already fail silently with `continue`. This change makes `get_fd_use_data_from_path` consistent with that behavior by removing the trace logging and dropping the now-unused `trace` import. ## Motivation Observed ~4000 noisy log messages per 100 invocations when trace logging was enabled, making trace output unusable for debugging. ## Testing - `cargo build` — clean compile, no unused-import warnings - `cargo test proc` — all 8 proc tests pass, including `test_get_fd_use_data` - `cargo clippy` — clean ## Checklist - [x] PR description is filled out - [x] Tests pass Co-authored-by: Claude Opus 4.8 <[email protected]>

…_enabled) (DataDog#1267) ## Summary Bumps the \`datadog-agent-config\` / \`dogstatsd\` / \`datadog-fips\` pin to \`8ce37eb\`, which contains two recently-merged upstream features: - [serverless-components#137](DataDog/serverless-components#137) — adds \`Config::dd_org_uuid: String\` (\`DD_ORG_UUID\` / yaml \`org_uuid\`) - [serverless-components#138](DataDog/serverless-components#138) — adds \`Config::logs_enabled: bool\` (\`DD_LOGS_ENABLED\` / yaml \`logs_enabled\`) ## What changes in bottlecap ### dd_org_uuid Drops the now-redundant local copy from \`LambdaConfig\`: - \`LambdaConfig\`: removed \`dd_org_uuid: String\`. - \`LambdaConfigSource\`: removed \`org_uuid: Option<String>\` and the \`merge_string!(self, dd_org_uuid, source, org_uuid)\` source-to-config renaming call in \`merge_from\`. - Consumers (\`secrets/decrypt.rs\`, \`secrets/delegated_auth/client.rs\`) switch from \`config.ext.dd_org_uuid\` → \`config.dd_org_uuid\`. - Tests assert on the upstream field. ### logs_enabled The lambda extension's existing OR-merge contract for log shipping is preserved **exactly**. The legacy semantics are: | \`DD_SERVERLESS_LOGS_ENABLED\` | \`DD_LOGS_ENABLED\` | resolved \`serverless_logs_enabled\` | |---|---|---| | unset | unset | **true** (default kept) | | any | true | true | | true | any | true | | false | false | false | | **unset** | **false** | **false** (alias alone can override default) | | false | unset | false | To preserve "was \`DD_LOGS_ENABLED\` explicitly set?" the alias source field is **retained** on \`LambdaConfigSource\` and the OR-merge in \`merge_from\` is kept. The upstream parsing of the same env var into \`config.logs_enabled\` runs in parallel — that field exists for any non-lambda consumer of the crate, but the lambda extension continues to gate log shipping on \`config.ext.serverless_logs_enabled\`. A regression test was added for the alias-only-false case (the trickiest invariant), so a future refactor that breaks it fails loudly. ## Why two PRs in one Both upstream PRs are merged and \`8ce37eb\` contains them as a single SHA. Bundling lets us bump once and avoid a transitional state where main is pinned to a SHA that has \`logs_enabled\` upstream but the lambda extension's source field still aliases it. ## Test plan - [x] \`cargo test --workspace --features default\` — all pass (lambda_config_tests now at 40, +2 for the new logs_enabled coverage) - [x] \`cargo clippy --workspace --all-targets --features default\` — clean - [x] \`cargo fmt -- --check\` — clean - [x] Regression test \`logs_enabled_alias_only_false_overrides_default\` exercises the invariant that \`DD_LOGS_ENABLED=false\` alone disables logs (overriding the default-true) - [x] No behavior change for customers — same env vars, same resolved field at lambda call sites

…#1269) ## Overview Fixes high-severity advisory [RUSTSEC-2026-0185](https://rustsec.org/advisories/RUSTSEC-2026-0185): *Remote memory exhaustion in quinn-proto from unbounded out-of-order stream reassembly*. Solution per the advisory is to upgrade to `>= 0.11.15`. `quinn-proto` is pulled in transitively via `reqwest -> quinn -> quinn-proto` across multiple workspace crates (`bottlecap`, `dogstatsd`, `datadog-fips`, `libddwaf`, `datadog-agent-config`). This blocks the v98 release — `cargo audit` is failing CI on `main`. `cargo audit` after the bump shows zero vulnerabilities; the remaining entries are pre-existing unmaintained-crate warnings (async-std, buf_redux, multipart, rustls-pemfile, safemem, twoway). ## Testing - `cargo audit` no longer reports a vulnerability.

## Overview Bump extension version to `98-next` in preparation for the v98 release. ## Testing No functional change.

…ck prefix (DataDog#1287) ## Overview Integration-suite deploys intermittently fail at `cdk deploy` (before any test runs) because of resource conflict. On some retries, the previous run resources were not cleaned up correctly. ## Fix * Include `CI_JOB_ID` in the identifier so every attempt gets unique names and there won't be resource conflict. * Also shorted 'integ' to 'it' so we are less likely to hit the 64 character limit on cdk resources names. Note - The integration tests are failing due to 429 error, too many requests. Will address this in follow up PR.

… 429s (DataDog#1288) ## Overview The integration tests intermittently fail due to not fetching telemetry data from Datadog due to 429, rate limit exceeded. This PR adds retries for fetching telemetry data. Since we are now retrying, the total time of a test might increase. So, this PR also increase the test timeout from 15 minutes to 30 minutes.

…og#1276) **Jira:** [SVLS-9380](https://datadoghq.atlassian.net/browse/SVLS-9380) (H1) · epic [SVLS-9378](https://datadoghq.atlassian.net/browse/SVLS-9378) ## What Switch the non-FIPS `default` feature from `reqwest/rustls-tls-native-roots` to `reqwest/rustls-tls-webpki-roots`. This removes the `rustls_native_certs::load_native_certs()` filesystem cert scan that ran on **every** `reqwest::Client::build()` — i.e. on the cold-start path for all three init reqwest clients (register in `main.rs`, shared flush in `http.rs`, secrets-decrypt in `secrets/decrypt.rs`). Cold-start hypothesis H1. FIPS builds are unchanged (keep native roots). ## Trade-off webpki trusts only the public Mozilla root bundle, so private/internal OS-installed CAs are no longer trusted implicitly. Affected setups (e.g. a TLS-intercepting `DD_PROXY_HTTPS`) supply the CA via `DD_TLS_CERT_FILE` (still layers on top of webpki) or use `DD_SKIP_SSL_VALIDATION`. Datadog public intake chains to public roots, so the default path is unaffected. ## Tested & verified - **Hermetic TLS tests** (`src/http.rs`): `tls_cert_file` trusts a private CA on top of webpki; webpki alone rejects it without the cert file; `skip_ssl_validation` escape hatch works; `load_custom_cert` parsing (single / multiple / empty / non-PEM). - **CI feature-graph guard** (`scripts/verify_tls_root_features.sh` + job): default build stays on webpki (no native-cert scan), FIPS stays on native roots — catches silent feature-graph drift. - **`cargo --unit-graph`**: native-cert scan is gone from all three reqwest clients' runtime path (native-roots survives only in a build-script dep); FIPS feature graph unchanged. - All GitHub Actions green (clippy `default` + `fips`, test, build, license, audit, the new TLS Root Features job). ## Not doing (by design) - **Secrets/AWS client** (`secrets/decrypt.rs` — KMS/SM/SSM/STS): intentionally keeps default roots, no proxy / `tls_cert_file` / `skip_ssl_validation`. These are direct AWS control-plane calls that chain to public AWS CAs; `DD_PROXY_HTTPS`/`tls_cert_file` are for Datadog egress; and we deliberately don't disable cert validation on credential/secret fetches. Only a fully air-gapped setup intercepting AWS traffic with a private CA would be affected — those should use VPC endpoints (public certs). Documented in code; revisit only on a real report. - **Trace client** (`traces/http_client.rs`, hyper/libdd-common): keeps native roots via its own trust path — out of scope here. ## Before all-region rollout Verify GovCloud / dedicated-region intake certs chain to the webpki bundle (deferred real-endpoint canary). [SVLS-9380]: https://datadoghq.atlassian.net/browse/SVLS-9380?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [SVLS-9378]: https://datadoghq.atlassian.net/browse/SVLS-9378?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

litianningdatadog and others added 30 commits February 24, 2026 15:46

fix: pin rustls-native-certs to <0.8.3 (DataDog#1047)

6e75acf

## Overview Regression on performance because `[email protected]` checks for multiple folders as opposed to only one ## Testing Manual testing in AWS Lambda and Self Monitoring Data

chore: sweep across qemu versions to see which ones do and dont work (D…

23eebef

…ataDog#1056)

chore: more fixes for serverless-init builds (DataDog#1057)

321fb09

fixing qemu flakiness.

lym953 and others added 28 commits May 29, 2026 13:05

fix(docs): remove serverless-init mention (DataDog#1243)

a5742fc

remove mention of serverless-init

chore: bump EXTENSION_VERSION to 98-next (DataDog#1268)

0d6285f

## Overview Bump extension version to `98-next` in preparation for the v98 release. ## Testing No functional change.

Merge branch 'main' of github.com:DataDog/datadog-lambda-extension

a58febb

smithclay approved these changes Jul 1, 2026

View reviewed changes

jaronoff97 merged commit ef70371 into main Jul 1, 2026
11 of 13 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

chore: merge main#18

chore: merge main#18
jaronoff97 merged 131 commits into
mainfrom
merge-main

jaronoff97 commented Jun 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

20 participants

Uh oh!

Conversation

jaronoff97 commented Jun 30, 2026

Overview

Testing

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

20 participants