fix(http): disable pooling on shared reqwest client (#1092)#1248
Merged
Conversation
The shared reqwest client used by the trace proxy, metrics, and logs flushers (bottlecap/src/http.rs) was still pooling connections with a 270s idle timeout. PR #1094 / v94 fixed this for the hyper client in traces/http_client.rs, but the reqwest client — which serves the proxy path that AppSec and many tracer configurations route through — was left pooled. In Lambda, the execution environment can be frozen for seconds to minutes between invocations. Pooled connections go stale during a freeze and fail on the next request after resume, surfacing as "Max retries exceeded, returning request error" followed by "TRACES | Request failed: No requests sent". Set pool_max_idle_per_host(0) on the reqwest client builder to match the pattern already used in traces/http_client.rs and in libdatadog's new_client_periodic. Each request opens a fresh connection; TLS session resumption inside the client still amortizes the handshake. Adds an inline test that spawns a keep-alive HTTP/1.1 server and counts TCP accepts across two sequential requests. With pooling on, the count is 1 (reuse); with pooling off, the count is 2 (fresh).
duncanista
requested review from
Copilot and
shreyamalpani
and removed request for
Copilot
June 9, 2026 20:09
shreyamalpani
approved these changes
Jun 9, 2026
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Fixes the remaining cases of #1092 — customers continuing to see
Max retries exceeded, returning request errorandTRACES | Request failed: No requests sentafter the v94 patch in #1094.PR #1094 disabled connection pooling on the hyper
HttpClientused by the trace flusher and stats flusher. It did not touch the reqwestClientinbottlecap/src/http.rs, which is used by:proxy_flusher) — the path AppSec and many tracer configurations route throughThat client was still configured with
pool_idle_timeout(270s)andtcp_keepalive(120s), leaving idle connections in the pool across Lambda freeze/resume cycles. Connections go stale during a freeze (the OS tears down the remote side while the local kernel still considers them established), and the next request after resume fails with broken pipe / RST → retries exhaust → the user-visible errors.Change
bottlecap/src/http.rs: replacepool_idle_timeout(270s) + tcp_keepalive(120s)withpool_max_idle_per_host(0). Mirrors the fix inbottlecap/src/traces/http_client.rsand the pattern in libdatadog'snew_client_periodic. Each request opens a fresh connection; TLS session resumption inside the client still amortizes the handshake.Trade-off: one extra TLS handshake per flush. Matches the pre-v93 behavior that customers had no complaints about.
Testing
shared_client_does_not_pool_connectionsinbottlecap/src/http.rs:get_client(...).opened=1, connection reused from pool) — proves it catches the regression.opened=2, fresh connection each time).cargo test --lib— 547 tests pass.cargo clippy --lib --tests -- -D warnings— clean.cargo fmt --check— clean.Other clients (unchanged)
HttpClienttraces/http_client.rsClient(shared)bottlecap/src/http.rsClient(Lambda Extensions API)bin/bottlecap/main.rs