DoorDash (APMS-19533) reports persistent UDS transport errors after the
trace-agent silently drops idle keep-alive connections under backpressure
- either "write: broken pipe" or "read: connection reset by peer" on the
next request that reuses a stale conn from the client pool. PR #4484
added retry loops to traces and stats, but both are gated on
sendRetries (default 0), so customers who don't explicitly opt in see
no retries.

Two layers of recovery, neither requires user configuration:

1. Mark POSTs replayable. Set Idempotency-Key on the trace and stats
   requests and set req.GetBody on the trace request (stats is
   *bytes.Buffer, which http.NewRequest already handles). Together
   these let net/http.persistConn.shouldRetryRequest treat the POST as
   idempotent and transparently retry on a fresh connection - covering
   the common errServerClosedIdle and read-after-write failures. See
   golang/go#19943.

2. Catch the residual mid-write window. stdlib still refuses to retry
   once any byte has been written, even with Idempotency-Key, because
   the error no longer classifies as nothingWrittenError. A small
   doWithStaleConnRetry helper handles that case: up to two extra
   application-level retries on EPIPE / ECONNRESET / net.ErrClosed,
   rewinding the body via req.GetBody. The agent dedupes traces by
   span ID and ignores Idempotency-Key, so a duplicate is harmless.

Also fixes a Go-precedence bug at writer.go:178 where
attempt+1%5 == 0 parsed as attempt + 1, so the periodic
"failure sending traces" log line never fired.

Verification: TestUDSTransportRecoversFromStaleIdleConn reproduces the
customer's failure on baseline (~30-45% of 400 concurrent POSTs fail
with the same error strings the customer reports) and passes 10/10
times with the fix applied.

Refs: APMS-19533
Refs: #4484

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

DoorDash (APMS-19533) reports persistent UDS transport errors after the
trace-agent silently drops idle keep-alive connections under backpressure
- either "write: broken pipe" or "read: connection reset by peer" on the
next request that reuses a stale conn from the client pool. PR #4484
added retry loops to traces and stats, but both are gated on
sendRetries (default 0), so customers who don't explicitly opt in see
no retries.

Two layers of recovery, neither requires user configuration:

1. Mark POSTs replayable. Set Idempotency-Key on the trace and stats
   requests and set req.GetBody on the trace request (stats is
   *bytes.Buffer, which http.NewRequest already handles). Together
   these let net/http.persistConn.shouldRetryRequest treat the POST as
   idempotent and transparently retry on a fresh connection - covering
   the common errServerClosedIdle and read-after-write failures. See
   golang/go#19943.

2. Catch the residual mid-write window. stdlib still refuses to retry
   once any byte has been written, even with Idempotency-Key, because
   the error no longer classifies as nothingWrittenError. A small
   doWithStaleConnRetry helper handles that case: up to two extra
   application-level retries on EPIPE / ECONNRESET / net.ErrClosed,
   rewinding the body via req.GetBody. The agent dedupes traces by
   span ID and ignores Idempotency-Key, so a duplicate is harmless.

Also fixes a Go-precedence bug at writer.go:178 where
attempt+1%5 == 0 parsed as attempt + 1, so the periodic
"failure sending traces" log line never fired.

Verification: TestUDSTransportRecoversFromStaleIdleConn reproduces the
customer's failure on baseline (~30-45% of 400 concurrent POSTs fail
with the same error strings the customer reports) and passes 10/10
times with the fix applied.

Refs: APMS-19533
Refs: #4484

The trace-agent recycles idle keep-alive UDS connections under backpressure.
The tracer's next request on a pooled-but-dead connection fails immediately
with `write: broken pipe` or `read: connection reset by peer`, dropping the
payload. Prior fix #4484 added retry loops gated on sendRetries (default 0),
so customers without WithSendRetries got no retry at all.

Fix — two layers:

1. Make each POST replayable so net/http transparently retries on a fresh
   connection before the error reaches the caller:
   - Set req.GetBody (http.NewRequest doesn't auto-detect the custom payload
     type) so the stdlib can rewind the body on retry.
   - Set an Idempotency-Key header so net/http.Request.isReplayable returns
     true and the stdlib's nothingWrittenError path silently retries.
   - Return the payload directly from GetBody (not via io.NopCloser) so
     payloadV1.Close() is called by the HTTP transport, completing the
     two-bit pool-return handoff (pv1StateBodyClosed via atomic.Or).

2. Add doWithStaleConnRetry: up to 3 extra attempts (4 total) on transient
   connection errors (EPIPE, ECONNRESET, ECONNABORTED, net.ErrClosed, plus
   a cross-platform string fallback for Windows WSA errors). Covers the
   residual mid-write window where stdlib refuses to replay because
   pc.nwrite > startBytesWritten (golang/go#19943).

Retrying is safe: the agent deduplicates traces by span ID and ignores the
Idempotency-Key header, so a duplicate payload is harmless.

Also adds internal/apps/staleidle-soak/: an end-to-end verification harness
that runs the fix against agent v7.77.1 over a real UDS socket. Scenarios
B.2/B.3 (stale-conn injection) show baseline dropping ~250+ traces per run;
patched delivers 0 dropped across 1.5M+ spans. B.4 (hung agent / timeout)
is a boundary scenario that both baseline and patched drop, documenting the
fix targets connection teardown, not agent unresponsiveness.

## Summary

- Make trace/stats POSTs replayable so `net/http` can transparently recover from idle UDS connections silently dropped by the agent: set `req.GetBody` (custom payload type that `http.NewRequest` doesn't auto-detect) and an `Idempotency-Key` header on both `httpTransport.send` and `httpTransport.sendStats`.
- Add a small `doWithStaleConnRetry` (up to 3 extra attempts (4 total)) on `EPIPE` / `ECONNRESET` / `net.ErrClosed` to cover the residual mid-write window where stdlib refuses to retry — because once any byte hits the wire, the error is no longer classified as `nothingWrittenError` (see [golang/go#19943](golang/go#19943)).
- Fix a Go-precedence bug at `writer.go:178` — `attempt+1%5 == 0` parsed as `attempt + 1`, so the periodic "failure sending traces" log never fired.
- New verification harness under `internal/apps/staleidle-soak/` that runs the fix end-to-end against the customer's exact agent version (`datadog/agent:7.77.1`) over a real UDS socket.

## Motivation

Following up on #4484, ([APMS-19533](https://datadoghq.atlassian.net/browse/APMS-19533)) still saw transport errors after upgrading to `v2.8.2`:

```
v2.8.2 ERROR: lost 2 traces: ... write unix @->/var/run/datadog/apm.socket: write: broken pipe
v2.8.2 ERROR: Error sending stats payload: ... read unix @->/var/run/datadog/apm.socket: read: connection reset by peer
```

PR #4484 added retry loops to `writer.go` and `stats.go`, but both are gated on `sendRetries` (default `0`). Customers who don't explicitly opt in get the loop budget of 1 attempt — i.e. no retry — so any stale-idle race drops a payload.

The fundamental issue: stdlib won't auto-replay a POST after a transport error unless it can both rewind the body and trust the call is idempotent. Without `GetBody` (custom payload) and without `Idempotency-Key`, `net/http.Request.isReplayable` returns `false` and stdlib silently swallows the recovery path.

## Why two layers

Empirically (see `TestUDSTransportRecoversFromStaleIdleConn`), `Idempotency-Key + GetBody` alone fixes ~80% of the failures. The remaining ~20% are `write: broken pipe` mid-request: stdlib sees `pc.nwrite > startBytesWritten` and won't classify it as `nothingWrittenError`, so it refuses to retry even though `isReplayable()` would now return true. The application-level retry covers that residual window.

Retrying is safe: the agent dedupes traces by span ID and ignores `Idempotency-Key`, so a duplicate payload is harmless. We only retry on the narrow set of errors that unambiguously signal a peer-side connection teardown.

## Test plan

### Layer A — unit test against synthetic close-every-conn server

- [x] `TestUDSTransportRecoversFromStaleIdleConn` (new) — UDS server that abruptly closes conns after responding (no `Connection: close` header), 400 concurrent POSTs across `send` + `sendStats`. On `main`: 30–45% of requests fail with the exact error strings the customer reports. With this PR: 10/10 runs green.
- [x] `go test -race -run "^(TestStatsFlushRetries|TestTraceWriter|TestApi|TestUDS|TestTraceCountHeader|TestDefaultHeaders|TestExternalEnvironment|TestFetchAgentFeaturesContainerTagsHash|TestWithUDS|TestWithHTTPClient|TestResolveAgentAddr|TestTransportResponse)$" ./ddtrace/tracer/` — all green.
- [x] `go build ./ddtrace/tracer/...` + `go vet ./ddtrace/tracer/...` — clean.

### Layer B — end-to-end harness against `datadog/agent:7.77.1` over real UDS

New under `internal/apps/staleidle-soak/`: a docker-compose'd agent pinned to the customer's version, a Go load driver, a UDS pass-through proxy that injects failure, and an orchestrator script that runs four scenarios on both branches for A/B comparison. B.1–B.3 are pass/fail; B.4 is a **boundary** scenario that documents the edge of what the fix covers.

| Scenario  | Side     | spans_created | flush_traces | traces_dropped | lost_trace_log |
|-----------|----------|---------------|--------------|----------------|----------------|
| B.1 healthy steady-state  | baseline | 299 980  | 299 980  | 0   | 0 |
| B.1 healthy steady-state  | **patched** | 299 979  | 299 979  | **0**   | **0** |
| B.2 stale-conn (60s)      | baseline | 300 000  | 299 700  | 300 | 1 |
| B.2 stale-conn (60s)      | **patched** | 299 973  | 299 973  | **0**   | **0** |
| B.3 stale-conn (5 min)    | baseline | 1 499 439 | 1 499 189 | 250 | 1 |
| B.3 stale-conn (5 min)    | **patched** | 1 499 951 | 1 499 951 | **0**   | **0** |
| B.4 **hung agent** (60s)  | baseline | 299 957 | 0 | 299 957 | 2 |
| B.4 **hung agent** (60s)  | patched | 299 950 | 0 | 299 950 | 2 |

**B.2 / B.3** inject the customer's failure mode — the agent silently closes idle keep-alive UDS conns (proxy closes each conn after one response). They reproduce the customer's exact log shape:

```
Datadog Tracer v2.10.0-dev ERROR: lost 250 traces: Post "http://UDS__tmp_staleidle-proxy_apm.socket/v0.4/traces": EOF
```

With the patch applied, every span across 1.5 M+ creations on the longest soak is delivered (`spans_created == flush_traces`), with zero error-log lines and zero `api_errors`.

**B.4** is deliberately *not* a pass/fail test. The proxy (`--mode hang`) accepts the request but never responds, simulating a hung/unresponsive agent. The tracer times out (`context deadline exceeded`) — a non-teardown error the fix intentionally does **not** retry (retrying against a wedged agent only stacks load). B.4 therefore drops on *both* baseline and patched, which is the point: it documents that the fix targets connection teardown (the customer's observed error family) and **not** agent unresponsiveness. The customer's reported errors (APMS-19533) are all teardown; none are timeouts.

The harness is checked in so anyone can re-run the same verification (see `internal/apps/staleidle-soak/README.md`).

## Refs

- APMS-19533
- #4484

[APMS-19533]: https://datadoghq.atlassian.net/browse/APMS-19533?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ



Co-authored-by: kemal.akkoyun <[email protected]>