fix: enforce span limit in curl_multi_exec and DDTrace\start_span code paths#3691
Conversation
DD_TRACE_SPANS_LIMIT was correctly enforced in the hook system (uhook.c, uhook_legacy.c, uhook_attributes.c) but bypassed in three other code paths, causing unbounded span creation and OOM when curl_multi_exec is called repeatedly (e.g. AWS SDK SQS workloads). Missing checks: - dd_multi_inject_headers() (ext/handlers_curl.c): PHP 8 curl multi creates one INTERNAL_SPAN per handle without checking the limit. This is the primary path causing the reported customer OOM. - dd_inject_distributed_tracing_headers_multi() (ext/handlers_curl_php7.c): same issue on PHP 7. - dd_start_span() / DDTrace_start_trace_span() (ext/ddtrace.c): PHP-level DDTrace\start_span() and DDTrace\start_trace_span() APIs open real spans without checking the limit. Fix: add !ddtrace_tracer_is_limited() guards to each path. The curl multi paths fall through to header injection only (no span) when limited, preserving distributed tracing propagation. The PHP-level APIs return a dummy span (matching the disabled-tracing behaviour). Fixes: APMS-18744
|
✨ Fix all issues with BitsAI or with Cursor
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3691 +/- ##
==========================================
- Coverage 62.17% 62.08% -0.09%
==========================================
Files 141 141
Lines 13352 13352
Branches 1746 1746
==========================================
- Hits 8302 8290 -12
- Misses 4259 4269 +10
- Partials 791 793 +2 see 2 files with indirect coverage changes Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
DDTrace\start_span() and DDTrace\start_trace_span() are user-facing APIs that intentionally bypass the span limit — the limit applies to auto-instrumentation (hooks) only. Guarding these APIs with ddtrace_tracer_is_limited() broke: - testTracerFlushedWhenSpanLimitExceeded: the test explicitly verifies that user-created spans work even when DD_TRACE_SPANS_LIMIT is hit. - Guzzle integration tests: isolateTracer() uses start_trace_span() internally to create isolated trace contexts; when closed_spans_count accumulated past the limit across PHPUnit tests, no new stack was created and all spans in the isolated test were lost. The curl multi code paths in handlers_curl.c and handlers_curl_php7.c remain fixed (the actual OOM culprit reported in APMS-18744).
bwoebi
left a comment
There was a problem hiding this comment.
I think we should do this, yes.
Benchmarks [ tracer ]Benchmark execution time: 2026-03-06 14:31:49 Comparing candidate commit bf6cd22 in PR branch Found 2 performance improvements and 0 performance regressions! Performance is the same for 190 metrics, 2 unstable metrics. scenario:MessagePackSerializationBench/benchMessagePackSerialization
scenario:MessagePackSerializationBench/benchMessagePackSerialization-opcache
|
…ans (APMS-19944) (#4030) * fix(curl): enforce DD_TRACE_SPANS_LIMIT for curl_multi_exec parent spans (APMS-19944) The curl_multi_exec integration opens its parent span via the user-facing DDTrace\start_span(), which intentionally bypasses DD_TRACE_SPANS_LIMIT (see #3691). Distributed-header injection then flags that span NOT_DROPPABLE, so the post-hook's try_drop_span() can never reclaim it. In long-running CLI workloads that repeatedly call curl_multi_exec (e.g. AWS SDK SQS batches), one un-droppable span accumulates per call, growing memory 1:1 with calls until OOM. Skip creating the parent span when the tracer is already limited. Distributed tracing headers are still injected by the C handler against the active span, so propagation is preserved, and the child request spans are already limited. Fixes: APMS-19944 * test(curl): drop flaky is_limited assertion from span-limit test dd_trace_tracer_is_limited() reflects the open+closed span counters at the moment of the call, which depends on end-of-run flush/cleanup timing and varies across CI environments and PHP versions (7.3/7.4/8.2-ASAN all reported bool(false) at script end). The span-count assertion (BOUNDED) is the real regression guard: with the fix the retained curl_multi_exec spans stay capped near DD_TRACE_SPANS_LIMIT, without it they scale 1:1 with iterations. Bump to 100 iterations for a clearer margin and drop the fragile check. * test(curl): remove comments from span-limit test * fix(curl): check span limit at top of curl_multi_exec hook Address review: move the ddtrace_tracer_is_limited() guard to the start of the begin hook so the limited path returns immediately, skipping the args lookup and ObjectKVStore reuse check. Spans for a multi-op are only created on the first curl_multi_exec call, so a new handle has nothing to reuse when limited; the boundary case (parent created just before the limit) is still finalized by CurlSpanInfo::__destruct().
Summary
DD_TRACE_SPANS_LIMITwas correctly enforced in the hook system (uhook.c,uhook_legacy.c,uhook_attributes.c) but bypassed in thecurl_multi_execcode paths, causing unbounded span creation and OOM when curl multi is called repeatedly (e.g. AWS SDK SQS workloads on ECS Fargate).Fixes APMS-18744.
Root cause
dd_multi_inject_headers()(ext/handlers_curl.c): PHP 8 curl multi creates oneDDTRACE_INTERNAL_SPANper handle on everycurl_multi_execcall without checking the limit. With ~100k SQS calls (each using curl multi internally), this produced 100k+ unconstrained spans — the primary cause of the reported OOM.dd_inject_distributed_tracing_headers_multi()(ext/handlers_curl_php7.c): same issue on PHP 7.Fix
Add
!ddtrace_tracer_is_limited()guards to both curl multi paths. When the span limit is reached, the paths fall through to header injection only (no span created) — distributed tracing propagation is preserved, only span tracking is skipped.