Skip to content

Fix #3135: Force flushing on shutdown of entry point processes#3398

Merged
bwoebi merged 2 commits into
masterfrom
bob/sync-force-flush
Sep 29, 2025
Merged

Fix #3135: Force flushing on shutdown of entry point processes#3398
bwoebi merged 2 commits into
masterfrom
bob/sync-force-flush

Conversation

@bwoebi

@bwoebi bwoebi commented Sep 1, 2025

Copy link
Copy Markdown
Collaborator

Cgroup roots as well as container root processes generally cause the container to terminate on end. This currently drops all pending traces on the floor.

Enforce a synchronous flush here, by default for PID=1 and PPID=1.

@bwoebi
bwoebi requested a review from a team as a code owner September 1, 2025 12:47
@codecov-commenter

codecov-commenter commented Sep 1, 2025

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.74%. Comparing base (4fb6495) to head (31e13fc).
⚠️ Report is 391 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3398      +/-   ##
==========================================
- Coverage   61.87%   61.74%   -0.13%     
==========================================
  Files         141      141              
  Lines       12481    12481              
  Branches     1630     1630              
==========================================
- Hits         7722     7706      -16     
- Misses       4038     4054      +16     
  Partials      721      721              

see 3 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4fb6495...31e13fc. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@pr-commenter

pr-commenter Bot commented Sep 1, 2025

Copy link
Copy Markdown

Benchmarks [ tracer ]

Benchmark execution time: 2025-09-22 15:40:53

Comparing candidate commit 31e13fc in PR branch bob/sync-force-flush with baseline commit 4fb6495 in branch master.

Found 1 performance improvements and 7 performance regressions! Performance is the same for 185 metrics, 1 unstable metrics.

scenario:MessagePackSerializationBench/benchMessagePackSerialization

  • 🟥 execution_time [+2.729µs; +4.231µs] or [+2.487%; +3.858%]

scenario:PDOBench/benchPDOOverhead

  • 🟥 execution_time [+8.539µs; +11.714µs] or [+3.230%; +4.431%]

scenario:PDOBench/benchPDOOverheadWithDBM

  • 🟥 execution_time [+7.746µs; +13.143µs] or [+2.952%; +5.009%]

scenario:PHPRedisBench/benchRedisOverhead

  • 🟥 execution_time [+33.139µs; +82.452µs] or [+3.227%; +8.029%]

scenario:SamplingRuleMatchingBench/benchRegexMatching1-opcache

  • 🟩 execution_time [-5.396µs; -5.231µs] or [-80.213%; -77.749%]

scenario:SamplingRuleMatchingBench/benchRegexMatching2-opcache

  • 🟥 execution_time [+158.166ns; +348.434ns] or [+2.355%; +5.188%]

scenario:SamplingRuleMatchingBench/benchRegexMatching3-opcache

  • 🟥 execution_time [+239.462ns; +393.538ns] or [+3.610%; +5.933%]

scenario:SamplingRuleMatchingBench/benchRegexMatching4-opcache

  • 🟥 execution_time [+202.804ns; +394.196ns] or [+3.030%; +5.890%]

@bwoebi
bwoebi force-pushed the bob/sync-force-flush branch 3 times, most recently from 1edd421 to 4d81d94 Compare September 4, 2025 18:23
@bwoebi
bwoebi requested review from a team as code owners September 4, 2025 18:23
@bwoebi
bwoebi force-pushed the bob/sync-force-flush branch from 4d81d94 to 0edd29d Compare September 5, 2025 12:59
Also avoid retrying a trace flush, that's pretty much pointless.

Signed-off-by: Bob Weinand <[email protected]>
Signed-off-by: Bob Weinand <[email protected]>
@bwoebi
bwoebi force-pushed the bob/sync-force-flush branch from 0edd29d to 31e13fc Compare September 22, 2025 14:35
Comment thread ext/signals.c
dd_signal_data.uc = uc;

if (ddtrace_sidecar) {
// Spawn a thread using clone() to perform sidecar cleanup asynchronously to avoid async unsafeness in the signal handler

@morrisonlevi morrisonlevi Sep 25, 2025

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand. We're supposed to be forcing a flush because the entry point (or a child of it) is shutting down. But this makes a thread... why would this have a realistic chance to complete before the process exits?

Presumably the profiler should so something similar, but it's less dire (only the most recent partial minute is lost, probably).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the thread continues executing and regularly shuts down - there's not a problem. (The signal handler resuming basically just continues normal execution.)
All this does is slightly deferring the signal delivery by the time it needs to flush the traces, essentially. dd_call_prev_handler only calls the prev handler after flushing.

Does that make sense?

@bwoebi
bwoebi merged commit 7ab3820 into master Sep 29, 2025
1863 of 1870 checks passed
@bwoebi
bwoebi deleted the bob/sync-force-flush branch September 29, 2025 18:30
@github-actions github-actions Bot added this to the 1.13.0 milestone Sep 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants