Skip to content

[NO-TICKET] Allow lowering profiling cpu sampling interval#5424

Merged
ivoanjo merged 2 commits into
masterfrom
ivoanjo/configurable-sampling-interval
Mar 11, 2026
Merged

[NO-TICKET] Allow lowering profiling cpu sampling interval#5424
ivoanjo merged 2 commits into
masterfrom
ivoanjo/configurable-sampling-interval

Conversation

@ivoanjo

@ivoanjo ivoanjo commented Mar 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

What does this PR do?

This PR adds an experimental setting experimental_cpu_sampling_interval_ms to the profiler's advanced configuration that allows lowering the CPU sampling interval from the default 10ms down to 1ms.

Motivation:

Fixes #5393

Some users need higher resolution CPU profiling data and are willing to accept increased overhead in exchange for more accurate sampling.

Change log entry

Yes. Allow lowering profiling cpu sampling interval

Additional Notes:

N/A

How to test the change?

I've added specs to cover the change, although the actual sampling rate is not tested integration-style. I somewhat hesitated in adding that since it can be a bit flaky if e.g. the profiler can't sample fast enough in CI.

@ivoanjo
[NO-TICKET] Allow profiling cpu sampling interval to be lowered
4c842b2 
**What does this PR do?**

This PR adds an experimental setting `experimental_cpu_sampling_interval_ms`
to the profiler's advanced configuration that allows lowering the CPU
sampling interval from the default 10ms down to 1ms.

**Motivation:**

Fixes #5393

Some users need higher resolution CPU profiling data and are willing to
accept increased overhead in exchange for more accurate sampling.

**Additional Notes:**

N/A

**How to test the change?**

I've added specs to cover the change, although the actual sampling
rate is not tested integration-style. I somewhat hesitated in adding
that since it can be a bit flaky if e.g. the profiler can't sample
fast enough in CI.
@ivoanjo
ivoanjo requested review from a team as code owners March 6, 2026 11:24
@github-actions github-actions Bot added core Involves Datadog core libraries profiling Involves Datadog profiling labels Mar 6, 2026
@github-actions

github-actions Bot commented Mar 6, 2026 •
edited
Loading

Copy link
Copy Markdown

Typing analysis

Note: Ignored files are excluded from the next sections.

Untyped methods

This PR introduces 6 partially typed methods, and clears 6 partially typed methods. It increases the percentage of typed methods from 61.1% to 61.12% (+0.02%).

Partially typed methods (+6-6) ❌ Introduced: 
sig/datadog/profiling/collectors/cpu_and_wall_time_worker.rbs:51
└── def stats: () -> ::Hash[::Symbol, untyped]
sig/datadog/profiling/collectors/cpu_and_wall_time_worker.rbs:52
└── def stats_and_reset_not_thread_safe: () -> ::Hash[::Symbol, untyped]
sig/datadog/profiling/collectors/cpu_and_wall_time_worker.rbs:53
└── def self._native_stats: (CpuAndWallTimeWorker self_instance) -> ::Hash[::Symbol, untyped]
sig/datadog/profiling/component.rbs:48
└── def self.dir_interruption_workaround_enabled?: (untyped settings, bool no_signals_workaround_enabled) -> bool
sig/datadog/profiling/component.rbs:49
└── def self.can_apply_exec_monkey_patch?: (untyped settings) -> bool
sig/datadog/profiling/component.rbs:50
└── def self.enable_gvl_profiling?: (untyped settings, Datadog::Core::Logger logger) -> bool
✅ Cleared: 
sig/datadog/profiling/collectors/cpu_and_wall_time_worker.rbs:49
└── def stats: () -> ::Hash[::Symbol, untyped]
sig/datadog/profiling/collectors/cpu_and_wall_time_worker.rbs:50
└── def stats_and_reset_not_thread_safe: () -> ::Hash[::Symbol, untyped]
sig/datadog/profiling/collectors/cpu_and_wall_time_worker.rbs:51
└── def self._native_stats: (CpuAndWallTimeWorker self_instance) -> ::Hash[::Symbol, untyped]
sig/datadog/profiling/component.rbs:47
└── def self.dir_interruption_workaround_enabled?: (untyped settings, bool no_signals_workaround_enabled) -> bool
sig/datadog/profiling/component.rbs:48
└── def self.can_apply_exec_monkey_patch?: (untyped settings) -> bool
sig/datadog/profiling/component.rbs:49
└── def self.enable_gvl_profiling?: (untyped settings, Datadog::Core::Logger logger) -> bool

If you believe a method or an attribute is rightfully untyped or partially typed, you can add # untyped:accept on the line before the definition to remove it from the stats.

@ivoanjo ivoanjo mentioned this pull request Mar 6, 2026
Closed
@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Mar 6, 2026 •
edited by datadog-prod-us1-3 Bot
Loading

Copy link
Copy Markdown
Contributor

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
• Patch Coverage: 92.98%
• Overall Coverage: 95.12%

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a163de4 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!

@pr-commenter

pr-commenter Bot commented Mar 6, 2026 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-03-10 15:55:42

Comparing candidate commit a163de4 in PR branch ivoanjo/configurable-sampling-interval with baseline commit 125a060 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 46 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

@ivoanjo

ivoanjo commented Mar 11, 2026

Copy link
Copy Markdown
Member Author

We might iterate a bit on this one still based of feedback, but for now merging this first version :)

@ivoanjo
ivoanjo merged commit f84399c into master Mar 11, 2026
631 checks passed
@ivoanjo
ivoanjo deleted the ivoanjo/configurable-sampling-interval branch March 11, 2026 12:02
@github-actions github-actions Bot added this to the 2.30.0 milestone Mar 11, 2026
@ivoanjo ivoanjo changed the title [NO-TICKET] Allow profiling cpu sampling interval to be lowered [NO-TICKET] Allow lowering profiling cpu sampling interval Mar 11, 2026
@ivoanjo

ivoanjo commented Mar 11, 2026

Copy link
Copy Markdown
Member Author

(Last-minute tweaking to name, the old one sounded clunky; also updated change log entry to match)

@Strech Strech mentioned this pull request Mar 19, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Strech Strech Strech approved these changes

Assignees

No one assigned

Labels

core Involves Datadog core libraries profiling Involves Datadog profiling

Projects

None yet

Milestone

2.30.0

Development

Successfully merging this pull request may close these issues.

[FEATURE]: Make profiling interval configurable

2 participants

@ivoanjo @Strech