Skip to content

Backport #3197 to 1.x-stable: Shut down profiler if any components failed#3216

Merged
ivoanjo merged 5 commits into
1.x-stablefrom
ivoanjo/backport-improve-profiler-error-handling
Oct 27, 2023
Merged

Backport #3197 to 1.x-stable: Shut down profiler if any components failed#3216
ivoanjo merged 5 commits into
1.x-stablefrom
ivoanjo/backport-improve-profiler-error-handling

Conversation

@ivoanjo

@ivoanjo ivoanjo commented Oct 24, 2023

Copy link
Copy Markdown
Member

This is a backport of #3197 to the 1.x-stable branch.

I merely rebased the commits on top of the other branch, with no other changes.

**What does this PR do?**

This PR changes the profiler `Collectors::CpuAndWallTimeWorker` and
`Scheduler` components to signal the top `Profiler` instance when they
failed, so the `Profiler` can stop the other component as well.

**Motivation:**

While never reported by a customer, it ocurred to me that right now if
one of the component fails, the other will keep working in a weird
state.

Of particular concern, if the scheduler failed, the `StackRecorder`
would trigger a memory leak: it would start accumulating samples and
never flush them.

This PR changes the profiler so that if one component stops with
a failure, the other will do so as well.

**Additional Notes:**

I'm not especially happy with how this change turned out.

1. The error handling in the scheduler somewhat duplicates similar
   logic in the `Core::Workers::Async` helper, which it uses.

   I wanted to avoid changing the shared helper just for this
   use-case, which I managed, but at the cost of a bit of
   duplication (including a duplicated log message, sigh).

2. I don't like the `on_failure_proc: nil`, but adding the
   argument would mean changing A LOT of tests, which is doable
   but has its own ugliness as well.

   I've considered passing the `on_failure_proc` via the constructor,
   but it's not entirely straightforward: the `Profiler` instance is
   created after the two others, so we'd need a way to break the loop.

In the end, I decided to move forward with something, as I think it's
important to plug this gap.

**How to test the change?**

This change includes test coverage.
I haven't yet gotten used to updating these as part of my usual
workflow :sweatsmile:.
@ivoanjo
ivoanjo requested review from a team as code owners October 24, 2023 16:45
@github-actions github-actions Bot added the profiling Involves Datadog profiling label Oct 24, 2023
@ivoanjo
ivoanjo merged commit f897407 into 1.x-stable Oct 27, 2023
@ivoanjo
ivoanjo deleted the ivoanjo/backport-improve-profiler-error-handling branch October 27, 2023 10:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

profiling Involves Datadog profiling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants