fix(profiler): decrement refcount given strong reference of frame object in python 3.11 by Yun-Kim · Pull Request #4901 · DataDog/dd-trace-py

Yun-Kim · 2023-01-13T15:58:56Z

Description

#4125 added the use of a getter function PyThread_GetFrame() to provide compatibility for Python 3.11 in the profiler (specifically the stack collector). However, PyThread_GetFrame() returns a strong reference to a PyFrameObject and its reference count was not decremented in that change. This means that the frame object will not be garbage collected and likely the cause behind the memory leak mentioned in #4899.

Checklist

Followed the library release note guidelines when writing a release note.
Add additional sections for feat and fix pull requests.
Library documentation and/or Datadog's documentation site is updated. Link to doc PR in description.

Relevant issue(s)

Fixes #4899

Testing strategy

Since this change ensures that all FrameType objects in the stack collector have their reference count properly decremented, a regression test was added to ensure that all FrameType objects from the stack collector are garbage collected. test_collect_ensure_all_frames_gc runs an arbitrary function _foo() multiple times, then makes sure that garbage collection has occurred. Finally, the test checks that all FrameType objects relating to that function _foo() have been collected by the garbage collector, or in other words are not present in the garbage collector.

Checking the garbage collector was the most direct way to test that this change indeed does avoid leaking any memory due to FrameType objects in the stack collector.

Reviewer Checklist

Title is accurate.
Description motivates each change.
No unnecessary changes were introduced in this PR.
Avoid breaking API changes unless absolutely necessary.
Tests provided or description of manual testing performed is included in the code or PR.
Release note has been added and follows the library release note guidelines, or else changelog/no-changelog label added.
All relevant GitHub issues are correctly linked.
Backports are identified and tagged with Mergifyio.

P403n1x87 · 2023-01-14T00:05:07Z

@Yun-Kim can we perhaps add a test case where we assert that

import gc
from types import FrameType

assert all(not isinstance(_, FrameType) for _ in gc.get_objects())

?

Yun-Kim · 2023-01-14T00:09:11Z

can we perhaps add a test case where we assert that
import gc
from types import FrameType

assert all(not isinstance(_, FrameType) for _ in gc.get_objects())
?

@P403n1x87 I think we can do that in #4895 instead of this PR if the pypi build error (with Project being found when we should expect a PyFrameObject) is what you're referencing.

P403n1x87 · 2023-01-16T15:56:29Z

@Yun-Kim I think it might be worth adding the test case here too. This is the PR that fixes the frame object leaks so it would be nice if we could add the test case that would catch any potential regressions.

brettlangdon · 2023-01-17T23:26:28Z

@Yun-Kim can you fill in the "testing strategy" details of the PR description? it would be good to have some human words describing how we decided to test it this way / are there other ways or ways that cannot be automated?

P403n1x87

👍 👍

…ion in Python 3.11 (#4937) ## Description #4901 added a fix to properly decrement frametype objects in Python 3.11 in our stack collector, with a regression test. We found that this regression test does not properly assert that frametype objects were all garbage collected, which this test fixes. Specifically, the previous change was looking for only frame objects related to a function call `_foo()`. However, after reverting the fix from #4901 this test still passed, because all frametype objects have been garbage collected. This change makes a broader assertion instead that all frametype objects have been garbage collected, which does expectedly fail if the fix from #4901 is reverted.  ## Checklist - [ ] Followed the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines) when writing a release note. - [ ] Add additional sections for `feat` and `fix` pull requests. - [ ] [Library documentation](https://github.com/DataDog/dd-trace-py/tree/1.x/docs) and/or [Datadog's documentation site](https://github.com/DataDog/documentation/) is updated. Link to doc PR in description.   ## Motivation  ## Design  ## Testing strategy   ## Relevant issue(s)  ## Testing strategy   ## Reviewer Checklist - [ ] Title is accurate. - [ ] Description motivates each change. - [ ] No unnecessary changes were introduced in this PR. - [ ] Avoid breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes unless absolutely necessary. - [ ] Tests provided or description of manual testing performed is included in the code or PR. - [ ] Release note has been added and follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines), or else `changelog/no-changelog` label added. - [ ] All relevant GitHub issues are correctly linked. - [ ] Change contains telemetry where appropriate (logs, metrics, etc.). - [ ] Telemetry is meaningful, actionable and does not have the potential to leak sensitive data. Co-authored-by: Brett Langdon <[email protected]>

…ect in python 3.11 (DataDog#4901) ## Description DataDog#4125 added the use of a getter function `PyThread_GetFrame()` to provide compatibility for Python 3.11 in the profiler (specifically the stack collector). However, `PyThread_GetFrame()` returns a strong reference to a `PyFrameObject` and its reference count was not decremented in that change. This means that the frame object will not be garbage collected and likely the cause behind the memory leak mentioned in DataDog#4899. ## Checklist - [ ] Followed the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines) when writing a release note. - [ ] Add additional sections for `feat` and `fix` pull requests. - [ ] [Library documentation](https://github.com/DataDog/dd-trace-py/tree/1.x/docs) and/or [Datadog's documentation site](https://github.com/DataDog/documentation/) is updated. Link to doc PR in description.   ## Motivation  ## Design  ## Testing strategy   ## Relevant issue(s)  ## Testing strategy   ## Reviewer Checklist - [ ] Title is accurate. - [ ] Description motivates each change. - [ ] No unnecessary changes were introduced in this PR. - [ ] Avoid breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes unless absolutely necessary. - [ ] Tests provided or description of manual testing performed is included in the code or PR. - [ ] Release note has been added and follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines), or else `changelog/no-changelog` label added. - [ ] All relevant GitHub issues are correctly linked. - [ ] Backports are identified and tagged with Mergifyio. Co-authored-by: Brett Langdon <[email protected]>

…ion in Python 3.11 (DataDog#4937) ## Description DataDog#4901 added a fix to properly decrement frametype objects in Python 3.11 in our stack collector, with a regression test. We found that this regression test does not properly assert that frametype objects were all garbage collected, which this test fixes. Specifically, the previous change was looking for only frame objects related to a function call `_foo()`. However, after reverting the fix from DataDog#4901 this test still passed, because all frametype objects have been garbage collected. This change makes a broader assertion instead that all frametype objects have been garbage collected, which does expectedly fail if the fix from DataDog#4901 is reverted.  ## Checklist - [ ] Followed the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines) when writing a release note. - [ ] Add additional sections for `feat` and `fix` pull requests. - [ ] [Library documentation](https://github.com/DataDog/dd-trace-py/tree/1.x/docs) and/or [Datadog's documentation site](https://github.com/DataDog/documentation/) is updated. Link to doc PR in description.   ## Motivation  ## Design  ## Testing strategy   ## Relevant issue(s)  ## Testing strategy   ## Reviewer Checklist - [ ] Title is accurate. - [ ] Description motivates each change. - [ ] No unnecessary changes were introduced in this PR. - [ ] Avoid breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes unless absolutely necessary. - [ ] Tests provided or description of manual testing performed is included in the code or PR. - [ ] Release note has been added and follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines), or else `changelog/no-changelog` label added. - [ ] All relevant GitHub issues are correctly linked. - [ ] Change contains telemetry where appropriate (logs, metrics, etc.). - [ ] Telemetry is meaningful, actionable and does not have the potential to leak sensitive data. Co-authored-by: Brett Langdon <[email protected]>

…ect in python 3.11 [backport] (#4951) ## Description Backports for #4901 and #4937 to 1.7.  ## Checklist - [ ] Followed the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines) when writing a release note. - [ ] Add additional sections for `feat` and `fix` pull requests. - [ ] [Library documentation](https://github.com/DataDog/dd-trace-py/tree/1.x/docs) and/or [Datadog's documentation site](https://github.com/DataDog/documentation/) is updated. Link to doc PR in description.   ## Motivation  ## Design  ## Testing strategy   ## Relevant issue(s)  ## Testing strategy   ## Reviewer Checklist - [ ] Title is accurate. - [ ] Description motivates each change. - [ ] No unnecessary changes were introduced in this PR. - [ ] Avoid breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes unless absolutely necessary. - [ ] Tests provided or description of manual testing performed is included in the code or PR. - [ ] Release note has been added and follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines), or else `changelog/no-changelog` label added. - [ ] All relevant GitHub issues are correctly linked. - [ ] Change contains telemetry where appropriate (logs, metrics, etc.). - [ ] Telemetry is meaningful, actionable and does not have the potential to leak sensitive data. Co-authored-by: Brett Langdon <[email protected]>

…ect in python 3.11 (#4901) ## Description #4125 added the use of a getter function `PyThread_GetFrame()` to provide compatibility for Python 3.11 in the profiler (specifically the stack collector). However, `PyThread_GetFrame()` returns a strong reference to a `PyFrameObject` and its reference count was not decremented in that change. This means that the frame object will not be garbage collected and likely the cause behind the memory leak mentioned in #4899. ## Checklist - [ ] Followed the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines) when writing a release note. - [ ] Add additional sections for `feat` and `fix` pull requests. - [ ] [Library documentation](https://github.com/DataDog/dd-trace-py/tree/1.x/docs) and/or [Datadog's documentation site](https://github.com/DataDog/documentation/) is updated. Link to doc PR in description.   ## Motivation  ## Design  ## Testing strategy   ## Relevant issue(s)  ## Testing strategy   ## Reviewer Checklist - [ ] Title is accurate. - [ ] Description motivates each change. - [ ] No unnecessary changes were introduced in this PR. - [ ] Avoid breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes unless absolutely necessary. - [ ] Tests provided or description of manual testing performed is included in the code or PR. - [ ] Release note has been added and follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines), or else `changelog/no-changelog` label added. - [ ] All relevant GitHub issues are correctly linked. - [ ] Backports are identified and tagged with Mergifyio. Co-authored-by: Brett Langdon <[email protected]>

…ion in Python 3.11 (#4937) ## Description #4901 added a fix to properly decrement frametype objects in Python 3.11 in our stack collector, with a regression test. We found that this regression test does not properly assert that frametype objects were all garbage collected, which this test fixes. Specifically, the previous change was looking for only frame objects related to a function call `_foo()`. However, after reverting the fix from #4901 this test still passed, because all frametype objects have been garbage collected. This change makes a broader assertion instead that all frametype objects have been garbage collected, which does expectedly fail if the fix from #4901 is reverted.  ## Checklist - [ ] Followed the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines) when writing a release note. - [ ] Add additional sections for `feat` and `fix` pull requests. - [ ] [Library documentation](https://github.com/DataDog/dd-trace-py/tree/1.x/docs) and/or [Datadog's documentation site](https://github.com/DataDog/documentation/) is updated. Link to doc PR in description.   ## Motivation  ## Design  ## Testing strategy   ## Relevant issue(s)  ## Testing strategy   ## Reviewer Checklist - [ ] Title is accurate. - [ ] Description motivates each change. - [ ] No unnecessary changes were introduced in this PR. - [ ] Avoid breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes unless absolutely necessary. - [ ] Tests provided or description of manual testing performed is included in the code or PR. - [ ] Release note has been added and follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines), or else `changelog/no-changelog` label added. - [ ] All relevant GitHub issues are correctly linked. - [ ] Change contains telemetry where appropriate (logs, metrics, etc.). - [ ] Telemetry is meaningful, actionable and does not have the potential to leak sensitive data. Co-authored-by: Brett Langdon <[email protected]>

Decrement reference count for strong reference of frame object

a8a3476

P403n1x87 reviewed Jan 13, 2023

View reviewed changes

Comment thread ddtrace/profiling/collector/stack.pyx

P403n1x87 previously approved these changes Jan 13, 2023

View reviewed changes

Add regression test for lazy PyFrameObjects in Python 3.11+

2b56056

Yun-Kim dismissed P403n1x87’s stale review via 2b56056 January 17, 2023 15:01

Yun-Kim marked this pull request as ready for review January 17, 2023 15:06

Yun-Kim requested a review from a team as a code owner January 17, 2023 15:06

P403n1x87 reviewed Jan 17, 2023

View reviewed changes

Comment thread tests/profiling/collector/test_stack.py Outdated

Yun-Kim and others added 5 commits January 17, 2023 10:46

Specific assertion on regression test

96ba7eb

Fix skip statement

fc930a0

Fix line numbers of failing tests due to added imports

152b061

Merge branch '1.x' into yunkim/fix-profiler-311-memory-leak

9563511

Merge branch '1.x' into yunkim/fix-profiler-311-memory-leak

b984b99

brettlangdon enabled auto-merge (squash) January 17, 2023 23:25

P403n1x87 approved these changes Jan 18, 2023

View reviewed changes

brettlangdon approved these changes Jan 18, 2023

View reviewed changes

Merge branch '1.x' into yunkim/fix-profiler-311-memory-leak

0cf1580

brettlangdon merged commit a01c18f into DataDog:1.x Jan 18, 2023

Yun-Kim mentioned this pull request Jan 18, 2023

chore(profiler): fix regression test for frame object garbage collection in Python 3.11 #4937

Merged

12 tasks

Yun-Kim deleted the yunkim/fix-profiler-311-memory-leak branch January 19, 2023 20:40

Yun-Kim mentioned this pull request Jan 23, 2023

fix(profiler): decrement refcount given strong reference of frame object in python 3.11 [backport] #4951

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(profiler): decrement refcount given strong reference of frame object in python 3.11 #4901

fix(profiler): decrement refcount given strong reference of frame object in python 3.11 #4901
brettlangdon merged 8 commits into
DataDog:1.xfrom
Yun-Kim:yunkim/fix-profiler-311-memory-leak

Yun-Kim commented Jan 13, 2023 •

edited by brettlangdon

Loading

Uh oh!

Uh oh!

P403n1x87 commented Jan 14, 2023

Uh oh!

Yun-Kim commented Jan 14, 2023 •

edited

Loading

Uh oh!

P403n1x87 commented Jan 16, 2023

Uh oh!

Uh oh!

brettlangdon commented Jan 17, 2023

Uh oh!

P403n1x87 left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Yun-Kim commented Jan 13, 2023 • edited by brettlangdon Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Relevant issue(s)

Testing strategy

Reviewer Checklist

Uh oh!

Uh oh!

P403n1x87 commented Jan 14, 2023

Uh oh!

Yun-Kim commented Jan 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

P403n1x87 commented Jan 16, 2023

Uh oh!

Uh oh!

brettlangdon commented Jan 17, 2023

Uh oh!

P403n1x87 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Yun-Kim commented Jan 13, 2023 •

edited by brettlangdon

Loading

Yun-Kim commented Jan 14, 2023 •

edited

Loading