feat(tracer): add Python 3.11 support#4125
Merged
Merged
Conversation
…rarily commented out profiler compilation
…regarding python version alphas
P403n1x87
reviewed
Aug 26, 2022
P403n1x87
reviewed
Aug 26, 2022
P403n1x87
reviewed
Aug 26, 2022
Closed
Contributor
|
Just a note that the |
… depend on bytecode
Contributor
Author
As discussed, the |
Kyle-Verhoog
previously approved these changes
Oct 25, 2022
Kyle-Verhoog
approved these changes
Oct 26, 2022
majorgreys
approved these changes
Oct 26, 2022
Contributor
Author
|
@Mergifyio backport 1.6 |
Contributor
✅ Backports have been createdDetails
|
mergify Bot
pushed a commit
that referenced
this pull request
Oct 26, 2022
Python 3.11 has made some new changes to their C internal API that we need to address in a couple of places, mostly in our profiler. I've moved the profiler changes into #4343 and changed this PR to focus on the tracer and integrations supporting Python 3.11. Note that some of the profiler changes were kept in this PR to ensure that the profiler can compile with Python 3.11 per our setup/install rules. The major tracer-related fix is: Python 3.11 moved _PyFloat_Pack8() into their internal C API and replaced it with PyFloat_Pack8() to their public C API (Link to CPython issue). We use it once in ddtrace.internal.pack_template.h where I've added a Python version check to use the correct corresponding function. Note: grpcio with pytest-asyncio is causing Python segmentation faults on the test_unary_exception and test_unary_cancellation test cases, and from the traceback it doesn't look like dd-trace-py is involved. You can reproduce this using this gist which does not involve dd-trace-py. As a result I am skipping the grpcio_asyncio tests with Python 3.11. Co-authored-by: Kyle Verhoog <[email protected]> (cherry picked from commit c94cc14) # Conflicts: # tox.ini
mergify Bot
added a commit
that referenced
this pull request
Oct 27, 2022
This is an automatic backport of pull request #4125 done by [Mergify](https://mergify.com). <details> <summary>Mergify commands and options</summary> <br /> More conditions and actions can be found in the [documentation](https://docs.mergify.com/). You can also trigger Mergify actions by commenting on this pull request: - `@Mergifyio refresh` will re-evaluate the rules - `@Mergifyio rebase` will rebase this PR on its base branch - `@Mergifyio update` will merge the base branch into this PR - `@Mergifyio backport <destination>` will backport this PR on `<destination>` branch Additionally, on Mergify [dashboard](https://dashboard.mergify.com/) you can: - look at your merge queues - generate the Mergify configuration with the config editor. Finally, you can contact us on https://mergify.com </details>
mergify Bot
pushed a commit
that referenced
this pull request
Oct 28, 2022
## Description Blocked by #4125. This PR removes the skipping of building Python 3.11 wheels in our build_deploy job. ## Checklist - [x] Title must conform to [conventional commit](https://github.com/conventional-changelog/commitlint/tree/master/%40commitlint/config-conventional). - [x] Add additional sections for `feat` and `fix` pull requests. - [x] Ensure tests are passing for affected code. - [x] [Library documentation](https://github.com/DataDog/dd-trace-py/tree/1.x/docs) and/or [Datadog's documentation site](https://github.com/DataDog/documentation/) is updated. Link to doc PR in description. ## Motivation ## Design ## Testing strategy ## Relevant issue(s) ## Testing strategy ## Reviewer Checklist - [x] Title is accurate. - [x] Description motivates each change. - [x] No unnecessary changes were introduced in this PR. - [x] PR cannot be broken up into smaller PRs. - [x] Avoid breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes unless absolutely necessary. - [x] Tests provided or description of manual testing performed is included in the code or PR. - [x] Release note has been added for fixes and features, or else `changelog/no-changelog` label added. - [x] All relevant GitHub issues are correctly linked. - [x] Backports are identified and tagged with Mergifyio. - [x] Add to milestone.
mergify Bot
pushed a commit
that referenced
this pull request
Oct 28, 2022
## Description Blocked by #4125. This PR removes the skipping of building Python 3.11 wheels in our build_deploy job. ## Checklist - [x] Title must conform to [conventional commit](https://github.com/conventional-changelog/commitlint/tree/master/%40commitlint/config-conventional). - [x] Add additional sections for `feat` and `fix` pull requests. - [x] Ensure tests are passing for affected code. - [x] [Library documentation](https://github.com/DataDog/dd-trace-py/tree/1.x/docs) and/or [Datadog's documentation site](https://github.com/DataDog/documentation/) is updated. Link to doc PR in description. ## Motivation ## Design ## Testing strategy ## Relevant issue(s) ## Testing strategy ## Reviewer Checklist - [x] Title is accurate. - [x] Description motivates each change. - [x] No unnecessary changes were introduced in this PR. - [x] PR cannot be broken up into smaller PRs. - [x] Avoid breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes unless absolutely necessary. - [x] Tests provided or description of manual testing performed is included in the code or PR. - [x] Release note has been added for fixes and features, or else `changelog/no-changelog` label added. - [x] All relevant GitHub issues are correctly linked. - [x] Backports are identified and tagged with Mergifyio. - [x] Add to milestone. (cherry picked from commit 62852e3)
10 tasks
10 tasks
Kyle-Verhoog
pushed a commit
that referenced
this pull request
Nov 3, 2022
…4448) From #4125, two sections in the stack collector were modified to reflect Python 3.11 changes. However, those changes are not the exact same logic as the former code, including version gating and exception representation. I'm specifically applying these changes only if Python 3.11 or higher are being run. Otherwise, it is being reverted to what it was prior to #4125. While I'm unsure what is exactly causing the issue, we saw an issue (only when the profiler was activated) when running dogweb CI where tests involving sqlalchemy failed on setup with sessions becoming immediately inactive before any queries were run. Furthermore, we saw failures in asyncpg framework tests and major memory/CPU usage spikes when #4125 was merged into 1.x, both of which went away with this fix.
mergify Bot
pushed a commit
that referenced
this pull request
Nov 3, 2022
…4448) From #4125, two sections in the stack collector were modified to reflect Python 3.11 changes. However, those changes are not the exact same logic as the former code, including version gating and exception representation. I'm specifically applying these changes only if Python 3.11 or higher are being run. Otherwise, it is being reverted to what it was prior to #4125. While I'm unsure what is exactly causing the issue, we saw an issue (only when the profiler was activated) when running dogweb CI where tests involving sqlalchemy failed on setup with sessions becoming immediately inactive before any queries were run. Furthermore, we saw failures in asyncpg framework tests and major memory/CPU usage spikes when #4125 was merged into 1.x, both of which went away with this fix. (cherry picked from commit 6bc6ff1)
Kyle-Verhoog
added a commit
that referenced
this pull request
Nov 3, 2022
…4448) (#4452) From #4125, two sections in the stack collector were modified to reflect Python 3.11 changes. However, those changes are not the exact same logic as the former code, including version gating and exception representation. I'm specifically applying these changes only if Python 3.11 or higher are being run. Otherwise, it is being reverted to what it was prior to #4125. While I'm unsure what is exactly causing the issue, we saw an issue (only when the profiler was activated) when running dogweb CI where tests involving sqlalchemy failed on setup with sessions becoming immediately inactive before any queries were run. Furthermore, we saw failures in asyncpg framework tests and major memory/CPU usage spikes when #4125 was merged into 1.x, both of which went away with this fix. (cherry picked from commit 6bc6ff1) Co-authored-by: Yun Kim <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Kyle Verhoog <[email protected]>
Merged
11 tasks
brettlangdon
added a commit
that referenced
this pull request
Jan 18, 2023
…ect in python 3.11 (#4901) ## Description #4125 added the use of a getter function `PyThread_GetFrame()` to provide compatibility for Python 3.11 in the profiler (specifically the stack collector). However, `PyThread_GetFrame()` returns a strong reference to a `PyFrameObject` and its reference count was not decremented in that change. This means that the frame object will not be garbage collected and likely the cause behind the memory leak mentioned in #4899. ## Checklist - [ ] Followed the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines) when writing a release note. - [ ] Add additional sections for `feat` and `fix` pull requests. - [ ] [Library documentation](https://github.com/DataDog/dd-trace-py/tree/1.x/docs) and/or [Datadog's documentation site](https://github.com/DataDog/documentation/) is updated. Link to doc PR in description. <!-- Copy and paste the relevant snippet based on the type of pull request --> <!-- START feat --> ## Motivation <!-- Expand on why the change is required, include relevant context for reviewers --> ## Design <!-- Include benefits from the change as well as possible drawbacks and trade-offs --> ## Testing strategy <!-- Describe the automated tests and/or the steps for manual testing. <!-- END feat --> <!-- START fix --> ## Relevant issue(s) <!-- Link the pull request to any issues related to the fix. Use keywords for links to automate closing the issues once the pull request is merged. --> ## Testing strategy <!-- Describe any added regression tests and/or the manual testing performed. --> <!-- END fix --> ## Reviewer Checklist - [ ] Title is accurate. - [ ] Description motivates each change. - [ ] No unnecessary changes were introduced in this PR. - [ ] Avoid breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes unless absolutely necessary. - [ ] Tests provided or description of manual testing performed is included in the code or PR. - [ ] Release note has been added and follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines), or else `changelog/no-changelog` label added. - [ ] All relevant GitHub issues are correctly linked. - [ ] Backports are identified and tagged with Mergifyio. Co-authored-by: Brett Langdon <[email protected]>
Yun-Kim
added a commit
to Yun-Kim/dd-trace-py
that referenced
this pull request
Jan 23, 2023
…ect in python 3.11 (DataDog#4901) ## Description DataDog#4125 added the use of a getter function `PyThread_GetFrame()` to provide compatibility for Python 3.11 in the profiler (specifically the stack collector). However, `PyThread_GetFrame()` returns a strong reference to a `PyFrameObject` and its reference count was not decremented in that change. This means that the frame object will not be garbage collected and likely the cause behind the memory leak mentioned in DataDog#4899. ## Checklist - [ ] Followed the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines) when writing a release note. - [ ] Add additional sections for `feat` and `fix` pull requests. - [ ] [Library documentation](https://github.com/DataDog/dd-trace-py/tree/1.x/docs) and/or [Datadog's documentation site](https://github.com/DataDog/documentation/) is updated. Link to doc PR in description. <!-- Copy and paste the relevant snippet based on the type of pull request --> <!-- START feat --> ## Motivation <!-- Expand on why the change is required, include relevant context for reviewers --> ## Design <!-- Include benefits from the change as well as possible drawbacks and trade-offs --> ## Testing strategy <!-- Describe the automated tests and/or the steps for manual testing. <!-- END feat --> <!-- START fix --> ## Relevant issue(s) <!-- Link the pull request to any issues related to the fix. Use keywords for links to automate closing the issues once the pull request is merged. --> ## Testing strategy <!-- Describe any added regression tests and/or the manual testing performed. --> <!-- END fix --> ## Reviewer Checklist - [ ] Title is accurate. - [ ] Description motivates each change. - [ ] No unnecessary changes were introduced in this PR. - [ ] Avoid breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes unless absolutely necessary. - [ ] Tests provided or description of manual testing performed is included in the code or PR. - [ ] Release note has been added and follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines), or else `changelog/no-changelog` label added. - [ ] All relevant GitHub issues are correctly linked. - [ ] Backports are identified and tagged with Mergifyio. Co-authored-by: Brett Langdon <[email protected]>
emmettbutler
pushed a commit
that referenced
this pull request
Jan 30, 2023
…ect in python 3.11 (#4901) ## Description #4125 added the use of a getter function `PyThread_GetFrame()` to provide compatibility for Python 3.11 in the profiler (specifically the stack collector). However, `PyThread_GetFrame()` returns a strong reference to a `PyFrameObject` and its reference count was not decremented in that change. This means that the frame object will not be garbage collected and likely the cause behind the memory leak mentioned in #4899. ## Checklist - [ ] Followed the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines) when writing a release note. - [ ] Add additional sections for `feat` and `fix` pull requests. - [ ] [Library documentation](https://github.com/DataDog/dd-trace-py/tree/1.x/docs) and/or [Datadog's documentation site](https://github.com/DataDog/documentation/) is updated. Link to doc PR in description. <!-- Copy and paste the relevant snippet based on the type of pull request --> <!-- START feat --> ## Motivation <!-- Expand on why the change is required, include relevant context for reviewers --> ## Design <!-- Include benefits from the change as well as possible drawbacks and trade-offs --> ## Testing strategy <!-- Describe the automated tests and/or the steps for manual testing. <!-- END feat --> <!-- START fix --> ## Relevant issue(s) <!-- Link the pull request to any issues related to the fix. Use keywords for links to automate closing the issues once the pull request is merged. --> ## Testing strategy <!-- Describe any added regression tests and/or the manual testing performed. --> <!-- END fix --> ## Reviewer Checklist - [ ] Title is accurate. - [ ] Description motivates each change. - [ ] No unnecessary changes were introduced in this PR. - [ ] Avoid breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes unless absolutely necessary. - [ ] Tests provided or description of manual testing performed is included in the code or PR. - [ ] Release note has been added and follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines), or else `changelog/no-changelog` label added. - [ ] All relevant GitHub issues are correctly linked. - [ ] Backports are identified and tagged with Mergifyio. Co-authored-by: Brett Langdon <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Python 3.11 has made some new changes to their C internal API that we need to address in a couple of places, mostly in our profiler. I've moved the profiler changes into #4343 and changed this PR to focus on the tracer and integrations supporting Python 3.11. Note that some of the profiler changes were kept in this PR to ensure that the profiler can compile with Python 3.11 per our setup/install rules.
The major tracer-related fix is:
_PyFloat_Pack8()into their internal C API and replaced it withPyFloat_Pack8()to their public C API (Link to CPython issue). We use it once inddtrace.internal.pack_template.hwhere I've added a Python version check to use the correct corresponding function.Note: grpcio with pytest-asyncio is causing Python segmentation faults on the
test_unary_exceptionandtest_unary_cancellationtest cases, and from the traceback it doesn't look like dd-trace-py is involved. You can reproduce this using this gist which does not involve dd-trace-py. As a result I am skipping the grpcio_asyncio tests with Python 3.11.Checklist
featandfixpull requests.Reviewer Checklist
changelog/no-changeloglabel added.