[Testing only] Add python cycle detection #143204

albanD · 2024-12-13T17:18:01Z

This is just a test PR to see how bad things go.
The detection works as expected and test_torch and test_autograd have been fixed.
Unfortunately we leak a lot more objects than I expected and enabling this is going to be a challenge.

Based on the CI results here, I will most likely make this an opt-in and only enable it on the relevant core and Dynamo tests we care about and fix those.

Stack from ghstack (oldest at bottom):

-> [Testing only] Add python cycle detection #143204

[ghstack-poisoned]

pytorch-bot · 2024-12-13T17:18:05Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/143204

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 43 New Failures

As of commit d8f83a8 with merge base 3cbc8c5 ():

NEW FAILURES - The following jobs have failed:

Lint / lintrunner-noclang / linux-job (gh)
>>> Lint for torch/testing/_internal/common_utils.py:
pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu) (gh)
distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce
pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, lf.linux.g4dn.12xlarge.nvidia.gpu) (gh)
distributed/tensor/test_op_strategy.py::TestEinsumStrategies::test_bmm_1d_mesh
pull / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 1, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
test_nn.py::TestFusionEval::test_fuse_module_eval_numerics
pull / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 2, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
test_quantization.py::TestQuantizedOps::test_cat_nhwc
pull / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 3, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
test_quantization.py::TestQuantizedOps::test_add_scalar_relu
pull / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 4, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
test_pruning_op.py::PruningOpTest::test_rowwise_prune_op_32bit_indices
pull / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 5, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
test_quantization.py::TestQuantizedOps::test_equal
pull / linux-focal-cuda12.4-py3.10-gcc9-sm89 / test (default, 1, 5, lf.linux.g6.4xlarge.experimental.nvidia.gpu) (gh)
test_nn.py::TestFusionEval::test_fuse_module_eval_numerics
pull / linux-focal-cuda12.4-py3.10-gcc9-sm89 / test (default, 2, 5, lf.linux.g6.4xlarge.experimental.nvidia.gpu) (gh)
test_quantization.py::TestQuantizedOps::test_mul_scalar_relu
pull / linux-focal-cuda12.4-py3.10-gcc9-sm89 / test (default, 3, 5, lf.linux.g6.4xlarge.experimental.nvidia.gpu) (gh)
test_quantization.py::TestQuantizedOps::test_avg_pool2d
pull / linux-focal-cuda12.4-py3.10-gcc9-sm89 / test (default, 4, 5, lf.linux.g6.4xlarge.experimental.nvidia.gpu) (gh)
test_pruning_op.py::PruningOpTest::test_rowwise_prune_op_32bit_indices
pull / linux-focal-cuda12.4-py3.10-gcc9-sm89 / test (default, 5, 5, lf.linux.g6.4xlarge.experimental.nvidia.gpu) (gh)
test_quantization.py::TestQuantizedOps::test_avg_pool2d_nhwc
pull / linux-focal-py3.13-clang10 / test (crossref, 1, 2, lf.linux.2xlarge) (gh)
test_nn.py::TestFusionEval::test_fuse_module_eval_numerics
pull / linux-focal-py3.13-clang10 / test (crossref, 2, 2, lf.linux.2xlarge) (gh)
test_pruning_op.py::PruningOpTest::test_rowwise_prune_op_32bit_indices
pull / linux-focal-py3.13-clang10 / test (default, 1, 5, lf.linux.4xlarge) (gh)
test_nn.py::TestFusionEval::test_fuse_module_eval_numerics
pull / linux-focal-py3.13-clang10 / test (default, 2, 5, lf.linux.4xlarge) (gh)
test_quantization.py::TestQuantizedOps::test_mul_scalar_relu
pull / linux-focal-py3.13-clang10 / test (default, 3, 5, lf.linux.4xlarge) (gh)
test_quantization.py::TestQuantizedOps::test_avg_pool2d
pull / linux-focal-py3.13-clang10 / test (default, 4, 5, lf.linux.4xlarge) (gh)
test_quantization.py::TestQuantizedOps::test_add_scalar_relu
pull / linux-focal-py3.13-clang10 / test (default, 5, 5, lf.linux.4xlarge) (gh)
test_quantization.py::TestQuantizedOps::test_avg_pool2d_nhwc
pull / linux-focal-py3.13-clang10 / test (dynamo_wrapped, 2, 3, lf.linux.2xlarge) (gh)
test_nn.py::TestFusionEval::test_fuse_module_eval_numerics
pull / linux-focal-py3.13-clang10 / test (dynamo_wrapped, 3, 3, lf.linux.2xlarge) (gh)
test_pruning_op.py::PruningOpTest::test_rowwise_prune_op_32bit_indices
pull / linux-focal-py3.9-clang10 / test (crossref, 1, 2, lf.linux.2xlarge) (gh)
'Test'
pull / linux-focal-py3.9-clang10 / test (crossref, 2, 2, lf.linux.2xlarge) (gh)
'test/test_jit.py::TestBackends::test_execution'
pull / linux-focal-py3.9-clang10 / test (default, 1, 5, lf.linux.4xlarge) (gh)
'Test'
pull / linux-focal-py3.9-clang10 / test (default, 2, 5, lf.linux.4xlarge) (gh)
export/test_export.py::TestExport::test_suggested_fixes_new_roots
pull / linux-focal-py3.9-clang10 / test (default, 4, 5, lf.linux.4xlarge) (gh)
'Test'
pull / linux-focal-py3.9-clang10 / test (default, 5, 5, lf.linux.4xlarge) (gh)
'test/test_jit.py::TestBackends::test_execution'
pull / linux-focal-py3.9-clang10 / test (dynamo_wrapped, 2, 3, lf.linux.2xlarge) (gh)
'Test'
pull / linux-focal-py3.9-clang10 / test (dynamo_wrapped, 3, 3, lf.linux.2xlarge) (gh)
test_jit_fuser_te.py::TestTEFuserDynamic::test_inlined_optimized_graph
pull / linux-focal-py3.9-clang10-onnx / test (default, 1, 2, lf.linux.2xlarge) (gh)
'Test'
pull / linux-focal-py3.9-clang10-onnx / test (default, 2, 2, lf.linux.2xlarge) (gh)
'Test'
pull / linux-jammy-py3.10-clang15-asan / test (default, 2, 6, lf.linux.4xlarge) (gh)
test_nn.py::TestFusionEval::test_fuse_module_eval_numerics
pull / linux-jammy-py3.10-clang15-asan / test (default, 3, 6, lf.linux.4xlarge) (gh)
test_quantization.py::TestQuantizedOps::test_max_pool1d
pull / linux-jammy-py3.10-clang15-asan / test (default, 4, 6, lf.linux.4xlarge) (gh)
test_quantization.py::TestQuantizedOps::test_equal
pull / linux-jammy-py3.10-clang15-asan / test (default, 5, 6, lf.linux.4xlarge) (gh)
test_quantization.py::TestQuantizedOps::test_cat
pull / linux-jammy-py3.10-clang15-asan / test (default, 6, 6, lf.linux.4xlarge) (gh)
test_quantization.py::TestQuantizedOps::test_cat_nhwc
pull / linux-jammy-py3.9-gcc11 / test (default, 1, 5, lf.linux.2xlarge) (gh)
'Test'
pull / linux-jammy-py3.9-gcc11 / test (default, 2, 5, lf.linux.2xlarge) (gh)
export/test_export.py::TestExport::test_suggested_fixes_new_roots
pull / linux-jammy-py3.9-gcc11 / test (default, 5, 5, lf.linux.2xlarge) (gh)
'test/test_jit.py::TestBackends::test_execution'
pull / linux-jammy-py3.9-gcc11 / test (distributed, 1, 2, lf.linux.2xlarge) (gh)
distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce
pull / linux-jammy-py3.9-gcc11 / test (distributed, 2, 2, lf.linux.2xlarge) (gh)
distributed/tensor/test_op_strategy.py::TestEinsumStrategies::test_bmm_1d_mesh
pull / linux-jammy-py3.9-gcc11 / test (jit_legacy, 1, 1, lf.linux.2xlarge) (gh)
'test/test_jit_legacy.py::TestBackends::test_execution'

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[ghstack-poisoned]

ghstack-source-id: 61cbf16 Pull Request resolved: #143204

[ghstack-poisoned]

ghstack-source-id: a603628 Pull Request resolved: #143204

[ghstack-poisoned]

ghstack-source-id: 01eddd1 Pull Request resolved: #143204

[ghstack-poisoned]

ghstack-source-id: 0ab2487 Pull Request resolved: #143204

github-actions · 2025-03-23T18:37:58Z

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

github-actions · 2025-06-08T19:34:01Z

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

Update

7a79140

[ghstack-poisoned]

albanD requested a review from a team as a code owner December 13, 2024 17:18

This was referenced Dec 13, 2024

move function before modifying it #143202

Closed

Fix Tensor clear to properly clear slots #143203

Closed

github-actions bot requested review from SherlockNoMad, antoniojkim, bdhirsh, ezyang and miladm December 13, 2024 17:19

albanD changed the title ~~Add python cycle detection~~ [Testing only] Add python cycle detection Dec 13, 2024

Update

aa7cba7

[ghstack-poisoned]

Update

c414a2e

[ghstack-poisoned]

albanD mentioned this pull request Dec 13, 2024

No actual change, just remove variable contain Tensors from global scope #143225

Closed

albanD added a commit that referenced this pull request Dec 13, 2024

Add python cycle detection

1734330

ghstack-source-id: 61cbf16 Pull Request resolved: #143204

Update

cc23b5b

[ghstack-poisoned]

albanD added the topic: not user facing topic category label Dec 16, 2024

Update

9c249b6

[ghstack-poisoned]

albanD mentioned this pull request Dec 16, 2024

Various fix for memory leak in test autograd and dataloader #143323

Closed

Update

82433bc

[ghstack-poisoned]

Update

a6c5c09

[ghstack-poisoned]

albanD added a commit that referenced this pull request Dec 17, 2024

Add python cycle detection

19246d0

ghstack-source-id: a603628 Pull Request resolved: #143204

Update

7e78170

[ghstack-poisoned]

albanD added a commit that referenced this pull request Jan 10, 2025

Add python cycle detection

1c98a0a

ghstack-source-id: 01eddd1 Pull Request resolved: #143204

Update

d8f83a8

[ghstack-poisoned]

albanD added a commit that referenced this pull request Jan 22, 2025

Add python cycle detection

3235501

ghstack-source-id: 0ab2487 Pull Request resolved: #143204

github-actions bot added the Stale label Mar 23, 2025

albanD removed the Stale label Apr 9, 2025

github-actions bot added the Stale label Jun 8, 2025

github-actions bot closed this Jul 8, 2025

github-actions bot deleted the gh/albanD/3/head branch August 8, 2025 02:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Testing only] Add python cycle detection #143204

[Testing only] Add python cycle detection #143204

Uh oh!

albanD commented Dec 13, 2024 •

edited

Loading

Uh oh!

pytorch-bot bot commented Dec 13, 2024 •

edited

Loading

Uh oh!

github-actions bot commented Mar 23, 2025

Uh oh!

github-actions bot commented Jun 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[Testing only] Add python cycle detection #143204

[Testing only] Add python cycle detection #143204

Uh oh!

Conversation

albanD commented Dec 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Dec 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/143204

❌ 43 New Failures

Uh oh!

github-actions bot commented Mar 23, 2025

Uh oh!

github-actions bot commented Jun 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

albanD commented Dec 13, 2024 •

edited

Loading

pytorch-bot bot commented Dec 13, 2024 •

edited

Loading