[wip] debug cpu/gpu backprop mismatch #139933

bobrenjc93 · 2024-11-06T23:06:18Z

Stack from ghstack (oldest at bottom):

=== Backprop works for cpu but not gpu ===

PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=2 fdb --debugger lldb debug /home/bobren/.conda/envs/pytorch/bin/python3 test/inductor/test_torchinductor_opinfo.py TestInductorOpInfoCUDA.test_comprehensive_linalg_norm_subgradients_at_zero_cuda_float64

PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=2 python test/inductor/test_torchinductor_opinfo.py TestInductorOpInfoCUDA.test_comprehensive_linalg_norm_subgradients_at_zero_cuda_float64

PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=2 python test/inductor/test_torchinductor_opinfo.py TestInductorOpInfoCPU.test_comprehensive_linalg_norm_subgradients_at_zero_cpu_float64

===

[ghstack-poisoned]

pytorch-bot · 2024-11-06T23:06:21Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/139933

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 51 New Failures, 2 Unrelated Failures

As of commit 3506555 with merge base ffb9790 ():

NEW FAILURES - The following jobs have failed:

Check Labels / Check labels (gh)
# This PR needs a release notes: label
Lint / lintrunner-clang / linux-job (gh)
>>> Lint for torch/csrc/autograd/python_engine.cpp:
Lint / lintrunner-noclang / linux-job (gh)
>>> Lint for torch/csrc/autograd/python_engine.cpp:
pull / cuda12.1-py3.10-gcc9-sm75 / test (pr_time_benchmarks, 1, 1, linux.g4dn.metal.nvidia.gpu) (gh)
REGRESSION: benchmark ('aotdispatcher_training_subclass_cpu', 'compile_time_instruction_count') failed, actual result 10642258913 is 3.02% higher than expected 10330000000 ±+1.50% if this is an expected regression, please update the expected results.
pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 1, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
'test/test_python_dispatch.py::TestPythonDispatch::test_none_wrapping'
pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 3, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
dynamo/test_dynamic_shapes.py::DynamicShapesMiscTests::test_closure_out_of_scope_cell_dynamic_shapes
pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 4, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
dynamo/test_higher_order_ops.py::HigherOrderOpTests::test_capture_global_num
pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 1, 5, lf.linux.g5.4xlarge.nvidia.gpu) (gh)
'test/test_python_dispatch.py::TestPythonDispatch::test_none_wrapping'
pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 2, 5, lf.linux.g5.4xlarge.nvidia.gpu) (gh)
dynamo/test_repros.py::ReproTests::test_issue175
pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 3, 5, lf.linux.g5.4xlarge.nvidia.gpu) (gh)
inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_lerp_cuda_float16
pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 4, 5, lf.linux.g5.4xlarge.nvidia.gpu) (gh)
inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_norm_cuda_float16
pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 5, 5, lf.linux.g5.4xlarge.nvidia.gpu) (gh)
inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_norm_subgradients_at_zero_cuda_float32
pull / linux-focal-py3_9-clang9-xla / test (xla, 1, 1, lf.linux.12xlarge) (gh)
ModuleNotFoundError: No module named 'torch.version'
pull / linux-focal-py3.11-clang10 / test (crossref, 1, 2, lf.linux.2xlarge) (gh)
'test/test_python_dispatch.py::TestPythonDispatch::test_none_wrapping'
pull / linux-focal-py3.11-clang10 / test (crossref, 2, 2, lf.linux.2xlarge) (gh)
'test/test_autograd.py::TestAutograd::test_custom_autograd_no_early_free'
pull / linux-focal-py3.11-clang10 / test (default, 1, 5, lf.linux.4xlarge) (gh)
'test/test_python_dispatch.py::TestPythonDispatch::test_none_wrapping'
pull / linux-focal-py3.11-clang10 / test (default, 2, 5, lf.linux.4xlarge) (gh)
inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCPU::test_comprehensive_nn_functional_local_response_norm_cpu_float16
pull / linux-focal-py3.11-clang10 / test (default, 3, 5, lf.linux.4xlarge) (gh)
'test/test_autograd.py::TestAutograd::test_custom_autograd_no_early_free'
pull / linux-focal-py3.11-clang10 / test (default, 4, 5, lf.linux.4xlarge) (gh)
dynamo/test_dynamic_shapes.py::DynamicShapesMiscTests::test_closure_out_of_scope_cell_dynamic_shapes
pull / linux-focal-py3.11-clang10 / test (default, 5, 5, lf.linux.4xlarge) (gh)
dynamo/test_dynamic_shapes.py::DynamicShapesMiscTests::test_config_obj_dynamic_shapes
pull / linux-focal-py3.11-clang10 / test (dynamo_wrapped, 1, 3, lf.linux.2xlarge) (gh)
'test/test_python_dispatch.py::TestPythonDispatch::test_none_wrapping'
pull / linux-focal-py3.11-clang10 / test (dynamo_wrapped, 2, 3, lf.linux.2xlarge) (gh)
'test/test_nn.py::TestNN::test_BCELoss_no_batch_dim_mean'
pull / linux-focal-py3.11-clang10 / test (dynamo_wrapped, 3, 3, lf.linux.2xlarge) (gh)
'test/test_autograd.py::TestAutograd::test_custom_autograd_no_early_free'
pull / linux-focal-py3.12-clang10 / test (default, 1, 5, lf.linux.4xlarge) (gh)
'test/test_python_dispatch.py::TestPythonDispatch::test_none_wrapping'
pull / linux-focal-py3.12-clang10 / test (default, 2, 5, lf.linux.4xlarge) (gh)
inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCPU::test_comprehensive_nn_functional_local_response_norm_cpu_float16
pull / linux-focal-py3.12-clang10 / test (default, 3, 5, lf.linux.4xlarge) (gh)
'test/test_autograd.py::TestAutograd::test_custom_autograd_no_early_free'
pull / linux-focal-py3.12-clang10 / test (default, 4, 5, lf.linux.4xlarge) (gh)
dynamo/test_dynamic_shapes.py::DynamicShapesMiscTests::test_closure_out_of_scope_cell_dynamic_shapes
pull / linux-focal-py3.12-clang10 / test (default, 5, 5, lf.linux.4xlarge) (gh)
dynamo/test_dynamic_shapes.py::DynamicShapesMiscTests::test_config_obj_dynamic_shapes
pull / linux-focal-py3.12-clang10 / test (dynamo_wrapped, 1, 3, lf.linux.2xlarge) (gh)
'test/test_python_dispatch.py::TestPythonDispatch::test_none_wrapping'
pull / linux-focal-py3.12-clang10 / test (dynamo_wrapped, 2, 3, lf.linux.2xlarge) (gh)
'test/test_nn.py::TestNN::test_BCELoss_no_batch_dim_mean'
pull / linux-focal-py3.12-clang10 / test (dynamo_wrapped, 3, 3, lf.linux.2xlarge) (gh)
'test/test_autograd.py::TestAutograd::test_custom_autograd_no_early_free'
pull / linux-focal-py3.9-clang10 / test (crossref, 1, 2, lf.linux.2xlarge) (gh)
'test/test_python_dispatch.py::TestPythonDispatch::test_none_wrapping'
pull / linux-focal-py3.9-clang10 / test (crossref, 2, 2, lf.linux.2xlarge) (gh)
'test/test_autograd.py::TestAutograd::test_custom_autograd_no_early_free'
pull / linux-focal-py3.9-clang10 / test (default, 1, 5, lf.linux.4xlarge) (gh)
'test/test_python_dispatch.py::TestPythonDispatch::test_none_wrapping'
pull / linux-focal-py3.9-clang10 / test (default, 2, 5, lf.linux.4xlarge) (gh)
inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCPU::test_comprehensive_nn_functional_local_response_norm_cpu_float16
pull / linux-focal-py3.9-clang10 / test (default, 3, 5, lf.linux.4xlarge) (gh)
inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCPU::test_comprehensive_nn_functional_local_response_norm_cpu_float32
pull / linux-focal-py3.9-clang10 / test (default, 4, 5, lf.linux.4xlarge) (gh)
inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCPU::test_comprehensive_nn_functional_local_response_norm_cpu_float64
pull / linux-focal-py3.9-clang10 / test (default, 5, 5, lf.linux.4xlarge) (gh)
dynamo/test_higher_order_ops.py::HigherOrderOpTests::test_capture_global_num
pull / linux-focal-py3.9-clang10 / test (dynamo_wrapped, 1, 3, lf.linux.2xlarge) (gh)
'test/test_python_dispatch.py::TestPythonDispatch::test_none_wrapping'
pull / linux-focal-py3.9-clang10 / test (dynamo_wrapped, 2, 3, lf.linux.2xlarge) (gh)
'test/test_cpp_extensions_jit.py::TestCppExtensionJIT::test_custom_compound_op_autograd'
pull / linux-focal-py3.9-clang10 / test (dynamo_wrapped, 3, 3, lf.linux.2xlarge) (gh)
'test/test_autograd.py::TestAutograd::test_custom_autograd_no_early_free'
pull / linux-jammy-py3.10-clang15-asan / test (default, 1, 6, lf.linux.4xlarge) (gh)
'test/test_python_dispatch.py::TestPythonDispatch::test_none_wrapping'
pull / linux-jammy-py3.10-clang15-asan / test (default, 2, 6, lf.linux.4xlarge) (gh)
'test/test_autograd.py::TestAutograd::test_custom_autograd_no_early_free'
pull / linux-jammy-py3.10-clang15-asan / test (default, 3, 6, lf.linux.4xlarge) (gh)
'test/functorch/test_eager_transforms.py::TestCompileTransformsCPU::test_compile_vmap_hessian_cpu'
pull / linux-jammy-py3.10-clang15-asan / test (default, 5, 6, lf.linux.4xlarge) (gh)
dynamo/test_inline_inbuilt_nn_modules.py::InlineInbuiltNNModulesMiscTests::test_closure_out_of_scope_cell_inline_inbuilt_nn_modules
pull / linux-jammy-py3.10-clang15-asan / test (default, 6, 6, lf.linux.4xlarge) (gh)
dynamo/test_dynamic_shapes.py::DynamicShapesMiscTests::test_closure_out_of_scope_cell_dynamic_shapes
pull / linux-jammy-py3.9-gcc11 / test (default, 1, 5, lf.linux.2xlarge) (gh)
'test/test_python_dispatch.py::TestPythonDispatch::test_none_wrapping'
pull / linux-jammy-py3.9-gcc11 / test (default, 2, 5, lf.linux.2xlarge) (gh)
inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCPU::test_comprehensive_nn_functional_local_response_norm_cpu_float16
pull / linux-jammy-py3.9-gcc11 / test (default, 3, 5, lf.linux.2xlarge) (gh)
'test/test_autograd.py::TestAutograd::test_custom_autograd_no_early_free'
pull / linux-jammy-py3.9-gcc11 / test (default, 4, 5, lf.linux.2xlarge) (gh)
dynamo/test_dynamic_shapes.py::DynamicShapesMiscTests::test_closure_out_of_scope_cell_dynamic_shapes
pull / linux-jammy-py3.9-gcc11 / test (default, 5, 5, lf.linux.2xlarge) (gh)
dynamo/test_dynamic_shapes.py::DynamicShapesMiscTests::test_config_obj_dynamic_shapes

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 2, 5, lf.linux.4xlarge.nvidia.gpu) (gh) (detected as infra flaky with no log or failing log classifier)
pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 5, 5, lf.linux.4xlarge.nvidia.gpu) (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2024-11-06T23:06:50Z

This PR needs a `release notes:` label

If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

=== Backprop works for cpu but not gpu === PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=2 fdb --debugger lldb debug /home/bobren/.conda/envs/pytorch/bin/python3 test/inductor/test_torchinductor_opinfo.py TestInductorOpInfoCUDA.test_comprehensive_linalg_norm_subgradients_at_zero_cuda_float64 PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=2 python test/inductor/test_torchinductor_opinfo.py TestInductorOpInfoCUDA.test_comprehensive_linalg_norm_subgradients_at_zero_cuda_float64 PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=2 python test/inductor/test_torchinductor_opinfo.py TestInductorOpInfoCPU.test_comprehensive_linalg_norm_subgradients_at_zero_cpu_float64 === [ghstack-poisoned]

ghstack-source-id: b9392a4 Pull Request resolved: #139933

=== Backprop works for cpu but not gpu === PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=2 fdb --debugger lldb debug /home/bobren/.conda/envs/pytorch/bin/python3 test/inductor/test_torchinductor_opinfo.py TestInductorOpInfoCUDA.test_comprehensive_linalg_norm_subgradients_at_zero_cuda_float64 PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=2 python test/inductor/test_torchinductor_opinfo.py TestInductorOpInfoCUDA.test_comprehensive_linalg_norm_subgradients_at_zero_cuda_float64 PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=2 python test/inductor/test_torchinductor_opinfo.py TestInductorOpInfoCPU.test_comprehensive_linalg_norm_subgradients_at_zero_cpu_float64 === [ghstack-poisoned]

ghstack-source-id: 0f165d8 Pull Request resolved: #139933

=== Backprop works for cpu but not gpu === PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=2 fdb --debugger lldb debug /home/bobren/.conda/envs/pytorch/bin/python3 test/inductor/test_torchinductor_opinfo.py TestInductorOpInfoCUDA.test_comprehensive_linalg_norm_subgradients_at_zero_cuda_float64 PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=2 python test/inductor/test_torchinductor_opinfo.py TestInductorOpInfoCUDA.test_comprehensive_linalg_norm_subgradients_at_zero_cuda_float64 PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=2 python test/inductor/test_torchinductor_opinfo.py TestInductorOpInfoCPU.test_comprehensive_linalg_norm_subgradients_at_zero_cpu_float64 === [ghstack-poisoned]

ghstack-source-id: e7a2416 Pull Request resolved: #139933

wip debug cpu/gpu backprop mismatch

79b825b

[ghstack-poisoned]

bobrenjc93 mentioned this pull request Nov 6, 2024

Trigger symfloat specialization in argument binding code #139454

Closed

bobrenjc93 changed the title ~~wip debug cpu/gpu backprop mismatch~~ [wip] debug cpu/gpu backprop mismatch Nov 6, 2024

bobrenjc93 mentioned this pull request Nov 6, 2024

specialize symfloats for wrapped_gradient in get_fake_value #139935

Closed

bobrenjc93 added a commit that referenced this pull request Nov 7, 2024

wip debug cpu/gpu backprop mismatch

ac511bf

ghstack-source-id: b9392a4 Pull Request resolved: #139933

bobrenjc93 added a commit that referenced this pull request Nov 8, 2024

wip debug cpu/gpu backprop mismatch

638de68

ghstack-source-id: 0f165d8 Pull Request resolved: #139933

bobrenjc93 added a commit that referenced this pull request Nov 10, 2024

wip debug cpu/gpu backprop mismatch

0c3e33d

ghstack-source-id: e7a2416 Pull Request resolved: #139933

bobrenjc93 closed this Nov 12, 2024

github-actions bot deleted the gh/bobrenjc93/113/head branch December 12, 2024 02:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[wip] debug cpu/gpu backprop mismatch #139933

[wip] debug cpu/gpu backprop mismatch #139933

Uh oh!

bobrenjc93 commented Nov 6, 2024 •

edited

Loading

Uh oh!

pytorch-bot bot commented Nov 6, 2024 •

edited

Loading

Uh oh!

github-actions bot commented Nov 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[wip] debug cpu/gpu backprop mismatch #139933

[wip] debug cpu/gpu backprop mismatch #139933

Uh oh!

Conversation

bobrenjc93 commented Nov 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/139933

❌ 51 New Failures, 2 Unrelated Failures

Uh oh!

github-actions bot commented Nov 6, 2024

This PR needs a release notes: label

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bobrenjc93 commented Nov 6, 2024 •

edited

Loading

pytorch-bot bot commented Nov 6, 2024 •

edited

Loading

This PR needs a `release notes:` label