Fix gradgrad issues with BatchNorm and Advanced Indexing (master) #2269

gchanan · 2017-08-01T22:47:32Z

This adds two tests for gradchecks:

That backward is reentrant
That input grad sizes match input sizez.
Uncovers an issue with BatchNorm which is now fixed
Uncovers an issue with advanced indexing which is also now fixed.

…t needed. These could cause a problem with double backwards because they were std::move'd in Backward.

… indexing case that fails check.

gchanan · 2017-08-01T22:48:12Z

@killeent as mentioned in #2268, we should (either in this PR or another) move the resizing logic for advanced indexing to C++.

soumith · 2017-08-02T01:46:19Z

will leave this for @apaszke @killeent review.

apaszke

LGTM

torch/csrc/autograd/functions/batch_normalization.h


  virtual void releaseVariables() override;

-  at::Tensor save_mean;


…bcc3b0 (pytorch#26309) Summary: Pull Request resolved: pytorch#26309 Previous import was 95252c2adec185e305e34486c6756ece9aa8f57f Included changes: - **[1316afc9](onnx/onnx@1316afc9)**: Update IR doc to clarify initializers are permitted as node inputs (pytorch#2320) <G. Ramalingam> - **[5e920d0c](onnx/onnx@5e920d0c)**: Avoid uses of special chars (pytorch#2315) <Wei-Sheng Chin> - **[2fa08b0f](onnx/onnx@2fa08b0f)**: Regenerate ONNX proto and add release date to ver 6 IR (pytorch#2316) <Wei-Sheng Chin> - **[adf9c7a3](onnx/onnx@adf9c7a3)**: Add description of default type about y_zero_point (pytorch#2110) <Takeshi Watanabe> - **[ee7072c7](onnx/onnx@ee7072c7)**: Support make_attribute empty string (pytorch#2129) <shjwudp> - **[f913b6e7](onnx/onnx@f913b6e7)**: More unsqueeze tests (pytorch#2200) <James Allingham> - **[57b51937](onnx/onnx@57b51937)**: Fix resize shape inference issue in opset10 (pytorch#2294) <Bowen Bao> - **[d7595f34](onnx/onnx@d7595f34)**: Sequence related ops (pytorch#2249) <Bowen Bao> - **[599f3da9](onnx/onnx@599f3da9)**: Add helper function update_inputs_outputs_dims to tools (pytorch#2148) <Bowen Bao> - **[3e6382bc](onnx/onnx@3e6382bc)**: Update documentation about required input output types (pytorch#2310) <G. Ramalingam> - **[0c765d9b](onnx/onnx@0c765d9b)**: Shape inference for NMS (pytorch#2269) <Hariharan Seshadri> - **[89266710](onnx/onnx@89266710)**: Fix extra collect_snippets warning (pytorch#2277) (pytorch#2307) <Lutz Roeder> Test Plan: ci Reviewed By: hl475 Differential Revision: D17403954 fbshipit-source-id: 8f7cbf77b7e09b73db2015f3e0f436772482b322

…bcc3b0 (#26309) Summary: Pull Request resolved: #26309 Previous import was 95252c2adec185e305e34486c6756ece9aa8f57f Included changes: - **[1316afc9](onnx/onnx@1316afc9)**: Update IR doc to clarify initializers are permitted as node inputs (#2320) <G. Ramalingam> - **[5e920d0c](onnx/onnx@5e920d0c)**: Avoid uses of special chars (#2315) <Wei-Sheng Chin> - **[2fa08b0f](onnx/onnx@2fa08b0f)**: Regenerate ONNX proto and add release date to ver 6 IR (#2316) <Wei-Sheng Chin> - **[adf9c7a3](onnx/onnx@adf9c7a3)**: Add description of default type about y_zero_point (#2110) <Takeshi Watanabe> - **[ee7072c7](onnx/onnx@ee7072c7)**: Support make_attribute empty string (#2129) <shjwudp> - **[f913b6e7](onnx/onnx@f913b6e7)**: More unsqueeze tests (#2200) <James Allingham> - **[57b51937](onnx/onnx@57b51937)**: Fix resize shape inference issue in opset10 (#2294) <Bowen Bao> - **[d7595f34](onnx/onnx@d7595f34)**: Sequence related ops (#2249) <Bowen Bao> - **[599f3da9](onnx/onnx@599f3da9)**: Add helper function update_inputs_outputs_dims to tools (#2148) <Bowen Bao> - **[3e6382bc](onnx/onnx@3e6382bc)**: Update documentation about required input output types (#2310) <G. Ramalingam> - **[0c765d9b](onnx/onnx@0c765d9b)**: Shape inference for NMS (#2269) <Hariharan Seshadri> - **[89266710](onnx/onnx@89266710)**: Fix extra collect_snippets warning (#2277) (#2307) <Lutz Roeder> Test Plan: ci Reviewed By: hl475 Differential Revision: D17403954 fbshipit-source-id: 78a9c3ecf5aa7f7a0ba8ea30286eab61ee903772

…pr_simplify" printing (pytorch#2269)

…ytorch#2269) Fixes SWDEV-536456 Fixes error post-ROCm#2256: ``` 00:12:44.248 pytorch#22 155.3 ERROR: Ignored the following versions that require a different python version: 0.52.0 Requires-Python >=3.6,<3.9; 0.52.0rc3 Requires-Python >=3.6,<3.9; 0.61.0 Requires-Python >=3.10; 0.61.0rc1 Requires-Python >=3.10; 0.61.0rc2 Requires-Python >=3.10; 0.61.1rc1 Requires-Python >=3.10; 0.61.2 Requires-Python >=3.10; 3.3 Requires-Python >=3.10; 3.3rc0 Requires-Python >=3.10; 3.4 Requires-Python >=3.10; 3.4.1 Requires-Python >=3.10; 3.4.2 Requires-Python >=3.10; 3.4rc0 Requires-Python >=3.10; 3.5 Requires-Python >=3.11; 3.5rc0 Requires-Python >=3.11; 8.2.0 Requires-Python >=3.10; 8.2.1 Requires-Python >=3.10 00:12:44.248 pytorch#22 155.3 ERROR: Could not find a version that satisfies the requirement numba==0.61.2 (from versions: 0.1, 0.2, 0.3, 0.5.0, 0.6.0, 0.7.0, 0.7.1, 0.7.2, 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.12.1, 0.12.2, 0.13.0, 0.13.2, 0.13.3, 0.13.4, 0.14.0, 0.15.1, 0.16.0, 0.17.0, 0.18.1, 0.18.2, 0.19.1, 0.19.2, 0.20.0, 0.21.0, 0.22.0, 0.22.1, 0.23.0, 0.23.1, 0.24.0, 0.25.0, 0.26.0, 0.27.0, 0.28.1, 0.29.0, 0.30.0, 0.30.1, 0.31.0, 0.32.0, 0.33.0, 0.34.0, 0.35.0, 0.36.1, 0.36.2, 0.37.0, 0.38.0, 0.38.1, 0.39.0, 0.40.0, 0.40.1, 0.41.0, 0.42.0, 0.42.1, 0.43.0, 0.43.1, 0.44.0, 0.44.1, 0.45.0, 0.45.1, 0.46.0, 0.47.0, 0.48.0, 0.49.0, 0.49.1rc1, 0.49.1, 0.50.0rc1, 0.50.0, 0.50.1, 0.51.0rc1, 0.51.0, 0.51.1, 0.51.2, 0.52.0rc2, 0.53.0rc1.post1, 0.53.0rc2, 0.53.0rc3, 0.53.0, 0.53.1, 0.54.0rc2, 0.54.0rc3, 0.54.0, 0.54.1rc1, 0.54.1, 0.55.0rc1, 0.55.0, 0.55.1, 0.55.2, 0.56.0rc1, 0.56.0, 0.56.2, 0.56.3, 0.56.4, 0.57.0rc1, 0.57.0, 0.57.1rc1, 0.57.1, 0.58.0rc1, 0.58.0rc2, 0.58.0, 0.58.1, 0.59.0rc1, 0.59.0, 0.59.1, 0.60.0rc1, 0.60.0) 00:12:44.248 pytorch#22 155.3 ERROR: No matching distribution found for numba==0.61.2 ``` Validation: * Docker image: http://rocm-ci.amd.com/job/mainline-framework-pytorch-internal-cs9-ci/132 * Wheels: http://rocm-ci.amd.com/job/mainline-pytorch_internal-manylinux-wheels/102/ From `registry-sc-harbor.amd.com/framework/compute-rocm-dkms-no-npi-hipclang:16180_ubuntu22.04_py3.9_pytorch_lw_rocm7.0_IT_py3.9_a11d94ad`: ``` root@f43861a0a856:/# pip list | egrep "numpy|pandas" numpy 2.0.2 pandas 2.2.3 root@f43861a0a856:/# python Python 3.9.23 (main, Jun 4 2025, 08:55:38) [GCC 11.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import torch >>> import numpy >>> import pandas root@f43861a0a856:/data/pytorch-micro-benchmarking# HIP_VISIBLE_DEVICES=1 python3 micro_benchmarking_pytorch.py --network resnet50 INFO: running forward and backward for warmup. INFO: running the benchmark.. OK: finished running benchmark.. --------------------SUMMARY-------------------------- Microbenchmark for network : resnet50 Num devices: 1 Dtype: FP32 Mini batch size [img] : 64 Time per mini-batch : 0.11354223489761353 Throughput [img/sec] : 563.6669038416574 ```

gchanan added 3 commits August 1, 2017 15:46

Remove save_mean/save_var from BatchNorm double backwards, as it's no…

d668183

…t needed. These could cause a problem with double backwards because they were std::move'd in Backward.

add reentrancy checking for gradcheck.

7750f92

Add tests that gradcheck grad sizes match input size and fix advanced…

c7e8dfb

… indexing case that fails check.

gchanan mentioned this pull request Aug 1, 2017

Segfault when double backward on BatchNorm2d #2264

Closed

apaszke approved these changes Aug 2, 2017

View reviewed changes

torch/csrc/autograd/functions/batch_normalization.h

virtual void releaseVariables() override;

at::Tensor save_mean;

This comment was marked as off-topic.

Sign in to view

gchanan merged commit ca98c65 into pytorch:master Aug 2, 2017

houseroad mentioned this pull request Sep 16, 2019

Automatic update of fbcode/onnx to 1316afc9f972f81340faa05763e2898f38bcc3b0 #26309

Closed

IvanYashchuk pushed a commit to IvanYashchuk/pytorch that referenced this pull request Jan 5, 2023

Allow users to specify which pass to trigger PYTORCH_NVFUSER_DUMP="ex…

74b773e

…pr_simplify" printing (pytorch#2269)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix gradgrad issues with BatchNorm and Advanced Indexing (master) #2269

Fix gradgrad issues with BatchNorm and Advanced Indexing (master) #2269

Uh oh!

gchanan commented Aug 1, 2017

Uh oh!

gchanan commented Aug 1, 2017

Uh oh!

soumith commented Aug 2, 2017

Uh oh!

apaszke left a comment •

edited

Loading

Uh oh!

This comment was marked as off-topic.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		virtual void releaseVariables() override;

		at::Tensor save_mean;

Fix gradgrad issues with BatchNorm and Advanced Indexing (master) #2269

Fix gradgrad issues with BatchNorm and Advanced Indexing (master) #2269

Uh oh!

Conversation

gchanan commented Aug 1, 2017

Uh oh!

gchanan commented Aug 1, 2017

Uh oh!

soumith commented Aug 2, 2017

Uh oh!

apaszke left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as off-topic.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

apaszke left a comment •

edited

Loading