Repository navigation
Commit 7ed69a4
committed
Clarify the accumulate path for zero_grad(set_to_none=False)
Review question: is the `grad is None` first-contribution test safe when the optimizer was
built with `zero_grad(set_to_none=False)`? It is: that setting leaves a zeroed tensor where
`set_to_none=True` leaves `None`, so the first contribution takes the add path and lands on
the same value, and nothing in the branch depends on `grad` being `None`. Say so where the
branch is, and pin it in the test.
`test_two_optimizer_steps_do_not_pollute_each_other` already ran both settings, but
`second != first` cannot show pollution - a first-step value carried into the second still
differs from the first. It now compares the second step's gradient against a fresh model on
the same weights and input, bit-exactly.
Verified with the new test over an out-of-tree copy of an installed TransformerEngine:
5 failed, 3 passed against the unfixed `backward_dw` (the same five this PR describes), and
8 passed with this PR's `backward_dw` in place.
Signed-off-by: 0z5a <[email protected]>1 parent 9087a7e commit 7ed69a4
2 files changed
Lines changed: 24 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
137 | 137 | | |
138 | 138 | | |
139 | 139 | | |
140 | | - | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
141 | 148 | | |
142 | 149 | | |
143 | 150 | | |
144 | 151 | | |
145 | | - | |
| 152 | + | |
146 | 153 | | |
147 | 154 | | |
148 | 155 | | |
149 | | - | |
| 156 | + | |
| 157 | + | |
150 | 158 | | |
151 | 159 | | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
152 | 167 | | |
153 | 168 | | |
154 | 169 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1966 | 1966 | | |
1967 | 1967 | | |
1968 | 1968 | | |
| 1969 | + | |
| 1970 | + | |
| 1971 | + | |
| 1972 | + | |
| 1973 | + | |
| 1974 | + | |
1969 | 1975 | | |
1970 | 1976 | | |
1971 | 1977 | | |
| |||
0 commit comments