Skip to content

Commit 7ed69a4

Browse files
committed
Clarify the accumulate path for zero_grad(set_to_none=False)
Review question: is the `grad is None` first-contribution test safe when the optimizer was built with `zero_grad(set_to_none=False)`? It is: that setting leaves a zeroed tensor where `set_to_none=True` leaves `None`, so the first contribution takes the add path and lands on the same value, and nothing in the branch depends on `grad` being `None`. Say so where the branch is, and pin it in the test. `test_two_optimizer_steps_do_not_pollute_each_other` already ran both settings, but `second != first` cannot show pollution - a first-step value carried into the second still differs from the first. It now compares the second step's gradient against a fresh model on the same weights and input, bit-exactly. Verified with the new test over an out-of-tree copy of an installed TransformerEngine: 5 failed, 3 passed against the unfixed `backward_dw` (the same five this PR describes), and 8 passed with this PR's `backward_dw` in place. Signed-off-by: 0z5a <[email protected]>
1 parent 9087a7e commit 7ed69a4

2 files changed

Lines changed: 24 additions & 3 deletions

File tree

‎tests/pytorch/test_tied_delayed_wgrad.py‎

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -137,18 +137,33 @@ def test_multiple_microbatches_accumulate():
137137

138138
@pytest.mark.parametrize("set_to_none", [True, False])
139139
def test_two_optimizer_steps_do_not_pollute_each_other(set_to_none):
140-
"""After a reset, the second step's grad must not inherit the first's."""
140+
"""After a reset, the second step's grad must be the second step's alone.
141+
142+
``zero_grad(set_to_none=False)`` leaves a zeroed tensor where ``set_to_none=True`` leaves
143+
``None``, so the first contribution of the step takes the add path instead of the assign
144+
path. Both settings have to deliver the same value, and ``second != first`` on its own
145+
would not show pollution: a first-step value carried into the second still differs from
146+
the first.
147+
"""
141148
model = _build(True, True)
142149
params = list({id(p): p for p in model.parameters()}.values())
143150
opt = torch.optim.SGD(params, lr=0.0)
144151

145-
_run(model, torch.randn(BATCH, SIZE, device=DEVICE, dtype=DTYPE), delay=True)
152+
_run(model, _x(), delay=True)
146153
first = model[0].weight.grad.detach().clone()
147154
opt.zero_grad(set_to_none=set_to_none)
148155

149-
_run(model, torch.randn(BATCH, SIZE, device=DEVICE, dtype=DTYPE), delay=True)
156+
second_x = _x()
157+
_run(model, second_x, delay=True)
150158
second = model[0].weight.grad.detach().clone()
151159

160+
# What the second step owes on its own: same weights, same input, first backward only.
161+
fresh = _build(True, True)
162+
fresh.load_state_dict(model.state_dict())
163+
_tie(fresh, True)
164+
_run(fresh, second_x.detach().clone().requires_grad_(True), delay=True)
165+
166+
torch.testing.assert_close(second, fresh[0].weight.grad, rtol=0, atol=0)
152167
assert not torch.equal(first, second)
153168
assert second.abs().sum() > 0
154169

‎transformer_engine/pytorch/module/base.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1966,6 +1966,12 @@ def backward_dw(self):
19661966
# module overwrite the contributions of the earlier ones, so
19671967
# accumulate instead: initialise on the first contribution of the
19681968
# step and add on every later one.
1969+
#
1970+
# The first contribution is assigned only when zero_grad left grad as
1971+
# None, which is what set_to_none=True does. set_to_none=False leaves a
1972+
# zeroed tensor instead, so that same contribution takes the add path and
1973+
# lands on the same value; nothing here needs grad to be None to start a
1974+
# step.
19691975
if weight_tensor.grad is None:
19701976
weight_tensor.grad = wgrad
19711977
else:

0 commit comments

Comments
 (0)