Repository navigation
Fix scalar parameter byte-view broadcasts - #8642
Conversation
Signed-off-by: Xucheng Zhou <[email protected]>
|
Current status (October 26): head October 6 CI handoff (historical)A consolidated CI handoff for my other pending fixes: #8728, #8726, #8715, #8709, #8619, #8512, #8511 and #8510 still have workflows awaiting maintainer approval and required checks Expected. Please approve the appropriate standard CI runs after reviewing their individual diffs. Thanks also to @ebarkhordar for the detailed CPU/PyTorch parity check on #8619; the CUDA-toolkit validation boundary remains as stated in that review. |
|
Hi, @pengdurice could you help accelerate the merge process? we have a lot of cases pending on this PR. Thx! CC @delock |
The CI tests are having some flakiness issues blocking several PRs's merge. Team is working on fixing the issue and will merge this PR when the CI issue is fixed. Thank you for your patience! |
bf222a2
gpu-test-labeler
Bot
added
the
skip-gpu-test
Fixes #8639.
PyTorch rejects reinterpreting a zero-dimensional tensor as
uint8when the element sizes differ. DeepSpeed uses that byte view in model and ZeRO parameter broadcasts, so a scalar parameter fails during initialization. Unsqueezing only scalar tensors creates a one-dimensional view of the same storage before the byte reinterpretation; non-scalar tensors keep the existing path.Tested with a two-rank CPU/Gloo regression on Python 3.12 and PyTorch 2.12.1: ZeRO stages 0 and 3 both pass with this change. The same stage-0 test fails on the unmodified base with the reported
self.dim() cannot be 0error. Repository formatting/static hooks,py_compile, andgit diff --checkpass. A newer Linux flake8 reports three warnings already present on the base; excluding only those unchanged warnings leaves no lint findings in the changed files. GPU distributed execution was not tested.Prepared with AI assistance; I reviewed the diff and ran the tests above.