Repository navigation
Description
Summary
compressed_tensors.offload.dispatch.dispatch_with_map calls model.get_submodule(name) for every name in a broadcast-from-rank-0 device map (dispatch_with_map, line 95). When the local model has None entries in an nn.ModuleList — the canonical sharded-MoE pattern where each rank owns a slice of routed experts and non-owned slots are placeholders — get_submodule raises:
AttributeError: `0` is not an nn.Module
…because nn.ModuleList.__getattr__ rejects None children, but the rank-0-authored device map still references those submodule names on the assumption that every rank's model is structurally identical.
Where it bites
For us: the AttributeError fires in the post-save from_accelerate(model) step of llmcompressor.transformers.compression.compressed_tensors_utils.save_pretrained_wrapper (around line 94). The save_pretrained itself completes successfully on rank 0; the artifact is on disk; then from_accelerate runs unconditionally on every rank, broadcasts rank 0's device map, and ranks 1+ crash because rank 0's device map references experts that exist only on rank 0.
Full stack (4-rank torchrun, DeepSeek-V4 sharded experts, 64 experts/rank):
File ".../compressed_tensors/offload/convert/from_accelerate.py", line 62, in from_accelerate
dispatch_with_map(model, *broadcast_obj)
File ".../compressed_tensors/offload/dispatch.py", line 95, in dispatch_with_map
module = model.get_submodule(name)
File ".../torch/nn/modules/module.py", line 735, in get_submodule
raise AttributeError("`" + item + "` is not an nn.Module")
AttributeError: `0` is not an nn.Module
Why it's a real bug
The device map is authoritative for rank 0's view of the module tree, not for every rank's view. With contiguous expert sharding (self.experts = nn.ModuleList([Expert(...) if owned else None for i in range(...)])), the non-rank-0 ranks have legitimate None slots that rank 0 doesn't see. Crashing on the lookup conflates "module doesn't exist on this rank" with "device map is malformed."
Proposed fix
Wrap the model.get_submodule(name) call in dispatch_with_map with try/except AttributeError and skip (with a debug log) when the module isn't present on the local rank. Pseudocode:
for name, device in device_map.items():
try:
module = model.get_submodule(name)
except AttributeError:
# Module not present on this rank (e.g. sharded MoE expert
# slot that's None locally). Device map is authored from
# rank 0's view; skip silently on ranks that don't own it.
continue
# ... existing dispatch logic ...This is the smaller of two possible fixes. The larger one — having from_accelerate build a per-rank device map by intersecting the broadcast map with each rank's actual module set — is more invasive and arguably a separate refactor.
Reproducer (small)
import torch.nn as nn
from compressed_tensors.offload.dispatch import dispatch_with_map
class M(nn.Module):
def __init__(self):
super().__init__()
# mimic sharded MoE: positions 0 and 2 are None on this rank
self.experts = nn.ModuleList([None, nn.Linear(8, 8), None, nn.Linear(8, 8)])
m = M()
device_map = {
"experts.0": "cuda:0", # not present locally
"experts.1": "cuda:0",
"experts.2": "cuda:0", # not present locally
"experts.3": "cuda:0",
}
dispatch_with_map(m, device_map) # currently raises AttributeError on experts.0Workaround in our repo (until upstream fix lands)
We monkey-patch modify_save_pretrained to skip from_accelerate entirely for the sharded-MoE save path. Our _CompatModel.save_pretrained writes per-rank subdirs and rank 0 merges; we don't need the accelerate round-trip after save. That's an artifact-specific workaround though; the upstream dispatch_with_map should still be robust to this case.
Related
- GPTQModifier hangs on multi-rank with sharded MoE experts (dist.reduce/broadcast on disjoint module sets) llm-compressor#2734 (multi-rank disjoint-module collective desync — adjacent class of bug at the collective level rather than the dispatch level).
- [Feature] Native intra-calibration resume in
SequentialPipelinellm-compressor#2741 (intra-calibration resume; not directly related but shares the operational context of large sharded-MoE calibration).
Activity
[Offload] Skip device-map entries with no local module in dispatch_wi…
[Offload] Skip device-map entries with no local module in dispatch_wi…