Skip to content

[Bug] dispatch_with_map raises AttributeError: '0' is not an nn.Module on sharded models (sharded MoE expert ModuleList) #711

Closed

Description

@pasta-paul

Summary

compressed_tensors.offload.dispatch.dispatch_with_map calls model.get_submodule(name) for every name in a broadcast-from-rank-0 device map (dispatch_with_map, line 95). When the local model has None entries in an nn.ModuleList — the canonical sharded-MoE pattern where each rank owns a slice of routed experts and non-owned slots are placeholders — get_submodule raises:

AttributeError: `0` is not an nn.Module

…because nn.ModuleList.__getattr__ rejects None children, but the rank-0-authored device map still references those submodule names on the assumption that every rank's model is structurally identical.

Where it bites

For us: the AttributeError fires in the post-save from_accelerate(model) step of llmcompressor.transformers.compression.compressed_tensors_utils.save_pretrained_wrapper (around line 94). The save_pretrained itself completes successfully on rank 0; the artifact is on disk; then from_accelerate runs unconditionally on every rank, broadcasts rank 0's device map, and ranks 1+ crash because rank 0's device map references experts that exist only on rank 0.

Full stack (4-rank torchrun, DeepSeek-V4 sharded experts, 64 experts/rank):

File ".../compressed_tensors/offload/convert/from_accelerate.py", line 62, in from_accelerate
  dispatch_with_map(model, *broadcast_obj)
File ".../compressed_tensors/offload/dispatch.py", line 95, in dispatch_with_map
  module = model.get_submodule(name)
File ".../torch/nn/modules/module.py", line 735, in get_submodule
  raise AttributeError("`" + item + "` is not an nn.Module")
AttributeError: `0` is not an nn.Module

Why it's a real bug

The device map is authoritative for rank 0's view of the module tree, not for every rank's view. With contiguous expert sharding (self.experts = nn.ModuleList([Expert(...) if owned else None for i in range(...)])), the non-rank-0 ranks have legitimate None slots that rank 0 doesn't see. Crashing on the lookup conflates "module doesn't exist on this rank" with "device map is malformed."

Proposed fix

Wrap the model.get_submodule(name) call in dispatch_with_map with try/except AttributeError and skip (with a debug log) when the module isn't present on the local rank. Pseudocode:

for name, device in device_map.items():
    try:
        module = model.get_submodule(name)
    except AttributeError:
        # Module not present on this rank (e.g. sharded MoE expert
        # slot that's None locally). Device map is authored from
        # rank 0's view; skip silently on ranks that don't own it.
        continue
    # ... existing dispatch logic ...

This is the smaller of two possible fixes. The larger one — having from_accelerate build a per-rank device map by intersecting the broadcast map with each rank's actual module set — is more invasive and arguably a separate refactor.

Reproducer (small)

import torch.nn as nn
from compressed_tensors.offload.dispatch import dispatch_with_map

class M(nn.Module):
    def __init__(self):
        super().__init__()
        # mimic sharded MoE: positions 0 and 2 are None on this rank
        self.experts = nn.ModuleList([None, nn.Linear(8, 8), None, nn.Linear(8, 8)])

m = M()
device_map = {
    "experts.0": "cuda:0",   # not present locally
    "experts.1": "cuda:0",
    "experts.2": "cuda:0",   # not present locally
    "experts.3": "cuda:0",
}
dispatch_with_map(m, device_map)  # currently raises AttributeError on experts.0

Workaround in our repo (until upstream fix lands)

We monkey-patch modify_save_pretrained to skip from_accelerate entirely for the sharded-MoE save path. Our _CompatModel.save_pretrained writes per-rank subdirs and rank 0 merges; we don't need the accelerate round-trip after save. That's an artifact-specific workaround though; the upstream dispatch_with_map should still be robust to this case.

Related

Activity

  1. added a commit that references this issue on Jun 16, 2026
    21c06ff
  2. added a commit that references this issue on Jun 30, 2026
    16521e1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions