Skip to content

fix(shrink): slice per-layer config lists with the pruned layers - #1256

Merged
MakazhanAlpamys merged 5 commits into
MakazhanAlpamys:mainfrom
drakeo338:claude/1241-fix
Sep 27, 2026
Merged

MakazhanAlpamys merged 5 commits into
MakazhanAlpamys:mainfrom
drakeo338:claude/1241-fix

Conversation

@drakeo338

Copy link
Copy Markdown
Contributor

Fixes #1241.

prune_model_layers sliced model.model.layers and updated num_hidden_layers, but left layer_types and SmolLM3's no_rope_layers at their original length. On Qwen2, Qwen3, Qwen2-MoE and SmolLM3, transformers 5 checks those lists against num_hidden_layers when saving, so a shrunk model failed to save and soup shrink exited 1 with an empty model/ directory. This slices both lists with the same kept layer indices used for the layers themselves, so a mixed sliding/full-attention list still lines up per surviving layer.

Type of change

  • Bug fix

Checklist

  • ruff check src/soup_cli/ scripts/ tests/ passes
  • pytest tests/ -v passes
  • Updated relevant docs (README.md and the matching page under docs/) if needed
  • Added a changelog fragment, or this change has no user-visible impact

…azhanAlpamys#1241)

prune_model_layers rebuilt model.model.layers from the kept indices and
patched config.num_hidden_layers, but left layer_types (Qwen2, Qwen3,
Qwen2-MoE, SmolLM3) and SmolLM3's no_rope_layers at the old length.
transformers 5 validates num_hidden_layers == len(layer_types) on save,
so soup shrink died after scoring with an empty model/ directory.

Both lists are now in a declared _PER_LAYER_CONFIG_ATTRS table and are
sliced with the same kept indices as the layers. A tail cut would be
wrong: a Qwen2 config with sliding windows has a mixed full/sliding
list, and SmolLM3's no_rope_layers has a 0 every fourth layer, so the
dropped block's entries differ from the last ones.

The tests prune each shrink-family config at the first allowed start,
the middle, ending at n-1 and at the largest block, then check the lists
and a save/reload round trip; a second test fails if a supported config
carries a per-layer list missing from the table, and a CLI test shrinks
a tiny local Qwen2 end to end without a download.

Closes MakazhanAlpamys#1241.

@MakazhanAlpamys MakazhanAlpamys left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@drakeo338, thank you for this, and welcome to Soup.

What is good here. I checked each of these myself.

  • The fix takes the route #1241 asked for. The per-layer lists live in one declared table (_PER_LAYER_CONFIG_ATTRS, src/soup_cli/utils/shrink.py:156), and they are sliced by the same kept indices as the layers, not cut at the tail. The guard test fails when a supported config carries a per-layer list that is missing from the table. Every acceptance box is covered, including all four drop positions and the mixed sliding/full Qwen2 list that exists to catch a tail cut.

  • It works through the real CLI, not only in the tests. I ran python -m soup_cli.cli shrink --drop-layers 3 --device cpu on three tiny local 8-layer models:

    • Qwen2 with use_sliding_window: true, which gives a mixed list;
    • a hybrid Qwen3.5 text model, where layer_types decides whether each layer is built with linear or full attention;
    • SmolLM3, which carries no_rope_layers.

    All three exit 0 on your branch and exit 1 on main. In each case:

    • the saved layer_types / no_rope_layers equal [orig[i] for i in kept];
    • the reload reports 0 missing and 0 unexpected keys;
    • every kept layer's tensors are identical to the original's (60, 64 and 45 tensors compared, 0 differ).
  • It covers more than the PR claims. With default configs, every causal-LM architecture in transformers 5.17 that the shrink allowlist accepts saves and reloads bit-identically after your change: Llama, DiffLlama, Qwen2, Qwen2-MoE, Qwen3, Qwen3-MoE, Qwen3-Next, Qwen3.5 text, Qwen3.5-MoE text, Qwen4-exp text, Llama4 text and SmolLM3. I did not build Mllama.

  • Your tests catch the fix being broken. I broke it five ways and re-ran your file (36 tests) each time:

    • deleting the slicing loop: 21 failed
    • tail truncation: 5 failed
    • head truncation: 5 failed
    • dropping no_rope_layers from the table: 5 failed
    • inverting the length guard: 21 failed

    On unmodified main your file gives 28 failed and 8 passed; the 8 are the Llama and Qwen3-MoE cases, which carry no per-layer list. On your branch merged with current main, your file gives 36 passed. Together with test_v07129.py, test_v07133.py and test_issue795_quantization_unhonoured_tasks.py, it is 358 passed, and ruff check is clean. The CLI test also runs its output through strip_ansi plus whitespace collapse. That is the only way it survives the coloured Linux and macOS runners.

One change is needed before this can merge.

  • The changelog fragment has to mention its own PR number (changelog.d/0.75.1/1256.fixed.md:1). The changelog validator requires #1256 in the text. On this branch tests/test_issue487_changelog_fragments.py::test_repository_fragments_match_the_newest_release fails (I ran it: 1 failed, 20 passed), so it would fail in every CI cell. Our credit format names both the issue and the PR: (#1241 by @drakeo338 in #1256). I made exactly that one edit locally and the file passes (21 passed). As written, (#1241 by @drakeo338) also reads as if you had filed #1241.

Suggestions (not blocking; the third one needs a decision).

  • Add a hybrid model to _CASES. Qwen3_5TextConfig or Qwen3NextConfig would do. In those models layer_types decides the module type of each layer, so a wrong slice silently changes the architecture rather than a flag. That makes it the strongest regression case, and your code already handles it.

  • max_window_layers keeps its old value. In my Qwen2 run it stays at 4, while the sliding block now starts at index 3. transformers 5.17 only reads it when layer_types is absent, so nothing breaks today. A consumer that derives the window from it would disagree, though. When the kept list is still full-then-sliding, it can be recomputed as the number of leading full-attention entries.

  • Index-valued layer lists. Some configs hold layer numbers rather than one entry per layer: Qwen-MoE mlp_only_layers / decoder_sparse_step and Llama4-text moe_layers (interleave_moe_layer_step). They are not per-layer lists, so the table correctly leaves them alone. But nothing remaps them either, and with non-default values the pruned model reloads with the MoE and dense MLPs in the wrong layers.

    Config After a (1, 1) drop, on reload
    Qwen2-MoE, mlp_only_layers=[3] 10 missing / 10 unexpected keys
    Llama4-text, interleave_moe_layer_step=2 18 missing / 18 unexpected keys

    from_pretrained random-initialises those weights without raising. On main, both configs fail at save on the layer_types length. With your fix the save succeeds, so for these two the failure goes from loud to silent. (Qwen3-MoE with decoder_sparse_step=2 has no layer_types and is already silently misaligned on main: 12 / 12.) The official Qwen MoE checkpoints use the defaults and are unaffected. A small guard would close this: in prune_model_layers, raise a ValueError naming the attribute when an index-valued list is non-default. Since this PR turns those two loud failures into silent ones, please add that guard here: it is a few lines, and one test per attribute (a non-default value is refused, the default still prunes) pins it.

For context only; nothing here is asked of this PR. Composite checkpoints such as qwen3_5 have no top-level num_hidden_layers, so soup shrink stops before it reaches the prune. That is an older, separate problem.

So there are two things before merge: the fragment line, and the index-list guard above. After that, this waits on the required checks, which have not run on the current head yet (13 pending).

mlp_only_layers, decoder_sparse_step (Qwen-MoE) and moe_layers,
interleave_moe_layer_step (Llama4-text) name layers by index. Slicing the
layers cannot remap them, so with a non-default value the pruned model
saved fine and reloaded with its MoE and dense MLPs on the wrong layers.
prune_model_layers now raises a ValueError naming the attribute instead.

Also adds a hybrid Qwen3.5 text config to the prune round-trip cases,
where layer_types decides each layer's attention module.
@drakeo338

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review. The fragment now reads (#1241 by @drakeo338 in #1256).

prune_model_layers now raises a ValueError naming the attribute when mlp_only_layers, decoder_sparse_step, moe_layers or interleave_moe_layer_step is non-default, before anything is sliced. Each has a test that a non-default value is refused (your Qwen2-MoE mlp_only_layers=[3] and Llama4-text interleave_moe_layer_step=2 among them) and that the default still prunes and reloads with no missing keys.

I also added Qwen3_5TextConfig to _CASES as the hybrid case.

@MakazhanAlpamys

Copy link
Copy Markdown
Owner

@drakeo338, thank you for turning this around so quickly, and for taking the guard on in this PR rather than leaving it for later.

What is good here. I checked each of these myself, on your branch merged with current main.

  • The fragment is fixed. It credits (#1241 by @drakeo338 in #1256) and passes the changelog validator.

  • The guard is in the right shape, and it refuses before anything is changed. The four attributes sit in one declared table next to the per-layer one. The refusal comes before the slice, and your refusal tests also check that the model still has all 6 layers afterwards. Through the real CLI, a tiny Qwen2-MoE with mlp_only_layers=[3] exits 1 with cannot prune a model whose config sets mlp_only_layers=[3]: ..., and no model/ directory is written.

  • The table is complete for everything soup shrink can reach. I checked every causal-LM config in transformers 5.17 that the shrink allowlist accepts and that has a top-level num_hidden_layers (12 model types).

    • No default config trips your guard.
    • The only other layer-numbered setting is no_rope_layer_interval (Llama4-text, SmolLM3). It is only used to generate no_rope_layers when that list is missing, and your slice already covers the list.
  • The hybrid case is in. With Qwen3_5TextConfig in _CASES, a wrong slice now changes which layers get linear or full attention. The tests still catch it.

  • Your tests catch the guard being broken. I broke it nine ways and re-ran your file (49 tests) each time. All nine were caught:

    • guard loop disabled: 4 failed
    • comparison inverted: 16 failed
    • guard moved after the slice: 4 failed
    • each of the four table entries removed: 1 failed each
    • moe_layers default changed to []: 2 failed
    • attribute name dropped from the message: 4 failed

    The first-round fix is still pinned, now including the hybrid case:

    • slicing loop disabled: 28 failed
    • tail truncation: 7 failed
    • head truncation: 7 failed
    • no_rope_layers dropped from the table: 5 failed
    • length check inverted: 28 failed
    • shrink.py reverted to main: 40 failed

    On unmodified main, your file gives 40 failed and 9 passed. On your branch merged with current main, it gives 392 passed together with test_v07129.py, test_v07133.py, test_issue795_quantization_unhonoured_tasks.py and test_issue487_changelog_fragments.py, and ruff check is clean.

Two things in the new guard need to change, plus one docs line.

  1. A shrunk Llama4-text model can no longer be shrunk again (src/soup_cli/utils/shrink.py:253-259). Llama4-text fills moe_layers with every layer by default, so for 6 layers it is [0, 1, 2, 3, 4, 5].

    • After a drop that list is not updated. The saved config says num_hidden_layers: 5 with moe_layers: [0, 1, 2, 3, 4, 5].
    • The next prune compares that list with the default for 5 layers and refuses.
    • Through the real CLI on a tiny default Llama4-text model, the first soup shrink --drop-layers 1 exits 0. soup shrink on its output then exits 1 with cannot prune a model whose config sets moe_layers=[0, 1, 2, 3, 4, 5]. Without the guard that second run would have gone ahead.

    When the prune goes ahead, the guard has already shown that every one of these attributes is at its default. So the fix is to keep each one at the default for the new layer count, right after the per-layer loop:

        # Every index-valued attribute is at its default (checked above); keep it the
        # default for the NEW layer count, e.g. Llama4-text ``moe_layers``.
        for attr, default_for in _LAYER_INDEX_CONFIG_DEFAULTS.items():
            if getattr(config, attr, None) is not None:
                setattr(config, attr, default_for(len(kept_idx)))
  2. The refusal comes after the expensive part, and --plan-only disagrees with it. Asking for the guard inside prune_model_layers was my suggestion, and it should stay there. The problem is when soup shrink gets to it.

    • prune_model_layers runs only after the command has loaded the full model (src/soup_cli/commands/shrink.py:329), scored every block (:334) and measured perplexity (:354). On the tiny Qwen2-MoE above, the error appears after Loading ..., Scoring importance over 4 calib prompts ... and the importance table. On a real MoE checkpoint, that means loading all of its weights before being told no.
    • --plan-only on the same model exits 0 with Would drop layers 1..1 (1 of 6): a plan the real run refuses.

    The check needs only config.json, and the command already has a "fail fast on the CONFIG before loading weights" step at line 318. So please move the loop into a small public helper in src/soup_cli/utils/shrink.py and call it from both places:

    def check_layer_index_config(config: object, n_layers: int) -> None:
        """Refuse a config whose MoE layout names layers by index.
    
        Any attribute in :data:`_LAYER_INDEX_CONFIG_DEFAULTS` that is not at its
        default for ``n_layers`` holds layer numbers a prune cannot remap.
        """
        for attr, default_for in _LAYER_INDEX_CONFIG_DEFAULTS.items():
            value = getattr(config, attr, None)
            if value is None:
                continue
            if isinstance(value, (list, tuple)):
                value = list(value)
            if value != default_for(n_layers):
                raise ValueError(
                    f"cannot prune a model whose config sets {attr}={value!r}: it holds "
                    "layer indices that would point at the wrong layers after the drop"
                )

    In prune_model_layers, the loop becomes check_layer_index_config(config, n_total). In _shrink_impl (src/soup_cli/commands/shrink.py:321-325), it becomes:

        from soup_cli.utils.shrink import arch_family_of_config, check_layer_index_config
    
        pre_config = AutoConfig.from_pretrained(model, trust_remote_code=trc)
        arch_family_of_config(pre_config)
        n_layers = int(pre_config.num_hidden_layers)
        check_layer_index_config(pre_config, n_layers)

    These two tests pin both points. They reuse _COMMON, _LLAMA4_TEXT and _QWEN2_MOE from your file:

    def test_a_default_moe_layout_stays_default_after_the_prune(tmp_path):
        """Llama4-text fills ``moe_layers`` with every layer by default. After the
        prune it must name every KEPT layer, or the saved config carries an index
        past ``num_hidden_layers`` and the shrunk model cannot be shrunk again."""
        import torch
        from transformers import AutoModelForCausalLM
    
        from soup_cli.utils.shrink import prune_model_layers
    
        config_name, extra = _LLAMA4_TEXT
        config = getattr(transformers, config_name)(**_COMMON, **extra)
        torch.manual_seed(0)
        model = AutoModelForCausalLM.from_config(config)
    
        prune_model_layers(model, 1, 1)
        assert model.config.moe_layers == list(range(5))
    
        model.save_pretrained(str(tmp_path))
        again = AutoModelForCausalLM.from_pretrained(str(tmp_path))
        prune_model_layers(again, 1, 1)
        assert again.config.num_hidden_layers == 4
    
    
    def test_shrink_cli_refuses_an_index_valued_layout_before_loading(tmp_path, monkeypatch):
        """The refusal needs only config.json, so it comes before the weights are
        loaded and scored, and ``--plan-only`` does not print a plan the real run
        would refuse."""
        import json
    
        from tokenizers import ByteLevelBPETokenizer
        from transformers import AutoModelForCausalLM, PreTrainedTokenizerFast
        from typer.testing import CliRunner
    
        from soup_cli.cli import app
        from tests.conftest import strip_ansi
    
        monkeypatch.chdir(tmp_path)
        lines = ["the quick brown fox jumps over the lazy dog", "pruning drops a block of layers"]
        bpe = ByteLevelBPETokenizer()
        bpe.train_from_iterator(lines, vocab_size=300, min_frequency=1, special_tokens=["<eos>"])
        tok = PreTrainedTokenizerFast(tokenizer_object=bpe, eos_token="<eos>", pad_token="<eos>")
        config_name, extra = _QWEN2_MOE
        config = getattr(transformers, config_name)(
            **{**_COMMON, "vocab_size": len(tok)}, **extra, mlp_only_layers=[3]
        )
        AutoModelForCausalLM.from_config(config).save_pretrained("moe")
        tok.save_pretrained("moe")
        (tmp_path / "calib.jsonl").write_text(
            "\n".join(json.dumps({"text": line}) for line in lines), encoding="utf-8"
        )
    
        r = CliRunner().invoke(
            app,
            ["shrink", "--model", "moe", "--drop-layers", "1", "--calib", "calib.jsonl",
             "--device", "cpu", "--plan-only"],
        )
        plain = " ".join(strip_ansi(r.output).split())
        assert r.exit_code == 1, (plain, repr(r.exception))
        assert "mlp_only_layers" in plain, plain
        assert "Scoring importance" not in plain, plain

    I tried exactly this change with these tests, on your branch merged with main:

    • your file gives 51 passed;
    • together with test_v07129.py, test_v07133.py and test_issue795_quantization_unhonoured_tasks.py, it is 371 passed;
    • ruff check is clean.

    Each test also fails where it should:

    • on your branch as pushed, both fail;
    • without the reset loop from point 1, the first one fails;
    • without the call in _shrink_impl, the second one fails;
    • on main, both fail.
  3. Please add one sentence to the docs. When soup shrink refuses someone's MoE model, they will look at the "Arch support (v1)" paragraph (docs/peft-and-efficiency.md:73). Something like:

    MoE configs that place their MoE layers by layer number (a non-default mlp_only_layers or decoder_sparse_step on Qwen MoE models, moe_layers or interleave_moe_layer_step on Llama4-text) are refused before the model is loaded, because a prune cannot renumber them. Per-layer lists such as layer_types and no_rope_layers are sliced along with the layers.

The required checks on the current head are queued (13 of 13, no results yet), and your branch merges cleanly with current main.

… before load

Reset every index-valued config attribute to its default for the new layer count so a shrunk Llama4-text model can be shrunk again. Move the refusal into check_layer_index_config and call it from the CLI's config-only pre-check so it runs before weights load and --plan-only agrees. Document the refusal.
@drakeo338

Copy link
Copy Markdown
Contributor Author

Pushed 2b9e773 with all three points: the reset loop after the per-layer slice, check_layer_index_config called from prune_model_layers and from _shrink_impl before the model loads, your two tests, and the docs sentence. Each new test fails without its half of the change.

@drakeo338

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed review — all three are addressed in 2b9e773:

  1. After the per-layer slice, every index-valued attribute in _LAYER_INDEX_CONFIG_DEFAULTS is reset to its default for the new layer count, so moe_layers is no longer left full and a shrunk Llama4-text model can be shrunk again.
  2. The index-config check is now a public helper check_layer_index_config(config, n_layers) in src/soup_cli/utils/shrink.py, called from prune_model_layers and also from _shrink_impl's config-only pre-check before weights load — so it refuses early and agrees with --plan-only.
  3. Added the explanatory sentence in docs/peft-and-efficiency.md.

Both suggested tests are pinned (test_a_default_moe_layout_stays_default_after_the_prune, test_shrink_cli_refuses_an_index_valued_layout_before_loading); test_issue1241_shrink_per_layer_config.py is at 51 passing.

@MakazhanAlpamys

Copy link
Copy Markdown
Owner

@drakeo338, thank you. All three points from my last review are done, and I checked each one by running it. I ran everything on your branch merged with current main.

  • A shrunk Llama4-text model can be shrunk again. I used a tiny default Llama4-text model with the real soup shrink --drop-layers 1.
    • The first run exits 0 and saves num_hidden_layers: 5 with moe_layers: [0, 1, 2, 3, 4].
    • Shrinking that output again also exits 0 and saves 4 layers with moe_layers: [0, 1, 2, 3]. On the previous head, this second run was refused.
  • The refusal now comes before anything expensive, and --plan-only agrees with it. I used a tiny Qwen2-MoE with mlp_only_layers=[3].
    • The real run exits 1 in about 11 seconds. Its only output line is Error: cannot prune a model whose config sets mlp_only_layers=[3]: ...: no Loading, no Scoring importance, and no model/ directory written.
    • --plan-only exits 1 with the same line. Before, it printed a plan the real run refused.
  • The docs sentence is in (docs/peft-and-efficiency.md:74), and it merges cleanly with the other edits main has made to that page.
  • Moving the check into check_layer_index_config was the right shape. prune_model_layers keeps it as a second line of defence, and your tests hold it there: deleting that call fails the 4 direct refusal tests.

What I ran.

  • Your test file with test_v07129.py, test_v07133.py, test_issue795_quantization_unhonoured_tasks.py and test_issue487_changelog_fragments.py: 394 passed. ruff check is clean.
  • On unmodified main, your test file gives 42 failed, 9 passed, and both new tests are among the failures.
  • I broke the change 10 ways and re-ran your file together with test_v07129.py (137 tests). Nine were caught:
What I broke Failed
reset loop deleted 1
reset to the old layer count 1
reset condition inverted 1
early check in _shrink_impl deleted 1
early check moved to after scoring 1
call in prune_model_layers deleted 4
helper loop emptied 5
helper comparison inverted 18
the round-1 slice turned into a tail cut 7

One optional line, and it is my omission rather than yours.

  • The gap. The CLI test I gave you checks that scoring does not start, but not that loading does not start. If the check is moved to just after _load_for_shrink (src/soup_cli/commands/shrink.py:330), every test still passes (137 passed). The docs sentence would then quietly become untrue.
  • The fix. One more assertion in test_shrink_cli_refuses_an_index_valued_layout_before_loading closes it:
    assert "mlp_only_layers" in plain, plain
    assert "Loading" not in plain, plain
    assert "Scoring importance" not in plain, plain
  • Checked. I tried it on your branch merged with main. The test passes as it is, and it fails when the check is moved after the load, deleted, or moved after scoring.
  • Not blocking. The refusal still happens before scoring, before the perplexity pass and before anything is written. Add it only if you want to.

Nothing else is needed from you. The required checks on 2b9e7737 have started and have no results yet; the branch merges cleanly with current main, so it now waits only on those checks.

@MakazhanAlpamys
MakazhanAlpamys merged commit cad2e48 into MakazhanAlpamys:main Sep 27, 2026
19 checks passed
@MakazhanAlpamys

Copy link
Copy Markdown
Owner

Merged as cad2e48a. Thank you, @drakeo338:

On the final head, merged with main, the new test file and four neighbours give 394 passed. The new file gives 42 failed / 9 passed on unmodified main. Of 10 mutations of this round's change, 9 fail a test; the one survivor is a gap in the test text the review itself supplied.

MakazhanAlpamys added a commit that referenced this pull request Sep 27, 2026
…laddinoglu (#1254) and Y.B. (#1256); Kratos (#1259)

Display names from `gh api users/<login>`.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

soup shrink crashes after scoring on Qwen2 / Qwen3 / Qwen2-MoE / SmolLM3: config.layer_types keeps the old length, so nothing is saved

2 participants