Repository navigation
fix(shrink): slice per-layer config lists with the pruned layers - #1256
Conversation
…azhanAlpamys#1241) prune_model_layers rebuilt model.model.layers from the kept indices and patched config.num_hidden_layers, but left layer_types (Qwen2, Qwen3, Qwen2-MoE, SmolLM3) and SmolLM3's no_rope_layers at the old length. transformers 5 validates num_hidden_layers == len(layer_types) on save, so soup shrink died after scoring with an empty model/ directory. Both lists are now in a declared _PER_LAYER_CONFIG_ATTRS table and are sliced with the same kept indices as the layers. A tail cut would be wrong: a Qwen2 config with sliding windows has a mixed full/sliding list, and SmolLM3's no_rope_layers has a 0 every fourth layer, so the dropped block's entries differ from the last ones. The tests prune each shrink-family config at the first allowed start, the middle, ending at n-1 and at the largest block, then check the lists and a save/reload round trip; a second test fails if a supported config carries a per-layer list missing from the table, and a CLI test shrinks a tiny local Qwen2 end to end without a download. Closes MakazhanAlpamys#1241.
MakazhanAlpamys
left a comment
There was a problem hiding this comment.
@drakeo338, thank you for this, and welcome to Soup.
What is good here. I checked each of these myself.
-
The fix takes the route #1241 asked for. The per-layer lists live in one declared table (
_PER_LAYER_CONFIG_ATTRS,src/soup_cli/utils/shrink.py:156), and they are sliced by the same kept indices as the layers, not cut at the tail. The guard test fails when a supported config carries a per-layer list that is missing from the table. Every acceptance box is covered, including all four drop positions and the mixed sliding/full Qwen2 list that exists to catch a tail cut. -
It works through the real CLI, not only in the tests. I ran
python -m soup_cli.cli shrink --drop-layers 3 --device cpuon three tiny local 8-layer models:- Qwen2 with
use_sliding_window: true, which gives a mixed list; - a hybrid Qwen3.5 text model, where
layer_typesdecides whether each layer is built with linear or full attention; - SmolLM3, which carries
no_rope_layers.
All three exit 0 on your branch and exit 1 on
main. In each case:- the saved
layer_types/no_rope_layersequal[orig[i] for i in kept]; - the reload reports 0 missing and 0 unexpected keys;
- every kept layer's tensors are identical to the original's (60, 64 and 45 tensors compared, 0 differ).
- Qwen2 with
-
It covers more than the PR claims. With default configs, every causal-LM architecture in transformers 5.17 that the shrink allowlist accepts saves and reloads bit-identically after your change: Llama, DiffLlama, Qwen2, Qwen2-MoE, Qwen3, Qwen3-MoE, Qwen3-Next, Qwen3.5 text, Qwen3.5-MoE text, Qwen4-exp text, Llama4 text and SmolLM3. I did not build Mllama.
-
Your tests catch the fix being broken. I broke it five ways and re-ran your file (36 tests) each time:
- deleting the slicing loop: 21 failed
- tail truncation: 5 failed
- head truncation: 5 failed
- dropping
no_rope_layersfrom the table: 5 failed - inverting the length guard: 21 failed
On unmodified
mainyour file gives 28 failed and 8 passed; the 8 are the Llama and Qwen3-MoE cases, which carry no per-layer list. On your branch merged with currentmain, your file gives 36 passed. Together withtest_v07129.py,test_v07133.pyandtest_issue795_quantization_unhonoured_tasks.py, it is 358 passed, andruff checkis clean. The CLI test also runs its output throughstrip_ansiplus whitespace collapse. That is the only way it survives the coloured Linux and macOS runners.
One change is needed before this can merge.
- The changelog fragment has to mention its own PR number (
changelog.d/0.75.1/1256.fixed.md:1). The changelog validator requires#1256in the text. On this branchtests/test_issue487_changelog_fragments.py::test_repository_fragments_match_the_newest_releasefails (I ran it: 1 failed, 20 passed), so it would fail in every CI cell. Our credit format names both the issue and the PR:(#1241 by @drakeo338 in #1256). I made exactly that one edit locally and the file passes (21 passed). As written,(#1241 by @drakeo338)also reads as if you had filed #1241.
Suggestions (not blocking; the third one needs a decision).
-
Add a hybrid model to
_CASES.Qwen3_5TextConfigorQwen3NextConfigwould do. In those modelslayer_typesdecides the module type of each layer, so a wrong slice silently changes the architecture rather than a flag. That makes it the strongest regression case, and your code already handles it. -
max_window_layerskeeps its old value. In my Qwen2 run it stays at 4, while the sliding block now starts at index 3. transformers 5.17 only reads it whenlayer_typesis absent, so nothing breaks today. A consumer that derives the window from it would disagree, though. When the kept list is still full-then-sliding, it can be recomputed as the number of leading full-attention entries. -
Index-valued layer lists. Some configs hold layer numbers rather than one entry per layer: Qwen-MoE
mlp_only_layers/decoder_sparse_stepand Llama4-textmoe_layers(interleave_moe_layer_step). They are not per-layer lists, so the table correctly leaves them alone. But nothing remaps them either, and with non-default values the pruned model reloads with the MoE and dense MLPs in the wrong layers.Config After a (1, 1)drop, on reloadQwen2-MoE, mlp_only_layers=[3]10 missing / 10 unexpected keys Llama4-text, interleave_moe_layer_step=218 missing / 18 unexpected keys from_pretrainedrandom-initialises those weights without raising. Onmain, both configs fail at save on thelayer_typeslength. With your fix the save succeeds, so for these two the failure goes from loud to silent. (Qwen3-MoE withdecoder_sparse_step=2has nolayer_typesand is already silently misaligned onmain: 12 / 12.) The official Qwen MoE checkpoints use the defaults and are unaffected. A small guard would close this: inprune_model_layers, raise aValueErrornaming the attribute when an index-valued list is non-default. Since this PR turns those two loud failures into silent ones, please add that guard here: it is a few lines, and one test per attribute (a non-default value is refused, the default still prunes) pins it.
For context only; nothing here is asked of this PR. Composite checkpoints such as qwen3_5 have no top-level num_hidden_layers, so soup shrink stops before it reaches the prune. That is an older, separate problem.
So there are two things before merge: the fragment line, and the index-list guard above. After that, this waits on the required checks, which have not run on the current head yet (13 pending).
…in the shrink fragment
mlp_only_layers, decoder_sparse_step (Qwen-MoE) and moe_layers, interleave_moe_layer_step (Llama4-text) name layers by index. Slicing the layers cannot remap them, so with a non-default value the pruned model saved fine and reloaded with its MoE and dense MLPs on the wrong layers. prune_model_layers now raises a ValueError naming the attribute instead. Also adds a hybrid Qwen3.5 text config to the prune round-trip cases, where layer_types decides each layer's attention module.
|
Thanks for the thorough review. The fragment now reads
I also added |
|
@drakeo338, thank you for turning this around so quickly, and for taking the guard on in this PR rather than leaving it for later. What is good here. I checked each of these myself, on your branch merged with current
Two things in the new guard need to change, plus one docs line.
The required checks on the current head are queued (13 of 13, no results yet), and your branch merges cleanly with current |
… before load Reset every index-valued config attribute to its default for the new layer count so a shrunk Llama4-text model can be shrunk again. Move the refusal into check_layer_index_config and call it from the CLI's config-only pre-check so it runs before weights load and --plan-only agrees. Document the refusal.
|
Pushed 2b9e773 with all three points: the reset loop after the per-layer slice, check_layer_index_config called from prune_model_layers and from _shrink_impl before the model loads, your two tests, and the docs sentence. Each new test fails without its half of the change. |
|
Thanks for the detailed review — all three are addressed in 2b9e773:
Both suggested tests are pinned ( |
|
@drakeo338, thank you. All three points from my last review are done, and I checked each one by running it. I ran everything on your branch merged with current
What I ran.
One optional line, and it is my omission rather than yours.
assert "mlp_only_layers" in plain, plain
assert "Loading" not in plain, plain
assert "Scoring importance" not in plain, plain
Nothing else is needed from you. The required checks on |
|
Merged as
On the final head, merged with |
…laddinoglu (#1254) and Y.B. (#1256); Kratos (#1259) Display names from `gh api users/<login>`. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Fixes #1241.
prune_model_layersslicedmodel.model.layersand updatednum_hidden_layers, but leftlayer_typesand SmolLM3'sno_rope_layersat their original length. On Qwen2, Qwen3, Qwen2-MoE and SmolLM3, transformers 5 checks those lists againstnum_hidden_layerswhen saving, so a shrunk model failed to save andsoup shrinkexited 1 with an emptymodel/directory. This slices both lists with the same kept layer indices used for the layers themselves, so a mixed sliding/full-attention list still lines up per surviving layer.Type of change
Checklist
ruff check src/soup_cli/ scripts/ tests/passespytest tests/ -vpassesREADME.mdand the matching page underdocs/) if needed