FIX Keep the order of rank_pattern and alpha_pattern keys on save by jayzuccarelli · Pull Request #3947 · huggingface/peft

jayzuccarelli · 2026-10-12T11:11:19Z

Fixes #3818 (approval: #3818 (comment); design per #3834 (comment))

When more than one key in rank_pattern / alpha_pattern matches a module, the first match in dict order wins. save_pretrained wrote the config with sort_keys=True, which also sorted the nested pattern dicts, so the winning key could change on reload: a different scaling for alpha_pattern, a size mismatch for rank_pattern.

save_pretrained still sorts all keys, nested dicts included, except for the dicts stored under rank_pattern and alpha_pattern, which keep their order. The exempted keys live in CONFIG_KEYS_WITH_PRESERVED_ORDER in config.py, so another key can be added there if needed. Matching is unchanged.

Backward compatibility: the load path does not change, so every checkpoint that loads today loads the same way. Checkpoints that were already saved with a changed order are not repaired, since the original order is not on disk. The only difference in new files is the key order inside rank_pattern and alpha_pattern; everything else in adapter_config.json is written as before.

Tests:

new test_overlapping_pattern_keys_order_survives_save_load in TestRankAndAlphaPattern, for rank_pattern and alpha_pattern with keys in non-alphabetical order. Both cases fail on main and pass here.
new test_save_pretrained_sorts_keys_except_pattern_keys in test_config.py: the pattern keys keep their order, and the rest of the file, including a nested loftq_config, is the same as with sort_keys=True.
pytest tests/test_config.py tests/test_tuners_utils.py tests/test_initialization.py: 2042 passed, 27 skipped, 2 xfailed
pytest tests/test_custom_models.py -k "save or hotswap or adalora or pattern": 3185 passed, 343 skipped
ruff check and ruff format: clean
MetaMathQA benchmark: not run (needs CUDA). The change only affects the key order in the saved config.

That json.dumps call is the only place in src/peft that writes the config.

AI assistance (Claude Code) was used for this PR. I reviewed every changed line and ran the tests above.

… the config Co-Authored-By: Claude Opus 5.5 <[email protected]>

chrikrah

@jayzuccarelli 5916eac can go in as is. @BenjaminBossan asked on #3834 whether anything in PEFT depends on the sorted order. You covered the writer there, so I read the other side, every reader of pattern-key order in src/peft. Nothing depends on it.

All 17 get_pattern_key(...) call sites take the first match: LoRA, LoHa, LoKr, DeLoRA, HiRA, WaveFT, EVA, CorDA, ASTRA and utils/hotswap.py. FourierFT, C3A and OSF match their own pattern dicts by hand, and first match wins there too. The only reader that is not first-match is mask_using_rank_pattern in AdaLoRA. It checks the first key for the adapter name, and model.py:312 strips that name from every key at once, so order cannot change the answer.

$ grep -rn "get_pattern_key(" src/peft | grep -v "def get_pattern_key" | wc -l    # PR head 5916eac
17
$ grep -rno "next(filter(lambda key: re.match" src/peft/tuners/fourierft/model.py src/peft/tuners/c3a/model.py
src/peft/tuners/fourierft/model.py:70:next(filter(lambda key: re.match
src/peft/tuners/c3a/model.py:66:next(filter(lambda key: re.match
$ grep -n "rank_pattern.items\|break" src/peft/tuners/osf/model.py
94:            for pattern, rank in osf_config.rank_pattern.items():
97:                    break
$ sed -n 507p src/peft/tuners/adalora/layer.py
        if self.adapter_name not in next(iter(rank_pattern.keys())):

@BenjaminBossan, does that cover the double-check you asked for, or is there a reader outside PEFT, such as diffusers' adapter loading, that you want checked before this goes in?

BenjaminBossan · 2026-10-13T02:35:04Z

@jayzuccarelli I haven't reviewed this PR yet, but wondered about a slightly different solution: Right now, you only re-order the top level entries and leave everything else in the existing order. This would lead to changes in the adapter_config.json that go beyond just rank_pattern and alpha_pattern. Maybe that is fine, but I wonder if it would work if we kept on sorting everything, included nested dicts, and only making an exception for alpha_pattern and rank_pattern. Since the naming convention for those are the same across PEFT methods that support them, they should all benefit from the fix.

If you think that works, please implement it in a way that makes it easy to add more keys that are exempted from the sorting if ever needed in the future.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

jayzuccarelli · 2026-10-13T07:38:14Z

@BenjaminBossan that works, and it is the smaller change to the saved file, so I switched to it in 5c17ba6.

save_pretrained sorts all keys again, nested dicts included, and only skips the dicts stored under rank_pattern and alpha_pattern. The exempted keys are a module-level set, CONFIG_KEYS_WITH_PRESERVED_ORDER in config.py, so adding another one later is a one-line change.

For the default config of every config class exported from peft (45 of them), the output is byte-identical to json.dumps(..., sort_keys=True), so the only difference in adapter_config.json is the key order inside those two dicts. I added a test for that in test_config.py, with a nested loftq_config to cover nested dicts. The round-trip test from the first commit is unchanged and still passes. I also updated the PR description.

BenjaminBossan

Thanks for implementing the suggestion, I believe this also should help reduce unnecessary diff noise when creating new PEFT checkpoints.

The changes generally look good, I have only a few comments left.

BenjaminBossan · 2026-10-13T07:56:44Z

 MIN_EXPECTED_CONFIG_KEYS = {"peft_type"}

+# The adapter_config.json is written with sorted keys, except for the dicts stored under these keys. For those, the
+# order matters: when more than one key matches a module, the first one wins, so the order has to survive a save/load


Let's be a bit more explicit in the comment by mentioning rank_pattern and alpha_pattern here. Otherwise, if we later add an item below and don't update the comment, it could state something incorrect.

Done in 301748f: the comment now names rank_pattern and alpha_pattern as the keys the reason applies to, and asks to extend it when a key is added.

BenjaminBossan · 2026-10-13T08:04:23Z


+    def test_save_pretrained_sorts_keys_except_pattern_keys(self, tmp_path):
+        # The config is written with sorted keys, nested dicts included, but the order of rank_pattern and
+        # alpha_pattern is kept, see #3818.


Let's quickly mention why it's okay to only test LoRA here.

Added in 301748f: all configs are saved by the same PeftConfigMixin.save_pretrained, and the methods that support these arguments use the same names.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

HuggingFaceDocBuilderDev · 2026-10-14T06:45:33Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

BenjaminBossan

Thanks for the last update, the PR now LGTM.

FIX Keep the order of rank_pattern and alpha_pattern keys when saving…

5916eac

… the config Co-Authored-By: Claude Opus 5.5 <[email protected]>

github-actions Bot added the triaged label Oct 12, 2026

chrikrah reviewed Oct 12, 2026

View reviewed changes

BenjaminBossan mentioned this pull request Oct 13, 2026

Add state dict regression cases for overlapping rank/alpha pattern keys #3834

Closed

Sort all config keys except rank_pattern and alpha_pattern

5c17ba6

Co-Authored-By: Claude Opus 5.5 <[email protected]>

BenjaminBossan requested changes Oct 13, 2026

View reviewed changes

Address review: clarify comments

301748f

Co-Authored-By: Claude Opus 5.5 <[email protected]>

BenjaminBossan approved these changes Oct 14, 2026

View reviewed changes

BenjaminBossan merged commit da5e2aa into huggingface:main Oct 14, 2026
10 checks passed

jayzuccarelli deleted the fix/pattern-order-roundtrip branch October 14, 2026 08:15

devYRPauli mentioned this pull request Oct 15, 2026

n_frequency_pattern, block_size_pattern and lambda_pattern lose their key order on save #3985

Open

Navigation Menu

Repository navigation

FIX Keep the order of rank_pattern and alpha_pattern keys on save - #3947

FIX Keep the order of rank_pattern and alpha_pattern keys on save#3947
BenjaminBossan merged 3 commits into
huggingface:mainfrom
jayzuccarelli:fix/pattern-order-roundtrip

jayzuccarelli commented Oct 12, 2026 •

edited

Loading

Uh oh!

chrikrah left a comment

Uh oh!

BenjaminBossan commented Oct 13, 2026

Uh oh!

jayzuccarelli commented Oct 13, 2026

Uh oh!

BenjaminBossan left a comment

Uh oh!

BenjaminBossan Oct 13, 2026

Uh oh!

jayzuccarelli Oct 13, 2026

Uh oh!

Uh oh!

BenjaminBossan Oct 13, 2026

Uh oh!

jayzuccarelli Oct 13, 2026

Uh oh!

HuggingFaceDocBuilderDev commented Oct 14, 2026

Uh oh!

BenjaminBossan left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

jayzuccarelli commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chrikrah left a comment

Choose a reason for hiding this comment

Uh oh!

BenjaminBossan commented Oct 13, 2026

Uh oh!

jayzuccarelli commented Oct 13, 2026

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

BenjaminBossan Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

jayzuccarelli Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

BenjaminBossan Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

jayzuccarelli Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

HuggingFaceDocBuilderDev commented Oct 14, 2026

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jayzuccarelli commented Oct 12, 2026 •

edited

Loading