FIX: Minimal fix for loading PEFT weights #42387

BenjaminBossan · 2025-11-25T10:08:28Z

What does this PR do?

After the weight conversion PR #41580, some adjustments were still required for loading PEFT weights. This PR presents a minimal fix to make it work again.

Besides renaming keys, this PR does not address possible conversions that might be required to be applied to the PEFT weights themselves (most wouldn't work anyway, but e.g. chunking should be possible to implement).

As for test, the existing test_peft_from_pretrained in test_peft_integration.py actually fails on main right now, this PR fixes it. As the tests are slow tests, normal CI won't pick this up though.

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved on slack.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

After the weight conversion PR huggingface#41580, some adjustments were still required for loading PEFT weights. This PR presents a minimal fix to make it work again. Besides renaming keys, this PR does not address possible conversions that might be required to be applied to the PEFT weights themselves (most wouldn't work anyway, but e.g. chunking should be possible to implement). As for test, the existing test_peft_from_pretrained in test_peft_integration.py actually fails on main right now, this PR fixes it. As the tests are slow tests, normal CI won't pick this up though.

HuggingFaceDocBuilderDev · 2025-11-25T10:17:51Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

BenjaminBossan · 2025-11-25T10:59:31Z

One failing test in CI, tests/models/mixtral/test_modeling_mixtral.py::MixtralModelTest::test_load_balancing_loss, should be unrelated. LMK if/how slow tests should be triggered. I ran them locally, there are 11 failing tests without this PR, they all pass with this PR:

FAILED tests/peft_integration/test_peft_integration.py::PeftIntegrationTester::test_peft_add_adapter_from_pretrained - AttributeError: 'list' object has no attribute 'items'
FAILED tests/peft_integration/test_peft_integration.py::PeftIntegrationTester::test_peft_from_pretrained - AttributeError: 'list' object has no attribute 'items'
FAILED tests/peft_integration/test_peft_integration.py::PeftIntegrationTester::test_peft_from_pretrained_hub_kwargs - AttributeError: 'list' object has no attribute 'items'
FAILED tests/peft_integration/test_peft_integration.py::PeftIntegrationTester::test_peft_from_pretrained_kwargs - AttributeError: 'list' object has no attribute 'items'
FAILED tests/peft_integration/test_peft_integration.py::PeftIntegrationTester::test_peft_load_adapter_training_inference_mode_false - AttributeError: 'list' object has no attribute 'items'
FAILED tests/peft_integration/test_peft_integration.py::PeftIntegrationTester::test_peft_load_adapter_training_inference_mode_true - AttributeError: 'list' object has no attribute 'items'
FAILED tests/peft_integration/test_peft_integration.py::PeftIntegrationTester::test_peft_pipeline - AttributeError: 'list' object has no attribute 'items'
FAILED tests/peft_integration/test_peft_integration.py::PeftIntegrationTester::test_peft_save_pretrained - AttributeError: 'list' object has no attribute 'items'
FAILED tests/peft_integration/test_peft_integration.py::PeftIntegrationTester::test_peft_save_quantized - AttributeError: 'list' object has no attribute 'items'
FAILED tests/peft_integration/test_peft_integration.py::PeftIntegrationTester::test_peft_save_quantized_regression - AttributeError: 'list' object has no attribute 'items'
FAILED tests/peft_integration/test_peft_integration.py::PeftIntegrationTester::test_peft_state_dict - AttributeError: 'list' object has no attribute 'items'

vasqu

Preapproving but we need to sync with the other PR before it gets re-broken

Trusting you on the functionality here!

vasqu · 2025-11-26T15:46:35Z

src/transformers/integrations/peft.py

+                    elif len(patterns) > 1 and len(replacements) > 1:
+                        raise ValueError(
+                            "WeightConversion incorrectly initialized, please open an issue and report this error "
+                            "here: https://github.com/huggingface/transformers/issues"
+                        )


There will be n:n patterns in #39585, so maybe we should add exceptions for those? Not sure when I get to merge that PR because handling memory issues on the converter which need to be resolved first

I changed the code to not raise in case of n:n

vasqu · 2025-11-26T15:50:16Z

src/transformers/integrations/peft.py

+                    patterns = weight_converter.source_keys
+                    replacements = weight_converter.target_keys


Let's be careful here and sync with #42396 cc @Cyrilvallez

BenjaminBossan

Preapproving but we need to sync with the other PR before it gets re-broken

I'll leave it up to the transformers devs to decide what to merge in what order, as I truly have no overview :) Please just ping me if this PR needs to be updated.

Trusting you on the functionality here!

At the end of the day, this functionality is pretty much a no-op for most cases. At the very least, the PR makes it so that loading works for those cases (right now it's broken). For full support of key mapping, there will be more work in the future, but from my POV, this PR is a strict improvement and (absent an alternative) should be in v5.

BenjaminBossan · 2025-11-26T16:09:58Z

src/transformers/integrations/peft.py

+                    elif len(patterns) > 1 and len(replacements) > 1:
+                        raise ValueError(
+                            "WeightConversion incorrectly initialized, please open an issue and report this error "
+                            "here: https://github.com/huggingface/transformers/issues"
+                        )


I changed the code to not raise in case of n:n

ArthurZucker

Can you update the renaming using the alts and globs!

ArthurZucker · 2025-11-27T14:15:46Z

src/transformers/integrations/peft.py

+                    # mapping is either 1:1 or 1:n or n:1 or n:n
+                    if len(patterns) > 1 and len(replacements) == 1:
+                        replacements = len(patterns) * replacements
+                    elif len(patterns) == 1 and len(replacements) > 1:
+                        patterns = len(replacements) * patterns
+                    elif len(patterns) > 1 and len(replacements) > 1 and len(patterns) != len(replacements):
+                        raise ValueError(
+                            "WeightConversion incorrectly initialized, please open an issue and report this error "
+                            "here: https://github.com/huggingface/transformers/issues"
+                        )
+
+                    for pattern, replacement in zip(patterns, replacements, strict=True):
+                        new_key, n_replace = re.subn(pattern, replacement, new_key)
+                        # Early exit of the loop
+                        if n_replace > 0:
+                            break


This is fine, but lets rather use this:

renamings = [entry for entry in weight_mapping if isinstance(entry, WeightRenaming)] rename_alt, _, rename_by_group = build_glob_alternation(renamings) renamed_key = rename_alt.sub(lambda m: repl(m, rename_by_group), original_key).replace("\\", "")

this is the only think you need please!

We can leave the conversion as a todo.

Let's wait for Cyril's PR, we need a self.weigh_mapping!

BenjaminBossan · 2025-11-27T14:48:20Z

@ArthurZucker Thanks for the feedback, this looks much more succinct, I changed the code accordingly. The failing test (tests/models/regnet/test_modeling_regnet.py::RegNetModelTest::test_can_load_ignoring_mismatched_shapes) looks like it's unrelated.

ArthurZucker · 2025-11-27T15:14:01Z

Merging, we still have todo for weight conversion

* FIX Minimal fix for loading PEFT weights After the weight conversion PR huggingface#41580, some adjustments were still required for loading PEFT weights. This PR presents a minimal fix to make it work again. Besides renaming keys, this PR does not address possible conversions that might be required to be applied to the PEFT weights themselves (most wouldn't work anyway, but e.g. chunking should be possible to implement). As for test, the existing test_peft_from_pretrained in test_peft_integration.py actually fails on main right now, this PR fixes it. As the tests are slow tests, normal CI won't pick this up though. * Allow n:n matching * Reviewer feedback

BenjaminBossan requested review from ArthurZucker and vasqu November 25, 2025 10:59

tomaarsen mentioned this pull request Nov 26, 2025

Allow fallback to loading from Auto"SubProcessor".from_pretrained when model_type can't be inferred from config #42402

Merged

vasqu approved these changes Nov 26, 2025

View reviewed changes

BenjaminBossan added 2 commits November 26, 2025 17:09

Allow n:n matching

423fbc6

Merge branch 'main' into fix-issue-weight-converter-peft

ee7ea24

BenjaminBossan commented Nov 26, 2025

View reviewed changes

ArthurZucker reviewed Nov 27, 2025

View reviewed changes

BenjaminBossan added 2 commits November 27, 2025 15:36

Reviewer feedback

5807a3c

Merge branch 'main' into fix-issue-weight-converter-peft

59f051b

ArthurZucker merged commit e31c5f6 into huggingface:main Nov 27, 2025
21 of 23 checks passed

BenjaminBossan deleted the fix-issue-weight-converter-peft branch November 27, 2025 15:14

albertvillanova mentioned this pull request Dec 4, 2025

CI fails with dev dependencies: AttributeError: 'list' object has no attribute 'items' huggingface/trl#4592

Closed

		patterns = weight_converter.source_keys
		replacements = weight_converter.target_keys

FIX: Minimal fix for loading PEFT weights #42387

FIX: Minimal fix for loading PEFT weights #42387

Uh oh!

Conversation

BenjaminBossan commented Nov 25, 2025

What does this PR do?

Before submitting

Uh oh!

HuggingFaceDocBuilderDev commented Nov 25, 2025

Uh oh!

BenjaminBossan commented Nov 25, 2025

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

vasqu Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

BenjaminBossan Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

vasqu Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

BenjaminBossan Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

ArthurZucker Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

BenjaminBossan commented Nov 27, 2025

Uh oh!

Uh oh!

ArthurZucker commented Nov 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants