Add Support for Tracking Parameter Names (named_parameters) in Optimizer State Dict #134107

ErezYosef · 2024-08-21T13:47:57Z

A proposal addressing Issue #1489: Optimizer should track parameter names and not id.

(also mentioned in here: [RFC] Introducing FQNs/clarity eyeglasses to optim state_dict

Summary

This PR introduces a backward-compatible enhancement where optimizers track parameter names instead of just their id.
Optimizers can be initialized with named_parameters() as:

optimizer = optim.SGD(model.named_parameters(), lr=0.01, momentum=0.9)

This allows for greater clarity and ease when handling optimizers, as the parameters' names are preserved within the optimizer’s state_dict as:

state_dict = 
{
    'state': {
    0: {'momentum_buffer': tensor(...), ...},
    1: {'momentum_buffer': tensor(...), ...},
    },
    'param_groups': [
        {
        'lr': 0.01,
        'weight_decay': 0,
        ...
        'params': [0,1]
        'param_names' ['layer.weight', 'layer.bias']  (optional)
        }
    ]
}

Loading state_dict is not changed (backward-compatible) and the param_names key will be ignored.

Key Features

Named Parameters in Optimizer Initialization:

Optimizers can accept the output of model.named_parameters() during initialization, allowing them to store parameter names directly.

Parameter Names in `state_dict`:

The parameter names are saved as a list in the optimizer’s state_dict with key param_names, alongside the params indices, ensuring seamless tracking of both names and parameters.

Backward Compatibility

No Breaking Changes:

This change is fully backward-compatible. The added param_names key in the optimizer's state_dict is ignored when loading a state to the optimizer.

Customization with Hooks:

For more control, the loaded state_dict can be modified using a custom register_load_state_dict_pre_hook, providing flexibility for different design needs.

Documentation Updates

Please refer to the documentation changes for more details on how this feature is implemented and how it can be used effectively.

Solution Example:

A suggested solution to the problem mentioned in #1489, for the same parameters but in a different order.
The following register_load_state_dict_pre_hook should be added to the optimizer before loading to enable loading the state dict :

def adapt_state_dict_ids(optimizer, state_dict):
    # assuming a single param group. 
    current_state_group = optimizer.state_dict()['param_groups'][0]
    loaded_state_group = state_dict['param_groups'][0]

    # same number of params, same names, only different ordering
    current_state_name_to_id_mapping = {}  # mapping --  param_name: id
    for i, name in enumerate(current_state_group['param_names']):
        current_state_name_to_id_mapping[name] = current_state_group['params'][i]

    # changing the ids of the loaded state dict to match the order of the given state dict.
    for i, name in enumerate(current_state_group['param_names']):
        loaded_state_group['params'][i] = current_state_name_to_id_mapping[name]

    return state_dict

In this code, the loaded state_dict ids are adapted to match the order of the current optimizer state_dict.
Both the previous and the current optimizers are required to be initiated with named_parameters() to have the 'param_names' key in the dict.

Note

This is my first contribution to PyTorch, and I wish to receive feedback or suggestions for improvement.

pytorch-bot · 2024-08-21T13:48:01Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/134107

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 8f3560a with merge base de4c2a3 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

linux-foundation-easycla · 2024-08-21T13:48:02Z

The committers listed above are authorized under a signed CLA.

✅ login: ErezYosef / name: Erez Yosef (18afbe2, 89dc602, 418becd, db0f35f, e8cb806, a3de0e1, 179bdcd, 327be61, 8ec59f4, 3f0abc3, 92c79fa, 7ce4644, 5730e9a, 76ebd3c, 8f3560a, 9d7a208)

vadimkantorov · 2024-08-21T22:51:55Z

Is this related?

Also allow dicts as type of params= field in param groups of optimizers #72146

ErezYosef · 2024-08-22T09:00:08Z

Is this related?

Also allow dicts as type of params= field in param groups of optimizers #72146

Issue #72146 mentions this problem of missing names, but suggests changing the state_dict['param_groups']['params'] from list to dict. It is an option but backward compatibility and the ordering of the parameters should be checked.

Also this problem is mentioned in Issue 71683 comment 1019456452 as said: "provide param_names list inside the param group" , as suggested here.

janeyx99 · 2024-08-22T17:07:42Z

hi @ErezYosef! Thanks for the well-written proposal and implementation attempt. I agree this is a good direction to go towards, where we gently enable accepting named_parameters without breaking BC.

Before I review your PR in full, it'd be essential to add testing and docs:

Add tests in test/test_optim.py to verify behavior. For example, I would expect that an optimizer given the named_parameters would be saved and checkpointed properly. We'd also want to make sure that nothing semantically changes for all our implementations when parameters are named_parameters().
Add documentation for each optimizer in their respective files (though should mostly be modularized in optimizer.py).
Add docs to optim.rst to show the use case people would use this for

ErezYosef · 2024-08-23T11:51:01Z

Thanks @janeyx99 .
I pushed the changes you requested.
Please review the test I added, I hope it is good.

Erez.

vadimkantorov · 2024-08-23T15:39:39Z

Maybe also this is related:

[proposal] "Name" string attribute for modules, parameters, buffers, tensors for more pleasant debugging (including distributed FSDP2 logging) #104247

could be an alternative, e.g. when .parameters() is called it would e.g. it could return param.with_name(name) which could then be used

With named_parameters() support of optimizers, maybe it's less pressing

ErezYosef · 2024-08-25T11:53:59Z

Maybe also this is related:

[proposal] "Name" string attribute for modules, parameters, buffers, tensors for more pleasant debugging (especially for graph printouts / export / studying compiled generated code) #104247

could be an alternative, e.g. when .parameters() is called it would e.g. it could return param.with_name(name) which could then be used

With named_parameters() support of optimizers, maybe it's less pressing

It looks like a very useful feature that could enhance debugging and the overall experience.
Once it is fully characterized and implemented the optimizers can also support it (according to the specific design choices that need to be made).

However, the current PR contributes to the existing code as it is, and enables more flexibility (without having major changes).
Therefore, I believe it can be merged independently of any potential future features.

Erez.

test/test_optim.py

ErezYosef · 2024-09-15T13:11:54Z

Hi @janeyx99

I added a few more tests + explanations and an example to present how to use the new feature.
Please check these tests and verify their correctness.

Thanks,
Erez.

janeyx99

Sorry this review took me so long--I just got back from hyperfocusing on the conference + traveling to the conference and am prioritizing responding to GH notifications now! I would like to get your PR in by the end of this week if possible.

Thanks for your work and for adding tests--I've added some more highlevel comments throughout. Mainly, I'm curious how far we want to support:

Do we want to support mixed names + no names within a param group? (I'm thinking no)
Do we want to support param groups with names + param groups without names within one optimizer? (Maybe, but if there's no real use case for this, I'd also argue to stay as simple as we can).
What would you guess is the main use case for using named parameters? Is it for easier filtering later? For debugging? For sharding the statedict? Whatever that is, I'd like to showcase an example in the docs.
What is the expected behavior of:

optim = AdamW(model.named_parameters())
named_sd = optim.state_dict()

optim2 = AdamW(model.parameters())
optim2.load_state_dict(named_sd)

and

optim = AdamW(model.parameters())
unnamed_sd = optim.state_dict()

optim2 = AdamW(model.named_parameters())
optim2.load_state_dict(unnamed_sd)

For each of the above, I'd love to get a simple test case going exhibiting the behavior. For example, if we want to error, we should add combinations of mixed names/no names in get_error_inputs_for_all_optims.

docs/source/optim.rst

test/test_optim.py

torch/optim/optimizer.py

test/test_optim.py

ErezYosef · 2024-10-01T14:10:12Z

Thanks so much for your detailed review, @janeyx99, and no worries about the timing!

Regarding your questions:

I completely agree with you. I’ll ensure that we don’t mix parameters with and without names within a parameter group.
Keeping things simple sounds like a good approach, and I can enforce consistent behavior across all groups. However, this would introduce a constraint that may not be strictly necessary.
Named parameters primarily offer flexibility for various use cases, including easier filtering, debugging, and managing the state_dict, as you mentioned. I’ll add examples to showcase how this can be useful.
The example I added in optim.rst (involving fc1 and fc2) is similar to the scenario that led me to open this PR (training of multi-experts).
It can also be used to apply something like "strict=False" strict keyword doesn't exist in the optimizer's load_state_dict #3852 (see added example)
The behavior in these cases would ignore the names in the state dict (for backward compatibility), treating them as the optimizer was initialized with model.parameters(). I’ll add tests to cover this as you suggested.

I’ll make sure to include test cases to capture these behaviors, including scenarios where we expect an error if mixed names are used.

Thanks again,
Erez.

janeyx99 · 2024-10-02T17:21:38Z

Ah thank you. Let me know when this PR is ready for another review--I've kicked off the CI for now.

ErezYosef · 2024-10-03T09:58:06Z

Ah thank you. Let me know when this PR is ready for another review--I've kicked off the CI for now.

Thanks,
I assume these commits address all your points.

Addressing the points:

I added the raise:

        if len(extracted_param_names) != 0:
            if len(extracted_param_names) == len(extracted_param_tensors):
                param_group['param_names'] = extracted_param_names
            else:
                raise ValueError("all optimizer params should be with/without names. Some param names are missing")

I added:

                ....
                raise ValueError("all optimizer param groups should be with/without names. "
                                 f'cannot add param group {current_group_txt} to the optimizer')

Examples were added to the doc
I add a test for the behavior:

            # Make sure that param_names are preserved when provided to at least one of the optimizers
            if is_named_optim0 or is_named_optim1:
                self.assertEqual(optimizer2.state_dict()['param_groups'][0]['param_names'],
                                 ['0.weight', '0.bias', '1.weight', '1.bias'])

Feel free to let me know if there's anything else to add.
Erez.

janeyx99

Looks really good! CI looks like a mess though--could you rebase and let's see if these tests all pass.

janeyx99 · 2024-10-04T18:50:28Z

docs/source/optim.rst

Suggested change

parameters (all should be :class:`~torch.nn.Parameter` s) to optimize or named parameters

(tuples of (str, :class:`~torch.nn.Parameter`)). Then,

parameters (all should be :class:`~torch.nn.Parameter` s) or named parameters

(tuples of (str, :class:`~torch.nn.Parameter`)) to optimize. Then,

janeyx99 · 2024-10-04T22:34:15Z

docs/source/optim.rst

Suggested change

--------------------------------------------

------------------------------------------------------------

this needs to match the line above

docs/source/optim.rst

janeyx99 · 2024-10-04T23:03:59Z

torch/optim/optimizer.py

Suggested change

If ``param_names`` exist in loaded state dict ``param_groups`` they will be saved or will override

If ``param_names`` exist in loaded state dict ``param_groups`` they will be saved and override

torch/optim/optimizer.py

janeyx99 · 2024-10-04T23:10:04Z

torch/optim/optimizer.py

yea i agree with you this might be an unnecessary constraint but if someone wants this feature later it shouldn't be awful to add it. thanks for the error here!

ErezYosef · 2024-10-06T07:06:35Z

Thanks.
I am not sure how the CI works. Hope it will work easily after a rebase.

Do you want me to rebase on upstream viable/strict? or which branch?
should it be forced?

Do you know if these commands will work?

git pull --rebase upstream viable/strict
git push -f

Erez.

janeyx99 · 2024-10-06T12:07:51Z

Rebasing onto viable/strict or main would both work. the commands you wrote should be fine—i typically go for a git fetch followed by a git rebase.

…ave-load equality

janeyx99

Approval contingent on green CI :D (and that you address the nits on wording)

docs/source/optim.rst

Co-authored-by: Jane (Yuan) Xu <[email protected]>

janeyx99 · 2024-10-09T15:12:58Z

docs/source/optim.rst

-and optimizer states from ``model`` into both ``fc1`` and ``fc2`` of ``model2``
-(and adjust them accordingly)::
+Let's say that ``model`` implements an expert (MoE), and we want to duplicate it and resume training
+for two experts, both initialized the same way as the ``fc`` layer. For the following ``model2`` wecreate two layers identical to ``fc`` and resume training by loading the model weights and optimizer states from ``model`` into both ``fc1`` and ``fc2`` of ``model2`` (and adjust them accordingly)::


Suggested change

for two experts, both initialized the same way as the ``fc`` layer. For the following ``model2`` wecreate two layers identical to ``fc`` and resume training by loading the model weights and optimizer states from ``model`` into both ``fc1`` and ``fc2`` of ``model2`` (and adjust them accordingly)::

for two experts, both initialized the same way as the ``fc`` layer. For the following ``model2`` we create two layers identical to ``fc`` and resume training by loading the model weights and optimizer states from ``model`` into both ``fc1`` and ``fc2`` of ``model2`` (and adjust them accordingly)::

Fixed in commit 9d7a208

janeyx99 · 2024-10-09T15:50:20Z

CI failures are real! hud.pytorch.org/pr/134107. Please verify the tests pass locally: python test/test_optim.py -k <test_name> before committing. lmk if you're stuck on any errors, the SparseAdam ones you can follow the example from the other tests to make the grads sparse 😛

ErezYosef · 2024-10-09T16:17:18Z

Thansk.

I can see how to make grad sparse. but it is a kind of copy-paste of test_can_load_older_state_dict so why is it works on the original test?
Not sure I understand this error:

X linux-jammy-py3.9-gcc11 / test (docs_test, 1, 1, linux.2xlarge)
    Queued: 0.0s, Duration: 10.0m, Test insights, Show artifacts, Raw logs

▼ /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/jit/_trace.py:791: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:

/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/jit/_trace.py:791: TracerWarning: 
Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:

2024-10-09T16:01:47.8645453Z Mismatched elements: 4 / 12 (33.3%)
2024-10-09T16:01:47.8646178Z Greatest absolute difference: 0.39807265996932983 at index (0, 2) (up to 1e-05 allowed)
2024-10-09T16:01:47.8647146Z Greatest relative difference: 1.0307998656385509 at index (0, 1) (up to 1e-05 allowed)
2024-10-09T16:01:47.8647768Z   _check_trace(
2024-10-09T16:01:47.8679892Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/jit/_trace.py:791: TracerWarning: Trace had nondeterministic nodes. Did you forget call .eval() on your model? Nodes:
2024-10-09T16:01:47.8681483Z 	%14 : Float(1, 4, strides=[4, 1], requires_grad=0, device=cpu) = aten::rand(%9, %10, %11, %12, %13) # <doctest default[0]>:2:0
2024-10-09T16:01:47.8682670Z This may cause errors in trace checking. To disable trace checking, pass check_trace=False to torch.jit.trace()
2024-10-09T16:01:47.8683459Z   _check_trace(
2024-10-09T16:01:47.8685149Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/jit/_trace.py:791: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:
2024-10-09T16:01:47.8686536Z Tensor-likes are not close!

What is its source and maybe you have a clue how to solve it?

Thanks,
Erez.

janeyx99 · 2024-10-09T16:56:41Z

Ah, it's skipped in our skip list: https://github.com/pytorch/pytorch/blob/main/torch/testing/_internal/common_optimizers.py#L2163-L2169 I think skipping it is fine as well, though I'd prefer it be in the test case, e.g.,

@optims([o for o in optims_db if not o.only_supports_sparse_grads])
def test...

janeyx99 · 2024-10-09T16:58:27Z

For the docs test, the error in jit is a red herring. The more relevant line is

2024-10-09T16:18:53.1562928Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/optim/optimizer.py:docstring of torch.optim.optimizer.Optimizer.load_state_dict:7: WARNING: Block quote ends without a blank line; unexpected unindent.

janeyx99 · 2024-10-11T17:48:52Z

For lint, I'd recommend pip install lintrunner and then running lintrunner --init; lintrunner -a to get all of lint to be fixed for ci.

ErezYosef · 2024-10-13T09:33:19Z

For lint, I'd recommend pip install lintrunner and then running lintrunner --init; lintrunner -a to get all of lint to be fixed for ci.

Thanks, @janeyx99. However, I'm running into some issues with this command, possibly related to its dependencies.

ErezYosef · 2024-10-14T15:37:31Z

Looks like we have green CI. 💯
What should I do next?

janeyx99 · 2024-10-14T16:48:44Z

Wonderful!

@pytorchbot merge

pytorchmergebot · 2024-10-14T16:50:34Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

ErezYosef · 2024-10-15T09:54:31Z

docs/source/optim.rst

+How to utilize named parameters to load optimizer state dict
+------------------------------------------------------------
+
+The function :func:`~Optimizer.load_state_dict` stores the optional ``param_names``content from the


Probably missing whitespace after param_names

if u want to open another pr to address, i'd approve it

@janeyx99

### Description: This PR addresses a minor [formatting issue identified in a previous contribution to the Optimizer documentation](#134107 (comment)). Specifically, it fixes the missing whitespace after `param_names` in the section on utilizing named parameters to load the optimizer state dict. You can find the related docs here: [Optimizer Documentation](https://pytorch.org/docs/main/optim.html#how-to-utilize-named-parameters-to-load-optimizer-state-dict). @janeyx99 Pull Request resolved: #138321 Approved by: https://github.com/janeyx99

ErezYosef requested review from albanD and janeyx99 as code owners August 21, 2024 13:47

pytorch-bot bot added the release notes: optim label Aug 21, 2024

pytorchbot added the open source label Aug 21, 2024

mikaylagawarecki added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Aug 23, 2024

janeyx99 reviewed Sep 5, 2024

View reviewed changes

test/test_optim.py Outdated Show resolved Hide resolved

janeyx99 reviewed Sep 30, 2024

View reviewed changes

janeyx99 reviewed Oct 4, 2024

View reviewed changes

ErezYosef added 8 commits October 6, 2024 15:27

Adding support for named_parameters in optimizer state dict.

db0f35f

Adding docs and tests.

18afbe2

Adding BC tests: same results with named_parameters, and satet_dict s…

3f0abc3

…ave-load equality

Adding docs explanations to utilize the new feature.

92c79fa

fix example typos

e8cb806

code adaptations and docs examples

a3de0e1

adding the required tests

89dc602

Fixes and improvements based on review feedback

179bdcd

janeyx99 approved these changes Oct 7, 2024

View reviewed changes

docs/source/optim.rst Outdated Show resolved Hide resolved

docs/source/optim.rst Outdated Show resolved Hide resolved

docs/source/optim.rst Outdated Show resolved Hide resolved

docs/source/optim.rst Outdated Show resolved Hide resolved

ErezYosef and others added 2 commits October 9, 2024 11:53

Apply suggestions from code review

327be61

Co-authored-by: Jane (Yuan) Xu <[email protected]>

Merge branch 'main' into optimizer_named_nightly

5730e9a

janeyx99 reviewed Oct 9, 2024

View reviewed changes

ErezYosef added 5 commits October 9, 2024 20:47

Lint tests fix

418becd

Tests fix

9d7a208

Lint minor fix

76ebd3c

test_fix

7ce4644

fixing indentation trying to make CI satisfied

8ec59f4

lint fix

8f3560a

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 14, 2024

pytorchmergebot added the merging label Oct 14, 2024

pytorchmergebot added the Merged label Oct 14, 2024

pytorchmergebot closed this in 197601e Oct 14, 2024

pytorchmergebot removed the merging label Oct 14, 2024

ErezYosef commented Oct 15, 2024

View reviewed changes

ErezYosef mentioned this pull request Oct 18, 2024

Documentation Update: Fix Missing Whitespace in Optimizer Docs #138321

Closed

ganow mentioned this pull request Feb 20, 2025

Fix problematic behavior of optimizer/scheduler in FeatureInversionTask KamitaniLab/bdpy#101

Merged

-parameters (all should be :class:`~torch.nn.Parameter` s) to optimize or named parameters
-(tuples of (str, :class:`~torch.nn.Parameter`)). Then,
+parameters (all should be :class:`~torch.nn.Parameter` s) or named parameters
+(tuples of (str, :class:`~torch.nn.Parameter`)) to optimize. Then,

	--------------------------------------------
	------------------------------------------------------------

	If ``param_names`` exist in loaded state dict ``param_groups`` they will be saved or will override
	If ``param_names`` exist in loaded state dict ``param_groups`` they will be saved and override

	for two experts, both initialized the same way as the ``fc`` layer. For the following ``model2`` wecreate two layers identical to ``fc`` and resume training by loading the model weights and optimizer states from ``model`` into both ``fc1`` and ``fc2`` of ``model2`` (and adjust them accordingly)::
	for two experts, both initialized the same way as the ``fc`` layer. For the following ``model2`` we create two layers identical to ``fc`` and resume training by loading the model weights and optimizer states from ``model`` into both ``fc1`` and ``fc2`` of ``model2`` (and adjust them accordingly)::

Add Support for Tracking Parameter Names (named_parameters) in Optimizer State Dict #134107

Add Support for Tracking Parameter Names (named_parameters) in Optimizer State Dict #134107

Uh oh!

Conversation

ErezYosef commented Aug 21, 2024

Summary

Key Features

Named Parameters in Optimizer Initialization:

Parameter Names in state_dict:

Backward Compatibility

No Breaking Changes:

Customization with Hooks:

Documentation Updates

Solution Example:

Note

Uh oh!

pytorch-bot bot commented Aug 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/134107

✅ No Failures

Uh oh!

linux-foundation-easycla bot commented Aug 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vadimkantorov commented Aug 21, 2024

Uh oh!

ErezYosef commented Aug 22, 2024

Uh oh!

janeyx99 commented Aug 22, 2024

Uh oh!

ErezYosef commented Aug 23, 2024

Uh oh!

vadimkantorov commented Aug 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ErezYosef commented Aug 25, 2024

Uh oh!

Uh oh!

ErezYosef commented Sep 15, 2024

Uh oh!

janeyx99 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ErezYosef commented Oct 1, 2024

Uh oh!

janeyx99 commented Oct 2, 2024

Uh oh!

ErezYosef commented Oct 3, 2024

Uh oh!

janeyx99 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ErezYosef commented Oct 6, 2024

Uh oh!

janeyx99 commented Oct 6, 2024

Uh oh!

janeyx99 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Parameter Names in `state_dict`:

pytorch-bot bot commented Aug 21, 2024 •

edited

Loading

linux-foundation-easycla bot commented Aug 21, 2024 •

edited

Loading

vadimkantorov commented Aug 23, 2024 •

edited

Loading

janeyx99 left a comment •

edited

Loading