System Info

• transformers version: 4.34.0
• Platform: Linux-5.15.90.2-microsoft-standard-WSL2-x86_64-with-glibc2.35
• Python version: 3.10.12
• Huggingface_hub version: 0.16.4
• Safetensors version: 0.3.3
• Accelerate version: 0.23.0
• Accelerate config: not found
• PyTorch version (GPU?): 2.2.0.dev20231005 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

Who can help?

@sanchit-gandhi

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Simply running:

from transformers import AutoProcessor, HubertModel
model = HubertModel.from_pretrained("facebook/hubert-base-ls960")

Produces the following warning:

Some weights of the model checkpoint at facebook/hubert-base-ls960 were not used when initializing HubertModel: ['encoder.pos_conv_embed.conv.weight_v', 'encoder.pos_conv_embed.conv.weight_g']
- This IS expected if you are initializing HubertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing HubertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of HubertModel were not initialized from the model checkpoint at facebook/hubert-base-ls960 and are newly initialized: ['encoder.pos_conv_embed.conv.parametrizations.weight.original0', 'encoder.pos_conv_embed.conv.parametrizations.weight.original1']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

What I gather from the PyTorch documentation
and updated code is that the PyTorch folks decided to migrate the weight_v and weight_g params of WeightNorm to original0 and original1.

Initially I thought the model was simply broken by this breaking change in PyTorch, however I was confused since I saw discussions that it should have been fixed by this PR in transformers, as discussed here: #24692

So I attached my debugger to _weight_norm_compat_hook, and sure enough it activated and seems to have migrated the state:
(during debug)

> state_dict[g_key]
tensor([[[0.3022, 0.1198, 0.1031, 0.1000, 0.0945, 0.0891, 0.0939, 0.0933, ...

(after model load, in Jupyter):

> model.encoder.pos_conv_embed.conv.parametrizations.weight.original0
Parameter containing:
tensor([[[0.3022, 0.1198, 0.1031, 0.1000, 0.0945, 0.0891, 0.0939, 0.0933, ...

So I'm pretty sure the warning is a false alarm, but I'm also confused since the migration happens before the warning is traced, so I wanted to check.

Expected behavior

No warning should have appeared.