System Info
- Transformers version: 4.40.0
Who can help?
@molbap @ArthurZucker
Information
Tasks
Reproduction
I tried to load https://huggingface.co/state-spaces/mamba2-130m into HF-compatible Mamba-2 (#32080), using the convert_mamba2_ssm_checkpoint_to_pytorch.py script. But the script assumes model weights to be in safetensors format:
|
with safe_open(mamba2_checkpoint_path, framework="pt") as f: |
|
for k in f.keys(): |
|
newk = k.removeprefix("model.") |
|
original_state_dict[newk] = f.get_tensor(k).clone() |
but the weight file is is in torch bin format and cannot be opened in this way.
Also, the script requires a tokenizer path:
|
parser.add_argument( |
|
"-c", |
|
"--tokenizer_model_path", |
|
type=str, |
|
required=True, |
|
help="Path to a `config.json` file corresponding to a Mamba2Config of the original mamba2_ssm model.", |
|
) |
but state-spaces/mamba2-130m reuses EleutherAI/gpt-neox-20b tokenizer instead of having its own.
Expected behavior
convert_mamba2_ssm_checkpoint_to_pytorch.py should be able to convert those Mamba-2 weights:
System Info
Who can help?
@molbap @ArthurZucker
Information
Tasks
examplesfolder (such as GLUE/SQuAD, ...)Reproduction
I tried to load https://huggingface.co/state-spaces/mamba2-130m into HF-compatible Mamba-2 (#32080), using the convert_mamba2_ssm_checkpoint_to_pytorch.py script. But the script assumes model weights to be in safetensors format:
transformers/src/transformers/models/mamba2/convert_mamba2_ssm_checkpoint_to_pytorch.py
Lines 32 to 35 in 984bc11
but the weight file is is in torch bin format and cannot be opened in this way.
Also, the script requires a tokenizer path:
transformers/src/transformers/models/mamba2/convert_mamba2_ssm_checkpoint_to_pytorch.py
Lines 55 to 61 in 984bc11
but
state-spaces/mamba2-130mreusesEleutherAI/gpt-neox-20btokenizer instead of having its own.Expected behavior
convert_mamba2_ssm_checkpoint_to_pytorch.pyshould be able to convert those Mamba-2 weights: