System Info

• transformers version: 4.30.0.dev0
• Platform: Linux-5.15.107+-x86_64-with-glibc2.31
• Python version: 3.10.11
• Huggingface_hub version: 0.14.1
• Safetensors version: 0.3.1
• PyTorch version (GPU?): 2.0.1+cu118 (True)
• Tensorflow version (GPU?): 2.12.0 (True)
• Flax version (CPU?/GPU?/TPU?): 0.6.9 (gpu)
• Jax version: 0.4.10
• JaxLib version: 0.4.10
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

@ArthurZucker @younesbelkada

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

import torch
from transformers import AutoModelForCausalLM, AutoConfig
from transformers import BitsAndBytesConfig

model_name = 'mosaicml/mpt-7b-instruct'

nf4_config = BitsAndBytesConfig(
   load_in_4bit=True,
   bnb_4bit_quant_type="nf4",
   bnb_4bit_use_double_quant=True,
   bnb_4bit_compute_dtype=torch.bfloat16
)

model_nf4 = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True, offload_folder="offload", 
                                                 offload_state_dict = True, quantization_config=nf4_config)

Error

ValueError: MPTForCausalLM does not support `device_map='auto'` yet.

I saw a similar issue #22188 for XGLMForCausalLM. I couldn't find MPTForCausalLM in the repository if the MPT model is not supported currently, is there any hack that I can use to get acclerate support ? Than

Expected behavior

Model gets loaded without any errors