from transformers import LlamaForCausalLM, AutoTokenizer, TextGenerationPipeline

model = LlamaForCausalLM.from_pretrained("daryl149/llama-2-7b-hf",load_in_8bit=True)
tokenizer = AutoTokenizer.from_pretrained("daryl149/llama-2-7b-hf")
pipeline = TextGenerationPipeline(model, tokenizer)
pipeline("Once upon a time,", max_new_tokens=100)

Recent popular LLM like llama-2 takes tons of memory. However, with quantization its memory requirement greatly reduce.
Huggingface allow me to convert llama-2 into quantized 8bit version by simply add an keyword argument load_in_8bit=True
Whish we can specify from_pretrained_kwargs in the model-setting.json
example model-setting.json :

{
    "name": "llama_mlserver",
    "implementation": "mlserver_huggingface.HuggingFaceRuntime",
    "parameters": {
        "extra": {
            "task": "text-generation",
            "pretrained_model": "daryl149/llama-2-7b-hf",
            "model_kwargs":{"load_in_8bit":true}
        }
    }
}