added model_kwargs to huggingface model#1417
Conversation
|
@adriangonz ,could you review it when you have a chance |
adriangonz
left a comment
adriangonz
left a comment
There was a problem hiding this comment.
Nice one @nanbo-liu! This would be a great addition to the HF runtime - thanks for contributing it 🚀
Changes look good - although, could you add some tests to validate that the model kwargs get used? Once tests are in, we should be good to land this one. 👍
|
@adriangonz ,added unit tests from the latest commit. |
adriangonz
left a comment
adriangonz
left a comment
There was a problem hiding this comment.
Brilliant! Thanks for the changes @nanbo-liu !
This should be good to go now 🚀
|
@adriangonz ,I rewrite the unit test a little bit. The previous version depends on package |
|
@adriangonz, just fixed the unit test bug. would you try again |
|
@adriangonz ,CICD keeps getting runtime error for |
|
Yuuup, it seems there's an unrelated error in the |
2 participants
This is for issue 1344
This allow
model-settings.jsonto takemodel_kwargs:example below:
{ "name": "llama", "implementation": "mlserver_huggingface.HuggingFaceRuntime", "parameters": { "extra": { "task": "text-generation", "pretrained_model": "daryl149/llama-2-7b-hf", "model_kwargs": { "device_map": "auto", "load_in_8bit": true } } } }