Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/mlserver/parallel/worker.py", line 158, in _process_model_update
await self._model_registry.load(model_settings)
File "/opt/conda/lib/python3.8/site-packages/mlserver/registry.py", line 293, in load
return await self._models[model_settings.name].load(model_settings)
File "/opt/conda/lib/python3.8/site-packages/mlserver/registry.py", line 148, in load
await self._load_model(new_model)
File "/opt/conda/lib/python3.8/site-packages/mlserver/registry.py", line 165, in _load_model
model.ready = await model.load()
File "/opt/conda/lib/python3.8/site-packages/mlserver_huggingface/runtime.py", line 27, in load
await asyncio.get_running_loop().run_in_executor(
File "/opt/conda/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/conda/lib/python3.8/site-packages/mlserver_huggingface/common.py", line 65, in load_pipeline_from_settings
hf_pipeline.tokenizer.pad_token_id = [str(eos_token_id)] # type: ignore
File "/opt/conda/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1218, in pad_token_id
self._pad_token = self.convert_ids_to_tokens(value) if value is not None else None
File "/opt/conda/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 313, in convert_ids_to_tokens
index = int(index)
ValueError: invalid literal for int() with base 10: 'None'
A couple thoughts I have on this code.
if settings.max_batch_size:
model = hf_pipeline.model
eos_token_id = model.config.eos_token_id
hf_pipeline.tokenizer.pad_token_id = [str(eos_token_id)] # type: ignore
When batch size is greater than 0 this line will often fail with the following error since many huggingface models do not have
eos_tokendefined in their config.Examples of Huggingface models that do not have
eos_tokenspecified in their configs.https://huggingface.co/dslim/bert-base-NER/tree/main
https://huggingface.co/hf-internal-testing/tiny-bert-for-token-classification
A couple thoughts I have on this code.
if settings.match_batch_size > 1because padding shouldn't be needed for batch size one.eos_tokenas thepad_tokenwon't change any behaviors of the model since it was not trained with that pad token?