Skip to content

Huggingface models frequently fail with batch size greater than 0. #1249

Description

@ajsalow

When batch size is greater than 0 this line will often fail with the following error since many huggingface models do not have eos_token defined in their config.

Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/mlserver/parallel/worker.py", line 158, in _process_model_update
    await self._model_registry.load(model_settings)
  File "/opt/conda/lib/python3.8/site-packages/mlserver/registry.py", line 293, in load
    return await self._models[model_settings.name].load(model_settings)
  File "/opt/conda/lib/python3.8/site-packages/mlserver/registry.py", line 148, in load
    await self._load_model(new_model)
  File "/opt/conda/lib/python3.8/site-packages/mlserver/registry.py", line 165, in _load_model
    model.ready = await model.load()
  File "/opt/conda/lib/python3.8/site-packages/mlserver_huggingface/runtime.py", line 27, in load
    await asyncio.get_running_loop().run_in_executor(
  File "/opt/conda/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/conda/lib/python3.8/site-packages/mlserver_huggingface/common.py", line 65, in load_pipeline_from_settings
    hf_pipeline.tokenizer.pad_token_id = [str(eos_token_id)]  # type: ignore
  File "/opt/conda/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1218, in pad_token_id
    self._pad_token = self.convert_ids_to_tokens(value) if value is not None else None
  File "/opt/conda/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 313, in convert_ids_to_tokens
    index = int(index)
ValueError: invalid literal for int() with base 10: 'None'

Examples of Huggingface models that do not have eos_token specified in their configs.
https://huggingface.co/dslim/bert-base-NER/tree/main
https://huggingface.co/hf-internal-testing/tiny-bert-for-token-classification

A couple thoughts I have on this code.

if settings.max_batch_size:
        model = hf_pipeline.model
        eos_token_id = model.config.eos_token_id
        hf_pipeline.tokenizer.pad_token_id = [str(eos_token_id)]  # type: ignore
  1. The if check could/should be if settings.match_batch_size > 1 because padding shouldn't be needed for batch size one.
  2. Can we say for certain adding the eos_token as the pad_token won't change any behaviors of the model since it was not trained with that pad token?
  3. Maybe it is safer to just check if the model has a `pad_token' and if not either raise and exception or log an error and continue but with batch size 1.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions