Background

What are PREDICTIVE_UNIT_PARAMETERS

This is a collection of parameters passed though the environment to a HuggingFace model. This is opposed to declaring them in a model-settings.json file as it is the case in Seldon Core v2. As of this moment, those parameters are injected via Seldon Core v1 only, where as Seldon Core v2 uses model-settings.json to provide the metadata for the model.

PREDICTIVE_UNIT_PARAMETERS are used only by the HuggingFace runtime and injected only by SCv1.

You can find the code for creating HuggingFaceSettings in ./runtimes/huggingface/ml-server-huggingface/settings.py fle along with functions for parsing those params from env vars or from model-settings.json

What is the problem

Currently, HuggingFaceSettings are created either from parsing the PREDICTIVE_UNIT_PARAMETERS from the environment OR from the model-settings.json. Meaning that if there is at least one parameter set in the env var, the model-settings.json extra parameters will be ignored. This makes it cumbersome when a deployment is created from the UI because additional params such as task, pretrained_model, pretrained_tokenizer, framework, etc. will have to be added one by one in the Wizard. Why they have to be added from the wizard and not just specified in model-settings.json - because currently SCv1 always injects model_uri param so the PREDICTIVE_UNIT_PARAMETERS env var so it's not empty. Because this var is not empty, the HF settings are initialised from it and the model-settings.json is ignored.

What needs to be done

When creating HuggingFace settings, env vars needs to be merged with params from model-settings.json, giving priority to env vars. For example:
If such env var exists:

PREDICTIVE_UNIT_PARAMETERS = [{"name":"model_uri","value":"/mnt/models","type":"STRING"}, {"name":"task_suffix","value":"else","type":"STRING"}]

and such model-settings.json file exists:

{
    "name": "transformer",
    "implementation": "mlserver_huggingface.HuggingFaceRuntime",
    "parameters": {
        "extra": {
            "task": "text-generation",
            "task_suffix": "something",
            "framework": "pt"
        }
    }
}

The outcome should be that the task parameter doesn't need to be specified in the wizard and
The HuggingFace settings should contain values: task = text-generation, task_suffix = else, framework = pt

Scope

This only relating to HuggingFace runtime and when it's used from SCv1 and only valid as long as SCv1 is still operational and related code is present in MLServer.