Skip to content

Commit f4c8347

Browse files
[Inference Endpoints] Allow container command/args without custom image + support them in update (#4628)
* Allow container command/args without custom image and add them to endpoint update * [Inference Endpoints] Address review: clarify container command/args replace semantics - --container-args/--container-command on update replace the current value rather than adding to it; say so in the help text instead of 'reset to the image default', which reads as restorative. Catalog-deployed endpoints already carry tuned engine flags in model.args, so overwriting them is lossy. - Drop 'Not limited to --custom-image deployments.' from the deploy help; removing the old 'Requires --custom-image.' already carries the message. - Align the container_command docstrings with container_args in create/update_inference_endpoint and InferenceEndpoint.update: model.command is as top-level as model.args and applies to managed engine images too. - Add a --container-args example to 'hf endpoints update'. - Regenerate the CLI reference. * [Inference Endpoints] Fix stale gating comment, cover the empty-args reset, fix doc example - The comment above the deploy validation said health route and port 'only make sense alongside a custom image'. They are BaseContainer fields that every image variant carries (a catalog vLLM image returns healthRoute and port), so the real reason to gate them is that this command can only build a custom image payload. - Nothing covered container_args=[] reaching the payload, which is what '--container-args ""' sends to clear the value. The CLI test mocks get_hf_api so it stops at shlex.split, and the existing SDK test only passes non-empty lists. - The guide applied vLLM tool-calling flags to a gpt2-large endpoint. --------- Co-authored-by: célina <[email protected]> Co-authored-by: Celina Hanouti <[email protected]>
1 parent 9a3222e commit f4c8347

8 files changed

Lines changed: 222 additions & 16 deletions

File tree

‎docs/source/en/guides/cli.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2324,6 +2324,13 @@ To deploy your own Docker image instead of a Hugging Face managed one, pass `--f
23242324
--type authenticated
23252325
```
23262326

2327+
`--container-args` and `--container-command` are not limited to custom images: they map to `model.args` and `model.command` in the API payload, which apply to managed images as well. On an existing endpoint both flags replace the current value rather than adding to it, so pass the full list you want and use an empty string to clear it. Endpoints deployed from the catalog already come with tuned engine flags, so check `hf endpoints describe` before overwriting them:
2328+
2329+
```bash
2330+
>>> hf endpoints update my-endpoint \
2331+
--container-args "--enable-auto-tool-choice --tool-call-parser lfm2"
2332+
```
2333+
23272334
### hf endpoints catalog
23282335

23292336
Use `hf endpoints catalog` to interact with the Inference Endpoints Model Catalog. Deploy models directly from the catalog with optimized configurations.

‎docs/source/en/guides/inference_endpoints.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,7 @@ By default the Inference Endpoint is built from a docker image provided by Huggi
100100

101101
The value to pass as `custom_image` is a dictionary containing a url to the docker container and configuration to run it. For more details about it, checkout the [Swagger documentation](https://api.endpoints.huggingface.cloud/#/v2%3A%3Aendpoint/create_endpoint).
102102

103-
For containers that need a custom entrypoint or runtime flags, pass `container_command` and/or `container_args` (each a list of tokens). They map to `model.command` and `model.args` in the API payload. The same is available from the CLI via `hf endpoints deploy --custom-image ... --container-command "..." --container-args "..."`.
103+
For containers that need a custom entrypoint or runtime flags, pass `container_command` and/or `container_args` (each a list of tokens). They map to `model.command` and `model.args` in the API payload. They are not tied to custom images: managed engine images (e.g. vLLM, SGLang) accept engine flags through `container_args` as well. The same is available from the CLI via `hf endpoints deploy ... --container-command "..." --container-args "..."`.
104104

105105
### Get or list existing Inference Endpoints
106106

@@ -248,6 +248,11 @@ InferenceEndpoint(name='my-endpoint-name', namespace='Wauplin', repository='gpt2
248248
# Update to larger instance
249249
>>> endpoint.update(accelerator="cpu", instance_size="x4", instance_type="intel-icl")
250250
InferenceEndpoint(name='my-endpoint-name', namespace='Wauplin', repository='gpt2-large', status='pending', url=None)
251+
252+
# Update the container arguments, e.g. engine flags on an endpoint running vLLM or SGLang.
253+
# Replaces the current value, it is not appended.
254+
>>> endpoint.update(container_args=["--enable-auto-tool-choice", "--tool-call-parser", "lfm2"])
255+
InferenceEndpoint(name='my-endpoint-name', namespace='Wauplin', repository='gpt2-large', status='pending', url=None)
251256
```
252257

253258
Or via CLI:
@@ -256,6 +261,7 @@ Or via CLI:
256261
hf endpoints update my-endpoint-name --repo gpt2-large
257262
hf endpoints update my-endpoint-name --min-replica 2 --max-replica 6
258263
hf endpoints update my-endpoint-name --accelerator cpu --instance-size x4 --instance-type intel-icl
264+
hf endpoints update my-endpoint-name --container-args "--enable-auto-tool-choice --tool-call-parser lfm2"
259265
```
260266

261267
### Delete the endpoint

‎docs/source/en/package_reference/cli.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1766,8 +1766,8 @@ $ hf endpoints deploy [OPTIONS] NAME
17661766
* `--custom-image TEXT`: Docker image URL for a custom container (e.g. 'nexagi/sglang:v0.5.12'). Requires '--framework custom'.
17671767
* `--health-route TEXT`: Health check route exposed by the custom container (e.g. '/health'). Requires --custom-image.
17681768
* `--port INTEGER`: Port the custom container listens on (e.g. 30000). Requires --custom-image.
1769-
* `--container-command TEXT`: Override the container entrypoint, as a quoted string split into tokens (e.g. "python -m sglang.launch_server"). Requires --custom-image.
1770-
* `--container-args TEXT`: Arguments appended to the container entrypoint, as a quoted string split into tokens (e.g. "--tp 8 --reasoning-parser qwen3"). Requires --custom-image.
1769+
* `--container-command TEXT`: Override the container entrypoint, as a quoted string split into tokens (e.g. "python -m sglang.launch_server").
1770+
* `--container-args TEXT`: Arguments appended to the container entrypoint, as a quoted string split into tokens (e.g. "--tp 8 --reasoning-parser qwen3").
17711771
* `-e, --env TEXT`: Set environment variables. E.g. --env ENV=value
17721772
* `--env-file TEXT`: Read in a file of environment variables.
17731773
* `-s, --secrets TEXT`: Set secret environment variables. E.g. --secrets SECRET=value or `--secrets HF_TOKEN` to pass your Hugging Face token.
@@ -1960,6 +1960,8 @@ $ hf endpoints update [OPTIONS] NAME
19601960
* `--framework TEXT`: The machine learning framework used for the model (e.g. 'custom').
19611961
* `--revision TEXT`: The specific model revision to deploy on the Inference Endpoint (e.g. '6c0e6080953db56375760c0471a8c5f2929baf11').
19621962
* `--task TEXT`: The task on which to deploy the model (e.g. 'text-classification').
1963+
* `--container-command TEXT`: Override the container entrypoint, as a quoted string split into tokens (e.g. "python -m sglang.launch_server"). Replaces the current value; pass an empty string to clear it.
1964+
* `--container-args TEXT`: Arguments appended to the container entrypoint, as a quoted string split into tokens (e.g. "--enable-auto-tool-choice --tool-call-parser lfm2"). Replaces the arguments currently set on the endpoint rather than adding to them, so include the ones you want to keep, run 'hf endpoints describe NAME' first to see them. Pass an empty string to clear them.
19631965
* `--min-replica INTEGER`: The minimum number of replicas (instances) to keep running for the Inference Endpoint.
19641966
* `--max-replica INTEGER`: The maximum number of replicas (instances) to scale to for the Inference Endpoint.
19651967
* `--scale-to-zero-timeout INTEGER`: The duration in minutes before an inactive endpoint is scaled to zero.
@@ -1970,6 +1972,7 @@ $ hf endpoints update [OPTIONS] NAME
19701972

19711973
Examples
19721974
$ hf endpoints update my-endpoint --min-replica 2
1975+
$ hf endpoints update my-endpoint --container-args "--enable-auto-tool-choice --tool-call-parser lfm2"
19731976

19741977
Learn more
19751978
Use `hf <command> --help` for more information about a command.

‎src/huggingface_hub/_inference_endpoints.py‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -267,6 +267,8 @@ def update(
267267
revision: str | None = None,
268268
task: str | None = None,
269269
custom_image: dict | None = None,
270+
container_command: list[str] | None = None,
271+
container_args: list[str] | None = None,
270272
secrets: dict[str, str] | None = None,
271273
) -> "InferenceEndpoint":
272274
"""Update the Inference Endpoint.
@@ -302,6 +304,12 @@ def update(
302304
custom_image (`dict`, *optional*):
303305
A custom Docker image to use for the Inference Endpoint. This is useful if you want to deploy an
304306
Inference Endpoint running on the `text-generation-inference` (TGI) framework (see examples).
307+
container_command (`list[str]`, *optional*):
308+
Override the container entrypoint command (maps to `model.command` in the API payload). Works with
309+
both managed engine images (e.g. vLLM, SGLang) and custom images.
310+
container_args (`list[str]`, *optional*):
311+
Arguments appended to the container entrypoint (maps to `model.args` in the API payload). Works with
312+
both managed engine images (e.g. vLLM, SGLang) and custom images.
305313
secrets (`dict[str, str]`, *optional*):
306314
Secret values to inject in the container environment.
307315
Returns:
@@ -322,6 +330,8 @@ def update(
322330
revision=revision,
323331
task=task,
324332
custom_image=custom_image,
333+
container_command=container_command,
334+
container_args=container_args,
325335
secrets=secrets,
326336
token=self._token, # type: ignore [arg-type]
327337
)

‎src/huggingface_hub/cli/inference_endpoints.py‎

Lines changed: 38 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -191,7 +191,7 @@ def deploy(
191191
"--container-command",
192192
help=(
193193
"Override the container entrypoint, as a quoted string split into tokens "
194-
'(e.g. "python -m sglang.launch_server"). Requires --custom-image.'
194+
'(e.g. "python -m sglang.launch_server").'
195195
),
196196
),
197197
] = None,
@@ -201,7 +201,7 @@ def deploy(
201201
"--container-args",
202202
help=(
203203
"Arguments appended to the container entrypoint, as a quoted string split into tokens "
204-
'(e.g. "--tp 8 --reasoning-parser qwen3"). Requires --custom-image.'
204+
'(e.g. "--tp 8 --reasoning-parser qwen3").'
205205
),
206206
),
207207
] = None,
@@ -219,9 +219,11 @@ def deploy(
219219
] = None,
220220
) -> None:
221221
"""Deploy an Inference Endpoint from a Hub repository."""
222-
# Custom-container knobs only make sense alongside a custom image.
223-
if custom_image is None and (health_route is not None or port is not None or container_command or container_args):
224-
raise CLIError("--health-route, --port, --container-command and --container-args require --custom-image.")
222+
# --health-route and --port are container-level fields, and the only container payload this
223+
# command can build is the custom one, so they need --custom-image. Container command/args are
224+
# top-level model fields and apply to managed engine images too, so they are not gated.
225+
if custom_image is None and (health_route is not None or port is not None):
226+
raise CLIError("--health-route and --port require --custom-image.")
225227
custom_image_dict: dict | None = None
226228
if custom_image is not None:
227229
custom_image_dict = {"url": custom_image}
@@ -346,7 +348,12 @@ def describe(
346348
out.dict(endpoint.raw)
347349

348350

349-
@ie_cli.command(examples=["hf endpoints update my-endpoint --min-replica 2"])
351+
@ie_cli.command(
352+
examples=[
353+
"hf endpoints update my-endpoint --min-replica 2",
354+
'hf endpoints update my-endpoint --container-args "--enable-auto-tool-choice --tool-call-parser lfm2"',
355+
]
356+
)
350357
def update(
351358
name: NameArg,
352359
namespace: NamespaceOpt = None,
@@ -392,6 +399,29 @@ def update(
392399
help="The task on which to deploy the model (e.g. 'text-classification').",
393400
),
394401
] = None,
402+
container_command: Annotated[
403+
str | None,
404+
Option(
405+
"--container-command",
406+
help=(
407+
"Override the container entrypoint, as a quoted string split into tokens "
408+
'(e.g. "python -m sglang.launch_server"). Replaces the current value; '
409+
"pass an empty string to clear it."
410+
),
411+
),
412+
] = None,
413+
container_args: Annotated[
414+
str | None,
415+
Option(
416+
"--container-args",
417+
help=(
418+
"Arguments appended to the container entrypoint, as a quoted string split into tokens "
419+
'(e.g. "--enable-auto-tool-choice --tool-call-parser lfm2"). Replaces the arguments currently '
420+
"set on the endpoint rather than adding to them, so include the ones you want to keep, run "
421+
"'hf endpoints describe NAME' first to see them. Pass an empty string to clear them."
422+
),
423+
),
424+
] = None,
395425
min_replica: Annotated[
396426
int | None,
397427
Option(
@@ -434,6 +464,8 @@ def update(
434464
framework=framework,
435465
revision=revision,
436466
task=task,
467+
container_command=shlex.split(container_command) if container_command is not None else None,
468+
container_args=shlex.split(container_args) if container_args is not None else None,
437469
accelerator=accelerator,
438470
instance_size=instance_size,
439471
instance_type=instance_type,

‎src/huggingface_hub/hf_api.py‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9416,11 +9416,11 @@ def create_inference_endpoint(
94169416
Inference Endpoint running on the `text-generation-inference` (TGI) framework or a custom container
94179417
(see examples).
94189418
container_command (`list[str]`, *optional*):
9419-
Override the container entrypoint command (maps to `model.command` in the API payload). Typically
9420-
used together with `custom_image`.
9419+
Override the container entrypoint command (maps to `model.command` in the API payload). Works with
9420+
both managed engine images (e.g. vLLM, SGLang) and custom images.
94219421
container_args (`list[str]`, *optional*):
9422-
Arguments appended to the container entrypoint (maps to `model.args` in the API payload). Typically
9423-
used together with `custom_image` to pass runtime flags to the container.
9422+
Arguments appended to the container entrypoint (maps to `model.args` in the API payload). Works with
9423+
both managed engine images (e.g. vLLM, SGLang) and custom images.
94249424
env (`dict[str, str]`, *optional*):
94259425
Non-secret environment variables to inject in the container environment.
94269426
secrets (`dict[str, str]`, *optional*):
@@ -9760,6 +9760,8 @@ def update_inference_endpoint(
97609760
revision: str | None = None,
97619761
task: str | None = None,
97629762
custom_image: dict | None = None,
9763+
container_command: list[str] | None = None,
9764+
container_args: list[str] | None = None,
97639765
env: dict[str, str] | None = None,
97649766
secrets: dict[str, str] | None = None,
97659767
# Route update
@@ -9811,6 +9813,12 @@ def update_inference_endpoint(
98119813
custom_image (`dict`, *optional*):
98129814
A custom Docker image to use for the Inference Endpoint. This is useful if you want to deploy an
98139815
Inference Endpoint running on the `text-generation-inference` (TGI) framework (see examples).
9816+
container_command (`list[str]`, *optional*):
9817+
Override the container entrypoint command (maps to `model.command` in the API payload). Works with
9818+
both managed engine images (e.g. vLLM, SGLang) and custom images.
9819+
container_args (`list[str]`, *optional*):
9820+
Arguments appended to the container entrypoint (maps to `model.args` in the API payload). Works with
9821+
both managed engine images (e.g. vLLM, SGLang) and custom images.
98149822
env (`dict[str, str]`, *optional*):
98159823
Non-secret environment variables to inject in the container environment
98169824
secrets (`dict[str, str]`, *optional*):
@@ -9865,6 +9873,10 @@ def update_inference_endpoint(
98659873
payload["model"]["task"] = task
98669874
if custom_image is not None:
98679875
payload["model"]["image"] = {"custom": custom_image}
9876+
if container_command is not None:
9877+
payload["model"]["command"] = container_command
9878+
if container_args is not None:
9879+
payload["model"]["args"] = container_args
98689880
if env is not None:
98699881
payload["model"]["env"] = env
98709882
if secrets is not None:

‎tests/test_cli.py‎

Lines changed: 80 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2632,8 +2632,11 @@ def test_deploy_custom_image(self, runner: CliRunner) -> None:
26322632
assert kwargs["env"] == {"MODEL_ID": "/repository"}
26332633
assert kwargs["type"] == "authenticated"
26342634

2635-
def test_deploy_custom_args_require_image(self, runner: CliRunner) -> None:
2635+
def test_deploy_container_args_without_custom_image(self, runner: CliRunner) -> None:
2636+
endpoint = Mock(raw={"name": "hub-args"})
26362637
with patch("huggingface_hub.cli.inference_endpoints.get_hf_api") as api_cls:
2638+
api = api_cls.return_value
2639+
api.create_inference_endpoint.return_value = endpoint
26372640
result = runner.invoke(
26382641
app,
26392642
[
@@ -2643,7 +2646,7 @@ def test_deploy_custom_args_require_image(self, runner: CliRunner) -> None:
26432646
"--repo",
26442647
"my-repo",
26452648
"--framework",
2646-
"custom",
2649+
"pytorch",
26472650
"--accelerator",
26482651
"gpu",
26492652
"--instance-size",
@@ -2655,7 +2658,38 @@ def test_deploy_custom_args_require_image(self, runner: CliRunner) -> None:
26552658
"--vendor",
26562659
"aws",
26572660
"--container-args",
2658-
"--tp 8",
2661+
"--enable-auto-tool-choice --tool-call-parser lfm2",
2662+
],
2663+
)
2664+
assert result.exit_code == 0, result.stdout
2665+
_, kwargs = api.create_inference_endpoint.call_args
2666+
assert kwargs["container_args"] == ["--enable-auto-tool-choice", "--tool-call-parser", "lfm2"]
2667+
assert "custom_image" not in kwargs
2668+
2669+
def test_deploy_port_and_health_route_require_image(self, runner: CliRunner) -> None:
2670+
with patch("huggingface_hub.cli.inference_endpoints.get_hf_api") as api_cls:
2671+
result = runner.invoke(
2672+
app,
2673+
[
2674+
"endpoints",
2675+
"deploy",
2676+
"my-endpoint",
2677+
"--repo",
2678+
"my-repo",
2679+
"--framework",
2680+
"custom",
2681+
"--accelerator",
2682+
"gpu",
2683+
"--instance-size",
2684+
"x8",
2685+
"--instance-type",
2686+
"nvidia-h200",
2687+
"--region",
2688+
"us-east-1",
2689+
"--vendor",
2690+
"aws",
2691+
"--port",
2692+
"8000",
26592693
],
26602694
)
26612695
assert result.exit_code != 0
@@ -2727,6 +2761,8 @@ def test_update(self, runner: CliRunner) -> None:
27272761
framework=None,
27282762
revision=None,
27292763
task=None,
2764+
container_command=None,
2765+
container_args=None,
27302766
accelerator="gpu",
27312767
instance_size="x4",
27322768
instance_type=None,
@@ -2739,6 +2775,47 @@ def test_update(self, runner: CliRunner) -> None:
27392775
)
27402776
assert '"name": "updated"' in result.stdout
27412777

2778+
def test_update_container_args(self, runner: CliRunner) -> None:
2779+
endpoint = Mock(raw={"name": "updated"})
2780+
with patch("huggingface_hub.cli.inference_endpoints.get_hf_api") as api_cls:
2781+
api = api_cls.return_value
2782+
api.update_inference_endpoint.return_value = endpoint
2783+
result = runner.invoke(
2784+
app,
2785+
[
2786+
"endpoints",
2787+
"update",
2788+
"my-endpoint",
2789+
"--container-args",
2790+
"--enable-auto-tool-choice --tool-call-parser lfm2",
2791+
],
2792+
)
2793+
assert result.exit_code == 0, result.stdout
2794+
_, kwargs = api.update_inference_endpoint.call_args
2795+
assert kwargs["container_args"] == ["--enable-auto-tool-choice", "--tool-call-parser", "lfm2"]
2796+
assert kwargs["container_command"] is None
2797+
assert '"name": "updated"' in result.stdout
2798+
2799+
def test_update_container_args_empty_string_resets(self, runner: CliRunner) -> None:
2800+
endpoint = Mock(raw={"name": "updated"})
2801+
with patch("huggingface_hub.cli.inference_endpoints.get_hf_api") as api_cls:
2802+
api = api_cls.return_value
2803+
api.update_inference_endpoint.return_value = endpoint
2804+
result = runner.invoke(
2805+
app,
2806+
[
2807+
"endpoints",
2808+
"update",
2809+
"my-endpoint",
2810+
"--container-args",
2811+
"",
2812+
],
2813+
)
2814+
assert result.exit_code == 0, result.stdout
2815+
_, kwargs = api.update_inference_endpoint.call_args
2816+
assert kwargs["container_args"] == []
2817+
assert kwargs["container_command"] is None
2818+
27422819
def test_delete(self, runner: CliRunner) -> None:
27432820
with patch("huggingface_hub.cli.inference_endpoints.get_hf_api") as api_cls:
27442821
api = api_cls.return_value

0 commit comments

Comments
 (0)