You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f4c8347
Browse filesBrowse the repository at this point in the historyBrowse files
[Inference Endpoints] Allow container command/args without custom image + support them in update (#4628)
* Allow container command/args without custom image and add them to endpoint update
* [Inference Endpoints] Address review: clarify container command/args replace semantics
- --container-args/--container-command on update replace the current value
rather than adding to it; say so in the help text instead of 'reset to the
image default', which reads as restorative. Catalog-deployed endpoints already
carry tuned engine flags in model.args, so overwriting them is lossy.
- Drop 'Not limited to --custom-image deployments.' from the deploy help; removing
the old 'Requires --custom-image.' already carries the message.
- Align the container_command docstrings with container_args in
create/update_inference_endpoint and InferenceEndpoint.update: model.command is
as top-level as model.args and applies to managed engine images too.
- Add a --container-args example to 'hf endpoints update'.
- Regenerate the CLI reference.
* [Inference Endpoints] Fix stale gating comment, cover the empty-args reset, fix doc example
- The comment above the deploy validation said health route and port 'only make
sense alongside a custom image'. They are BaseContainer fields that every image
variant carries (a catalog vLLM image returns healthRoute and port), so the real
reason to gate them is that this command can only build a custom image payload.
- Nothing covered container_args=[] reaching the payload, which is what
'--container-args ""' sends to clear the value. The CLI test mocks get_hf_api so
it stops at shlex.split, and the existing SDK test only passes non-empty lists.
- The guide applied vLLM tool-calling flags to a gpt2-large endpoint.
---------
Co-authored-by: célina <[email protected]>
Co-authored-by: Celina Hanouti <[email protected]>
Copy file name to clipboardExpand all lines: docs/source/en/guides/cli.md
+7Lines changed: 7 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2324,6 +2324,13 @@ To deploy your own Docker image instead of a Hugging Face managed one, pass `--f
2324
2324
--type authenticated
2325
2325
```
2326
2326
2327
+
`--container-args` and `--container-command` are not limited to custom images: they map to `model.args` and `model.command` in the API payload, which apply to managed images as well. On an existing endpoint both flags replace the current value rather than adding to it, so pass the full list you want and use an empty string to clear it. Endpoints deployed from the catalog already come with tuned engine flags, so check `hf endpoints describe` before overwriting them:
Use `hf endpoints catalog` to interact with the Inference Endpoints Model Catalog. Deploy models directly from the catalog with optimized configurations.
Copy file name to clipboardExpand all lines: docs/source/en/guides/inference_endpoints.md
+7-1Lines changed: 7 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -100,7 +100,7 @@ By default the Inference Endpoint is built from a docker image provided by Huggi
100
100
101
101
The value to pass as `custom_image` is a dictionary containing a url to the docker container and configuration to run it. For more details about it, checkout the [Swagger documentation](https://api.endpoints.huggingface.cloud/#/v2%3A%3Aendpoint/create_endpoint).
102
102
103
-
For containers that need a custom entrypoint or runtime flags, pass `container_command` and/or `container_args` (each a list of tokens). They map to `model.command` and `model.args` in the API payload. The same is available from the CLI via `hf endpoints deploy --custom-image ... --container-command "..." --container-args "..."`.
103
+
For containers that need a custom entrypoint or runtime flags, pass `container_command` and/or `container_args` (each a list of tokens). They map to `model.command` and `model.args` in the API payload. They are not tied to custom images: managed engine images (e.g. vLLM, SGLang) accept engine flags through `container_args` as well. The same is available from the CLI via `hf endpoints deploy ... --container-command "..." --container-args "..."`.
Copy file name to clipboardExpand all lines: docs/source/en/package_reference/cli.md
+5-2Lines changed: 5 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1766,8 +1766,8 @@ $ hf endpoints deploy [OPTIONS] NAME
1766
1766
*`--custom-image TEXT`: Docker image URL for a custom container (e.g. 'nexagi/sglang:v0.5.12'). Requires '--framework custom'.
1767
1767
*`--health-route TEXT`: Health check route exposed by the custom container (e.g. '/health'). Requires --custom-image.
1768
1768
*`--port INTEGER`: Port the custom container listens on (e.g. 30000). Requires --custom-image.
1769
-
*`--container-command TEXT`: Override the container entrypoint, as a quoted string split into tokens (e.g. "python -m sglang.launch_server"). Requires --custom-image.
1770
-
*`--container-args TEXT`: Arguments appended to the container entrypoint, as a quoted string split into tokens (e.g. "--tp 8 --reasoning-parser qwen3"). Requires --custom-image.
1769
+
*`--container-command TEXT`: Override the container entrypoint, as a quoted string split into tokens (e.g. "python -m sglang.launch_server").
1770
+
*`--container-args TEXT`: Arguments appended to the container entrypoint, as a quoted string split into tokens (e.g. "--tp 8 --reasoning-parser qwen3").
1771
1771
*`-e, --env TEXT`: Set environment variables. E.g. --env ENV=value
1772
1772
*`--env-file TEXT`: Read in a file of environment variables.
1773
1773
*`-s, --secrets TEXT`: Set secret environment variables. E.g. --secrets SECRET=value or `--secrets HF_TOKEN` to pass your Hugging Face token.
@@ -1960,6 +1960,8 @@ $ hf endpoints update [OPTIONS] NAME
1960
1960
*`--framework TEXT`: The machine learning framework used for the model (e.g. 'custom').
1961
1961
*`--revision TEXT`: The specific model revision to deploy on the Inference Endpoint (e.g. '6c0e6080953db56375760c0471a8c5f2929baf11').
1962
1962
*`--task TEXT`: The task on which to deploy the model (e.g. 'text-classification').
1963
+
*`--container-command TEXT`: Override the container entrypoint, as a quoted string split into tokens (e.g. "python -m sglang.launch_server"). Replaces the current value; pass an empty string to clear it.
1964
+
*`--container-args TEXT`: Arguments appended to the container entrypoint, as a quoted string split into tokens (e.g. "--enable-auto-tool-choice --tool-call-parser lfm2"). Replaces the arguments currently set on the endpoint rather than adding to them, so include the ones you want to keep, run 'hf endpoints describe NAME' first to see them. Pass an empty string to clear them.
1963
1965
*`--min-replica INTEGER`: The minimum number of replicas (instances) to keep running for the Inference Endpoint.
1964
1966
*`--max-replica INTEGER`: The maximum number of replicas (instances) to scale to for the Inference Endpoint.
1965
1967
*`--scale-to-zero-timeout INTEGER`: The duration in minutes before an inactive endpoint is scaled to zero.
@@ -1970,6 +1972,7 @@ $ hf endpoints update [OPTIONS] NAME
0 commit comments