Skip to content

Commit b4f4e92

Browse files
AlexKerWauplinhanouticelinagithub-actions[bot]
authored
[Inference] Add Baseten as inference provider (#3414)
* finished * Update src/huggingface_hub/inference/_providers/__init__.py * Update src/huggingface_hub/inference/_providers/baseten.py * Apply style fixes --------- Co-authored-by: AlexKer <[email protected]> Co-authored-by: Lucain <[email protected]> Co-authored-by: célina <[email protected]> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
1 parent b9dddd9 commit b4f4e92

7 files changed

Lines changed: 48 additions & 34 deletions

File tree

‎docs/source/en/guides/inference.md‎

Lines changed: 31 additions & 31 deletions
Large diffs are not rendered by default.

‎docs/source/en/package_reference/cli.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3082,7 +3082,7 @@ $ hf models list [OPTIONS] [REPO_ID]
30823082
* `--gated / --no-gated`: Filter by gated status. '--gated' for gated only, '--no-gated' for non-gated only.
30833083
* `--apps TEXT`: Filter by app(s) that can run the model, e.g. 'ollama' or 'vllm'.
30843084
* `--num-parameters TEXT`: Filter by parameter count, e.g. 'min:6B,max:128B'.
3085-
* `--inference-provider [cerebras|cohere|deepinfra|fal-ai|featherless-ai|fireworks-ai|groq|hf-inference|novita|nscale|openai|ovhcloud|publicai|replicate|scaleway|together|wavespeed|zai-org]`: Filter by inference provider(s) serving the model, e.g. 'fireworks-ai'.
3085+
* `--inference-provider [baseten|cerebras|cohere|deepinfra|fal-ai|featherless-ai|fireworks-ai|groq|hf-inference|novita|nscale|openai|ovhcloud|publicai|replicate|scaleway|together|wavespeed|zai-org]`: Filter by inference provider(s) serving the model, e.g. 'fireworks-ai'.
30863086
* `--warm`: Only list models currently served by at least one inference provider.
30873087
* `--sort [created_at|downloads|last_modified|likes|trending_score]`: Sort results.
30883088
* `--limit INTEGER`: Limit the number of results. [default: 30]

‎src/huggingface_hub/inference/_client.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -135,7 +135,7 @@ class InferenceClient:
135135
Note: for better compatibility with OpenAI's client, `model` has been aliased as `base_url`. Those 2
136136
arguments are mutually exclusive. If a URL is passed as `model` or `base_url` for chat completion, the `(/v1)/chat/completions` suffix path will be appended to the URL.
137137
provider (`str`, *optional*):
138-
Name of the provider to use for inference. Can be `"cerebras"`, `"cohere"`, `"deepinfra"`, `"fal-ai"`, `"featherless-ai"`, `"fireworks-ai"`, `"groq"`, `"hf-inference"`, `"novita"`, `"nscale"`, `"openai"`, `"ovhcloud"`, `"publicai"`, `"replicate"`, `"scaleway"`, `"together"`, `"wavespeed"` or `"zai-org"`.
138+
Name of the provider to use for inference. Can be `"baseten"`, `"cerebras"`, `"cohere"`, `"deepinfra"`, `"fal-ai"`, `"featherless-ai"`, `"fireworks-ai"`, `"groq"`, `"hf-inference"`, `"novita"`, `"nscale"`, `"openai"`, `"ovhcloud"`, `"publicai"`, `"replicate"`, `"scaleway"`, `"together"`, `"wavespeed"` or `"zai-org"`.
139139
Defaults to "auto": automatic routing, which defaults to "fastest" provider; you can
140140
switch to "cheapest" or "preferred" provider order at https://hf.co/settings/inference-providers.
141141
If model is a URL or `base_url` is passed, then `provider` is not used.

‎src/huggingface_hub/inference/_generated/_async_client.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@ class AsyncInferenceClient:
125125
Note: for better compatibility with OpenAI's client, `model` has been aliased as `base_url`. Those 2
126126
arguments are mutually exclusive. If a URL is passed as `model` or `base_url` for chat completion, the `(/v1)/chat/completions` suffix path will be appended to the URL.
127127
provider (`str`, *optional*):
128-
Name of the provider to use for inference. Can be `"cerebras"`, `"cohere"`, `"deepinfra"`, `"fal-ai"`, `"featherless-ai"`, `"fireworks-ai"`, `"groq"`, `"hf-inference"`, `"novita"`, `"nscale"`, `"openai"`, `"ovhcloud"`, `"publicai"`, `"replicate"`, `"scaleway"`, `"together"`, `"wavespeed"` or `"zai-org"`.
128+
Name of the provider to use for inference. Can be `"baseten"`, `"cerebras"`, `"cohere"`, `"deepinfra"`, `"fal-ai"`, `"featherless-ai"`, `"fireworks-ai"`, `"groq"`, `"hf-inference"`, `"novita"`, `"nscale"`, `"openai"`, `"ovhcloud"`, `"publicai"`, `"replicate"`, `"scaleway"`, `"together"`, `"wavespeed"` or `"zai-org"`.
129129
Defaults to "auto": automatic routing, which defaults to "fastest" provider; you can
130130
switch to "cheapest" or "preferred" provider order at https://hf.co/settings/inference-providers.
131131
If model is a URL or `base_url` is passed, then `provider` is not used.

‎src/huggingface_hub/inference/_providers/__init__.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
from huggingface_hub.utils import logging
88

99
from ._common import AutoRouterConversationalTask, TaskProviderHelper, _fetch_inference_provider_mapping
10+
from .baseten import BasetenConversationalTask
1011
from .cerebras import CerebrasConversationalTask
1112
from .cohere import CohereConversationalTask
1213
from .deepinfra import (
@@ -67,6 +68,7 @@
6768

6869

6970
PROVIDER_T = Literal[
71+
"baseten",
7072
"cerebras",
7173
"cohere",
7274
"deepinfra",
@@ -92,6 +94,9 @@
9294
CONVERSATIONAL_AUTO_ROUTER = AutoRouterConversationalTask()
9395

9496
PROVIDERS: dict[PROVIDER_T, dict[str, TaskProviderHelper]] = {
97+
"baseten": {
98+
"conversational": BasetenConversationalTask(),
99+
},
95100
"cerebras": {
96101
"conversational": CerebrasConversationalTask(),
97102
},
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
from ._common import BaseConversationalTask
2+
3+
4+
class BasetenConversationalTask(BaseConversationalTask):
5+
def __init__(self):
6+
super().__init__(provider="baseten", base_url="https://inference.baseten.co")

‎tests/test_inference_client.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,9 @@
6363

6464
# Avoid calling APIs in VCRed tests
6565
_RECOMMENDED_MODELS_FOR_VCR = {
66+
"baseten": {
67+
"conversational": "moonshotai/Kimi-K3",
68+
},
6669
"cerebras": {
6770
"conversational": "meta-llama/Llama-3.3-70B-Instruct",
6871
},

0 commit comments

Comments
 (0)