Skip to content
xorbitsaiPublic

About

Open-source inference platform for language, vision, embedding, reranking, image, audio, and video models. Deploy with multiple engines across heterogeneous hardware, from a laptop to a distributed cluster. Serve models through OpenAI-compatible APIs.

Topics

Resources

Code of conduct

Security policy

Stars

9.7k stars

Watchers

58 watching

Forks

Repository files navigation

Xinference

Xinference (Xorbits Inference) is an open-source distributed inference platform.

Deploy and run models on heterogeneous hardware, from a local machine to a distributed cluster. Xinference includes 300+ built-in models for language, vision, embeddings, reranking, image, audio, video, and experimental world generation. Register your own models, including traditional ML models through the Flexible type. Manage deployments in the Web UI and use models through OpenAI-compatible APIs, the Python SDK, or the CLI.

Quick start · Key features · Comparison · Self-hosting · Documentation · Model catalog · Releases · Xinference Enterprise

GitHub release PyPI version Docker tag Python versions Python CI

Apache 2.0 license GitHub stars GitHub forks Contributors Docker pulls Discord community Telegram community Follow Xinference on X

Languages · English · 日本語 · 한국어 · Deutsch · Français · Español · Italiano · Português · 繁體中文 · 简体中文

Quick start

1. Start the server

Xinference downloads the model and installs any needed engine dependencies on the first launch.

pip install xinference
xinference-local --host 127.0.0.1

Open http://127.0.0.1:9997. On a fresh instance, use the setup token printed in the server log to create an administrator account, then sign in. First-run setup

2. Launch a chat model

In Launch Model, search for qwen3.5 and open its launch form. Click Recommend configuration to fill in settings for your hardware. Review the recommendation and click Deploy.

Full Qwen3.5 launch form with the recommended configuration applied

3. Make your first request

Open the deployed model in Running Models to try a prompt in the Web UI. The first instance uses qwen3.5 as its Model UID. If your instance shows a different UID, use that value for model below. Create a key under API Key Management to call the API.

export XINFERENCE_API_KEY="<your-api-key>"
curl --fail-with-body http://127.0.0.1:9997/v1/chat/completions \
  -H "Authorization: Bearer $XINFERENCE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5",
    "messages": [{"role": "user", "content": "Translate this sentence into Spanish: The model is ready."}],
    "enable_thinking": false,
    "max_tokens": 96
  }'

The qwen3.5 chat playground showing a completed response

Key features

Area What Xinference supports
Model catalog 300+ built-in models across language, vision, embeddings, reranking, images, audio, video, and experimental world generation. Register your own models, including Flexible models for traditional ML. Model catalog
Engines and hardware Choose vLLM, SGLang, TRT-LLM, Transformers, llama.cpp, or MLX for supported LLMs. Hardware support varies by engine and includes CPU, GPUs, Apple Silicon, and supported MLU configurations. Backends · MLU support
Distributed serving Run on one machine, add workers and replicas, or run a large model across workers with a supported engine. Compatible deployments support heterogeneous prefill/decode separation. Distributed inference · P/D separation
APIs and tools OpenAI- and Anthropic-compatible endpoints, with streaming and tool calling where supported. Deploy and manage models through the Web UI, CLI, or Python SDK. Client guide · Tool calling
Performance and operations Benchmark running chat models in the Web UI. Supported deployments can share prefix caches with Xavier, change selected runtime settings, and reload replicas one at a time. Performance tests · Runtime configuration
Languages Use the Web UI in English, Simplified Chinese, Traditional Chinese, Japanese, or Korean. The README is available in ten languages.

How Xinference compares

Capability Xinference LocalAI KServe NVIDIA Dynamo
Standalone (no Kubernetes) Supported Supported No documented support Supported
Multi-node inference Supported Supported Supported Supported
CPU / GPU / Apple Metal Supported Supported Partial or via integration Partial or via integration
llama.cpp / vLLM / SGLang Supported Supported Partial or via integration Partial or via integration
CUDA ↔ Metal prefill/decode Supported No documented support No documented support No documented support
Mixed engines under one model ID Supported Partial or via integration Partial or via integration Partial or via integration
Rolling reconfiguration with GPU weight reuse Supported No documented support No documented support No documented support
Model catalog Supported 300+ Supported Partial or via integration Partial or via integration
Model deployment Web UI Supported Supported Partial or via integration No documented support
Embeddings and reranking Supported Supported Supported Supported
Image and video generation Supported Supported No documented support Supported
Speech recognition and synthesis Supported Supported No documented support Partial or via integration

Supported · Partial or via integration · No documented support

Scope and sources

Based on documented features in the open-source releases. Engine and hardware support varies by model. CUDA ↔ Metal prefill/decode requires a compatible model and engine pair.

Xinference can serve replicas with different engines concurrently under one model ID. LocalAI uses an ordered failover chain under one model name and sends requests to the first available target. KServe and Dynamo can route across separate deployments. GPU weight reuse applies to supported vLLM and SGLang configurations.

Recent updates

Read the v4.2.0 release notes

Built-in models added in the last two releases (v4.1 and v4.2)

Model performance tests

Open Performance Tests, pick a running chat model, and set the concurrency levels you want to test. Compare throughput, time to first token, and time per output token before choosing a serving configuration. Benchmark guide

Web UI benchmark charts comparing throughput and latency at different concurrency levels

Example from a local Qwen3.5-0.8B run. Performance depends on the hardware, engine, and workload.

Using Xinference

If you want to… Start here
Install on Linux, macOS, or Windows One-command installer · Installation guide
Run in a container or on Kubernetes Docker · Kubernetes
Register your own model Custom models
Connect an application OpenAI-compatible APIs · Xagent · Dify

Apps and frameworks · LangChain · LlamaIndex · FastGPT · RAGFlow · MaxKB · Chatbox

Xinference Enterprise

Deploy Xinference Enterprise in your own cloud or data center. Xinference Cloud offers managed dedicated deployments, and the Model API provides hosted model endpoints. Enterprise overview

Business enquiries · [email protected]

Getting involved

Platform What it's for
GitHub Issues Report bugs and request features.
Discord Ask questions and discuss Xinference with other users.
Telegram Chat with the Xinference community.
X Follow project news and releases.

Citation

If you use Xinference in your research, please cite Xinference: Making Large Model Serving Easy.

@inproceedings{lu2024xinference,
    title = "Xinference: Making Large Model Serving Easy",
    author = "Lu, Weizheng and Xiong, Lingfeng and Zhang, Feng and Qin, Xuye and Chen, Yueguo",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.emnlp-demo.30",
    pages = "291--300",
}

Contributors

Thank you to everyone who helps build Xinference. Open your first pull request to add a model, improve the docs, or fix a bug. Read the contribution guide.

Xinference contributor avatar wall

About

Open-source inference platform for language, vision, embedding, reranking, image, audio, and video models. Deploy with multiple engines across heterogeneous hardware, from a laptop to a distributed cluster. Serve models through OpenAI-compatible APIs.

Topics

Resources

Code of conduct

Security policy

Stars

9.7k stars

Watchers

58 watching

Forks

Releases

Packages

Used by

Contributors

Languages