Xinference (Xorbits Inference) is an open-source distributed inference platform.
Deploy and run models on heterogeneous hardware, from a local machine to a distributed cluster. Xinference includes 300+ built-in models for language, vision, embeddings, reranking, image, audio, video, and experimental world generation. Register your own models, including traditional ML models through the Flexible type. Manage deployments in the Web UI and use models through OpenAI-compatible APIs, the Python SDK, or the CLI.
Quick start · Key features · Comparison · Self-hosting · Documentation · Model catalog · Releases · Xinference Enterprise
Languages · English · 日本語 · 한국어 · Deutsch · Français · Español · Italiano · Português · 繁體中文 · 简体中文
Xinference downloads the model and installs any needed engine dependencies on the first launch.
pip install xinference
xinference-local --host 127.0.0.1Open http://127.0.0.1:9997. On a fresh instance, use the setup token printed in the server log to create an administrator account, then sign in. First-run setup
In Launch Model, search for qwen3.5 and open its launch form. Click Recommend configuration to fill in settings for your hardware. Review the recommendation and click Deploy.
Open the deployed model in Running Models to try a prompt in the Web UI. The first instance uses qwen3.5 as its Model UID. If your instance shows a different UID, use that value for model below. Create a key under API Key Management to call the API.
export XINFERENCE_API_KEY="<your-api-key>"
curl --fail-with-body http://127.0.0.1:9997/v1/chat/completions \
-H "Authorization: Bearer $XINFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen3.5",
"messages": [{"role": "user", "content": "Translate this sentence into Spanish: The model is ready."}],
"enable_thinking": false,
"max_tokens": 96
}'| Area | What Xinference supports |
|---|---|
| Model catalog | 300+ built-in models across language, vision, embeddings, reranking, images, audio, video, and experimental world generation. Register your own models, including Flexible models for traditional ML. Model catalog |
| Engines and hardware | Choose vLLM, SGLang, TRT-LLM, Transformers, llama.cpp, or MLX for supported LLMs. Hardware support varies by engine and includes CPU, GPUs, Apple Silicon, and supported MLU configurations. Backends · MLU support |
| Distributed serving | Run on one machine, add workers and replicas, or run a large model across workers with a supported engine. Compatible deployments support heterogeneous prefill/decode separation. Distributed inference · P/D separation |
| APIs and tools | OpenAI- and Anthropic-compatible endpoints, with streaming and tool calling where supported. Deploy and manage models through the Web UI, CLI, or Python SDK. Client guide · Tool calling |
| Performance and operations | Benchmark running chat models in the Web UI. Supported deployments can share prefix caches with Xavier, change selected runtime settings, and reload replicas one at a time. Performance tests · Runtime configuration |
| Languages | Use the Web UI in English, Simplified Chinese, Traditional Chinese, Japanese, or Korean. The README is available in ten languages. |
Supported ·
Partial or via integration ·
No documented support
Scope and sources
Based on documented features in the open-source releases. Engine and hardware support varies by model. CUDA ↔ Metal prefill/decode requires a compatible model and engine pair.
Xinference can serve replicas with different engines concurrently under one model ID. LocalAI uses an ordered failover chain under one model name and sends requests to the first available target. KServe and Dynamo can route across separate deployments. GPU weight reuse applies to supported vLLM and SGLang configurations.
- Xinference · Model catalog · Backends · Distributed inference · Cross-engine P/D on CUDA and Metal · Replica engine overrides · Rolling reconfiguration
- LocalAI · Model and deployment features · Backends and hardware · Distributed mode · Model aliases and failover
- KServe · Deployment and hardware · Built-in generative runtimes · Kubeflow model catalog · Kubeflow Web UI · Canary deployments
- NVIDIA Dynamo · Deployment, hardware, and engines · Deployment recipes · Image, video, and speech generation · Embedding and reranking support · Multi-deployment routing
Built-in models added in the last two releases (v4.1 and v4.2)
- Built-in support for Audio8-TTS-Realistic-0.6b: #6003
- Built-in support for MiniCPM-OCR-1.0: #5916
- Built-in support for OvisOCR3: #5914
- Built-in support for Unlimited-OCR-2: #5913
- Built-in support for FLUX 3 Image Klein: #5792
- Built-in support for FIBO 2.0 series (Dev, Edit Turbo): #5790
- Built-in support for Ming-3.0 TTS series (Tiny, 0.5B): #5788
- Built-in support for Xiaomi-OCR-0: #5764
- Built-in support for MOSS-Image-SFT: #5755
- Built-in support for DeepSeek-V4.1-Flash-1017: #5753
- Built-in support for Iris-3B series (Base, Turbo): #5686
- Built-in support for MiniCPM-V-4.7 series (1B, 35B-A3B): #5676
- Built-in support for Qwen3.8-Flash-Next series (Base, FP8): #5673
- Built-in support for GLM-5.3 series (BF16, FP8): #5666
- Built-in support for Qwen-Image-2.1-Turbo: #5665
- Built-in support for DeepSeek-V4.1-Flash: #5661
Open Performance Tests, pick a running chat model, and set the concurrency levels you want to test. Compare throughput, time to first token, and time per output token before choosing a serving configuration. Benchmark guide
Example from a local Qwen3.5-0.8B run. Performance depends on the hardware, engine, and workload.
| If you want to… | Start here |
|---|---|
| Install on Linux, macOS, or Windows | One-command installer · Installation guide |
| Run in a container or on Kubernetes | Docker · Kubernetes |
| Register your own model | Custom models |
| Connect an application | OpenAI-compatible APIs · Xagent · Dify |
Apps and frameworks · LangChain · LlamaIndex · FastGPT · RAGFlow · MaxKB · Chatbox
Deploy Xinference Enterprise in your own cloud or data center. Xinference Cloud offers managed dedicated deployments, and the Model API provides hosted model endpoints. Enterprise overview
Business enquiries · [email protected]
| Platform | What it's for |
|---|---|
| GitHub Issues | Report bugs and request features. |
| Discord | Ask questions and discuss Xinference with other users. |
| Telegram | Chat with the Xinference community. |
| X | Follow project news and releases. |
If you use Xinference in your research, please cite Xinference: Making Large Model Serving Easy.
@inproceedings{lu2024xinference,
title = "Xinference: Making Large Model Serving Easy",
author = "Lu, Weizheng and Xiong, Lingfeng and Zhang, Feng and Qin, Xuye and Chen, Yueguo",
booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
month = nov,
year = "2024",
address = "Miami, Florida, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.emnlp-demo.30",
pages = "291--300",
}Thank you to everyone who helps build Xinference. Open your first pull request to add a model, improve the docs, or fix a bug. Read the contribution guide.



