The open-source LLM proxy for small businesses, development teams and growing organisations. One endpoint in front of 12 local inference backends plus any OpenAI-compatible server, with load balancing, automatic failover, Claude Code support and a built-in dashboard. Deploy in minutes with Docker.
Need hands-on tuning? See Optimise.
Get up and running with Olla in minutes using containers.
A read-only dashboard is built into Olla and served on the same port as the proxy. There is no extra service to run, and no curling JSON to find out which backend is slow.
Olla health-checks every backend and routes around the ones that fail. Circuit breakers and exponential backoff stop a struggling node being hammered while it recovers, and it rejoins on its own once it is healthy again.
Olla serves the Anthropic Messages API alongside the OpenAI API. Requests pass straight through to backends that speak Anthropic natively and are translated to OpenAI chat for the rest, so Claude Code and other Anthropic clients can run against your local models.
Built as a dedicated LLM proxy, Olla routes every request from your tools to the right backend, and steps around the ones that fail, from a single binary.
Olla discovers the models on every backend and lists them through one API, merging models that share a name. Where runtimes name the same model differently, such as qwen3.8:27b on Ollama and Qwen/Qwen3.8-27B on vLLM, a model alias maps them to one name so a request can land on any of them.
Model UnificationPick priority for tiered hardware, round-robin for an even spread, or least-connections, the default, for latency-sensitive work. When a backend fails before the reply starts, Olla moves the request to the next one, and it never replays a request once the response has begun. The proxy.retry settings control that failover.
Load BalancingOne Go binary with per-endpoint connection pooling, object pooling to keep garbage-collection pressure low and lock-free statistics. It adds well under a millisecond to a request and idles at around 20 MB of memory.
PerformanceContinuous health checks and circuit breakers detect failing backends and route around them. Exponential backoff gives a struggling node room to recover, and healed endpoints rejoin automatically.
Health MonitoringPer-IP and global rate limits, applied as independent tiers and shared by the Claude Code routes, and request size limits protect your backends. Backends that need credentials get per-endpoint bearer, API key or basic auth, kept out of status output, and CORS can be switched on for browser clients.
SecurityA Prometheus endpoint exposes per-endpoint health, latency, routing and security metrics, ready for Grafana. Every proxied request is logged with its status, request and failure counts tally each backend attempt once with a backend 5xx counted as a failure, and the built-in dashboard shows the behaviour of the whole herd live.
ObservabilityReal herds run more than one kind of server. Olla keeps them working together.
Model aliases map differently named copies of a model across Ollama, vLLM, LM Studio and oMLX to one name your clients use.
Read moreHugging Face Text Embeddings Inference is a backend, and embed and rerank requests without a model name go to a host that can serve them.
Read moreOpt in to keep a multi-turn conversation on the backend that already holds its KV-cache, so follow-up turns can reuse the cached prompt instead of recomputing it.
Read moreoMLX and vLLM-MLX sit alongside vLLM, SGLang and llama.cpp, so a Mac Studio or MacBook can join the same herd as your GPU servers.
Read moreOllama vision and tool-calling models are recognised, so requests with images or tools prefer the hosts that can handle them.
Read moreolla --validate-config reports pass, warn or fail with exit codes for CI, and flags misspelt keys with the line they are on.
Read moreEvery figure below comes from a real benchmark run on a single dev box, not a spec sheet. A small Go binary, low overhead, and routing that stays out of your way.
Measured on Olla v0.0.28, Windows 11 dev box (Ryzen 9 5950X); mock backends isolate the proxy from inference. Conditions stated - see the full benchmarks.
Scrape /internal/metrics for per-endpoint health, latency, routing and security metrics, ready for Grafana.
Every proxied request is logged with its status, from startup through health checks to completion.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β β β β£β£β β β β β β£β£β β β β β β’°β‘β’Ήβ‘β β β β’°β‘β’Ήβ‘β‘ β β βββββββ βββ βββ ββββββ β β’Έβ‘β£Έβ‘·β β β »β’Ύβ£β£Έβ‘ β β ββββββββββββ βββ ββββββββ β’ β‘Ύβ β β β β β β β β β’·β‘ β β βββ ββββββ βββ ββββββββ β£Ώβ β’β£β’β£ β£€β£β‘β£ β‘β β£Ώ β β βββ ββββββ βββ ββββββββ β’»β£β β β‘β β’€β β’³β β β’ β‘Ώβ‘ β β ββββββββββββββββββββββββββββ βββ β£Όβ β β β ³β €β ¬β €β β β β β£· β β β’Έβ‘β β β β β β β β β β’Έβ‘ β β github.com/thushan/olla v0.1.0 β’Έβ‘ β β β β β β β β β β’β‘Ώ β ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Olla works with 12 local inference backends plus any OpenAI-compatible server. The ik_llama.cpp fork is supported via the llama.cpp backend.
See IntegrationsOlla is built for small businesses, development teams, startups and growing organisations that want reliable routing across their own inference servers without enterprise complexity. It suits a single workstation through to a handful of GPU servers. For multi-node GPU fleets that need a control plane, per-GPU telemetry and an operator console, FoundryOS is now in early access.
Olla is a high-performance, open-source LLM proxy and load balancer. It puts one endpoint in front of 12 local inference backends plus any OpenAI-compatible server, including Ollama, LM Studio, vLLM, SGLang, llama.cpp and oMLX. It provides unified model discovery, automatic failover, a built-in dashboard and Prometheus metrics.
Olla can be installed using Docker or Podman with a single command: docker run -t --name olla -p 40114:40114 ghcr.io/thushan/olla:v0.1.0. This pulls the image and starts the proxy on port 40114 (4 OLLA), see our Quickstart Guide.
Yes, Olla is completely free and open source software under the Apache License, Version 2.0. The source code is available on GitHub.
Olla is distributed under the Apache-2.0. Learn about Olla Licensing.
Olla supports 12 local inference backends plus any OpenAI-compatible server. The named backends are Ollama, LM Studio, vLLM, vLLM-MLX, SGLang, llama.cpp, Lemonade, LMDeploy, LiteLLM, Docker Model Runner, oMLX and Hugging Face Text Embeddings Inference (TEI). The ik_llama.cpp fork works through the llama.cpp backend. Learn more about Olla Integrations.
Yes, in beta. Olla serves the Anthropic Messages API alongside the OpenAI API. Requests pass straight through to backends that support it natively, including vLLM, SGLang, LMDeploy, llama.cpp, LM Studio, Ollama and oMLX, and are translated to OpenAI chat for the rest. Claude Code traffic takes the same rate limits as the proxy routes, so many sessions behind one IP may need the per-IP limit raised. See Claude Code with Olla.
Yes. The release binaries and Docker image include a read-only dashboard at /internal/ui/ on the proxy's own port, showing endpoint health, latency, traffic and loaded models. For alerting and history, scrape the Prometheus endpoint at /internal/metrics. See the Dashboard guide.
Olla provides a unified model registry with model aliases, load balancing (priority, round-robin and least-connections), health checks and circuit breakers with automatic failover and configurable retries, opt-in KV-cache-aware sticky sessions, rate limiting, a built-in dashboard and a Prometheus metrics endpoint. Olla also has an extensive list of other features you can learn more in the Olla Concepts.
Yes, Olla is specifically designed to work with local LLM deployments. It can route requests across local inference nodes running Ollama, LM Studio, or other local backends, providing load balancing and failover capabilities for self-hosted AI infrastructure.
Olla is open source and growing fast. Perfect for small businesses, development teams, and organisations building reliable, unified AI infrastructure at any scale.
Growing community with regular updates and improvements.
Star on GitHubWe welcome contributions of all sizes. Every improvement matters!
Get started with Olla today and experience intelligent load balancing for your AI workloads.
