Your app shouldn't die because Groq returned a 429.
llm-free-cascade sends one chat completion through a chain of LLM providers
that have a free tier, and falls through to the next one when the current one
is rate-limited, out of quota, slow or down. It reads why a call failed, so a
rate-limited key rests for exactly as long as the provider says, and a limit
that is per model falls to the next model on the same key before leaving the
provider.
Zero dependencies. Node 18+. One generate() call, 15 providers known out of the box.
const { LLMCascade } = require('llm-free-cascade');
const cascade = LLMCascade.fromEnv(); // GROQ_API_KEY, GEMINI_API_KEY, MISTRAL_API_KEY, ...
const { text, provider, attempts } = await cascade.generate({
system: 'You are a helpful assistant.',
user: 'Explain photosynthesis in one sentence.',
});
// provider -> whichever one answered; attempts -> who it skipped, and whynpm install llm-free-cascade- Reset-aware cooldowns. Reads
Retry-Afterand "try again in 2m59s" text. A per-minute limit rests for a minute, a per-day limit until the provider's reset (Gemini: midnight Pacific). The chain wraps back to your fastest provider quickly instead of benching it for ten minutes. - Per-key and per-model rotation. Several keys per provider are used round-robin. Limits are usually per model, so a 429 on one model falls to the next on the same key first.
- Failure classification. A prompt that is too long skips one provider for one call. Timeouts and 5xx trip a circuit breaker. A bad key leaves rotation alone, without benching the provider.
- Hard bounds. Per-attempt
timeoutMs(idle timeout for streams) and a whole-calldeadlineMs. - Parse-and-retry. Pass
json: trueand aparsefunction. A response that fails to parse counts as a provider failure and the next one is tried. - Streaming, multi-turn
messages, tokenusage,shadowCostUsd, and a paid last-resort provider (DeepSeek, Anthropic) at the end of the chain. - Custom providers. Any OpenAI-compatible endpoint (a self-hosted gateway, LiteLLM, OmniRoute) joins the chain with one config entry.
- Visibility.
stats(),onAttempt,probe()and a CLI, so you can answer "which key served what today, and when does it reset?"
$ npx llm-free-cascade status
llm-free-cascade — 3 provider(s) configured, in this order:
1. groq 2 key(s) [tier 1]
models: openai/gpt-oss-120b -> qwen/qwen3.8-27b -> openai/gpt-oss-20b
limits: 30 RPM, 1000 RPD, 8000 TPM, 200000 TPD, per model, per organization
2. gemini 1 key(s) [tier 1, may train on your prompts]
models: gemini-3.6-flash
3. mistral 1 key(s) [tier 2]
models: mistral-small-latest
npx llm-free-cascade probe # one tiny live request through every key
npx llm-free-cascade probe --watch 300 # ...repeated every 5 minutes
npx llm-free-cascade keys # which providers you're missing, and where to sign up
npx llm-free-cascade free-models # OpenRouter's free models right nowRun the cascade as a local OpenAI-compatible endpoint and point any tool that
takes a custom base URL at it (Python openai, LangChain, Open WebUI,
Continue, Aider, ...):
npx llm-free-cascade serve # http://127.0.0.1:8787/v1from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8787/v1", api_key="unused")
r = client.chat.completions.create(model="cascade", messages=[{"role": "user", "content": "hi"}])model: "cascade" uses the whole chain; a provider name ("groq") pins one.
Supported: text messages, stream: true, max_tokens, temperature, top_p,
stop, seed, response_format: json_object, GET /v1/models. Tool calls,
images, n > 1 and non-default penalties return a clear 400 rather than being
ignored.
Safe by default: it listens on 127.0.0.1 only and rejects other Host headers
and non-JSON requests from web pages. To listen on another address
(--host 0.0.0.0) you must set LLM_CASCADE_SERVER_KEY; clients then send it
as their API key. Details: Guide → OpenAI-compatible server.
// Several keys for one provider (or GEMINI_API_KEY=key1,key2 in the environment)
new LLMCascade({ keys: { gemini: [key1, key2] } });
// Pick and order the chain
new LLMCascade({ keys: { ... }, order: ['groq', 'mistral', 'gemini'] });
// Bound latency: 8 s per attempt, 25 s for the whole call
new LLMCascade({ keys: { ... }, timeoutMs: 8000, deadlineMs: 25_000 });
// Multi-turn chat
await cascade.generate({ system, messages: [{ role: 'user', content: 'Hi' }] });
// Streaming
const { stream } = await cascade.generate({ system, user, stream: true });
for await (const chunk of stream) process.stdout.write(chunk);
// Add a gateway or any other OpenAI-compatible endpoint
new LLMCascade({ keys: { ... }, custom: { mygw: { baseUrl: 'https://gw.example/v1', models: ['auto'], apiKey } } });Ollama runs models on your own machine: no quota, no key, slower. Add it as the last tier so the cascade never runs dry:
ollama pull llama3.2new LLMCascade({
keys: { ... },
custom: {
ollama: {
baseUrl: 'http://localhost:11434/v1',
model: 'llama3.2',
keyless: true,
tier: 5, // after every free cloud provider
},
},
providerTimeoutMs: { ollama: 120_000 }, // local models can be slow on CPU
});Or without code (also works with serve):
LLM_CUSTOM_PROVIDERS='{"ollama":{"baseUrl":"http://localhost:11434/v1","model":"llama3.2","keyless":true,"tier":5}}'.
See the guide for details.
When every provider fails, generate() rejects with an LLMCascadeError
carrying code, a suggested statusCode and redacted failures.
Groq, Google Gemini, Mistral, NVIDIA NIM, Z.ai (Zhipu), OpenRouter, SiliconFlow, ModelScope, Cloudflare Workers AI, Cohere, Pollinations, OpenCode Zen, OVHcloud, plus DeepSeek and Anthropic as paid fallbacks. The full table with limits, caveats and signup links is on the project page and in the guide.
Free tiers change without notice (Cerebras, SambaNova and GitHub Models all cut
theirs in 2026), so each entry in src/providers.json
records verifiedAt and where its numbers came from. Read a provider's own
terms before relying on it, and before opening several free accounts to get
around a limit: many providers forbid that. Rotating across providers and
models you already have keys for is the intended use.
LiteLLM, Portkey and OmniRoute are much larger gateways with dashboards and far more providers, and are the better choice if you need those. This package is a small zero-dependency library you call in process, with an optional local server, for stacking free tiers you already have keys for.
- Guide: every option, in depth (cooldowns, timeouts, errors, hooks, stats, custom providers, the OpenAI-compatible server, cost and input hygiene).
- Project page: the
provider table, generated from
providers.json. - Changelog · Contributing: fixing a stale model name is a one-file change.
MIT