Developed by Sandeep Hipparagi.
Every prompt that reaches an LLM costs two independent resources: the tier of the model you send it to, and the test-time reasoning effort you let that model spend. Almost every router in the wild optimises one dimension — cheap model versus expensive model — and then throws a fixed reasoning budget on top. That is why a greeting can burn 8,000 thinking tokens on a frontier model while a proof request gets a one-shot answer from a mini model.
Inference-Pareto-Router treats routing as a point in a 2D space. It scores a
prompt on two orthogonal axes, logic (verifiability, algorithmic depth, math/code
structure) and knowledge (entity richness, domain nuance, world knowledge), and
returns a coordinate (ModelTier, ReasoningEffort) together with the exact,
provider-ready request body that implements it — in roughly 0.1 ms at the median,
with no network call in the decision path. The in-repo 100-scenario benchmark
reports mean 0.099 ms and p99 0.181 ms; the first call in a process pays a one-off
warmup of a few milliseconds.
- Installation
- Quickstart · Constraints and the routing decision
- Architecture
- How the 2D decision is computed
- Runnable examples
- Strategy presets
- Benchmarks
- Provider routing table
- Features
- CLI
- What is and is not implemented
- Development, testing and CI
- Attribution and license
pip install pareto-routerThe repository is named Inference-Pareto-Router; the PyPI distribution is
pareto-router.pip install pareto-routeris the canonical command.
Requires Python 3.11+. With uv or pipx:
uv add pareto-router
pipx run pareto-router "Why is my asyncio task never awaited?"Install the development toolchain (pytest, pytest-asyncio, mypy):
pip install "pareto-router[dev]"Fifteen lines: set your cost/latency/quality constraints, score a prompt, and get both the routing decision and the provider-ready payload.
from pareto_router import (HeuristicEvaluator, ParetoPolicy, ProviderNormalizer,
ProviderType, RoutingStrategy)
policy = ParetoPolicy(
logic_threshold_low=0.15, # below this: no reasoning at all (STANDARD mode)
logic_threshold_high=0.75, # above this: always FRONTIER
knowledge_threshold_high=0.65, # above this: always FRONTIER
overthinking_threshold=0.25, # clamp reasoning on greetings / trivial asks
default_strategy=RoutingStrategy.PARETO_COST,
)
scores = HeuristicEvaluator().evaluate_sync("Debug this race condition in a Rust mutex.")
decision = policy.evaluate_coordinates(scores, strategy=RoutingStrategy.PARETO_QUALITY)
payload = ProviderNormalizer().to_provider_payload(
decision, ProviderType.OPENAI, [{"role": "user", "content": "Debug this race condition."}])
print(decision.tier, decision.effort_level, decision.reasoning_budget_tokens)
print(payload.model, payload.body.get("reasoning_effort"))
# -> ModelTier.FRONTIER EffortLevel.HIGH 25464
# -> o1 highpayload is a complete, dispatchable request: payload.model, payload.body
(including the provider-specific reasoning knob) and payload.headers. Hand it to
your own client, or use the gateway wrapper below.
ParetoRouter exposes an OpenAI-shaped surface for exercising routing behaviour in
tests and demos:
from pareto_router import ParetoRouter, RoutingStrategy
router = ParetoRouter(strategy=RoutingStrategy.PARETO_BALANCED) # reads OPENAI_API_KEY
resp = router.chat.completions.create(
messages=[{"role": "user", "content": "Summarise this error log in one line."}],
)
print(resp.routing_decision.tier, resp.routing_decision.effort_level, resp.model)Read What is and is not implemented before using
this in production — the transport layer in 0.1.x is a deterministic stub, not a
live provider call.
pareto_router/
├── core/
│ ├── schemas.py Pydantic v2 models: ModelTier, ModelMode, EffortLevel,
│ │ RoutingStrategy, EvaluationScores, RoutingDecision, ProviderPayload
│ ├── policy.py ParetoPolicy - the 2D decision function (pure, no I/O)
│ └── normalizer.py ProviderModelCatalog + ProviderNormalizer - (tier, mode) to
│ a vendor model id and a ready request body
├── evaluator/
│ ├── base.py BaseEvaluator contract; measures latency around _compute_scores
│ ├── heuristic.py HeuristicEvaluator - in-box, no network
│ ├── onnx_evaluator.py ONNXEvaluator - local classifier via onnxruntime
│ └── decision_model.py DecisionModelClient - external logits service
├── gateway/
│ ├── client.py ParetoRouter / ParetoChatCompletions - OpenAI-shaped surface
│ ├── cascade.py SpeculativeCascade - escalate on verifier failure
│ └── verifiers.py JsonSchema / PythonSyntax / Regex / Custom verifiers
├── telemetry/metrics.py PrometheusMetricsExporter (exposition + JSON summary)
├── benchmarks/ scenarios.py, run_benchmarks.py - the 100-scenario suite
└── cli.py the `pareto-router` console script
The pipeline is one direction and has exactly one impure stage:
prompt text
│
▼ evaluator (pluggable, the only stage that may do I/O)
EvaluationScores { logic, knowledge, latency_ms }
│
▼ ParetoPolicy (pure function of scores + strategy preset)
RoutingDecision { tier, effort_score, effort_level, mode, reasoning_budget_tokens }
│
▼ ProviderNormalizer (catalogue lookup + vendor knob mapping)
ProviderPayload { provider, model, body, headers }
│
▼ your transport (httpx, openai SDK, your own gateway)
provider API
Because the decision stage is pure and the catalogue is data, you can unit-test
routing behaviour exhaustively without a provider key, and swap models by passing a
custom ProviderModelCatalog.
Three scripts in examples/ run with no API key:
python examples/01_route_and_inspect.py # one prompt, full decision breakdown
python examples/02_compare_strategies.py # the same prompts under all three presets
python examples/03_dispatch_payload.py # build the payload; --live to dispatch it02_compare_strategies.py prints the tier/effort each preset chooses:
prompt cost balanced quality
hi there compact/none compact/none compact/none
Summarise this error log in one line. compact/none compact/none compact/none
Prove that sqrt(2) is irrational. mid/none mid/low mid/low
Refactor this 400-line React component... mid/none frontier/none frontier/none
The router is a pure function of two scores and one strategy preset. There is no model call in the decision path, which is why it costs microseconds rather than milliseconds.
l measures how much verifiable structure a prompt has — proofs, code, multi-step
arithmetic, explicit constraints. k measures how much breadth of knowledge or
nuance it needs — named entities, domain jargon, cultural or stylistic depth.
Scores come from a pluggable evaluator (HeuristicEvaluator ships in-box;
ONNXEvaluator and DecisionModelClient allow an external classifier).
Frontier reasoning models measurably degrade on trivial inputs. If the evaluator
flags low-entropy input (is_low_entropy, is_greeting, is_aesthetic) and
l < overthinking_threshold, effort is pinned to 0.0 and the decision records
overthinking_penalized = True.
For everything else, effort is a rescaled distance above the low threshold:
where k > knowledge_threshold_high and l > 0.50 (a
"hard-and-broad" boost), and
| Effort score | EffortLevel |
Mode |
|---|---|---|
≤ 0.001 |
NONE |
STANDARD (thinking disabled) |
(0.001, 0.35] |
LOW |
REASONING |
(0.35, 0.70] |
MEDIUM |
REASONING |
(0.70, 1.0] |
HIGH |
REASONING |
When mode is REASONING, the effort score becomes an integer thinking budget:
A preset shifts every threshold at once, moving the operating point along the frontier without changing the algorithm:
| Threshold | PARETO_BALANCED |
PARETO_COST |
PARETO_QUALITY |
|---|---|---|---|
logic_threshold_low |
0.15 | 0.25 | 0.10 |
logic_threshold_high |
0.75 | 0.85 | 0.60 |
knowledge_threshold_low |
0.35 | 0.45 | 0.25 |
knowledge_threshold_high |
0.65 | 0.75 | 0.50 |
overthinking_threshold |
0.25 | 0.35 | 0.15 |
| Effort bias | — | −0.10 | +0.10 |
PARETO_COST needs more evidence before it escalates and trims effort;
PARETO_QUALITY escalates earlier and adds 0.10 to every effort score.
Measured on the in-repo 100-scenario synthetic suite
(python -m pareto_router.benchmarks.run_benchmarks, results in
benchmark_results.json):
| Metric | Result |
|---|---|
| Scenarios evaluated | 100 |
| Mean decision latency | 0.099 ms |
| p50 / p90 / p95 / p99 | 0.086 / 0.143 / 0.159 / 0.181 ms |
| Max observed | 0.404 ms |
| 15 ms latency budget | Passed (151× headroom at p99) |
| Tier distribution | compact 21 · mid 41 · frontier 38 |
| Effort distribution | none 63 · low 16 · medium 12 · high 9 |
| Overthinking clamp accuracy | 16 of 20 expected (80%) |
| Estimated cost saving vs. always-frontier-high | 71.5% |
| Estimated latency saving vs. always-frontier-high | 70.1% |
The saving figures are modelled against a baseline that sends every prompt to a frontier model with high reasoning effort. They are estimates from the tier/effort distribution, not billed invoices — reproduce them with the benchmark module.
(tier, mode) resolves to a concrete model per provider via ProviderModelCatalog,
which you can override to track your own deployments:
| Tier | Mode | OpenAI | Anthropic | Gemini |
|---|---|---|---|---|
| COMPACT | standard | gpt-4o-mini |
claude-3-5-haiku-20241022 |
gemini-2.5-flash |
| COMPACT | reasoning | o3-mini |
claude-3-7-sonnet-20250219 |
gemini-2.5-flash |
| MID | standard | gpt-4o |
claude-3-5-sonnet-20241022 |
gemini-2.5-flash |
| MID | reasoning | o3-mini |
claude-3-7-sonnet-20250219 |
gemini-2.5-pro |
| FRONTIER | standard | gpt-4o |
claude-3-5-sonnet-20241022 |
gemini-2.5-pro |
| FRONTIER | reasoning | o1 |
claude-3-7-sonnet-20250219 |
gemini-2.5-pro |
- Sub-millisecond, dependency-light decision path. Pydantic v2 + NumPy only for
the core;
httpx,openaiandonnxruntimeare used by the optional transport and ONNX evaluator. - Pluggable evaluation.
HeuristicEvaluator(in-box, no network),ONNXEvaluator(local classifier),DecisionModelClient(external logits service) all satisfy the sameBaseEvaluatorcontract. - Sliding-window context triage. The last
sliding_window_sizeturns inform the score without re-scoring the whole transcript. - Speculative cascade.
SpeculativeCascadeplus verifiers (JsonSchemaVerifier,PythonSyntaxVerifier,RegexVerifier,CustomVerifier) escalate a failed compact attempt to a stronger coordinate. - Streaming shapes.
ParetoStream/ParetoAsyncStreamexposeChatCompletionChunk-compatible iteration with the routing decision attached. - Prometheus telemetry.
PrometheusMetricsExporterrecords tier dispatches, overthinking clamps and evaluation overhead in exposition or JSON form. - Strict typing.
mypy --strictover bothpareto_routerandtestsin CI.
Installed as the pareto-router console script:
pareto-router "Explain why a B-tree beats a hash index for range scans"
pareto-router --provider anthropic "hi there" # overthinking clamp fires
cat prompt.txt | pareto-router --provider gemini # reads stdinIt prints the two axis scores, the decision latency, the chosen tier, mode, effort level and token budget, and the exact model plus payload extras that would be dispatched.
Honest boundaries, so you do not discover them in production:
| Capability | Status |
|---|---|
| Two-axis scoring, decision maths, tier/effort/budget selection | Implemented and tested |
| Provider payload translation (model + reasoning knobs + headers) | Implemented and tested |
| CLI triage | Implemented |
| Benchmark suite and latency budget check | Implemented |
| Cascade + verifier escalation logic | Implemented (executes against the stub executor) |
| Prometheus/JSON telemetry | Implemented |
Live provider transport in ParetoRouter.create() |
Not implemented in 0.1.x — returns a deterministic stub InferenceResponse so routing can be tested without spending tokens. Dispatch payload with your own client for real inference. |
| Adaptive feedback loop from observed cost/quality | Planned |
git clone https://github.com/Sandeep-Hipparagi/Inference-Pareto-Router.git
cd Inference-Pareto-Router
uv sync --all-extras --dev
uv run pytest -v # test suite
uv run mypy --strict pareto_router tests
uv run python -m pareto_router.benchmarks.run_benchmarks
python examples/01_route_and_inspect.pyCI (.github/workflows/ci-publish.yml) runs on every push and pull request to
main: a Python 3.11/3.12 matrix that installs with uv, runs mypy --strict
over package and tests, runs pytest, then executes the benchmark suite with
its latency-budget assertion. On a v* tag it builds the sdist and wheel and
publishes to PyPI via trusted publishing (OIDC), falling back to PYPI_API_TOKEN
when configured.
Designed and developed by Sandeep Hipparagi.
Licensed under the Apache License 2.0 — see LICENSE. You may use, modify and redistribute it, including commercially, with an express patent grant; retain the copyright and attribution notices.
Contributions are welcome: open an issue describing the routing behaviour you need, and include a benchmark scenario that demonstrates it.