One command against any OpenAI-compatible chat endpoint. One dated, evidence-backed diagnostic card.
routecheck runs a fixed battery of behavioural probes (QFS-CORE, C1–C10)
against a running /v1/chat/completions route and writes a card. It
targets the widely-implemented OpenAI-compatible shape (llama.cpp server
and Ollama are the routes it has been run against live so far; authed and
remote endpoints are supported and covered by the offline suite). It
writes:
card.json— machine-readable results, every field tagged with its provenance,card.md— the same results in plain English,raw/— the complete raw request/response body for every probe, so any claim on the card can be checked against what actually went over the wire.
Naming note (once, so nothing looks inconsistent): routecheck began life
under the internal codename qwenfield, and two identifiers deliberately
keep that history so every card ever written stays comparable — the
internal package is src/qwenfield/, card.json's machine instrument
field says "qwenfield", and the protocol id is QFS-1.0. These are
frozen identifiers, not branding; the sha256-pinned canonical prompts also
predate the rename and are frozen byte-for-byte by design.
| Probe | Question it answers about this endpoint, on this date |
|---|---|
| C1 Identity | What exactly was tested: model id, served file name/size (sha256 optional), context size, server build when exposed, full command line, date. A card without a complete identity block is never written. |
| C2 Load/placement | Operator-supplied only (load time, VRAM/RSS, placement recipe) — the tool talks to an already-running route and says SKIPPED honestly when this data isn't handed in. |
| C3 Speed | Decode tokens/sec medians (warm, ≥3 reps, cold rep discarded) for a fixed prose prompt and a fixed structured prompt. Server-reported timings are authoritative when present; client wall-clock is a labeled diagnostic. |
| C4 Prefill | Prompt-processing tokens/sec on a fixed ~8K-token passage, same median rules. |
| C5 Thinking budget | Does visible content come back empty at small max_tokens (60/500/2000/4096 ladder)? Reasoning-capable endpoints can spend the whole budget on hidden reasoning and return an empty answer with finish_reason=length. C5 measures the smallest probed budget at which the endpoint visibly answers, and that measured floor is used to size every other test in the same run. |
| C6 Thinking toggle | Is the enable_thinking request field honoured? Checks for a separate reasoning field, for <think>-tag leakage into visible content, and for reasoning text duplicated untagged into the answer. The verdict names this exact mechanism — a route using a different toggle is not exercised. |
| C7 Tool calls | One fixed tool + one fixed user message, two shots. Pass requires exactly one tool call, the right function, schema-conformant arguments, and the expected explicit values. |
| C8 Structured output | A fixed JSON Schema embedded in a fixed prompt, 3 reps, conformance checked (required keys, types, extra-key policy). |
| C9 Honesty probe | Asks about a nonexistent file and a nonexistent benchmark; a keyword heuristic flags hedging vs. confident fabrication. Explicitly not a semantic judge — the raw response is retained for human review, and the card's own grading_method field says exactly how it was graded. |
| C10 Needle retrieval | A seeded, programmatically generated haystack sized to ~75% of the served context, one planted fact, one question. The server's own usage.prompt_tokens is the recorded authoritative length. |
No install step, no packaging layer, no dependencies beyond Python 3.10+
and the requests package — everything else is stdlib. Two supported
paths, deliberately boring:
git cloneand run in place. Clone the repo, run./routecheckfrom the checkout. Done.- Plain directory copy. Copy the whole directory anywhere (USB stick,
scp, air-gapped box) and run./routecheckfrom the copy. Nothing depends on the directory's location or on.git/being present.
Either way, ./smoke.sh verifies a fresh copy in seconds (--version,
--help, prompt-manifest integrity), and make test runs the full
offline suite.
git clone <repo> && cd <repo>
./routecheck run \
--base-url http://127.0.0.1:8080/v1 \
--model-id my-served-model \
--ctx-size 32768 \
--out runs/my-model/$(date +%F)/
That's the whole thing. --ctx-size is required unless the route's
/props endpoint reports n_ctx (llama.cpp server does; then the
server's own value is preferred and recorded as such).
Against an authenticated endpoint:
export ROUTECHECK_API_KEY=... # sent as "Authorization: Bearer <key>"
./routecheck run \
--base-url https://api.example.com/v1 \
--model-id some-model \
--ctx-size 32768 \
--allow-remote \
--out runs/some-model/$(date +%F)/
(QWENFIELD_API_KEY, the pre-rename env var, is still honoured as a
legacy fallback; ROUTECHECK_API_KEY wins when both are set.)
Non-loopback URLs require the explicit --allow-remote flag. The API key
is never logged and never written into card.json, card.md, or any
raw/ artifact; the recorded command line shows [REDACTED] in its
place, and proxy env vars / ~/.netrc are never inherited.
Useful flags: --reps N (≥3), --seed, --timeout, --hash-gguf,
--gpu-probe (opt-in nvidia-smi line), operator-supplied C2 metadata
(--load-time-s, --vram-mb, --rss-mb, --placement-recipe,
--load-log). See --help.
Excerpts from a live run against a 27B model served by Ollama on a single consumer GPU (2026-08-09; excerpted verbatim — some field labels have since been tightened by a wording pass, numbers and structure unchanged):
## Identity block (C1)
- **Model id:** `qwen3.6:27b`
- **Base URL:** `http://127.0.0.1:11434/v1`
- **Ctx size (REQUIRED):** 32768 (provenance=STATED, source=--ctx-size flag)
- **GPU (nvidia-smi):** `NVIDIA GeForce RTX 5090, 32607 MiB, 20247 MiB, 580.173.02` (provenance=RECORDED)
### C5 - Thinking-budget map
| max_tokens | finish_reason | content chars | empty visible content |
|---|---|---|---|
| 60 | length | 0 | True |
| 500 | length | 0 | True |
| 2000 | stop | 357 | False |
| 4096 | stop | 357 | False |
### C7 - Tools (2 shots)
- shot 1: pass=True, finish_reason=tool_calls, tool_calls_count=1, values_ok=True
- shot 2: pass=True, finish_reason=tool_calls, tool_calls_count=1, values_ok=True
### C8 - Structured output (3 reps, protocol-fixed)
- conformance: 3/3
Read: on this endpoint, on this date, answers at 60 and 500 output tokens came back empty (the budget went to reasoning before any visible text); 2000 was enough. Tool-call shape and structured output both conformed. The same run also recorded a needle miss at ~28.5K prompt tokens — a finding, retained with its full raw request and response, not an embarrassment to hide.
- Every card field carries provenance.
RECORDED(raw artifact retained on disk — the CLI refuses to write a card whose RECORDED claims don't resolve to real files),STATED(observed or operator-supplied, no artifact), orSKIPPED(not attempted, with the reason stated). - INVALID is its own verdict. An empty answer at
finish_reason=lengthmeasures nothing about the model's answer. It is not scored as a fail — the card marks the test INVALID, says why, and tells you not to quote it. - A failed probe is never a placeholder. Connection refused, HTTP errors, malformed JSON, unexpected exceptions — all recorded with their raw evidence; one test's failure never drops the rest of the run.
- Prompts are integrity-checked before any network call. Every prompt
file is sha256-pinned in a manifest, and the manifest itself is pinned in
code. A modified prompt refuses to run; an explicit
--prompts-diroverride runs but the card is conspicuously flagged non-canonical. - Scope is stated on the card. Measurements describe the endpoint and configuration in the identity block, on the card's date — not the model in general.
- Nothing about quality or intelligence. No benchmark scores, no win rates, no rankings. It probes mechanical contract behaviour: budgets, toggles, tool shape, schema conformance, retrieval of one planted fact, throughput.
- Nothing universal about a model. A card describes one served configuration (quant, server, flags, context) at one point in time. A different quant or server of the same model can behave differently — that's much of the point of measuring.
- C9 is a heuristic, not a judge. Keyword flags over the raw response; check the retained response before repeating any honesty verdict.
- C10 is synthetic filler, not natural prose — it tests retrieval of a planted fact, not general long-context comprehension.
- C6 tests one toggle mechanism (
enable_thinking). A route using a different mechanism will look like a non-working toggle; the card says so rather than guessing. - C2 (load time/VRAM) can't be measured over HTTP by a tool that didn't start the server; it's operator-supplied or honestly SKIPPED.
- Speed numbers are not a leaderboard. Medians of 3+ warm reps on fixed prompts; server-reported and client-side figures are kept separate and labeled.
make test
331 offline tests: no GPU, no model, no network beyond in-process loopback mock servers. The suite deliberately proves the negative paths — tampered manifests refused, empty-answer traps detected both ways, symlink escapes refused, RECORDED-without-evidence refused, API keys absent from every written file.
MIT (see LICENSE).