Every app gets a local-first LLM with one API call — the SDK figures out the device, the model, the runtime, and when to fall back.
Modern LLM apps are thin clients over expensive, high-latency, privacy-hostile cloud APIs. The models and engines needed to run capable LLMs on consumer hardware already exist — what's missing is the orchestration layer between apps and that machinery. This project is that layer, built measurement-first: every design constant traces to an experiment logged in harness/FINDINGS.md.
Measured so far on a fanless M4 MacBook Air, a GTX 1050 Ti Windows laptop, and an iPhone 16. Some things that turned out not to be true:
- Published runtime rankings don't transfer between Macs, let alone to a phone (F1/F7).
- A fanless laptop's ~30% sustained clamp is a chassis property, not a local-inference one — a fan-cooled machine is flat (F21).
- A reasoning model's "accuracy" is mostly its token budget: 73% at 512 tokens, 98% at 2048, with one genuinely wrong answer in 370 (F17/F18).
- Memory is not what stops a phone running a 4B model — its weights are mmap'd and barely count against the kill threshold. Prefill latency and heat stop it (F37/F39).
- Four constants imported from credible sources failed on contact with hardware. None was wrong where it was measured; each described a regime we weren't in (F23, F24, F32, F34→F37).
- The routing policy was wrong three times in a row, and every failure was found by using it rather than by testing it — a synthetic suite that drove the router with a laptop profile passed throughout (F46/F47/F48).
▶ Watch the 24-second demo — answering with no network.
 |
 |
| Every turn shows the routing decision. Here the request is priced at ~0.7% of battery, over the 0.5% budget — with no cloud key configured it says so and runs locally anyway. | And what it cost. 2,046 tokens, 104.9s, 19.5 tok/s, 1.14% of battery. A router that spends your battery invisibly is the thing this project set out not to build. |
 |
 |
| The benchmark tab. Jetsam headroom is the number that matters on iOS, not RAM: 2,807 MB usable, and a loaded 1.7B costs 569 MB of footprint because weights are mmap'd (F37/F39). | Per-prompt-class results. Prefill is reported separately from decode, because on a phone prefill is what you feel: 1,027 tok/s prefill and 64.5 tok/s decode on a ~1,900-token prompt. |
Rows export as JSONL in the same shape as harness/results/, so a phone
run and a laptop run are directly comparable.
| Path | What it is |
|---|---|
| RESEARCH-AND-PLAN.md | Deep-researched, source-verified state of the art + the phased plan (with plain-English explanations) |
| harness/ | llmbench — benchmark harness (TTFT, decode/prefill, memory, sustained-thermal, quality eval with CIs) and whatfits — device profiler / model recommender with live memory-pressure probing |
| harness/FINDINGS.md | The lab notebook: 48 findings across 18 sessions on three devices, including nine corrections of our own earlier conclusions |
| sdk/ | LocalLLMKit — Swift SDK (macOS + iOS): profile → select → fetch → launch → stream, in one call. Server-based engines (llama.cpp, MLX) on macOS; an in-process llama.cpp engine on both |
| ios-harness/ | llmbench-on-iOS — the same benchmark protocol on a phone, plus a chat demo that shows each turn's routing decision and battery cost |
| docs/ | Design docs, including the mobile design doc whose research assumptions the device runs then corrected |
# What's the best local LLM for this machine?
cd harness && uv sync
uv run whatfits # static ranking + recommendation
uv run whatfits --probe Qwen3-8B # live fit verdict (loads the model,
# watches system memory pressure)
uv run whatfits --device iphone16 # what fits a measured phone
# Benchmark it yourself (needs `brew install llama.cpp` for the servers)
uv run llmbench fetch && uv run llmbench run
uv run llmbench sustain --minutes 10 # thermal / steady-state
uv run llmbench eval # quality eval, truncation-aware, Wilson CIs
uv run python -m llmbench.listeval # per-item accuracy inside enumerated answers
# The SDK — no brew needed, llama.cpp is linked as an XCFramework
cd ../sdk && swift run localllm-demo --engine inprocess "Hello!"
swift run localllm-chat # multi-turn, with routing decisions shown
swift run localllm-prefill Qwen3-1.7B # prefill vs decode throughput
swift run localllm-soak --minutes 60 # leak + thermal soak, gated on generation time
# On the phone
open ../ios-harness/llmbench-ios.xcodeproj # see ios-harness/README.md- "Best runtime" is not a constant. On a fanless M4 Air, llama.cpp wins time-to-first-token, MLX wins sustained throughput — and the published M2 Ultra ranking didn't transfer. Selection must be per-device, per-workload. (F1, F7)
- Fanless Macs throttle ~30% after ~90 seconds, with a universal boost → minute-2 crater → recovery signature. Peak benchmarks are fiction for long sessions; the profiler predicts burst and sustained separately. (F5, F8)
- "Does it fit?" is a runtime question. No per-process metric can
answer it (mmap'd weights are evictable page cache; Metal buffers are
wired) — so
whatfits --probetrial-loads the model and watches the system's own vitals. (F10, F12) - Quality is inseparable from token budget for thinking models. The 8B answered 100% of items it completed but truncated 5–11% at a 1024-token budget — and truncation rates differ per runtime on the same model. Naive evals mistake budget artifacts for intelligence. (F14, F17)
All phases have working v0s. 52 findings, 21 SDK tests, measured on three devices.
- Phase 0 — benchmark harness: done, and validated cross-platform. The same harness runs unmodified on macOS, Windows/NVIDIA and iOS.
- Phase 1 —
whatfitsprofiler: VRAM-aware for discrete GPUs, andwhatfits --device iphone16models a measured phone from the desktop. - Phase 2 — SDK core (
LocalLLMKit): llama.cpp and MLX server engines on macOS, plus an in-process llama.cpp engine for macOS and iOS — linked as a pinned XCFramework, so there is nobrew installprerequisite and no server process to orphan. - Phase 3 — hybrid routing: prices each request as prefill + decode against a latency budget and a battery budget, using per-device constants that were measured rather than assumed.
- Phase 4 — mobile: in-process engine (4a),
llmbench-on-iOS(4b), on-device model store (4c) and the chat demo (4d) all shipped. The app runs a 1.7B at ~40 tok/s on an iPhone 16 and shows each turn's routing decision and battery cost.
Open, honestly: 4d's gate is a week of real-world routing logs, which only usage produces. Android (same C core via JNI) and mlx-swift as a second in-process engine are both unstarted. Whether a small model can hold a tool-calling loop together on-device is unmeasured, so no agent layer exists yet. The mobile model store has no download resume.