Experiments running coding assistants on top of open-weight LLMs hosted locally on a Mac.
The idea: instead of relying on hosted frontier APIs, host public/open models like Qwen, Kimi K2, GLM, and friends locally — using Ollama (or similar runtimes like LM Studio, llama.cpp, vLLM, MLX) — and then point coding assistants at them to see how usable they are for real day-to-day coding work.
🖥️ Hardware note — everything here runs on a Mac Studio. All experiments are done on a Mac Studio (Apple M4 Max, 64 GB unified memory). Commands, model sizes, and quantization choices in this repo are written for that machine. We deliberately do not provide equivalent instructions for other GPUs (NVIDIA/AMD), other Macs, or cloud instances — but conceptually everything should transfer: pick a runtime, pull a model that fits your memory/VRAM budget, point your coding assistant at the local OpenAI-compatible endpoint. Sizes and quant levels are the only things you'd retune.
- Privacy & offline — code never leaves the machine.
- Cost — no per-token API bills while experimenting.
- Learning — understand how far open models have come for agentic coding, tool use, and long-context tasks.
- Apple Silicon — modern Macs (M-series, unified memory) can run surprisingly capable models locally.
- Stand up open models locally and benchmark them for coding tasks.
- Wire those local endpoints into coding assistants / agents.
- Compare models on real workflows: code generation, refactoring, debugging, tool calling, repo Q&A.
- Document what works, what doesn't, and the hardware/quantization trade-offs.
| Model family | Notes |
|---|---|
Qwen3.6 35B-A3B (qwen3.6:35b) |
⭐ Current local workhorse on this machine. 36B-total MoE (~3B active), Q4_K_M ≈ 23 GB, 256K context, tool-calling + thinking. Fast and capable for agentic coding — see benchmarks below. |
| Qwen (Qwen2.5-Coder, etc.) | Strong open coding models, multiple sizes; Qwen2.5-Coder 32B was the previous default |
| GLM (GLM-4.5 / GLM-4.5-Air / CodeGeeX) | Capable general + coding models; GLM-4.5-Air is the locally-feasible variant |
| Kimi K2 (Moonshot AI) | Large open MoE (~1T params). |
| others | Add as we test (Nemotron 3 Nano, North Mini Code, DeepSeek-Coder, Llama, Mistral, Codestral, ...) |
The exact list will grow/change as we run experiments. The local picks here track the "best small local coding models" landscape — see Sebastian Raschka's "Using Local Coding Agents", which recommends Qwen3.6 35B-A3B, Cohere's North Mini Code, and Nemotron 3 Nano in this size class.
⚠️ Caveat — Kimi K2 does not run locally on this hardware. It's a ~1T-parameter MoE model. Our Mac Studio has 64 GB, and even the largest configurable Mac Studio (M3 Ultra, 512 GB unified memory) can't host it — not even a heavily quantized version — once you account for weights plus KV cache / context. We'll only use Kimi K2 through Moonshot's hosted API (OpenAI-compatible) for comparison. Local hosting in this repo targets models that actually fit, e.g. Qwen2.5-Coder and GLM at practical quant sizes.
Primary runtime: Ollama on macOS.
# install (macOS)
brew install ollama
# or download the app from https://ollama.com
# start the server
ollama serve
# pull and run our current local workhorse (fits comfortably in 64 GB)
ollama pull qwen3.6:35b
ollama run qwen3.6:35bOllama exposes an OpenAI-compatible API at http://localhost:11434/v1, which most coding assistants can be pointed at.
Alternatives we may also try: LM Studio, llama.cpp, MLX, vLLM.
Quick smoke test (single Python function, with the model's thinking pass enabled), Mac Studio M4 Max / 64 GB:
| Metric | Value |
|---|---|
| Generation (eval) rate | ~90 tok/s |
| Prompt eval rate | ~75 tok/s |
| Cold load time (first call) | ~9 s |
| Model on disk | 23 GB (Q4_K_M) |
| Context length | 262,144 tokens |
For reference, Raschka measured ~40 tok/s for this model on a Mac Mini M4 — the extra memory bandwidth of the M4 Max roughly doubles throughput here, comfortably above the "20–30 tok/s is fine for agent work" bar.
ℹ️
qwen3.6:35bis a thinking model — it emits a visible reasoning trace before its answer, which improves quality but adds latency/tokens per turn. Whether that reasoning is shown/used depends on the harness you point at it.
We wanted GLM-5.2 as the headline GLM example, but it's a 756B-parameter model and Ollama only publishes it as a cloud model:
# Runs on Ollama's CLOUD, not on this Mac Studio. Requires `ollama signin`.
ollama run glm-5.2:cloud:cloud means inference happens on Ollama/Z.ai servers — your machine is just the client. There is no local tag that fits a 64 GB (or even a 512 GB) Mac.
For a GLM you can actually host locally, drop down to a smaller variant such as GLM-4.5-Air (~106B MoE — runnable only at aggressive quantization and tight even then on 64 GB) or a small dense GLM-4 model. In practice, the comfortable local coding workhorse on this 64 GB machine is Qwen3.6 35B-A3B (qwen3.6:35b, ~23 GB at Q4) — hence the example above.
- CLI / agentic assistants that support a custom OpenAI-compatible base URL (e.g. Qwen-Code, Cline, Aider, OpenCode)
- Claude Code pointed at the local model via a translation proxy — see below
- Editor integrations (e.g. Continue, Cline, Aider)
- Custom scripts / agents
Claude Code talks the Anthropic Messages API, while Ollama serves the OpenAI API — so they don't connect directly. The bridge is a small proxy that exposes an Anthropic-compatible endpoint backed by Ollama. LiteLLM is the standard choice.
# 1. Make sure the model is served locally
ollama serve # http://localhost:11434
ollama pull qwen3.6:35b
# 2. Run LiteLLM as an Anthropic-compatible proxy in front of Ollama
pip install "litellm[proxy]"
litellm --model ollama/qwen3.6:35b --port 4000
# → exposes http://localhost:4000 speaking the Anthropic API
# 3. Point Claude Code at the proxy and launch
export ANTHROPIC_BASE_URL="http://localhost:4000" # note: no /v1 — Claude Code appends the path
export ANTHROPIC_API_KEY="dummy" # required by Claude Code even though the proxy ignores it
claude --model ollama/qwen3.6:35bVerify the active endpoint/model from inside Claude Code with /status.
Caveats (these matter more with local models than with Anthropic's API):
- Tool use — Claude Code is heavily tool-driven (file edits, bash, etc.).
qwen3.6:35bis tool-capable, but smaller local models are hit-or-miss; expect more retries/misfires than with frontier models. Test on a small task first. - No prompt caching — disabled against a non-Anthropic endpoint, so long sessions re-send context each turn (slower, but no $ cost locally).
- Thinking model latency —
qwen3.6:35bemits a reasoning trace before acting, which adds tokens/latency per agent step. - Persistent config — instead of env vars you can set
env.ANTHROPIC_BASE_URL/env.ANTHROPIC_API_KEYandmodelin~/.claude/settings.json.
⚠️ This is for experimentation — running Claude Code on a local open model is not an officially supported Anthropic configuration, and the experience won't match Claude models. The point of this repo is to see how close open models get.
.
├── README.md
├── models/ # notes & configs per model
├── runtimes/ # ollama / lmstudio / vllm setup notes
├── assistants/ # configs for each coding assistant
└── experiments/ # writeups, benchmarks, findings
🚧 Early / exploratory. This is a personal experimentation repo — expect rough notes.
MIT