|
| 1 | +# Running Local Ollama Models in the AILANG Eval Rig — Operations Runbook |
| 2 | + |
| 3 | +Hard-earned lessons from the 2026-05-22/23 debugging sessions. Consolidates |
| 4 | +findings that are currently scattered across commits + design docs + chat |
| 5 | +history into one document you can hand to a new person (or your future |
| 6 | +self) for onboarding a new model. |
| 7 | + |
| 8 | +## TL;DR setup checklist (new model, e.g. qwen3-coder:30b) |
| 9 | + |
| 10 | +```bash |
| 11 | +# 1. Pull the base model |
| 12 | +ollama pull qwen3-coder:30b |
| 13 | + |
| 14 | +# 2. Build an AILANG-tuned variant (sampling fixes + output cap) |
| 15 | +ollama create qwen3-coder:30b-ailang \ |
| 16 | + -f tools/ollama/qwen3-coder-30b-ailang.modelfile # adapt from gemma4-26b-ailang.modelfile |
| 17 | + |
| 18 | +# 3. Register in opencode.jsonc with options.name="Ollama" (CRITICAL — see #971 below) |
| 19 | +# and per-model options block (temperature, max_tokens, etc.) |
| 20 | +$EDITOR ~/.config/opencode/opencode.jsonc |
| 21 | + |
| 22 | +# 4. Add models.yml entry routing to the variant |
| 23 | +$EDITOR internal/eval_harness/models.yml |
| 24 | + |
| 25 | +# 5. (Re)install ailang — auto-symlinks to ~/.local/bin for opencode-compat |
| 26 | +make install |
| 27 | + |
| 28 | +# 6. Sanity-check: opencode should see ailang AND your variant |
| 29 | +opencode mcp list # ailang-docs should be ✓ connected |
| 30 | +opencode models | grep ollama |
| 31 | +opencode run --format json --dangerously-skip-permissions \ |
| 32 | + --model "ollama/qwen3-coder:30b-ailang" "Run: ailang --version. Output DONE." |
| 33 | + |
| 34 | +# 7. Warm up the rig |
| 35 | +.claude/skills/local-ollama-eval/scripts/warmup_rig.sh ollama/qwen3-coder:30b-ailang |
| 36 | + |
| 37 | +# 8. Single-benchmark smoke |
| 38 | +make eval-smoke MODELS=opencode-qwen3-coder-30b-ailang \ |
| 39 | + EXTRA="-trials 1 -parallel 1 -agent-timeout 1200 -langs ailang \ |
| 40 | + -benchmarks fizzbuzz -output /tmp/quick_check" |
| 41 | + |
| 42 | +# 9. If it passes, full smoke tier with N=3 trials |
| 43 | +make eval-smoke MODELS=opencode-qwen3-coder-30b-ailang \ |
| 44 | + EXTRA="-trials 3 -parallel 1 -agent-timeout 1200 -langs ailang \ |
| 45 | + -output eval_results/rotation/$(date +%Y-%m-%d)/$(date +%H%M)_<model>_smoke_n3" |
| 46 | +``` |
| 47 | + |
| 48 | +## The mental model |
| 49 | + |
| 50 | +``` |
| 51 | + eval-suite (Go) |
| 52 | + │ |
| 53 | + │ spawns ONE opencode subprocess per benchmark trial (parallel = N from -parallel flag) |
| 54 | + ▼ |
| 55 | + opencode CLI (Node) |
| 56 | + │ |
| 57 | + │ reads ~/.config/opencode/opencode.jsonc (provider config, options, MCP servers, bash permissions) |
| 58 | + │ reads ~/.config/opencode/agents/*.md (per-agent system prompts; we use default "build" agent) |
| 59 | + │ spawns tool subprocesses (bash, read, write, edit) — sanitized PATH, NOT inheriting ~/go/bin |
| 60 | + │ sends OpenAI-compat /v1/chat/completions to provider |
| 61 | + ▼ |
| 62 | + ollama serve (Go) — process under launchd |
| 63 | + │ |
| 64 | + │ loads model into VRAM (kept resident via OLLAMA_KEEP_ALIVE=-1) |
| 65 | + │ serves one request at a time (OLLAMA_NUM_PARALLEL=1) |
| 66 | + │ Modelfile params apply: top_k, repeat_penalty, min_p, num_predict |
| 67 | + ▼ |
| 68 | + gemma4:26b-ailang (the variant we built) |
| 69 | +``` |
| 70 | + |
| 71 | +Four configuration surfaces, each in a different place: |
| 72 | + |
| 73 | +| Surface | What it controls | Lives at | |
| 74 | +|---|---|---| |
| 75 | +| Modelfile | ollama-native params (top_k, repeat_penalty, num_predict, min_p) | `tools/ollama/<model>-ailang.modelfile` | |
| 76 | +| opencode.jsonc model options | OpenAI-compat params (temperature, top_p, frequency_penalty, max_tokens) + MCP wiring + bash perms | `~/.config/opencode/opencode.jsonc` | |
| 77 | +| models.yml | harness routing (api_name, agent_cli, timeouts, budgets) | `internal/eval_harness/models.yml` | |
| 78 | +| ollama plist env | server-wide config (NUM_PARALLEL, KV_CACHE_TYPE, FLASH_ATTENTION, KEEP_ALIVE) | `~/Library/LaunchAgents/dev.ollama.serve.plist` | |
| 79 | + |
| 80 | +A mistake at any surface is silent in the others. The most painful traps below. |
| 81 | + |
| 82 | +## Hard-won lessons — DO NOT re-discover these |
| 83 | + |
| 84 | +### 1. opencode's child-shell PATH does NOT include `~/go/bin` |
| 85 | + |
| 86 | +**Symptom**: bash tool fails with `command not found: ailang`. Model falls into `find / -name ailang`, `ls -R /`, or guessing `/usr/local/bin/ailang` / `/opt/homebrew/bin/ailang` paths. Sessions hang for many minutes per pathological command. |
| 87 | + |
| 88 | +**Why**: opencode launches bash subprocesses with a sanitized PATH (`~/.local/bin:/usr/local/bin:/usr/bin:/bin:...:/opt/homebrew/bin`). Setting PATH in the executor env doesn't help — opencode overrides. |
| 89 | + |
| 90 | +**Fix**: `make install` now auto-symlinks `~/.local/bin/ailang -> ~/go/bin/ailang`. If you suspect the symlink is missing, `ls -la ~/.local/bin/ailang` to check. If gone, just `make install` again. |
| 91 | + |
| 92 | +### 2. opencode silently drops model options without `options.name` |
| 93 | + |
| 94 | +**Symptom**: opencode.jsonc has `provider.ollama.models["foo"].options = {temperature: 0.5, ...}` but generation behaves like default sampling. Temperature, max_tokens, frequency_penalty all ignored. |
| 95 | + |
| 96 | +**Why**: [sst/opencode#971](https://github.com/sst/opencode/issues/971) — `@ai-sdk/openai-compatible` provider requires `options.name` to be set OR all per-model options are dropped. Easy trap because the option block looks correct. |
| 97 | + |
| 98 | +**Fix**: |
| 99 | +```jsonc |
| 100 | +"provider": { |
| 101 | + "ollama": { |
| 102 | + "options": { |
| 103 | + "name": "Ollama", // <-- THIS is required |
| 104 | + "baseURL": "http://localhost:11434/v1" |
| 105 | + }, |
| 106 | + ... |
| 107 | + } |
| 108 | +} |
| 109 | +``` |
| 110 | + |
| 111 | +**Verify**: run the same prompt twice through opencode at `temperature: 0.5`. If output is identical, options pass through. If wildly different, they're being dropped. |
| 112 | + |
| 113 | +### 3. ollama's `/v1/chat/completions` only honors a subset of sampling params |
| 114 | + |
| 115 | +OpenAI-compat surface passes: `temperature`, `top_p`, `frequency_penalty`, `presence_penalty`, `max_tokens`, `seed`, `stop`, `stream`. |
| 116 | + |
| 117 | +Silently drops: `top_k`, `repeat_penalty`, `min_p`, `num_predict`, `repeat_last_n`. |
| 118 | + |
| 119 | +**Fix**: put dropped params in a Modelfile and `ollama create` a variant. The Modelfile params apply at model-load time and aren't affected by request-level options. |
| 120 | + |
| 121 | +### 4. gemma4:26b is prone to token-repetition collapse at default sampling |
| 122 | + |
| 123 | +**Symptom**: a single LLM response generates 30k+ tokens of repeated garbage (e.g., `v_h_y_f_d_v_h_y_f_d_...`). Per-call wall time blows up to 9+ minutes. |
| 124 | + |
| 125 | +**Why**: Google's default `temperature=1.0` with no repetition penalty is a known bug ([google-deepmind/gemma#622](https://github.com/google-deepmind/gemma/issues/622)) on the 26b MoE variant. Affects tool-argument generation particularly. |
| 126 | + |
| 127 | +**Fix**: |
| 128 | +- Modelfile: `repeat_penalty 1.1`, `min_p 0.05`, `num_predict 4096` (hard cap) |
| 129 | +- opencode.jsonc options: `temperature 0.5`, `frequency_penalty 0.3`, `max_tokens 4096` |
| 130 | + |
| 131 | +Trade-off: lower temperature reduces peak coding ability for Gemma 4 (per Unsloth's HF #21 thread) but eliminates collapses. We trade peak for reliability. |
| 132 | + |
| 133 | +### 5. Bash permission denylist works EVEN with `--dangerously-skip-permissions` |
| 134 | + |
| 135 | +Deny rules always deny, regardless of the dangerously flag. So you can run unattended (no permission prompts) AND still block pathological commands. |
| 136 | + |
| 137 | +```jsonc |
| 138 | +"permission": { |
| 139 | + "bash": { |
| 140 | + "*": "allow", |
| 141 | + "find / *": "deny", |
| 142 | + "find /System*": "deny", |
| 143 | + "ls -R /*": "deny", // trailing * catches pipelines like `ls -R / | grep X` |
| 144 | + "grep -r /*": "deny", |
| 145 | + "rm -rf /*": "deny", |
| 146 | + "sudo *": "deny" |
| 147 | + } |
| 148 | +} |
| 149 | +``` |
| 150 | + |
| 151 | +**Wildcard gotcha**: `"ls -R /": "deny"` matches the literal but NOT `ls -R / | grep ailang` (pipeline form). The trailing `*` in `"ls -R /*"` is required. |
| 152 | + |
| 153 | +### 6. `-agent-parallel` was a DEAD flag (removed 2026-05-23) |
| 154 | + |
| 155 | +The harness used to advertise `-agent-parallel` as a concurrency knob; it was decoration only. The real dispatch semaphore is `-parallel` (default 10). Mismatching these silently oversubscribes ollama's NUM_PARALLEL=1 queue, causing 15-minute TTFT timeouts. |
| 156 | + |
| 157 | +The flag has been deleted as of [commit 98c0c408](commits/98c0c408). The status banner now prints `Dispatch parallelism: N (-parallel flag)` to unambiguously identify the correct knob. |
| 158 | + |
| 159 | +### 7. opencode default agent has compaction baked in |
| 160 | + |
| 161 | +opencode's "build" agent auto-compacts when context is "full" — meaning at the model's max context window (e.g. 262k for gemma4:26b). Our typical agentic sessions hit ~36k input tokens stably (verified across 56 prior sessions in chains DB), well below the threshold. Compaction does NOT save us when individual responses go runaway (the 32k repetition case). Output capping via Modelfile `num_predict` is the right control there. |
| 162 | + |
| 163 | +### 8. AILANG MCP server vs CLI fallback |
| 164 | + |
| 165 | +The remote MCP at `mcp.ailang.sunholo.com` exposes 14+ structured tools (`ailang.prompt_get`, `ailang.stdlib_search`, `ailang.example_for_concept`, etc.). With `options.name="Ollama"` set and the MCP block in opencode.jsonc, **opencode picks up MCP tools automatically and adds them to the agent's tool catalog**. |
| 166 | + |
| 167 | +Verified live: with the v0.10.0-slim seed prompt that names the MCP tools, gemma4:26b uses them — observed `ailang-docs_stdlib_search` calls with queries like `"if"` and `"for"`. The agent is genuinely discovering syntax dynamically. |
| 168 | + |
| 169 | +If MCP is unreachable, opencode logs the error and continues; agent falls back to shelling out to `ailang docs` / `ailang examples` CLI. |
| 170 | + |
| 171 | +## Healthy-rig signatures (what "working" looks like) |
| 172 | + |
| 173 | +Watch the ollama log (`/tmp/ollama-serve-launchd.log`) during a run: |
| 174 | + |
| 175 | +| Pattern | Interpretation | |
| 176 | +|---|---| |
| 177 | +| Per-call durations 1s — 60s, most under 30s | ✓ Normal agentic loop | |
| 178 | +| Occasional 60-120s call (warm cache miss after gap) | ✓ Normal — model loading or large prefill | |
| 179 | +| Sub-second call (`1.7s`, `2.3s`) | ✓ Best — cache hit, no marginal prefill | |
| 180 | +| Same 200 status, slowly-growing durations (5s → 10s → 30s → 5min) | ✗ Queue oversubscription — check `-parallel` | |
| 181 | +| 200 status with `15m0s` duration | ✗ TTFT timeout — model didn't start producing for 15min | |
| 182 | +| 500 status | ✗ Real error — check ollama server log for OOM or model unload | |
| 183 | + |
| 184 | +Watch chains DB (`observatory.db`): |
| 185 | + |
| 186 | +```bash |
| 187 | +sqlite3 -readonly ~/.ailang/state/observatory.db <<'SQL' |
| 188 | +SELECT |
| 189 | + duration_ms, |
| 190 | + json_extract(attributes, '$.opencode.input_tokens') AS in_tok, |
| 191 | + json_extract(attributes, '$.opencode.output_tokens') AS out_tok |
| 192 | +FROM spans WHERE name = 'opencode.step' |
| 193 | +ORDER BY start_time DESC LIMIT 10; |
| 194 | +SQL |
| 195 | +``` |
| 196 | + |
| 197 | +| Pattern | Interpretation | |
| 198 | +|---|---| |
| 199 | +| input_tokens stable around 30-40k across steps | ✓ opencode compaction working | |
| 200 | +| output_tokens < 4096 (the Modelfile cap) per step | ✓ num_predict cap honored | |
| 201 | +| output_tokens jumping to 30k+ in one step | ✗ Sampling collapse — check Modelfile + opencode.jsonc options | |
| 202 | +| input_tokens growing each step (40k → 60k → 80k) | ⚠ History accumulating; eventual prefill ceiling | |
| 203 | + |
| 204 | +## Pre-flight checklist before a long rotation |
| 205 | + |
| 206 | +```bash |
| 207 | +# 1. Rig precondition (verify_setup.sh wraps these) |
| 208 | +curl -sf http://localhost:11434/api/tags >/dev/null # ollama UP |
| 209 | +curl -sf http://localhost:1957/health >/dev/null # ailang server UP |
| 210 | +launchctl list | grep dev.ailang.rig-watchdog # watchdog registered |
| 211 | +ollama list | grep <your-model> # variant exists |
| 212 | + |
| 213 | +# 2. Verify env config is sane |
| 214 | +ollama show <your-model> --modelfile | grep -E "PARAMETER" # sampling params baked in |
| 215 | +opencode mcp list | grep "ailang-docs.*connected" # MCP up |
| 216 | +opencode models | grep <your-model> # opencode sees model |
| 217 | + |
| 218 | +# 3. Warmup |
| 219 | +.claude/skills/local-ollama-eval/scripts/warmup_rig.sh ollama/<your-model> |
| 220 | + |
| 221 | +# 4. Single-trial sanity (1 benchmark, agent mode) |
| 222 | +make eval-smoke MODELS=<your-model> \ |
| 223 | + EXTRA="-trials 1 -parallel 1 -agent-timeout 1200 -langs ailang \ |
| 224 | + -benchmarks fizzbuzz -output /tmp/preflight_check" |
| 225 | +# Expect: opencode subprocess produces a solution.ail, ailang check / run ran, exits in <10 min |
| 226 | + |
| 227 | +# 5. Full rotation |
| 228 | +make eval-smoke MODELS=<your-model> \ |
| 229 | + EXTRA="-trials 3 -parallel 1 -agent-timeout 1200 -langs ailang \ |
| 230 | + -output eval_results/rotation/$(date +%Y-%m-%d)/$(date +%H%M)_<model>_smoke_n3" |
| 231 | +``` |
| 232 | + |
| 233 | +## Adding a model — checklist of all surfaces |
| 234 | + |
| 235 | +When you add a new model to the rig, you must touch all four configuration surfaces. Missing any one is silent failure. |
| 236 | + |
| 237 | +| Surface | What | Where | Failure mode if missing | |
| 238 | +|---|---|---|---| |
| 239 | +| 1 | Modelfile (sampling) | `tools/ollama/<model>-ailang.modelfile` | runs with bad defaults, may collapse | |
| 240 | +| 2 | Build variant | `ollama create <model>-ailang -f <modelfile>` | model not available | |
| 241 | +| 3 | opencode provider model | `~/.config/opencode/opencode.jsonc` `provider.ollama.models["<model>-ailang"]` with options block | opencode doesn't know about it | |
| 242 | +| 4 | models.yml entry | `internal/eval_harness/models.yml` with `agent_cli: "opencode"`, `agent_model_name: "ollama/<model>-ailang"` | harness can't route | |
| 243 | + |
| 244 | +## When something goes wrong — diagnostic flowchart |
| 245 | + |
| 246 | +**Sessions hang for many minutes**: |
| 247 | +1. Check `~/.local/bin/ailang` symlink exists. If not: `make install`. |
| 248 | +2. Check opencode bash logs for `find /`, `ls -R /`, or thrashing on PATH discovery. Update bash denylist. |
| 249 | +3. Check `/tmp/ollama-serve-launchd.log` for individual call durations. If 15m timeouts, check `-parallel` value (must be 1 to match server NUM_PARALLEL=1). |
| 250 | + |
| 251 | +**Sessions complete but produce 30k+ token outputs**: |
| 252 | +1. Check Modelfile has `repeat_penalty` and `num_predict` baked in: `ollama show <model> --modelfile`. |
| 253 | +2. Check opencode.jsonc has `options.name="Ollama"` set (sst/opencode#971). |
| 254 | +3. Check `temperature` is not 1.0 (Google default) for models with known repetition issues. |
| 255 | + |
| 256 | +**Pass rate is 0% on all benchmarks**: |
| 257 | +1. Read solution.ail from a result file — does it look like syntactically-valid AILANG? |
| 258 | +2. If no — it's a model capability ceiling. The prompt is being honored, the model just isn't good enough yet. Try a stronger model (qwen3-coder:30b, or wait for AILANG-finetuned variants). |
| 259 | +3. If solution.ail looks reasonable but the harness reports compile_error — check `eval_mode` in the result JSON. If `standard` not `agent`, the harness ran in standard mode (probably forgot `-agent` flag — should be auto-injected by `make eval-smoke` per [eval.mk fix](commits/07cdcbbd)). |
| 260 | + |
| 261 | +**MCP server unreachable**: |
| 262 | +1. `curl -sf https://mcp.ailang.sunholo.com/mcp/ -X POST -H "Content-Type: application/json" -H "Accept: application/json, text/event-stream" -d '{"jsonrpc":"2.0","method":"initialize","id":1,"params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"probe","version":"0"}}}'` |
| 263 | +2. If down, agent falls back to CLI tools (`ailang docs`, `ailang examples`) — slightly worse but functional. |
| 264 | +3. Cloud Run cold-start takes 1-5s; `warmup_rig.sh` pre-warms it. |
| 265 | + |
| 266 | +**ollama VRAM is way bigger than expected**: |
| 267 | +1. Check `OLLAMA_NUM_PARALLEL` env var — should be `1` for our rig. Higher values multiply KV-cache by N. |
| 268 | +2. Check `OLLAMA_KV_CACHE_TYPE` — should be `q8_0` to halve KV memory footprint. |
| 269 | +3. Check `OLLAMA_FLASH_ATTENTION` — required for q8 KV to actually take effect. |
| 270 | + |
| 271 | +## Why we made each choice (one-liners) |
| 272 | + |
| 273 | +- **`-parallel 1`** — single-GPU rig; ollama serializes anyway; oversubscription causes 15min TTFT |
| 274 | +- **`OLLAMA_KEEP_ALIVE=-1`** — keeps the ~100k-token opencode framework prefix in KV cache across trials |
| 275 | +- **`OLLAMA_NUM_PARALLEL=1`** — per-slot KV-cache pre-allocation at 110k context = 6 GB per slot; we want one |
| 276 | +- **`OLLAMA_KV_CACHE_TYPE=q8_0`** — halves KV memory (~25 GB → ~12 GB) for ~5% gen slowdown |
| 277 | +- **Modelfile `repeat_penalty=1.1`** — prevents repetition collapses; kept ≤1.1 because >1.1 hurts code (variable names legitimately repeat) |
| 278 | +- **opencode `temperature=0.5`** — Gemma 4 26b MoE collapses at the Google default 1.0; 0.5 is the agentic-loop middle ground |
| 279 | +- **`max_tokens=4096`** — hard cap per LLM response (matches Modelfile num_predict); bounds worst-case wall time at ~68s |
| 280 | +- **`options.name="Ollama"`** — sst/opencode#971 trap; without this, all per-model options are silently dropped |
| 281 | +- **`~/.local/bin/ailang` symlink** — opencode's child shell PATH excludes `~/go/bin`; symlink makes ailang visible to the bash tool |
| 282 | +- **bash denylist `find /*`** — prevents pathological filesystem exploration even with `--dangerously-skip-permissions` |
| 283 | +- **AILANG MCP wiring** — gives the agent first-class structured tools for syntax discovery, better than parsing prose CLI output |
| 284 | + |
| 285 | +## Acknowledged limits we don't yet have a fix for |
| 286 | + |
| 287 | +- **gemma4:26b doesn't converge on fizzbuzz** within a 5-min single-trial budget. Code is structurally close (recursive, uses map, has the right shape) but has small API errors each iteration. Model-capability ceiling. Try qwen3-coder:30b for direct comparison. |
| 288 | +- **Per-turn latency variance is real**. A "cold" session with a fresh framework prefix prefill is ~30-90s; a "warm" session with cache hits is sub-second. The first benchmark of a rotation pays the cold cost. |
| 289 | +- **opencode session-end is not always clean**. Sessions that hit token limits or agent iteration caps may not exit immediately. The `agent-timeout` wall-clock cap is the final safety net. |
0 commit comments