Skip to content

Commit 84f20a5

Browse files
docs(skill): rig operations runbook — consolidated lessons from 2026-05-22/23
Pulls together everything we learned today into one document so a future person (or future Claude) onboarding a new model doesn't have to re-discover any of it from scattered commit messages. Sections: - TL;DR setup checklist (9-step add-a-new-model workflow) - The mental model (eval-suite → opencode → ollama → variant; 4 config surfaces) - 8 hard-won lessons: 1. opencode child-shell PATH excludes ~/go/bin (auto-fixed by `make install`) 2. anomalyco/opencode#971 — options.name="Ollama" required or options silently dropped 3. /v1/chat/completions only honors a subset of sampling params (Modelfile for the rest) 4. gemma4:26b token-repetition collapse at default sampling (Modelfile + opencode options) 5. bash denylist works even with --dangerously-skip-permissions (deny rules always deny) 6. -agent-parallel was dead code (removed); use -parallel 7. opencode auto-compaction triggers at max ctx, doesn't help runaway responses 8. AILANG MCP wiring (and the verified-live MCP tool usage by gemma4:26b) - Healthy-rig signatures (per-call duration distributions, chains-DB token patterns) - Pre-flight checklist before a long rotation - Per-surface change matrix when adding a model - Diagnostic flowchart (hangs / runaway outputs / 0% pass rate / MCP down / VRAM bloat) - One-line justification for each design choice - Acknowledged remaining limits (gemma4:26b doesn't converge on fizzbuzz in 5min) SKILL.md now flags the runbook as 'READ FIRST when adding a new model' in the Resources section. Co-Authored-By: Claude Opus 4.7 <[email protected]>
1 parent a1fbdb0 commit 84f20a5

2 files changed

Lines changed: 290 additions & 0 deletions

File tree

‎.claude/skills/local-ollama-eval/SKILL.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -335,6 +335,7 @@ A proper `ailang eval-rotation` daemon (sketch in the M-EVAL-LOCAL-OLLAMA design
335335

336336
## Resources
337337

338+
- **Operations runbook (READ FIRST when adding a new model)**: [`resources/rig_operations_runbook.md`](resources/rig_operations_runbook.md) — consolidated lessons from the 2026-05-22/23 debugging sessions. Covers the four configuration surfaces (Modelfile, opencode.jsonc, models.yml, ollama plist), 8 hard-won lessons (PATH symlink, options.name trap, sampling collapse, bash denylist, etc.), the diagnostic flowchart, and a pre-flight checklist before any long rotation.
338339
- **Candidate models list**: [`resources/candidate_models.md`](resources/candidate_models.md) — what to pull next
339340
- **opencode config sample**: [`resources/opencode_jsonc_example.txt`](resources/opencode_jsonc_example.txt)
340341
- **Troubleshooting**: [`resources/troubleshooting.md`](resources/troubleshooting.md)
Lines changed: 289 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,289 @@
1+
# Running Local Ollama Models in the AILANG Eval Rig — Operations Runbook
2+
3+
Hard-earned lessons from the 2026-05-22/23 debugging sessions. Consolidates
4+
findings that are currently scattered across commits + design docs + chat
5+
history into one document you can hand to a new person (or your future
6+
self) for onboarding a new model.
7+
8+
## TL;DR setup checklist (new model, e.g. qwen3-coder:30b)
9+
10+
```bash
11+
# 1. Pull the base model
12+
ollama pull qwen3-coder:30b
13+
14+
# 2. Build an AILANG-tuned variant (sampling fixes + output cap)
15+
ollama create qwen3-coder:30b-ailang \
16+
-f tools/ollama/qwen3-coder-30b-ailang.modelfile # adapt from gemma4-26b-ailang.modelfile
17+
18+
# 3. Register in opencode.jsonc with options.name="Ollama" (CRITICAL — see #971 below)
19+
# and per-model options block (temperature, max_tokens, etc.)
20+
$EDITOR ~/.config/opencode/opencode.jsonc
21+
22+
# 4. Add models.yml entry routing to the variant
23+
$EDITOR internal/eval_harness/models.yml
24+
25+
# 5. (Re)install ailang — auto-symlinks to ~/.local/bin for opencode-compat
26+
make install
27+
28+
# 6. Sanity-check: opencode should see ailang AND your variant
29+
opencode mcp list # ailang-docs should be ✓ connected
30+
opencode models | grep ollama
31+
opencode run --format json --dangerously-skip-permissions \
32+
--model "ollama/qwen3-coder:30b-ailang" "Run: ailang --version. Output DONE."
33+
34+
# 7. Warm up the rig
35+
.claude/skills/local-ollama-eval/scripts/warmup_rig.sh ollama/qwen3-coder:30b-ailang
36+
37+
# 8. Single-benchmark smoke
38+
make eval-smoke MODELS=opencode-qwen3-coder-30b-ailang \
39+
EXTRA="-trials 1 -parallel 1 -agent-timeout 1200 -langs ailang \
40+
-benchmarks fizzbuzz -output /tmp/quick_check"
41+
42+
# 9. If it passes, full smoke tier with N=3 trials
43+
make eval-smoke MODELS=opencode-qwen3-coder-30b-ailang \
44+
EXTRA="-trials 3 -parallel 1 -agent-timeout 1200 -langs ailang \
45+
-output eval_results/rotation/$(date +%Y-%m-%d)/$(date +%H%M)_<model>_smoke_n3"
46+
```
47+
48+
## The mental model
49+
50+
```
51+
eval-suite (Go)
52+
│
53+
│ spawns ONE opencode subprocess per benchmark trial (parallel = N from -parallel flag)
54+
▼
55+
opencode CLI (Node)
56+
│
57+
│ reads ~/.config/opencode/opencode.jsonc (provider config, options, MCP servers, bash permissions)
58+
│ reads ~/.config/opencode/agents/*.md (per-agent system prompts; we use default "build" agent)
59+
│ spawns tool subprocesses (bash, read, write, edit) — sanitized PATH, NOT inheriting ~/go/bin
60+
│ sends OpenAI-compat /v1/chat/completions to provider
61+
▼
62+
ollama serve (Go) — process under launchd
63+
│
64+
│ loads model into VRAM (kept resident via OLLAMA_KEEP_ALIVE=-1)
65+
│ serves one request at a time (OLLAMA_NUM_PARALLEL=1)
66+
│ Modelfile params apply: top_k, repeat_penalty, min_p, num_predict
67+
▼
68+
gemma4:26b-ailang (the variant we built)
69+
```
70+
71+
Four configuration surfaces, each in a different place:
72+
73+
| Surface | What it controls | Lives at |
74+
|---|---|---|
75+
| Modelfile | ollama-native params (top_k, repeat_penalty, num_predict, min_p) | `tools/ollama/<model>-ailang.modelfile` |
76+
| opencode.jsonc model options | OpenAI-compat params (temperature, top_p, frequency_penalty, max_tokens) + MCP wiring + bash perms | `~/.config/opencode/opencode.jsonc` |
77+
| models.yml | harness routing (api_name, agent_cli, timeouts, budgets) | `internal/eval_harness/models.yml` |
78+
| ollama plist env | server-wide config (NUM_PARALLEL, KV_CACHE_TYPE, FLASH_ATTENTION, KEEP_ALIVE) | `~/Library/LaunchAgents/dev.ollama.serve.plist` |
79+
80+
A mistake at any surface is silent in the others. The most painful traps below.
81+
82+
## Hard-won lessons — DO NOT re-discover these
83+
84+
### 1. opencode's child-shell PATH does NOT include `~/go/bin`
85+
86+
**Symptom**: bash tool fails with `command not found: ailang`. Model falls into `find / -name ailang`, `ls -R /`, or guessing `/usr/local/bin/ailang` / `/opt/homebrew/bin/ailang` paths. Sessions hang for many minutes per pathological command.
87+
88+
**Why**: opencode launches bash subprocesses with a sanitized PATH (`~/.local/bin:/usr/local/bin:/usr/bin:/bin:...:/opt/homebrew/bin`). Setting PATH in the executor env doesn't help — opencode overrides.
89+
90+
**Fix**: `make install` now auto-symlinks `~/.local/bin/ailang -> ~/go/bin/ailang`. If you suspect the symlink is missing, `ls -la ~/.local/bin/ailang` to check. If gone, just `make install` again.
91+
92+
### 2. opencode silently drops model options without `options.name`
93+
94+
**Symptom**: opencode.jsonc has `provider.ollama.models["foo"].options = {temperature: 0.5, ...}` but generation behaves like default sampling. Temperature, max_tokens, frequency_penalty all ignored.
95+
96+
**Why**: [sst/opencode#971](https://github.com/sst/opencode/issues/971) — `@ai-sdk/openai-compatible` provider requires `options.name` to be set OR all per-model options are dropped. Easy trap because the option block looks correct.
97+
98+
**Fix**:
99+
```jsonc
100+
"provider": {
101+
"ollama": {
102+
"options": {
103+
"name": "Ollama", // <-- THIS is required
104+
"baseURL": "http://localhost:11434/v1"
105+
},
106+
...
107+
}
108+
}
109+
```
110+
111+
**Verify**: run the same prompt twice through opencode at `temperature: 0.5`. If output is identical, options pass through. If wildly different, they're being dropped.
112+
113+
### 3. ollama's `/v1/chat/completions` only honors a subset of sampling params
114+
115+
OpenAI-compat surface passes: `temperature`, `top_p`, `frequency_penalty`, `presence_penalty`, `max_tokens`, `seed`, `stop`, `stream`.
116+
117+
Silently drops: `top_k`, `repeat_penalty`, `min_p`, `num_predict`, `repeat_last_n`.
118+
119+
**Fix**: put dropped params in a Modelfile and `ollama create` a variant. The Modelfile params apply at model-load time and aren't affected by request-level options.
120+
121+
### 4. gemma4:26b is prone to token-repetition collapse at default sampling
122+
123+
**Symptom**: a single LLM response generates 30k+ tokens of repeated garbage (e.g., `v_h_y_f_d_v_h_y_f_d_...`). Per-call wall time blows up to 9+ minutes.
124+
125+
**Why**: Google's default `temperature=1.0` with no repetition penalty is a known bug ([google-deepmind/gemma#622](https://github.com/google-deepmind/gemma/issues/622)) on the 26b MoE variant. Affects tool-argument generation particularly.
126+
127+
**Fix**:
128+
- Modelfile: `repeat_penalty 1.1`, `min_p 0.05`, `num_predict 4096` (hard cap)
129+
- opencode.jsonc options: `temperature 0.5`, `frequency_penalty 0.3`, `max_tokens 4096`
130+
131+
Trade-off: lower temperature reduces peak coding ability for Gemma 4 (per Unsloth's HF #21 thread) but eliminates collapses. We trade peak for reliability.
132+
133+
### 5. Bash permission denylist works EVEN with `--dangerously-skip-permissions`
134+
135+
Deny rules always deny, regardless of the dangerously flag. So you can run unattended (no permission prompts) AND still block pathological commands.
136+
137+
```jsonc
138+
"permission": {
139+
"bash": {
140+
"*": "allow",
141+
"find / *": "deny",
142+
"find /System*": "deny",
143+
"ls -R /*": "deny", // trailing * catches pipelines like `ls -R / | grep X`
144+
"grep -r /*": "deny",
145+
"rm -rf /*": "deny",
146+
"sudo *": "deny"
147+
}
148+
}
149+
```
150+
151+
**Wildcard gotcha**: `"ls -R /": "deny"` matches the literal but NOT `ls -R / | grep ailang` (pipeline form). The trailing `*` in `"ls -R /*"` is required.
152+
153+
### 6. `-agent-parallel` was a DEAD flag (removed 2026-05-23)
154+
155+
The harness used to advertise `-agent-parallel` as a concurrency knob; it was decoration only. The real dispatch semaphore is `-parallel` (default 10). Mismatching these silently oversubscribes ollama's NUM_PARALLEL=1 queue, causing 15-minute TTFT timeouts.
156+
157+
The flag has been deleted as of [commit 98c0c408](commits/98c0c408). The status banner now prints `Dispatch parallelism: N (-parallel flag)` to unambiguously identify the correct knob.
158+
159+
### 7. opencode default agent has compaction baked in
160+
161+
opencode's "build" agent auto-compacts when context is "full" — meaning at the model's max context window (e.g. 262k for gemma4:26b). Our typical agentic sessions hit ~36k input tokens stably (verified across 56 prior sessions in chains DB), well below the threshold. Compaction does NOT save us when individual responses go runaway (the 32k repetition case). Output capping via Modelfile `num_predict` is the right control there.
162+
163+
### 8. AILANG MCP server vs CLI fallback
164+
165+
The remote MCP at `mcp.ailang.sunholo.com` exposes 14+ structured tools (`ailang.prompt_get`, `ailang.stdlib_search`, `ailang.example_for_concept`, etc.). With `options.name="Ollama"` set and the MCP block in opencode.jsonc, **opencode picks up MCP tools automatically and adds them to the agent's tool catalog**.
166+
167+
Verified live: with the v0.10.0-slim seed prompt that names the MCP tools, gemma4:26b uses them — observed `ailang-docs_stdlib_search` calls with queries like `"if"` and `"for"`. The agent is genuinely discovering syntax dynamically.
168+
169+
If MCP is unreachable, opencode logs the error and continues; agent falls back to shelling out to `ailang docs` / `ailang examples` CLI.
170+
171+
## Healthy-rig signatures (what "working" looks like)
172+
173+
Watch the ollama log (`/tmp/ollama-serve-launchd.log`) during a run:
174+
175+
| Pattern | Interpretation |
176+
|---|---|
177+
| Per-call durations 1s — 60s, most under 30s | ✓ Normal agentic loop |
178+
| Occasional 60-120s call (warm cache miss after gap) | ✓ Normal — model loading or large prefill |
179+
| Sub-second call (`1.7s`, `2.3s`) | ✓ Best — cache hit, no marginal prefill |
180+
| Same 200 status, slowly-growing durations (5s → 10s → 30s → 5min) | ✗ Queue oversubscription — check `-parallel` |
181+
| 200 status with `15m0s` duration | ✗ TTFT timeout — model didn't start producing for 15min |
182+
| 500 status | ✗ Real error — check ollama server log for OOM or model unload |
183+
184+
Watch chains DB (`observatory.db`):
185+
186+
```bash
187+
sqlite3 -readonly ~/.ailang/state/observatory.db <<'SQL'
188+
SELECT
189+
duration_ms,
190+
json_extract(attributes, '$.opencode.input_tokens') AS in_tok,
191+
json_extract(attributes, '$.opencode.output_tokens') AS out_tok
192+
FROM spans WHERE name = 'opencode.step'
193+
ORDER BY start_time DESC LIMIT 10;
194+
SQL
195+
```
196+
197+
| Pattern | Interpretation |
198+
|---|---|
199+
| input_tokens stable around 30-40k across steps | ✓ opencode compaction working |
200+
| output_tokens < 4096 (the Modelfile cap) per step | ✓ num_predict cap honored |
201+
| output_tokens jumping to 30k+ in one step | ✗ Sampling collapse — check Modelfile + opencode.jsonc options |
202+
| input_tokens growing each step (40k → 60k → 80k) | ⚠ History accumulating; eventual prefill ceiling |
203+
204+
## Pre-flight checklist before a long rotation
205+
206+
```bash
207+
# 1. Rig precondition (verify_setup.sh wraps these)
208+
curl -sf http://localhost:11434/api/tags >/dev/null # ollama UP
209+
curl -sf http://localhost:1957/health >/dev/null # ailang server UP
210+
launchctl list | grep dev.ailang.rig-watchdog # watchdog registered
211+
ollama list | grep <your-model> # variant exists
212+
213+
# 2. Verify env config is sane
214+
ollama show <your-model> --modelfile | grep -E "PARAMETER" # sampling params baked in
215+
opencode mcp list | grep "ailang-docs.*connected" # MCP up
216+
opencode models | grep <your-model> # opencode sees model
217+
218+
# 3. Warmup
219+
.claude/skills/local-ollama-eval/scripts/warmup_rig.sh ollama/<your-model>
220+
221+
# 4. Single-trial sanity (1 benchmark, agent mode)
222+
make eval-smoke MODELS=<your-model> \
223+
EXTRA="-trials 1 -parallel 1 -agent-timeout 1200 -langs ailang \
224+
-benchmarks fizzbuzz -output /tmp/preflight_check"
225+
# Expect: opencode subprocess produces a solution.ail, ailang check / run ran, exits in <10 min
226+
227+
# 5. Full rotation
228+
make eval-smoke MODELS=<your-model> \
229+
EXTRA="-trials 3 -parallel 1 -agent-timeout 1200 -langs ailang \
230+
-output eval_results/rotation/$(date +%Y-%m-%d)/$(date +%H%M)_<model>_smoke_n3"
231+
```
232+
233+
## Adding a model — checklist of all surfaces
234+
235+
When you add a new model to the rig, you must touch all four configuration surfaces. Missing any one is silent failure.
236+
237+
| Surface | What | Where | Failure mode if missing |
238+
|---|---|---|---|
239+
| 1 | Modelfile (sampling) | `tools/ollama/<model>-ailang.modelfile` | runs with bad defaults, may collapse |
240+
| 2 | Build variant | `ollama create <model>-ailang -f <modelfile>` | model not available |
241+
| 3 | opencode provider model | `~/.config/opencode/opencode.jsonc` `provider.ollama.models["<model>-ailang"]` with options block | opencode doesn't know about it |
242+
| 4 | models.yml entry | `internal/eval_harness/models.yml` with `agent_cli: "opencode"`, `agent_model_name: "ollama/<model>-ailang"` | harness can't route |
243+
244+
## When something goes wrong — diagnostic flowchart
245+
246+
**Sessions hang for many minutes**:
247+
1. Check `~/.local/bin/ailang` symlink exists. If not: `make install`.
248+
2. Check opencode bash logs for `find /`, `ls -R /`, or thrashing on PATH discovery. Update bash denylist.
249+
3. Check `/tmp/ollama-serve-launchd.log` for individual call durations. If 15m timeouts, check `-parallel` value (must be 1 to match server NUM_PARALLEL=1).
250+
251+
**Sessions complete but produce 30k+ token outputs**:
252+
1. Check Modelfile has `repeat_penalty` and `num_predict` baked in: `ollama show <model> --modelfile`.
253+
2. Check opencode.jsonc has `options.name="Ollama"` set (sst/opencode#971).
254+
3. Check `temperature` is not 1.0 (Google default) for models with known repetition issues.
255+
256+
**Pass rate is 0% on all benchmarks**:
257+
1. Read solution.ail from a result file — does it look like syntactically-valid AILANG?
258+
2. If no — it's a model capability ceiling. The prompt is being honored, the model just isn't good enough yet. Try a stronger model (qwen3-coder:30b, or wait for AILANG-finetuned variants).
259+
3. If solution.ail looks reasonable but the harness reports compile_error — check `eval_mode` in the result JSON. If `standard` not `agent`, the harness ran in standard mode (probably forgot `-agent` flag — should be auto-injected by `make eval-smoke` per [eval.mk fix](commits/07cdcbbd)).
260+
261+
**MCP server unreachable**:
262+
1. `curl -sf https://mcp.ailang.sunholo.com/mcp/ -X POST -H "Content-Type: application/json" -H "Accept: application/json, text/event-stream" -d '{"jsonrpc":"2.0","method":"initialize","id":1,"params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"probe","version":"0"}}}'`
263+
2. If down, agent falls back to CLI tools (`ailang docs`, `ailang examples`) — slightly worse but functional.
264+
3. Cloud Run cold-start takes 1-5s; `warmup_rig.sh` pre-warms it.
265+
266+
**ollama VRAM is way bigger than expected**:
267+
1. Check `OLLAMA_NUM_PARALLEL` env var — should be `1` for our rig. Higher values multiply KV-cache by N.
268+
2. Check `OLLAMA_KV_CACHE_TYPE` — should be `q8_0` to halve KV memory footprint.
269+
3. Check `OLLAMA_FLASH_ATTENTION` — required for q8 KV to actually take effect.
270+
271+
## Why we made each choice (one-liners)
272+
273+
- **`-parallel 1`** — single-GPU rig; ollama serializes anyway; oversubscription causes 15min TTFT
274+
- **`OLLAMA_KEEP_ALIVE=-1`** — keeps the ~100k-token opencode framework prefix in KV cache across trials
275+
- **`OLLAMA_NUM_PARALLEL=1`** — per-slot KV-cache pre-allocation at 110k context = 6 GB per slot; we want one
276+
- **`OLLAMA_KV_CACHE_TYPE=q8_0`** — halves KV memory (~25 GB → ~12 GB) for ~5% gen slowdown
277+
- **Modelfile `repeat_penalty=1.1`** — prevents repetition collapses; kept ≤1.1 because >1.1 hurts code (variable names legitimately repeat)
278+
- **opencode `temperature=0.5`** — Gemma 4 26b MoE collapses at the Google default 1.0; 0.5 is the agentic-loop middle ground
279+
- **`max_tokens=4096`** — hard cap per LLM response (matches Modelfile num_predict); bounds worst-case wall time at ~68s
280+
- **`options.name="Ollama"`** — sst/opencode#971 trap; without this, all per-model options are silently dropped
281+
- **`~/.local/bin/ailang` symlink** — opencode's child shell PATH excludes `~/go/bin`; symlink makes ailang visible to the bash tool
282+
- **bash denylist `find /*`** — prevents pathological filesystem exploration even with `--dangerously-skip-permissions`
283+
- **AILANG MCP wiring** — gives the agent first-class structured tools for syntax discovery, better than parsing prose CLI output
284+
285+
## Acknowledged limits we don't yet have a fix for
286+
287+
- **gemma4:26b doesn't converge on fizzbuzz** within a 5-min single-trial budget. Code is structurally close (recursive, uses map, has the right shape) but has small API errors each iteration. Model-capability ceiling. Try qwen3-coder:30b for direct comparison.
288+
- **Per-turn latency variance is real**. A "cold" session with a fresh framework prefix prefill is ~30-90s; a "warm" session with cache hits is sub-second. The first benchmark of a rotation pays the cold cost.
289+
- **opencode session-end is not always clean**. Sessions that hit token limits or agent iteration caps may not exit immediately. The `agent-timeout` wall-clock cap is the final safety net.

0 commit comments

Comments
 (0)