Skip to content

Commit ec0c902

Browse files
fix(local-models): one output budget per model, and stop pinning num_ctx below the prompt
Follow-up to the thinking-budget fix. The premise this was filed on was wrong in two of three places, so it was measured instead of assumed: harness | assumed | measured | how pi | 16384 | 16384 ok | pi's default; its config carried no maxTokens opencode | 4096 | ~32000 | banked sessions: qwen3.6 output to 31,703 and | | | `length` at exactly 32,000 — the per-model | | | max_tokens in opencode.jsonc never reaches the | | | wire (anomalyco/opencode#971 class) motoko | 8192 | 32768 | motoko-local-* declare 32768; the 8192 belonged | | | to the pi-/opencode- rows So models.yml disagreed with ITSELF: 32768 on motoko-local-*, 8192 on pi-/opencode-*, same model, same wire. Aligned at 32768 — the value the lane with the most banked local data already used, so motoko does not move, opencode moves ~2%, and only pi actually changes (16384 -> 32768). The feared baseline disruption was mostly an artifact of the wrong starting numbers. Not raised further: qwen3.6 hits `length` at ~32k about 0.5% of the time, but those are runaways, not normal turns, and on a time-boxed rig a bigger ceiling mostly buys longer runaways. MaxTokensPerBench covers the cumulative case. gemma4:26b unchanged (not established as a heavy reasoner in our data); gemma4:26b-ailang's 4096 is genuinely Modelfile-justified and untouched — the qwen Modelfiles have no num_predict at all, so that rationale never applied. Found en route and fixed here because it makes the above coherent: both option maps in internal/ai/ollama hardcoded num_ctx 8192 ("Reasonable context window", 2026-05-05, before the local rotation existed). num_ctx bounds prompt AND generation together, and the harness's agentic prompts for these models measure 28k-44k tokens — so the ceiling sat below the prompt and ollama context-shifted, while the same map asked for up to 32768 tokens of generation inside that 8192. Now omitted by default so ollama sizes from the model (measured: 262144), which is what the /v1 lanes already get. AILANG_OLLAMA_NUM_CTX pins it for VRAM. Scope, so this is not over-read: motoko's TOOL-CALLING turns never came through here (they route to /v1); what did is Generate, tool-less chat, and the legacy native tool path — including motoko's compaction_ai, i.e. the calls whose whole job is to digest a long context. TestPiModelsConfigMatchesRegistry now covers ollama rows and fails on registry self-disagreement; both new arms were proven to fail on the real historical bugs before landing. New wire-level tests assert the num_ctx bytes with a positive control. BASELINE BOUNDARY 2026-08-13: local qwen rows banked earlier ran on a per-harness budget and are not comparable across harnesses. Co-Authored-By: Claude Opus 5 <[email protected]>
1 parent 1bfe90b commit ec0c902

9 files changed

Lines changed: 395 additions & 80 deletions

File tree

‎.claude/rules/dev-workflow.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ make check-file-sizes # Fails if >800 lines
3939
| `AILANG_TRACE_MAX_SPANS=N` | Per-trace span budget (default 500). Overflow emits `trace.truncated` rollup |
4040
| `AILANG_NO_TRACE=1` | Back-compat alias for `AILANG_TRACE=off` |
4141
| `AILANG_EVAL_MAX_RSS=8G` | Eval memory cap per generated-code run (process-group RSS, default 8G, `off` disables). Breach → tree killed, banked as `resource_limit` |
42+
| `AILANG_OLLAMA_NUM_CTX=N` | Pin ollama's `num_ctx`. **Unset (default) = send no `num_ctx`**, so ollama sizes context from the model (measured 2026-08-13: 262144 for qwen3.6:35b-a3b-mxfp8) — matching the `/v1` lanes pi and opencode already use. Both option maps in `internal/ai/ollama` previously hardcoded **8192**, below the 28k–44k-token prompts the eval harness sends these models. Affects the non-tool paths only (`Generate`, tool-less chat, and the legacy native tool path incl. motoko's `compaction_ai`); motoko's tool-calling turns route via `/v1`, where `num_ctx` is not expressible. Raise/lower only for VRAM: the KV cache scales with it |
4243
| `AILANG_OLLAMA_V1_STREAM=1` | Opt into the streaming ollama `/v1` path (v0.34.0, default **off**). Exactly `"1"` opts in — flag-off wire bytes are unchanged. Changes what `AILANG_OLLAMA_HTTP_TIMEOUT_SEC` means (see below) |
4344
| `AILANG_OLLAMA_IDLE_TIMEOUT_SEC=120` | Streaming only. Max silence **between** bytes before the stream is failed with a typed `idle-timeout`. Default 120. Unparseable/`<= 0` falls back to the default |
4445
| `AILANG_OLLAMA_TTFT_TIMEOUT_SEC=600` | Streaming only. Max silence **before the first byte** (cold 35B load under GPU contention). Default 600. Unparseable/`<= 0` falls back to the default |

‎changelogs/v0.18-current.md‎

Lines changed: 28 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -41,10 +41,34 @@ costume.
4141
`--thinking` (`executor.Task.ReasoningEffort`), with an unknown level a loud pre-dispatch
4242
error. Unset stays unset: no reasoning control is sent, which is NOT the same as `off`.
4343

44-
Not changed, deliberately: the ollama rows in the pi config. Their ceiling is VRAM-bound and
45-
the three local harnesses disagree today (pi 16384, opencode 4096, motoko 8192 from the
46-
registry). Aligning them moves a live measurement baseline mid-rotation, so it is a decision
47-
rather than a cleanup.
44+
**Local (ollama) models, second pass — the three harnesses are now aligned, and two of the
45+
three numbers we thought we had were wrong.** The first pass deferred this as "VRAM-bound,
46+
moving it moves a baseline". Measured instead of assumed: opencode's per-model
47+
`max_tokens: 4096` is **not reaching the wire** (banked sessions show qwen3.6 outputs to
48+
31,703 tokens and `length` finishes at exactly 32,000), and motoko was on **32768**, not
49+
8192 — the 8192 belonged to the `pi-`/`opencode-` rows. So models.yml disagreed with itself:
50+
32768 on `motoko-local-*` and 8192 on `pi-`/`opencode-*`, same model, same wire.
51+
52+
Aligned at **32768** — the value the lane with the most banked local data already used, so
53+
motoko does not move, opencode moves ~2%, and only pi actually changes (16384 → 32768).
54+
`TestPiModelsConfigMatchesRegistry` now covers ollama rows and fails on registry
55+
self-disagreement. ⚠️ Baseline boundary **2026-08-13**: local qwen rows banked earlier ran
56+
on a per-harness budget and are not comparable across harnesses. `gemma4:26b` is unchanged
57+
(not established as a heavy reasoner in our data); `gemma4:26b-ailang`'s 4096 is genuinely
58+
deliberate — it matches `num_predict` in its Modelfile — and is untouched.
59+
60+
- **`num_ctx` was pinned to 8192 on the AILANG ollama path, below the prompts we send.**
61+
Both option maps in `internal/ai/ollama` hardcoded it ("Reasonable context window",
62+
2026-05-05, written before the local-model rotation existed). `num_ctx` bounds prompt AND
63+
generation together, while the harness's agentic prompts for these models measure
64+
**28k–44k tokens** (banked opencode sessions, first step-finish) — so ollama context-shifted
65+
and the model answered having seen a fraction of the task, while the same map asked for up
66+
to 32768 tokens of generation inside that 8192. Now omitted by default, letting ollama size
67+
context from the model (measured 262144), which is what the `/v1` lanes already get;
68+
`AILANG_OLLAMA_NUM_CTX` pins it when VRAM demands. Scope, so it is not over-read: motoko's
69+
**tool-calling turns never came through here** (they route to `/v1`); what did is
70+
`Generate`, tool-less chat, and the legacy native tool path — including motoko's
71+
`compaction_ai` summaries, i.e. the calls whose whole job is to digest a long context.
4872

4973
### Security
5074

‎internal/ai/ollama/client.go‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -131,8 +131,13 @@ func (c *Client) Generate(ctx context.Context, req *ai.Request) (*ai.Response, e
131131

132132
// Build options
133133
options := map[string]interface{}{
134-
"seed": int64(42), // Deterministic by default
135-
"num_ctx": 8192, // Reasonable context window
134+
"seed": int64(42), // Deterministic by default
135+
}
136+
// num_ctx is omitted by default so ollama sizes context from the model —
137+
// see resolveOllamaNumCtx (step.go) for why the old hardcoded 8192 sat below
138+
// the prompts we actually send.
139+
if n, ok := resolveOllamaNumCtx(); ok {
140+
options["num_ctx"] = n
136141
}
137142

138143
// Set max output tokens if specified

‎internal/ai/ollama/numctx_test.go‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
package ollama
2+
3+
import (
4+
"context"
5+
"io"
6+
"net/http"
7+
"net/http/httptest"
8+
"strings"
9+
"testing"
10+
11+
"github.com/sunholo-data/ailang/internal/ai"
12+
)
13+
14+
// TestResolveOllamaNumCtx_DefaultOmits pins the fix for a config that could not
15+
// work: both option maps in this package hardcoded num_ctx 8192 while the eval
16+
// harness sends these models 28k-44k-token prompts, and asked for up to 32768
17+
// tokens of generation inside that same 8192. Omitting the option lets ollama
18+
// size context from the model (measured 262144 for qwen3.6:35b-a3b-mxfp8), which
19+
// is what the /v1 lanes — opencode and pi — already get.
20+
func TestResolveOllamaNumCtx_DefaultOmits(t *testing.T) {
21+
t.Setenv("AILANG_OLLAMA_NUM_CTX", "")
22+
if n, ok := resolveOllamaNumCtx(); ok {
23+
t.Errorf("default must send NO num_ctx so the server sizes it; got %d", n)
24+
}
25+
}
26+
27+
func TestResolveOllamaNumCtx_EnvPins(t *testing.T) {
28+
t.Setenv("AILANG_OLLAMA_NUM_CTX", "65536")
29+
n, ok := resolveOllamaNumCtx()
30+
if !ok || n != 65536 {
31+
t.Errorf("resolveOllamaNumCtx() = (%d, %v), want (65536, true)", n, ok)
32+
}
33+
}
34+
35+
// A junk or non-positive value must fall back to "let the server decide" rather
36+
// than to some invented number — an unparseable override is not a request for a
37+
// small context.
38+
func TestResolveOllamaNumCtx_RejectsJunk(t *testing.T) {
39+
for _, v := range []string{"0", "-1", "lots", "8192.5"} {
40+
t.Setenv("AILANG_OLLAMA_NUM_CTX", v)
41+
if n, ok := resolveOllamaNumCtx(); ok {
42+
t.Errorf("AILANG_OLLAMA_NUM_CTX=%q should be ignored, got %d", v, n)
43+
}
44+
}
45+
}
46+
47+
// captureOllamaBody stands up a fake ollama and returns the request body the
48+
// client actually sent.
49+
func captureOllamaBody(t *testing.T, send func(c *Client) error) string {
50+
t.Helper()
51+
var body string
52+
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
53+
b, _ := io.ReadAll(r.Body)
54+
body = string(b)
55+
w.Header().Set("Content-Type", "application/x-ndjson")
56+
_, _ = w.Write([]byte(`{"message":{"role":"assistant","content":"ok"},"done":true}` + "\n"))
57+
}))
58+
defer srv.Close()
59+
t.Setenv("OLLAMA_HOST", srv.URL)
60+
c, err := NewClient(WithEndpoint(srv.URL))
61+
if err != nil {
62+
t.Fatalf("NewClient: %v", err)
63+
}
64+
if err := send(c); err != nil {
65+
t.Fatalf("send: %v", err)
66+
}
67+
return body
68+
}
69+
70+
// TestOllamaWire_NoNumCtxByDefault asserts on the BYTES, not the resolver: both
71+
// option maps in this package used to pin num_ctx to 8192, which sat below the
72+
// 28k-44k prompts the eval harness sends these models — so the model answered
73+
// having seen a fraction of the task, while the same map asked for up to 32768
74+
// tokens of generation inside that 8192. The wire must now carry no num_ctx, so
75+
// ollama sizes context from the model itself.
76+
func TestOllamaWire_NoNumCtxByDefault(t *testing.T) {
77+
t.Setenv("AILANG_OLLAMA_NUM_CTX", "")
78+
79+
t.Run("generate path", func(t *testing.T) {
80+
body := captureOllamaBody(t, func(c *Client) error {
81+
_, err := c.Generate(context.Background(), &ai.Request{Model: "qwen3.5", UserPrompt: "hi"})
82+
return err
83+
})
84+
if strings.Contains(body, "num_ctx") {
85+
t.Errorf("Generate must not pin num_ctx; body: %s", body)
86+
}
87+
})
88+
89+
t.Run("native chat path", func(t *testing.T) {
90+
t.Setenv("AILANG_OLLAMA_NATIVE_TOOLS", "1")
91+
body := captureOllamaBody(t, func(c *Client) error {
92+
_, err := c.Step(context.Background(), &ai.Request{
93+
Model: "qwen3.5",
94+
Messages: []ai.Message{{Role: "user", Content: "hi"}},
95+
Tools: []ai.ToolSchema{{Name: "noop", Description: "n", Parameters: `{"type":"object"}`}},
96+
})
97+
return err
98+
})
99+
if strings.Contains(body, "num_ctx") {
100+
t.Errorf("native /api/chat must not pin num_ctx; body: %s", body)
101+
}
102+
})
103+
}
104+
105+
// The override still has to reach the wire, or it is not a control.
106+
func TestOllamaWire_NumCtxOverrideReachesWire(t *testing.T) {
107+
t.Setenv("AILANG_OLLAMA_NUM_CTX", "65536")
108+
body := captureOllamaBody(t, func(c *Client) error {
109+
_, err := c.Generate(context.Background(), &ai.Request{Model: "qwen3.5", UserPrompt: "hi"})
110+
return err
111+
})
112+
if !strings.Contains(body, `"num_ctx":65536`) {
113+
t.Errorf("AILANG_OLLAMA_NUM_CTX=65536 must appear on the wire; body: %s", body)
114+
}
115+
}

‎internal/ai/ollama/step.go‎

Lines changed: 41 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,43 @@ const defaultOllamaMaxTokens = 16384
9595
// 2. the caller's reqMax when it already meets the floor;
9696
// 3. the floor (defaultOllamaMaxTokens) — so an unset/small default (e.g. motoko's
9797
// std/ai 4096) can't truncate a reasoning model mid-thought.
98+
//
99+
// resolveOllamaNumCtx reports the num_ctx to send, and whether to send one at all.
100+
//
101+
// Default: DON'T send it. Ollama then sizes the context from the model itself —
102+
// measured 2026-08-13, `ollama ps` reports CONTEXT 262144 for
103+
// qwen3.6:35b-a3b-mxfp8 — which is exactly what the opencode and pi lanes get,
104+
// since they reach ollama over /v1 where num_ctx is not expressible. Those are
105+
// the two local lanes with the healthiest tool-emission record.
106+
//
107+
// Both option maps in this package used to hardcode 8192 ("Reasonable context
108+
// window", 2026-05-05, M-AI-TOOL-LOOP M4 — written before the local-model
109+
// rotation existed). num_ctx bounds prompt AND generation together, while the
110+
// eval harness's agentic prompts for these models measure 28k-44k tokens (banked
111+
// opencode sessions, first step-finish, 21-day window). So the ceiling sat below
112+
// the prompt: ollama context-shifts, and the model answers having seen a fraction
113+
// of the task. The same maps then asked for num_predict up to 32768 — four times
114+
// more generation than the whole context allowed. That combination cannot work,
115+
// whichever number is right.
116+
//
117+
// Scope, so this is not over-read: motoko's TOOL-CALLING turns never came through
118+
// here (Step routes those to /v1 unless AILANG_OLLAMA_NATIVE_TOOLS=1). What did:
119+
// single-shot Generate calls, multi-turn chat without tools, and the legacy
120+
// native tool path — which includes motoko's compaction_ai summaries, i.e. the
121+
// calls whose entire job is to digest a long context.
122+
//
123+
// AILANG_OLLAMA_NUM_CTX pins a value when VRAM demands one — the KV cache scales
124+
// with it (the rig runs OLLAMA_KV_CACHE_TYPE=q8_0 + flash attention, which is what
125+
// makes the full context affordable at all).
126+
func resolveOllamaNumCtx() (int, bool) {
127+
if v := os.Getenv("AILANG_OLLAMA_NUM_CTX"); v != "" {
128+
if n, err := strconv.Atoi(v); err == nil && n > 0 {
129+
return n, true
130+
}
131+
}
132+
return 0, false
133+
}
134+
98135
func resolveOllamaMaxTokens(reqMax int) int {
99136
if v := os.Getenv("AILANG_OLLAMA_MAX_TOKENS"); v != "" {
100137
if n, err := strconv.Atoi(v); err == nil && n > 0 {
@@ -374,8 +411,10 @@ func (c *Client) Step(ctx context.Context, req *ai.Request) (*ai.Response, error
374411
}
375412

376413
options := map[string]interface{}{
377-
"seed": int64(42),
378-
"num_ctx": 8192,
414+
"seed": int64(42),
415+
}
416+
if n, ok := resolveOllamaNumCtx(); ok {
417+
options["num_ctx"] = n
379418
}
380419
if req.MaxTokens > 0 {
381420
options["num_predict"] = req.MaxTokens

‎internal/eval_harness/models.yml‎

Lines changed: 24 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1952,7 +1952,22 @@ models:
19521952
agent_cli: "opencode"
19531953
agent_model_name: "ollama/qwen3.5:35b-a3b-mxfp8"
19541954
model_family: "qwen-3-5"
1955-
max_output_tokens: 8192
1955+
# ALIGNED 8192 -> 32768 (2026-08-13). ONE output budget per MODEL, across
1956+
# every harness. The registry itself declared two different values for this
1957+
# same model — 32768 on the motoko-local-* rows, 8192 on the pi-/opencode-
1958+
# rows — so any pi-vs-opencode-vs-motoko comparison was partly measuring the
1959+
# harness's budget rather than the model. qwen3.x emit thousands of <think>
1960+
# tokens BEFORE the tool call, so a small budget truncates into a 0-tool-call
1961+
# no-op (wire-proven 2026-06-19; see defaultOllamaMaxTokens). All three
1962+
# harnesses reach ollama over /v1, where the server sizes context from the
1963+
# model itself — measured 262144 on 2026-08-13 — so 32768 of generation fits
1964+
# beside a 28-44k prompt with room to spare. 32768 = the motoko-local value,
1965+
# i.e. the lane with the most banked local data does NOT move.
1966+
# ⚠️ BASELINE BOUNDARY: local qwen rows banked BEFORE 2026-08-13 ran on a
1967+
# per-harness budget (pi 16384 via its own default, opencode ~32000, motoko
1968+
# 32768) and are not comparable across harnesses. See
1969+
# tools/pi-extensions/README.md.
1970+
max_output_tokens: 32768
19561971
ttft_timeout: 900
19571972
generation_timeout: 1200
19581973
budgets:
@@ -1989,7 +2004,8 @@ models:
19892004
agent_cli: "pi"
19902005
agent_model_name: "ollama/qwen3.5:35b-a3b-mxfp8"
19912006
model_family: "qwen-3-5"
1992-
max_output_tokens: 8192
2007+
# Aligned 2026-08-13 — one budget per model; rationale on opencode-qwen3-5-35b-a3b-mxfp8.
2008+
max_output_tokens: 32768
19932009
ttft_timeout: 900
19942010
generation_timeout: 1200
19952011
budgets:
@@ -2019,7 +2035,8 @@ models:
20192035
agent_cli: "opencode"
20202036
agent_model_name: "ollama/qwen3.6:35b-a3b-mxfp8"
20212037
model_family: "qwen-3-6"
2022-
max_output_tokens: 8192
2038+
# Aligned 2026-08-13 — one budget per model; rationale on opencode-qwen3-5-35b-a3b-mxfp8.
2039+
max_output_tokens: 32768
20232040
ttft_timeout: 900
20242041
generation_timeout: 1200
20252042
budgets:
@@ -2042,7 +2059,8 @@ models:
20422059
agent_cli: "pi"
20432060
agent_model_name: "ollama/qwen3.6:35b-a3b-mxfp8"
20442061
model_family: "qwen-3-6"
2045-
max_output_tokens: 8192
2062+
# Aligned 2026-08-13 — one budget per model; rationale on opencode-qwen3-5-35b-a3b-mxfp8.
2063+
max_output_tokens: 32768
20462064
ttft_timeout: 900
20472065
generation_timeout: 1200
20482066
budgets:
@@ -2067,7 +2085,8 @@ models:
20672085
agent_cli: "opencode"
20682086
agent_model_name: "ollama/qwen3.5:35b-a3b-nvfp4"
20692087
model_family: "qwen-3-5"
2070-
max_output_tokens: 8192
2088+
# Aligned 2026-08-13 — one budget per model; rationale on opencode-qwen3-5-35b-a3b-mxfp8.
2089+
max_output_tokens: 32768
20712090
ttft_timeout: 900
20722091
generation_timeout: 1200
20732092
budgets:

0 commit comments

Comments
 (0)