Skip to content

Commit 29f02c7

Browse files
sampling-config: gemma4:26b-ailang variant + opencode option pass-through
Root cause analysis from the 2026-05-23 sampling-collapse incident: opencode sessions on gemma4:26b were burning 5-15 minutes per call on degenerate token-repetition loops (single 32k-token argument observed in a write tool call: "/var/folders/hd68_v_h_y_f_d_v_h_y_f_d_..." repeated 31,814 chars). Per upstream bug github.com/google-deepmind/gemma#622, this is a known issue in the gemma4 26b MoE variant with Google's default sampling params (temperature=1.0, no repetition penalty). Fix uses a two-layer approach because ollama's /v1/chat/completions endpoint only honors a subset of sampling options: Layer 1: tools/ollama/gemma4-26b-ailang.modelfile (NEW) Bakes in ollama-native params that do NOT pass through /v1: top_k=64 (Google official for Gemma 4) repeat_penalty=1.1, repeat_last_n=256 min_p=0.05 (Unsloth safety floor) num_predict=4096 (hard cap on single-generation output) Build: ollama create gemma4:26b-ailang -f .../gemma4-26b-ailang.modelfile Layer 2: ~/.config/opencode/opencode.jsonc (separately committed) Sets OpenAI-compat names that DO pass through /v1: temperature=0.5 (Gemma 4 agentic-loop middle ground; lower than Google's 1.0 default for reliability) top_p=0.95 (Google official) frequency_penalty=0.3 (OpenAI-compat counterpart to repeat_penalty) max_tokens=4096 CRITICAL: provider.ollama.options.name="Ollama" must be set or per-model options are silently dropped (anomalyco/opencode#971). Easy trap. models.yml: adds opencode-gemma4-26b-ailang entry pointing at the variant. Existing opencode-gemma4-26b entry kept as A/B baseline (Google defaults). Verified live (single-fizzbuzz, 2026-05-23 14:46): - Determinism check ("What is 2+2?" twice) returned exactly "4" both times at 15-16s, confirming options pass through (not just dropped) - No 32k-token repetition loops observed in any opencode.step - Per-call durations: 1.7s - 1m43s (mostly under 30s), vs 552s monsters under old config - Model produced syntactically plausible AILANG (one real scoping bug, no collapse-into-gibberish behavior) Remaining separate issue (NOT sampling-related): opencode session hangs after writing solution.ail in agent mode for both old and new configs. Tracked separately — not blocked by this commit. Refs anomalyco/opencode#971, google-deepmind/gemma#622 Co-Authored-By: Claude Opus 4.7 <[email protected]>
1 parent f6bbb48 commit 29f02c7

2 files changed

Lines changed: 89 additions & 6 deletions

File tree

‎internal/eval_harness/models.yml‎

Lines changed: 49 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1038,7 +1038,7 @@ models:
10381038
opencode-gemma4-26b:
10391039
api_name: "gemma4:26b"
10401040
provider: "ollama"
1041-
description: "Gemma 4 26B (Q4_K_M, dense, 17GB) via opencode+Ollama — eval-grade local model"
1041+
description: "Gemma 4 26B (Q4_K_M, dense, 17GB) via opencode+Ollama — DEFAULT SAMPLING (Google's temp=1.0). Kept for A/B baseline only; production rotations should use opencode-gemma4-26b-ailang below."
10421042
env_var: ""
10431043
agent_cli: "opencode"
10441044
agent_model_name: "ollama/gemma4:26b"
@@ -1054,13 +1054,56 @@ models:
10541054
input_per_1k: 0.0
10551055
output_per_1k: 0.0
10561056
notes: |
1057-
Gemma 4 26B (25.8B params, Q4_K_M dense, 17GB) via Ollama.
1058-
Fits Macs with ≥32GB RAM. 262k context, vision + tools + thinking capabilities.
1059-
Primary benchmark model for open-source / offline agent eval.
1057+
Gemma 4 26B (25.8B params, Q4_K_M dense, 17GB) via Ollama with Google's
1058+
DEFAULT sampling parameters (temperature=1.0, no repetition penalty).
1059+
Empirically demonstrated 2026-05-23 to fall into token-collapse loops
1060+
under agentic tool-call workloads (32k-token argument repetition incident,
1061+
upstream bug: github.com/google-deepmind/gemma#622).
1062+
Kept as the baseline for A/B comparison ONLY. For production rotations
1063+
use opencode-gemma4-26b-ailang (below) which pairs a tuned Modelfile
1064+
with opencode sampling options.
10601065
Pull: ollama pull gemma4:26b
10611066
Requires: ollama serve + ~/.config/opencode/opencode.jsonc Ollama provider block.
1062-
NOTE 2026-05-22: api_name updated from "gemma4:26b-a4b-it-q4_K_M" (MoE variant)
1063-
to "gemma4:26b" (dense Q4_K_M) — the default Ollama-library tag.
1067+
1068+
opencode-gemma4-26b-ailang:
1069+
api_name: "gemma4:26b-ailang"
1070+
provider: "ollama"
1071+
description: "Gemma 4 26B with AILANG-tuned sampling (Modelfile + opencode options) — production rotation model as of 2026-05-23"
1072+
env_var: ""
1073+
agent_cli: "opencode"
1074+
agent_model_name: "ollama/gemma4:26b-ailang"
1075+
max_output_tokens: 4096 # belt-and-suspenders cap; primary cap is num_predict in the Modelfile
1076+
ttft_timeout: 900
1077+
generation_timeout: 1200
1078+
budgets:
1079+
hard_timeout_secs: 2400
1080+
pricing:
1081+
input_per_1k: 0.0
1082+
output_per_1k: 0.0
1083+
notes: |
1084+
Variant of gemma4:26b with sampling parameters tuned for agentic AILANG
1085+
code generation, derived from the 2026-05-23 sampling-collapse incident.
1086+
1087+
Modelfile (tools/ollama/gemma4-26b-ailang.modelfile) bakes in ollama-native
1088+
params that do NOT pass through /v1/chat/completions:
1089+
top_k=64 (Google official)
1090+
repeat_penalty=1.1 (community consensus for agentic tool-calls;
1091+
kept ≤1.1 since >1.1 hurts code per smcleod.net guide)
1092+
repeat_last_n=256
1093+
min_p=0.05 (Unsloth-recommended safety floor)
1094+
num_predict=4096 (hard cap on single-generation output)
1095+
1096+
opencode.jsonc model options (must travel via OpenAI-compatible names):
1097+
temperature=0.5 (Gemma 4 agentic-loop middle ground; Google official is
1098+
1.0 but the 26b MoE has documented collapse issues at 1.0)
1099+
top_p=0.95 (Google official)
1100+
frequency_penalty=0.3 (OpenAI-compat counterpart of repeat_penalty)
1101+
max_tokens=4096
1102+
1103+
Build the variant: ollama create gemma4:26b-ailang -f tools/ollama/gemma4-26b-ailang.modelfile
1104+
Verify config: opencode.jsonc must have provider.ollama.options.name="Ollama"
1105+
(sst/opencode#971 — options silently dropped without this)
1106+
A/B against opencode-gemma4-26b (default sampling) to quantify the improvement.
10641107
10651108
# =============================================================================
10661109
# OpenRouter-routed OS via opencode harness (M-AI-EFFECT-MODES, May 2026)
Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Gemma 4 26B — AILANG-eval-tuned variant
2+
#
3+
# Why this variant exists: Google's default Gemma 4 sampling params
4+
# (temperature=1.0, no repetition penalty) cause occasional token-collapse
5+
# loops in agentic tool-call scenarios. Empirically observed 2026-05-23:
6+
# gemma4:26b emitted the string `v_h_y_f_d_` 32,000 times in a single
7+
# write-tool filePath argument, burning 9 minutes of GPU time. See the
8+
# upstream bug at https://github.com/google-deepmind/gemma/issues/622
9+
# ("Gemma 4 token repetition collapse … affects 31b Dense and 26b MoE").
10+
#
11+
# The OpenAI-compatible /v1/chat/completions endpoint does NOT pass through
12+
# `top_k`, `repeat_penalty`, `min_p`, `num_predict` (per Ollama OpenAI
13+
# compatibility docs). They MUST be set in a Modelfile to actually take
14+
# effect. The opencode.jsonc sets the OpenAI-compat names (temperature,
15+
# top_p, frequency_penalty, max_tokens) which DO pass through.
16+
#
17+
# Build: ollama create gemma4:26b-ailang -f tools/ollama/gemma4-26b-ailang.modelfile
18+
# Verify: ollama show gemma4:26b-ailang --modelfile
19+
# Use: reference as `gemma4:26b-ailang` in opencode + models.yml
20+
21+
FROM gemma4:26b
22+
23+
# Google official top-k for Gemma 4 (Ollama gemma4:26b params blob).
24+
PARAMETER top_k 64
25+
26+
# Community consensus value for agentic tool-call loops with Gemma 3/4
27+
# (mychen76/gemma3_cline_roocode_qat ModelFile, Ollama issue #15502 mitigation).
28+
# Kept at 1.1, NOT higher, because smcleod.net 2025-04 LLM sampling guide
29+
# notes >1.1 hurts code because variable names should legitimately repeat.
30+
PARAMETER repeat_penalty 1.1
31+
PARAMETER repeat_last_n 256
32+
33+
# Safety-floor under top_p. Unsloth's Gemma run guide notes min_p=0.01
34+
# "works well"; we use 0.05 as a slightly stricter floor for agentic loops.
35+
PARAMETER min_p 0.05
36+
37+
# Cap any single generation at 4096 tokens. The repetition-collapse incident
38+
# emitted 32,000 tokens in one tool argument. Even with sampling fixes,
39+
# this is a hard belt-and-suspenders backstop.
40+
PARAMETER num_predict 4096

0 commit comments

Comments
 (0)