Skip to content

Commit 09ddede

Browse files
mogulOpenCode Agent
andcommitted
docs(opencode): keep unobserved reasoning indeterminate, require target-specific evidence
Co-authored-by: OpenCode Agent <[email protected]>
1 parent a54f752 commit 09ddede

3 files changed

Lines changed: 74 additions & 31 deletions

File tree

‎integrations/isolation/acq-kits/opencode/README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,9 @@ reasoning effort or variants, but OpenCode may derive selectable variants from
5959
reasoning-effort variants, per v1.18.35 `transform.ts` `variants()`); check
6060
this before any `reasoning: true` value is committed; the synthetic-image
6161
procedure covers image input only and cannot substantiate `reasoning`, `audio`,
62-
`video`, or `pdf`. See
62+
`video`, or `pdf`. On OpenCode 2.0.30 the `reasoning` model field is dropped
63+
(variants are independent of it); see
64+
[reasoning validation](docs/reasoning-validation.md). See
6365
[ADR 0006](../../docs/decisions/0006-neutral-model-capability-metadata.md) and
6466
the [synthetic-image validation procedure](docs/synthetic-image-validation.md).
6567

‎integrations/isolation/acq-kits/opencode/docs/reasoning-validation.md‎

Lines changed: 54 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -73,10 +73,12 @@ probe session against the gateway. Samples were small (see the limits section).
7373
`none`, and `thinking: {type: "disabled"}`, were accepted without error, with no observable change in tokens or
7474
correctness. `low` and `high` were tried on all four models, `none` and
7575
`thinking: disabled` on `claude-sonnet-5.5` and `gemini-3.5-flash` only,
76-
two runs per setting. A negative control needs a way to switch thinking off or
77-
a model known not to think. Only `gpt_5_2_default_v2` behaves as a
78-
non-thinker, and that comes from observation, not from any switch. An
79-
explicit `false` has nothing to compare against.
76+
two runs per setting. A control needs a way to switch thinking on or off
77+
on the target itself. Only `gpt_5_2_default_v2` behaves as a
78+
non-thinker, and that comes from observation, not from any switch, and it
79+
says nothing about the other models. Without a working switch, every model
80+
stays *Indeterminate*, including `gpt_5_2_default_v2`: its short wrong
81+
answers could equally be a reasoning model with thinking off by default.
8082

8183
## What a published value would mean
8284

@@ -90,15 +92,17 @@ probe session against the gateway. Samples were small (see the limits section).
9092
On the pinned version a value is inert and was seen to log a warning per model
9193
(`--print-logs`, debug level, local stand-in gateway). `false` has no use that `absent` does not already serve: no consumer
9294
examined here reads "known not to reason", and the evidence for it would be the weakest
93-
(absence of a signal the gateway never sends). Publishing `true` would be a
95+
(absence of a signal the gateway never sends, which cannot tell "does not
96+
reason" from "reasons but is not reported"). Publishing `true` would be a
9497
claim about a vendor behavior that no check in this path can see, and on 1.x
9598
may (unverified) add variants and change requests, which someone must accept
9699
(the OpenCode kit owner, and the owner of the data change).
97100

98101
## Procedure (for a future run)
99102

100-
Run only if the gateway starts exposing a reasoning signal, or a consumer is
101-
found that reads the field. Otherwise stop at the decision rule.
103+
Run only if the gateway starts exposing a reasoning signal, a consumer is
104+
found that reads the field, or the gateway or vendor states explicitly that a
105+
model does not support reasoning. Otherwise stop at the decision rule.
102106

103107
### Preconditions
104108

@@ -131,24 +135,42 @@ found that reads the field. Otherwise stop at the decision rule.
131135
`reasoning_tokens` in usage, completion tokens, `total_tokens` minus
132136
`prompt_tokens`, and whether the answer is correct.
133137
4. **Repeat** at least five times per prompt with fresh random tasks.
134-
5. **Classify** each model:
135-
- *Reasons, observed*: a reasoning field or `reasoning_tokens` is present on
136-
every run, **and** the non-thinker control (below) shows no such signal on
137-
the same task. A documented off switch that removes the signal and lowers
138-
correctness on every run strengthens this but does not replace it.
139-
- *Does not reason, observed*: the signal is absent on every run, the task
140-
fails on every run, **and** a documented on switch demonstrably changes the
141-
signal on a model that does think (so the probe is shown able to see it).
142-
- *Indeterminate*: anything else, including token accounting alone.
138+
5. **Classify** each model. Every class needs evidence about **that model**.
139+
A result on another model (a positive control, a non-thinker, a switch that
140+
works elsewhere) shows only that the probe can see a signal; it never
141+
establishes a value for the target.
142+
- *Reasons, observed* (supports `true`): on the target, a reasoning field or
143+
`reasoning_tokens` is present on every run, **and** the target's own
144+
documented off switch removes or sharply reduces that signal on the same
145+
task (or a non-thinker control run on the target itself shows none). Correctness
146+
falling when the switch is off strengthens this but does not replace it.
147+
- *Does not reason, known-unsupported* (supports `false`): only an explicit,
148+
target-specific statement that the model does not support reasoning: the
149+
gateway's or vendor's documentation for that model, or an explicit
150+
gateway response for that model (for example an error naming the
151+
reasoning parameter as unsupported for it). An OFF/ON run with no
152+
difference is **not** enough, because a switch the gateway ignores looks
153+
identical to a model that does not reason. A missing signal, failed
154+
arithmetic or low token counts are never enough.
155+
- *Indeterminate*: everything else. In particular: a null or negative
156+
observation (no signal, wrong answers, small token counts) on a model whose
157+
own positive control is missing, or was run only on a different model;
158+
a model that may have thinking disabled by default, suppressed telemetry
159+
or weak accuracy; and token accounting alone. Indeterminate means the
160+
field stays **absent** (unknown).
143161

144162
### Controls
145163

146164
- A trivial prompt, to give each model's baseline token count.
147-
- A model that behaves as a non-thinker on the same task (in the 2026-11-04
148-
session, `gpt_5_2_default_v2`), required for either published value, to show that the probe can tell the difference.
149-
- A switch (on or off) that the gateway documents for a model that thinks and
150-
that demonstrably changes the signal. Without one, neither class of
151-
"observed" above can be reached for `false`, so `false` cannot be published.
165+
- A switch on the **target model** (on or off) that the gateway documents for
166+
that model and that demonstrably changes the signal. Without one,
167+
*Reasons, observed* cannot be reached for that model, so `true` cannot be
168+
published for it. `false` does not come from a control at all: it needs the
169+
explicit target-specific statement described above.
170+
- A non-thinker on the same task (in the 2026-11-04 session,
171+
`gpt_5_2_default_v2`) shows only that the probe can tell two models apart.
172+
It is a calibration, not evidence about any other model, and it cannot move
173+
another model out of *Indeterminate*.
152174

153175
### Decision rule
154176

@@ -158,10 +180,11 @@ found that reads the field. Otherwise stop at the decision rule.
158180
value must change something a consumer reads, and whoever owns that
159181
consumer must accept the changes in variants, requests, cost and latency.
160182
- Publish `reasoning: false` only for a model classified *Does not reason,
161-
observed* **and** only if a consumer is shown to need the distinction from
162-
absent.
163-
- Everything else stays **absent**. As of the 2026-11-04 session this is every
164-
model tested, and every untested model.
183+
known-unsupported* from target-specific evidence **and** only if a consumer
184+
is shown to need the distinction from absent. A null observation never yields
185+
`false`.
186+
- Everything else, including every *Indeterminate* model, stays **absent**. As
187+
of the 2026-11-04 session this is every model tested, and every untested model.
165188

166189
### Evidence file
167190

@@ -185,7 +208,12 @@ version differs from the one under test.
185208

186209
- That a model "really reasons" internally. Token accounting and the presence of
187210
a stream field are observations about the gateway's reporting, not about
188-
the model.
211+
the model. Reporting telemetry does not establish internal reasoning, and
212+
its absence does not establish that a model does not reason.
213+
- That a model does not reason. A reasoning-capable model may have thinking off
214+
by default, suppressed telemetry, or wrong answers. Only target-specific
215+
evidence can move a model to `false`; a positive control on another model
216+
cannot.
189217
- Behavior of any OpenCode version other than the one run, including 1.x.
190218
- Behavior of consumers not examined here, notably the Quickstart projection.
191219
- Anything about audio, video, pdf or image input.

‎integrations/isolation/docs/decisions/0006-neutral-model-capability-metadata.md‎

Lines changed: 17 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -155,7 +155,7 @@ substantiates image input only.
155155
| `inputModalities` includes `image` | The synthetic-image procedure for that model: an exact-token read, repeated over several independently generated images, and a text-only control (explicit `["text"]`), repeated on the same images, that never returns the token. A single run is not enough. |
156156
| `inputModalities` includes `audio`, `video`, or `pdf` | Its own probe, with a known-content input of that type that cannot be answered from text or the model id, repeated samples, and a text-only control. An image probe says nothing about these. No such probe exists yet, so these values are not published. |
157157
| `inputModalities: ["text"]` (known text-only) | A negative result is also a claim. Record the failed attempts and the exact rejection or error, so a transient outage is not published as "unsupported". |
158-
| `reasoning: true` or `false` | No procedure can currently substantiate it on OpenCode 2.0.30 or the USAi gateway: OpenCode drops the field, and the gateway exposes no reasoning signal or working control. See [`reasoning-validation.md`](../../acq-kits/opencode/docs/reasoning-validation.md). **Decision rule: no value is published and the field stays absent for every model.** Revisit only if a future OpenCode or gateway exposes an observable control, meaning a consumer that reads the field and changes behavior with it, or a gateway signal (a reasoning stream field or `reasoning_tokens`) plus a documented switch that demonstrably changes it. An image probe says nothing about this. |
158+
| `reasoning: true` or `false` | No procedure can currently substantiate it on OpenCode 2.0.30 or the USAi gateway: OpenCode drops the field, and the gateway exposes no reasoning signal or working control. See [`reasoning-validation.md`](../../acq-kits/opencode/docs/reasoning-validation.md). **Decision rule: no value is published and the field stays absent for every model.** Revisit only if a future OpenCode or gateway exposes an observable control, meaning a consumer that reads the field and changes behavior with it, or a gateway signal (a reasoning stream field or `reasoning_tokens`) plus a documented switch that demonstrably changes it **on the target model** (for `true`), or an explicit gateway or vendor statement for the target model (for `false`). A null observation, or a control that works only on another model, leaves the model indeterminate and the field absent. An image probe says nothing about this. |
159159

160160
### Evidence record
161161

@@ -276,8 +276,9 @@ kept as written.
276276

277277
The owner of the epic accepted keeping `reasoning` **absent for every model**.
278278
No `reasoning: true` or `false` value is published, and none will be unless a
279-
future OpenCode or gateway exposes an observable control (see the evidence
280-
table).
279+
future OpenCode or gateway exposes an observable control, or the gateway or
280+
vendor states explicitly that a given model does not support reasoning (see the
281+
evidence table).
281282

282283
### Evidence
283284

@@ -297,7 +298,16 @@ USAi gateway; full method and limits are in
297298
the stand-in gateway only.
298299
- The gateway returned no reasoning stream field and no `reasoning_tokens` for
299300
the four models examined, and the effort and disable switches tried had no
300-
observable effect, so there is no usable negative control.
301+
observable effect, so there is no usable positive or negative control on
302+
any model. Every model is therefore *Indeterminate*: a null observation
303+
(no signal, wrong answers, low token counts) is not evidence that a model
304+
does not reason, and a control working on another model cannot establish a
305+
value for the target. `true` needs target-specific positive evidence (a signal plus that
306+
model's own switch changing it). `false` (known-unsupported) needs an explicit
307+
target-specific statement from the gateway or vendor that the model does
308+
not support reasoning; an OFF/ON run with no difference is not enough,
309+
because an ignored switch looks the same. The
310+
gateway's reporting telemetry does not establish internal reasoning.
301311

302312
The samples are small (four gateway models in depth, single-digit runs, one
303313
date). This is evidence for the decision, not proof.
@@ -324,6 +334,9 @@ per the kit README) was **not tested**.
324334
- The `reasoning` field and its type stay in the neutral and USAi schemas as
325335
optional. Absent means unknown.
326336
- Consumers MUST treat absent `reasoning` as unknown, not as "does not reason".
337+
- Unsuccessful or unobserved reasoning (no signal, failed task, or a positive
338+
control that exists only on another model) stays *Indeterminate* and the
339+
field stays absent. `false` is never concluded from a null observation.
327340
- Producers should not supply `reasoning` for OpenCode 2.0.30 consumers unless
328341
the behavior is re-verified. The renderer does not enforce this today; it
329342
would emit a supplied value.

0 commit comments

Comments
 (0)