You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9da68bb
Browse filesBrowse the repository at this point in the historyBrowse files
docs(isolation): tighten ADR 0006 evidence and trust claims
- Correct the rejected-alternative premise: under ADR 0004 the provider
snapshot is host-materialized and read-only to the guest; the argument is
provider neutrality, not snapshot trust.
- Add field- and modality-specific evidence requirements. The synthetic-image
procedure now covers image input only, with repeated samples and a repeated
text-only control; reasoning, audio, video and pdf need their own procedures
and stay absent until then. Note that a partial inputModalities list narrows
what OpenCode sends.
- Re-describe sources.capabilitiesVerified as a catalog-wide release reminder,
not evidence. Require a per-model, per-field evidence record in the data PR
description plus a committed file under the opencode kit's
docs/capability-evidence/. The guard test is unchanged apart from titles,
comments and names that say what it checks.
Co-authored-by: OpenCode Agent <[email protected]>
Copy file name to clipboardExpand all lines: integrations/isolation/docs/decisions/0006-neutral-model-capability-metadata.md
+69-14Lines changed: 69 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -58,7 +58,7 @@ Conclusions:
58
58
- The harm risk is wrong values: rendering `["text"]` wrongly (for example
59
59
from a mis-mapped upstream entry) disables images the gateway accepts, on
60
60
every OpenCode line. That is why **unknown must stay absent** in committed
61
-
data until verified, and why the renderer never defaults a value. This
61
+
data until per-model evidence exists, and why the renderer never defaults a value. This
62
62
record makes the data expressible; it does not commit any values.
63
63
64
64
Image-reading accuracy also varied by model family on a hard synthetic task
@@ -81,7 +81,8 @@ photos, other harnesses, repeat runs and OpenCode 1.x were not tested.
81
81
default. The flag matches each USAi id to a models.dev entry with the
82
82
builder's precision-guarded id matching, so "never inferred from id or name"
83
83
applies to the renderer and normalizer; the flag's output is unverified until
84
-
the live synthetic-image run.
84
+
per-model, per-field evidence exists (see
85
+
[Evidence required before a value is published](#evidence-required-before-a-value-is-published)).
85
86
- Declaring a capability must not silently turn on request-time behavior
86
87
(reasoning effort, thinking budgets, variants). The renderer emits none of
87
88
these; note that OpenCode itself may derive selectable variants from
@@ -101,7 +102,12 @@ photos, other harnesses, repeat runs and OpenCode 1.x were not tested.
101
102
additive, optional change.
102
103
3.**Put harness-native syntax (OpenCode `modalities`) in the provider's
103
104
snapshot.** Violates the Layer 3 boundary: provider inputs must not carry
104
-
harness-specific options, and the snapshot lives on an agent-writable path.
105
+
harness-specific options, and the provider-neutral snapshot would then be
106
+
usable by only one harness. The snapshot's trust properties are not the
107
+
argument: under ADR 0004 (Layer 1 and Security model) it is
108
+
host-materialized and read-only to the guest.
109
+
Only the OpenCode config the renderer writes in the guest is guest-writable,
110
+
and it is not an authority for model state.
105
111
4.**Infer capabilities in the renderer from model ids or names.** Rejected:
106
112
unverifiable and wrong for renamed or gateway-specific deployments.
107
113
@@ -123,12 +129,58 @@ leaves the harness-specific translation inside the OpenCode kit.
123
129
The one sanctioned exception is the USAi catalog builder's explicit
124
130
`--include-capabilities` flag, which is off by default, labels its output
125
131
`sources.capabilities` as unverified, and whose output must not be committed
126
-
until the synthetic-image procedure has verified it.
132
+
until each value it carries has the evidence described under
133
+
[Evidence required before a value is published](#evidence-required-before-a-value-is-published).
134
+
Values without that evidence stay absent.
135
+
- A present `inputModalities` array asserts the listed modalities only. A
136
+
modality that was not probed is not asserted, but the OpenCode mapping treats
137
+
an unlisted modality as disabled, so publishing a partial list narrows what
138
+
OpenCode will send. The review must accept that narrowing for each model, or
139
+
the field stays absent for it.
127
140
- A capability field is a statement of fact only. It does not select a
128
141
reasoning effort, budget, variant, or any other request setting.
129
142
- Both the USAi catalog schema and the neutral schema carry the same enum; a
130
143
test asserts they cannot diverge.
131
144
145
+
### Evidence required before a value is published
146
+
147
+
Evidence is specific to a model, a field, and (for `inputModalities`) a
148
+
modality. Evidence for one does not carry over to another, and a value without
149
+
its own evidence stays **absent** (unknown). The synthetic-image procedure
150
+
substantiates image input only.
151
+
152
+
| Value | Evidence required before it is published |
153
+
|---|---|
154
+
|`inputModalities` includes `image`| The synthetic-image procedure for that model: an exact-token read, repeated over several independently generated images, and a text-only control (explicit `["text"]`), repeated on the same images, that never returns the token. A single run is not enough. |
155
+
|`inputModalities` includes `audio`, `video`, or `pdf`| Its own probe, with a known-content input of that type that cannot be answered from text or the model id, repeated samples, and a text-only control. An image probe says nothing about these. No such probe exists yet, so these values are not published. |
156
+
|`inputModalities: ["text"]` (known text-only) | A negative result is also a claim. Record the failed attempts and the exact rejection or error, so a transient outage is not published as "unsupported". |
157
+
|`reasoning: true` or `false`| A separate procedure that checks the **target OpenCode version's** behavior, not just the gateway's: whether the model reasons at all, which variants OpenCode derives from the field (1.x derives low/medium/high variants for openai-compatible models), and request-time effects such as effort or thinking-budget parameters sent to the gateway. An image probe says nothing about this. No such procedure exists yet, so `reasoning` is not published. |
158
+
159
+
### Evidence record
160
+
161
+
`sources.capabilitiesVerified` is a catalog-wide string shaped like
162
+
`<date>; opencode <version>`. It is a **release reminder**, not evidence: it
163
+
cannot show which model, field, or modality was verified. The guard test only
164
+
checks its shape, so a well-formed string proves nothing about the data.
165
+
166
+
The data-change review is therefore required to find, for every populated
167
+
`reasoning` and `inputModalities` value, a traceable per-model, per-field record:
168
+
169
+
- The data pull request description states each model id, field, and value, with
170
+
the procedure used, the sample count, the control result, the OpenCode version,
171
+
and the date.
172
+
- A durable evidence file is committed in the same pull request at
0 commit comments