Skip to content

Commit 9da68bb

Browse files
committed
docs(isolation): tighten ADR 0006 evidence and trust claims
- Correct the rejected-alternative premise: under ADR 0004 the provider snapshot is host-materialized and read-only to the guest; the argument is provider neutrality, not snapshot trust. - Add field- and modality-specific evidence requirements. The synthetic-image procedure now covers image input only, with repeated samples and a repeated text-only control; reasoning, audio, video and pdf need their own procedures and stay absent until then. Note that a partial inputModalities list narrows what OpenCode sends. - Re-describe sources.capabilitiesVerified as a catalog-wide release reminder, not evidence. Require a per-model, per-field evidence record in the data PR description plus a committed file under the opencode kit's docs/capability-evidence/. The guard test is unchanged apart from titles, comments and names that say what it checks. Co-authored-by: OpenCode Agent <[email protected]>
1 parent a6820b8 commit 9da68bb

6 files changed

Lines changed: 157 additions & 62 deletions

File tree

‎integrations/isolation/acq-kits/opencode/README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,9 @@ outside the closed modality allowlist fail the render. The renderer emits no
5757
reasoning effort or variants, but OpenCode may derive selectable variants from
5858
`reasoning: true` (1.x does so for openai-compatible models: low/medium/high
5959
reasoning-effort variants, per v1.18.35 `transform.ts` `variants()`); check
60-
this before any `reasoning: true` value is committed. See
60+
this before any `reasoning: true` value is committed; the synthetic-image
61+
procedure covers image input only and cannot substantiate `reasoning`, `audio`,
62+
`video`, or `pdf`. See
6163
[ADR 0006](../../docs/decisions/0006-neutral-model-capability-metadata.md) and
6264
the [synthetic-image validation procedure](docs/synthetic-image-validation.md).
6365

‎integrations/isolation/acq-kits/opencode/docs/synthetic-image-validation.md‎

Lines changed: 55 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,19 @@ recognition works or is fixed.
88
## Purpose
99

1010
Establish, per USAi model, whether the gateway actually accepts and uses an
11-
image, so that `inputModalities` values are committed from observation rather
12-
than from upstream (models.dev) data or model names. See
11+
image, so that the `image` entry of `inputModalities` is committed from
12+
observation rather than from upstream (models.dev) data or model names. See
1313
`../../../docs/decisions/0006-neutral-model-capability-metadata.md`.
1414

15+
## Scope: image input only
16+
17+
This procedure can substantiate **image input and nothing else**. It cannot show
18+
that a model supports `reasoning`, `audio`, `video`, or `pdf` input, and a pass
19+
here must not be used to publish any of those. Each of them needs its own
20+
procedure and evidence, as set out in the "Evidence required before a value is
21+
published" section of the ADR. Until such a procedure has been run for a given
22+
model and field, that value stays **absent** (unknown).
23+
1524
## Preconditions
1625

1726
- A sandbox created from the release candidate bundle (kit pins updated by the
@@ -20,7 +29,8 @@ than from upstream (models.dev) data or model names. See
2029
it into a command, prompt, file, or log; never print the environment.
2130
- Network egress exactly as the kits already allow-list; TLS verification
2231
unchanged.
23-
- Human approval to spend a handful of requests per model.
32+
- Human approval to spend roughly ten or more requests per model (at least
33+
three images across each configuration).
2434

2535
## What unset means
2636

@@ -41,7 +51,7 @@ On v2, "unset" is therefore not a control for "no image support"; it already
4151
behaves like the image-capable configuration, so the text-only control must use
4252
an explicit `["text"]`. Explicit `modalities` give version-independent behavior
4353
and are the safe contract for any consumer still on 1.x, but unknown models stay
44-
absent in committed data until verified here.
54+
absent in committed data until evidenced here.
4555

4656
## The synthetic image
4757

@@ -61,51 +71,71 @@ not in the prompt.
6171
2. **Text control.** Send a plain text prompt; expect a normal answer. A
6272
failure here is an auth/routing problem, not an image problem — stop.
6373
3. **Image probe.** Attach the synthetic image and ask: "Reply with only the
64-
characters shown in the image." Record the model's reply verbatim.
74+
characters shown in the image." Record the model's reply verbatim. Repeat
75+
with at least three independently generated images (a new random token each
76+
time); one correct reply is not enough, because a single run can be luck or a
77+
cached answer.
6578
4. **Classify** the outcome:
6679
- *Accepted and correct* — reply equals the token.
6780
- *Accepted, wrong* — reply is fluent but not the token (image ignored or
6881
hallucinated). Treat as **not** image-capable.
6982
- *Rejected* — OpenCode or the gateway returns an error or strips the
7083
attachment. Record the exact message.
71-
5. **Configurations.** Run steps 3–4 with `modalities.input` unset (the
84+
5. **Configurations.** Repeat steps 3–4 (on the full set of at least three
85+
images) with `modalities.input` unset (the
7286
current committed state; on v2 this already means text and image, on 1.x
7387
it would mean text only), with
7488
`["text","image"]` set explicitly, and with `["text"]` set explicitly via
7589
throwaway local overrides. Unset versus explicit `["text","image"]` should
7690
match; the explicit `["text"]` run is the control that shows the metadata,
7791
rather than the gateway, governs OpenCode's behavior.
78-
6. **Negative control.** The explicit `["text"]` run must not return the
79-
token. A model that returns the token under `["text"]` indicates the
92+
6. **Negative control.** The explicit `["text"]` run, repeated on the same
93+
images, must never return the token. A model that returns the token under `["text"]` indicates the
8094
probe or harness is not honoring the setting; stop and investigate.
8195

8296
## Evidence to record
8397

8498
For each model: model id, OpenCode version, step-1 config facts, the verbatim
85-
replies, the classification, and the date. Keep the evidence table in a
86-
durable, self-contained place: the pull request description or tracking issue
87-
that records the run, with enough detail to be understood on its own (fully
88-
qualified references, no bare issue numbers). Only the resulting
89-
`sources.capabilitiesVerified` record, not the raw transcript, is committed.
99+
replies for every sample, the classification, the text-only control result, and
100+
the date. The evidence is per model and per field. Record it in two places, both
101+
self-contained (fully qualified references, no bare issue numbers):
102+
103+
- The data pull request description that commits the values.
104+
- A committed evidence file, at
105+
`capability-evidence/<YYYY-MM-DD>-opencode-<version>.md` in this kit's `docs`
106+
directory (the convention the ADR defines), with one table row per model,
107+
field, and value. It holds results and verbatim replies, not credentials or
108+
raw transcripts.
109+
110+
`sources.capabilitiesVerified` is only a catalog-wide release reminder shaped
111+
`<date>; opencode <version>`. It is not evidence, and committing it does not make
112+
any value verified. The reviewer checks each populated value against the
113+
evidence file.
90114

91115
## Decision rule
92116

93-
- Commit `inputModalities` including `image` only for models classified
94-
*Accepted and correct* with explicit `["text","image"]`. Accuracy matters:
95-
a model that reads the image but misreads the token is *Accepted, wrong* and
117+
- Commit `image` in `inputModalities` only for models classified *Accepted and
118+
correct* on every repeated sample with explicit `["text","image"]`, and whose
119+
explicit `["text"]` control did not return the token. Accuracy matters: a
120+
model that reads the image but misreads the token is *Accepted, wrong* and
96121
stays absent.
97-
- Models that fail, or were not tested, stay **absent** (unknown).
98-
- When capability values are committed, record the verification in
122+
- Models that fail, or were not tested, stay **absent** (unknown). This
123+
procedure publishes `image` only: do not add `audio`, `video`, `pdf`, or
124+
`reasoning` from it.
125+
- When capability values are committed, also record the catalog-wide reminder in
99126
`sources.capabilitiesVerified` of the USAi catalog as
100127
`<YYYY-MM-DD>; opencode <version>` (for example `2026-10-25; opencode
101-
2.0.30`) so the repository guard test passes honestly. The test requires
102-
this structure, not any non-empty string.
103-
- Before any `reasoning: true` value is committed, check how the target
104-
OpenCode version treats it: in OpenCode 1.x, `reasoning: true` on an
105-
openai-compatible model makes OpenCode derive selectable low/medium/high
106-
reasoning-effort variants. Confirm that is acceptable (or not wanted) first.
128+
2.0.30`) so the repository guard test passes. The test requires this shape,
129+
not any non-empty string, and it does not check the evidence file. The
130+
evidence file and pull request description are what substantiate the values.
131+
- `reasoning` needs a separate procedure that checks the **target OpenCode
132+
version's** behavior: whether the model reasons, which variants OpenCode
133+
derives from the field (1.x derives low/medium/high reasoning-effort variants
134+
for openai-compatible models), and request-time effects such as effort or
135+
thinking-budget parameters. Do not publish `reasoning: true` or `false` until
136+
that procedure exists and has been run for the model.
107137
- Generate candidate values with the catalog builder's opt-in
108-
`--include-capabilities` flag, then keep only the verified ones.
138+
`--include-capabilities` flag, then keep only the values that have evidence.
109139

110140
## Stop conditions
111141

‎integrations/isolation/docs/decisions/0006-neutral-model-capability-metadata.md‎

Lines changed: 69 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,7 @@ Conclusions:
5858
- The harm risk is wrong values: rendering `["text"]` wrongly (for example
5959
from a mis-mapped upstream entry) disables images the gateway accepts, on
6060
every OpenCode line. That is why **unknown must stay absent** in committed
61-
data until verified, and why the renderer never defaults a value. This
61+
data until per-model evidence exists, and why the renderer never defaults a value. This
6262
record makes the data expressible; it does not commit any values.
6363

6464
Image-reading accuracy also varied by model family on a hard synthetic task
@@ -81,7 +81,8 @@ photos, other harnesses, repeat runs and OpenCode 1.x were not tested.
8181
default. The flag matches each USAi id to a models.dev entry with the
8282
builder's precision-guarded id matching, so "never inferred from id or name"
8383
applies to the renderer and normalizer; the flag's output is unverified until
84-
the live synthetic-image run.
84+
per-model, per-field evidence exists (see
85+
[Evidence required before a value is published](#evidence-required-before-a-value-is-published)).
8586
- Declaring a capability must not silently turn on request-time behavior
8687
(reasoning effort, thinking budgets, variants). The renderer emits none of
8788
these; note that OpenCode itself may derive selectable variants from
@@ -101,7 +102,12 @@ photos, other harnesses, repeat runs and OpenCode 1.x were not tested.
101102
additive, optional change.
102103
3. **Put harness-native syntax (OpenCode `modalities`) in the provider's
103104
snapshot.** Violates the Layer 3 boundary: provider inputs must not carry
104-
harness-specific options, and the snapshot lives on an agent-writable path.
105+
harness-specific options, and the provider-neutral snapshot would then be
106+
usable by only one harness. The snapshot's trust properties are not the
107+
argument: under ADR 0004 (Layer 1 and Security model) it is
108+
host-materialized and read-only to the guest.
109+
Only the OpenCode config the renderer writes in the guest is guest-writable,
110+
and it is not an authority for model state.
105111
4. **Infer capabilities in the renderer from model ids or names.** Rejected:
106112
unverifiable and wrong for renamed or gateway-specific deployments.
107113

@@ -123,12 +129,58 @@ leaves the harness-specific translation inside the OpenCode kit.
123129
The one sanctioned exception is the USAi catalog builder's explicit
124130
`--include-capabilities` flag, which is off by default, labels its output
125131
`sources.capabilities` as unverified, and whose output must not be committed
126-
until the synthetic-image procedure has verified it.
132+
until each value it carries has the evidence described under
133+
[Evidence required before a value is published](#evidence-required-before-a-value-is-published).
134+
Values without that evidence stay absent.
135+
- A present `inputModalities` array asserts the listed modalities only. A
136+
modality that was not probed is not asserted, but the OpenCode mapping treats
137+
an unlisted modality as disabled, so publishing a partial list narrows what
138+
OpenCode will send. The review must accept that narrowing for each model, or
139+
the field stays absent for it.
127140
- A capability field is a statement of fact only. It does not select a
128141
reasoning effort, budget, variant, or any other request setting.
129142
- Both the USAi catalog schema and the neutral schema carry the same enum; a
130143
test asserts they cannot diverge.
131144

145+
### Evidence required before a value is published
146+
147+
Evidence is specific to a model, a field, and (for `inputModalities`) a
148+
modality. Evidence for one does not carry over to another, and a value without
149+
its own evidence stays **absent** (unknown). The synthetic-image procedure
150+
substantiates image input only.
151+
152+
| Value | Evidence required before it is published |
153+
|---|---|
154+
| `inputModalities` includes `image` | The synthetic-image procedure for that model: an exact-token read, repeated over several independently generated images, and a text-only control (explicit `["text"]`), repeated on the same images, that never returns the token. A single run is not enough. |
155+
| `inputModalities` includes `audio`, `video`, or `pdf` | Its own probe, with a known-content input of that type that cannot be answered from text or the model id, repeated samples, and a text-only control. An image probe says nothing about these. No such probe exists yet, so these values are not published. |
156+
| `inputModalities: ["text"]` (known text-only) | A negative result is also a claim. Record the failed attempts and the exact rejection or error, so a transient outage is not published as "unsupported". |
157+
| `reasoning: true` or `false` | A separate procedure that checks the **target OpenCode version's** behavior, not just the gateway's: whether the model reasons at all, which variants OpenCode derives from the field (1.x derives low/medium/high variants for openai-compatible models), and request-time effects such as effort or thinking-budget parameters sent to the gateway. An image probe says nothing about this. No such procedure exists yet, so `reasoning` is not published. |
158+
159+
### Evidence record
160+
161+
`sources.capabilitiesVerified` is a catalog-wide string shaped like
162+
`<date>; opencode <version>`. It is a **release reminder**, not evidence: it
163+
cannot show which model, field, or modality was verified. The guard test only
164+
checks its shape, so a well-formed string proves nothing about the data.
165+
166+
The data-change review is therefore required to find, for every populated
167+
`reasoning` and `inputModalities` value, a traceable per-model, per-field record:
168+
169+
- The data pull request description states each model id, field, and value, with
170+
the procedure used, the sample count, the control result, the OpenCode version,
171+
and the date.
172+
- A durable evidence file is committed in the same pull request at
173+
`integrations/isolation/acq-kits/opencode/docs/capability-evidence/<YYYY-MM-DD>-opencode-<version>.md`.
174+
It holds one table row per model, field, and value with the same facts. It
175+
holds verbatim results, not credentials or raw transcripts.
176+
- The date in `sources.capabilitiesVerified` matches the date of the most recent
177+
evidence file, and every populated value has a row in some evidence file. A
178+
value with no row is removed in review.
179+
180+
This is a review requirement. The guard test does not parse the evidence file,
181+
because that would add an unvalidated file format; a later change may validate
182+
it once the format has been used.
183+
132184
### OpenCode mapping (kit-owned)
133185

134186
| Neutral | OpenCode model config |
@@ -170,9 +222,9 @@ rollout order, not on negotiation machinery:
170222
neutral schema (including Quickstart's provider-input fixtures and any
171223
Prime Agent consumer, both outside this change) must adopt the new schema
172224
before step 3.
173-
3. Only then commit capability values into the USAi snapshot, after the
174-
gateway-side synthetic-image validation has recorded which models
175-
actually accept images.
225+
3. Only then commit capability values into the USAi snapshot, and only those
226+
values that have per-model, per-field evidence under the section above.
227+
Anything else stays absent.
176228

177229
No `schemaVersion` bump is made. If a later change is *not* additive (renames,
178230
required fields, changed meaning of absence) it needs a new version and a new
@@ -192,12 +244,14 @@ record.
192244
- Strict `additionalProperties: false` makes rollout order matter (above).
193245
- models.dev is the only automated source; it can be wrong for the gateway.
194246
The builder therefore does not copy it unless `--include-capabilities` is
195-
passed, labels that output in `sources.capabilities`, and a guard test
196-
refuses capability values in any committed catalog or snapshot unless
197-
`sources.capabilitiesVerified` holds a structured record
198-
(`<date>; opencode <version>`). A malformed upstream entry is treated as
199-
unknown with a warning; malformed data in this repository's own committed
200-
inputs fails the build.
247+
passed, and labels that output in `sources.capabilities`. A guard test refuses
248+
capability values in any committed catalog or snapshot unless
249+
`sources.capabilitiesVerified` has the shape `<date>; opencode <version>`.
250+
That string is a catalog-wide release reminder, not evidence of any
251+
per-model, per-field verification. Evidence is required in the data-change
252+
review (see [Evidence record](#evidence-record)). A malformed upstream entry is
253+
treated as unknown with a warning; malformed data in this repository's own
254+
committed inputs fails the build.
201255
- The legacy `emitters/opencode.mjs` GENERATED-block emitter does not emit the
202256
new fields. It exists only as a byte-exact contract check for the retired
203257
config and is intentionally left as is.
@@ -218,6 +272,7 @@ record.
218272
- `../provider-input-contract.md` — host validation invocation and the
219273
pinned-release rule.
220274
- `../../acq-kits/opencode/docs/synthetic-image-validation.md` — the live
221-
procedure that must precede any capability value being committed.
275+
procedure for image input only. Other fields and modalities need their own
276+
evidence.
222277
- Tracking: GSA-TTS/agentic-coding-patterns#366 within
223278
GSA-TTS/agentic-coding-patterns#357.

‎integrations/providers/usai/README.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -48,11 +48,12 @@ and `["text"]` mean known-unsupported and known text-only. The builder does
4848
**not** copy them by default. Passing `--include-capabilities` to a live-feed
4949
build copies them from the matched models.dev entry and labels that provenance
5050
in `sources.capabilities`; models.dev describes the upstream provider, not
51-
necessarily the USAi gateway, so that output must not be committed until it is
52-
confirmed against the gateway (then record `sources.capabilitiesVerified` as
53-
`<date>; opencode <version>`). Malformed upstream values are treated as unknown
54-
with a warning. Nothing is inferred from
55-
model ids or names. See
51+
necessarily the USAi gateway, so that output must not be committed until each
52+
value is confirmed against the gateway with per-model, per-field evidence (a
53+
data PR description plus a committed evidence file; see the ADR). The
54+
`sources.capabilitiesVerified` string (`<date>; opencode <version>`) is only a
55+
catalog-wide release reminder, not evidence. Malformed upstream values are
56+
treated as unknown with a warning. Nothing is inferred from model ids or names. See
5657
[ADR 0006](../../isolation/docs/decisions/0006-neutral-model-capability-metadata.md).
5758

5859
## Safety guarantee

‎integrations/providers/usai/scripts/build-catalog.mjs‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -31,10 +31,12 @@
3131
// matched models.dev entry into live-feed output. Off by
3232
// default because models.dev describes the upstream
3333
// provider, not what the USAi gateway forwards; output
34-
// produced with this flag must not be committed until it
35-
// is confirmed by the live synthetic-image procedure
36-
// (see the opencode kit's synthetic-image-validation.md)
37-
// and sources.capabilitiesVerified is recorded. With the
34+
// produced with this flag must not be committed until each
35+
// value has per-model, per-field evidence (the synthetic-
36+
// image procedure covers image input only; see ADR 0006
37+
// and the opencode kit's synthetic-image-validation.md).
38+
// sources.capabilitiesVerified is a catalog-wide release
39+
// reminder, not that evidence. With the
3840
// flag, --check against a capability-free catalog.json
3941
// reports drift by design.
4042
//

0 commit comments

Comments
 (0)