Skip to content

Commit 243c3f4

Browse files
authored
Merge pull request #1286 from Wibias/feat/cl-00-compatibility-contracts
docs(lab): define compatibility evidence and conformance contracts
2 parents e61f63f + fe7ae54 commit 243c3f4

9 files changed

Lines changed: 3972 additions & 0 deletions
Lines changed: 306 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,306 @@
1+
# OpenCodex Compatibility Lab / EvalGrid
2+
3+
Status: CL-00 architecture authority
4+
Authority baseline: `upstream/dev` at `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296`
5+
Package/runtime at baseline: OpenCodex `2.10.2`, Bun `1.3.14`
6+
7+
## Purpose
8+
9+
Compatibility Lab turns compatibility claims into bounded, reproducible
10+
evidence. It tests OpenCodex protocol behavior, exact configured routes, and
11+
later execution-grounded task outcomes without becoming a provider registry, a
12+
user-policy system, or a production router.
13+
14+
This directory is the programme authority. The contracts frozen by CL-00 are:
15+
16+
- [Architecture and evidence](./010_architecture_and_evidence_contract.md)
17+
- [Scenario model and initial catalogue](./020_scenario_contract_and_catalogue.md)
18+
- [Protocol V1 manifest authority](./021_protocol_v1_manifest_authority.md)
19+
- [Protocol V1 canonical cases](./022_protocol_v1_cases.json)
20+
- [Historical incident corpus](./030_incident_corpus.md)
21+
- [Security and privacy](./040_security_and_privacy.md)
22+
- [CL-00 independent acceptance review](./050_cl00_acceptance_review.md)
23+
- [PR stack status](./001_pr_stack_status.md)
24+
25+
Later phases may add implementation detail, but must amend these contracts
26+
explicitly rather than silently changing their meaning.
27+
28+
## Live repository truth
29+
30+
CL-00 audited the live `dev` tree before defining new authority.
31+
32+
### Shipped and authoritative
33+
34+
- Provider declarations and model metadata:
35+
`src/providers/registry.ts`, `src/providers/derive.ts`, `src/types.ts`;
36+
route-time claimed capability assembly in `src/routing/capability.ts` also
37+
consumes provider config, cached Codex catalog rows and native metadata.
38+
Generated fallback metadata lives in `src/generated/model-metadata.ts` and is
39+
sourced by `scripts/model-metadata.source.json`.
40+
- Routing Profile public/config types: `OcxRoutingProfileConfig` in
41+
`src/types.ts`; validation, normalization, revision hashing, persistence and
42+
resolution in `src/routing/profile.ts`.
43+
- Deterministic profile evaluation and route traces:
44+
`src/routing/evaluator.ts`, `src/routing/trace.ts`, and
45+
`src/router.ts`.
46+
- Profile management CRUD and dry-run:
47+
`src/server/management/routing-profile-routes.ts`.
48+
- Dashboard profile editor, dry-run and routing analytics:
49+
`gui/src/pages/RoutingProfiles.tsx`, mounted under Models -> Routing.
50+
- Canonical append-only request/usage evidence and rebuildable history
51+
projection:
52+
`src/usage/log.ts`, `src/routing/history/indexer.ts`, and
53+
`src/routing/analytics.ts`.
54+
- Why-this-route evidence:
55+
`RouteDecisionTraceV1`, request-history explain endpoints in
56+
`src/server/management/request-history-routes.ts`, and CLI explain support.
57+
The GUI Logs modal renders only a compact route summary through
58+
`gui/src/pages/log-route-decision.ts`; there is no GUI request-history browser
59+
or full trace + attempts + outcome view on this baseline.
60+
- Existing diagnostics are narrower than Compatibility Lab:
61+
the default `ocx doctor` path is observe-only environment/OAuth/runtime
62+
diagnosis; explicit `--fix-codex-runtime` may persist a repair. In contrast,
63+
`POST /api/providers/test` performs a bounded live `/models` connectivity
64+
check only when applicable; forward providers return configured status and
65+
static catalogues return not-applicable without network access.
66+
- Protocol behavior is already covered by many focused tests under `tests/`,
67+
but those tests are not a versioned scenario catalogue or evidence ledger.
68+
69+
### Not shipped
70+
71+
- No Compatibility Lab runner, scenario registry, evidence ledger, SQLite
72+
projection, CLI, management API, or UI exists.
73+
- Generated Cursor agent protobufs include task/grind/subagent message types,
74+
but OpenCodex has no native Agent Fabric task persistence, harness handoff,
75+
portable task-state model, or management API. Router Intelligence's own
76+
master plan explicitly excluded Agent Fabric.
77+
- No Routing Profile compatibility fields exist.
78+
79+
Current routing nuances that later phases must preserve rather than
80+
over-describe:
81+
82+
- selection traces and execution attempts are separate; the explain API merges
83+
trace + `attempts[]` + final outcome at read time;
84+
- `optimize.latency` is currently a declaration-priority share, while observed
85+
latency contributes through health evidence rather than an independent
86+
top-level score;
87+
- cost evidence is commonly unknown on the live pre-dispatch path because
88+
request usage is not yet available;
89+
- profile dry-run is evaluation-only and never dispatches upstream;
90+
- an unknown canonical `policy/<id>` currently falls through to ordinary model
91+
routing rather than failing closed.
92+
93+
Consequently, CL-00 defines future contracts and integration seams only. It
94+
does not rename existing Router Intelligence concepts or describe speculative
95+
Agent Fabric endpoints as current behavior.
96+
97+
## Architectural invariant
98+
99+
```text
100+
Provider Registry
101+
↓
102+
Compatibility Lab
103+
↓
104+
Compatibility Graph / Verified Evidence
105+
↓
106+
Routing Profiles
107+
↓
108+
Router Intelligence
109+
↓
110+
Selected Model / Route
111+
↓
112+
Agent Fabric / Real Execution
113+
└──────────────→ execution-grounded outcomes back to Lab
114+
```
115+
116+
The arrows are data dependencies, not ownership transfers.
117+
118+
### Provider Registry
119+
120+
The Provider Registry declares what a provider/model is believed to support and
121+
supplies defaults used to construct an effective route. The shipped
122+
`candidateCapabilityEvidence()` also combines explicit provider config, cached
123+
catalog rows, adapter-level inference and native-model metadata. These local
124+
declarations are claims. They may seed `CLAIMED`; they cannot by themselves
125+
produce `PROBED`, `VERIFIED`, `DEGRADED`, or `UNSUPPORTED`.
126+
127+
The Lab may snapshot a registry claim with its source revision for
128+
reproducibility. It must not create a parallel provider catalogue or write
129+
provider declarations back into the registry.
130+
131+
Registry-owned runtime defaults such as model wire selection, discovery policy,
132+
upstream streaming, reasoning replay, service-tier support, and item-ID repair
133+
are intentionally not all persisted to `config.json`. Claim snapshots capture
134+
the effective sources; they do not freeze runtime defaults into user config.
135+
136+
### Compatibility Lab
137+
138+
The Lab owns versioned scenarios, immutable compatibility evidence, failure
139+
attribution, freshness, derived verdicts, and regression history. It may
140+
project evidence into a compatibility graph keyed by exact route subject and
141+
evidence layer.
142+
143+
The Lab never chooses a production candidate, mutates a Routing Profile,
144+
changes provider metadata, or turns a probe result directly into a route.
145+
146+
### Routing Profiles
147+
148+
Routing Profiles remain the sole compatibility-policy surface. Future
149+
compatibility requirements extend `OcxRoutingProfileConfig`, its
150+
normalizer/revision, the existing evaluator, the existing management
151+
CRUD/dry-run endpoints, and the Models dashboard editor. Existing combo and
152+
account-pool controls retain their separate non-compatibility responsibilities.
153+
There will be no compatibility-specific profile store, evaluator, or editor.
154+
155+
### Router Intelligence
156+
157+
Router Intelligence combines the selected profile with current capability,
158+
compatibility, health, quota, cost, and latency evidence and makes the
159+
deterministic route decision. Its existing `RouteDecisionTraceV1` remains the
160+
authority for explaining that decision. Compatibility inputs will later add
161+
bounded evidence to that trace rather than introduce a second explanation
162+
record.
163+
164+
### Agent Fabric
165+
166+
Agent Fabric is a future producer of task-effectiveness observations. It owns
167+
real task execution and its sandbox. The Lab accepts only structured outcome
168+
data and sanitized content-addressed artifact references; it does not copy task
169+
repositories, prompts, worktrees, or hidden reasoning.
170+
171+
Because a native Agent Fabric is not present on the CL-00 baseline, this
172+
programme freezes the consumer semantics, not a fictitious production API. A
173+
later producer contract must identify its schema version, task class, exact
174+
route subject, deterministic verifier results, timing, resource limits,
175+
outcome, and sanitized artifact references. Existing request-grounded evidence
176+
may be linked through `RouteDecisionTraceV1`, `PersistedUsageAttempt`, and the
177+
final request outcome; prompt-bearing `responses-state.json` and generated
178+
Cursor task protobufs are not Lab feeds.
179+
180+
## Evidence-layer invariant
181+
182+
Every scenario and observation has exactly one layer:
183+
184+
1. `protocol_conformance`: whether OpenCodex translates and preserves a
185+
protocol contract correctly.
186+
2. `live_route_compatibility`: whether an exact
187+
provider/model/adapter/configuration route works now.
188+
3. `task_effectiveness`: whether that route produces verifier-confirmed
189+
outcomes for a versioned class of coding work.
190+
191+
Verdicts are projected per `(subject, layer, suite)`. Evidence from one layer
192+
may be shown as a prerequisite or correlated signal, but cannot promote or
193+
degrade another layer's verdict. There is no universal compatibility score.
194+
195+
## Persistence authority
196+
197+
Future implementation uses the existing OpenCodex config root returned by
198+
`getConfigDir()` (`OPENCODEX_HOME`, default `~/.opencodex`) and owns:
199+
200+
```text
201+
~/.opencodex/lab/
202+
compatibility.jsonl
203+
compatibility.sqlite
204+
artifacts/
205+
```
206+
207+
- `compatibility.jsonl` is the canonical append-only evidence/event ledger.
208+
- `compatibility.sqlite` is a disposable query projection rebuilt from JSONL.
209+
- `artifacts/` contains bounded, sanitized, content-addressed artifacts.
210+
- Scenario/suite manifests and synthetic fixture/source anchors are
211+
content-addressed contract artifacts retained with the observations that
212+
reference them.
213+
- Verdicts are derived projections, never mutable canonical booleans.
214+
- Corrections append invalidation events or a new claim snapshot with explicit
215+
`supersedes[]`; prior bytes are not edited.
216+
- The Lab does not copy `usage.jsonl` or routing-history rows. When useful, an
217+
observation references an existing request ID or route decision ID.
218+
- Agent Fabric supplies structured outcome data/references, never repositories
219+
or prompt transcripts.
220+
221+
This location follows current repository state-root conventions. No filename
222+
or location change from the proposed architecture was justified by the audit.
223+
224+
## Routing Profiles boundary for CL-06
225+
226+
CL-06 must add optional compatibility controls alongside existing capability,
227+
health, quota, cost, and latency policy:
228+
229+
- required compatibility suites;
230+
- minimum compatibility status;
231+
- maximum evidence age;
232+
- unknown-evidence behavior;
233+
- degraded-evidence behavior.
234+
235+
`minimum compatibility status` is not a total ordering across all verdicts.
236+
Only `PROBED` and `VERIFIED` are positive thresholds. `DEGRADED` is governed by
237+
its explicit behavior, `UNKNOWN`/`CLAIMED`/`BLOCKED` by unknown-evidence
238+
behavior, and `UNSUPPORTED` fails a required suite.
239+
240+
The exact future flow is:
241+
242+
```text
243+
Routing Profile
244+
↓
245+
Configured candidates
246+
↓
247+
Hard capability gates
248+
↓
249+
Compatibility requirements / penalties
250+
↓
251+
Eligible candidates
252+
↓
253+
Health / quota / cost / latency scoring
254+
↓
255+
Deterministic winner
256+
```
257+
258+
All compatibility fields are optional. Profiles that omit them retain their
259+
current validation, revision, eligibility and scoring behavior. A profile
260+
evaluation reads an existing projection only. No compatibility probe, network
261+
request, task, or projection rebuild may run synchronously on the production
262+
request path.
263+
264+
## Programme phases
265+
266+
Only CL-00 is authorized by this document at present.
267+
268+
| Phase | Purpose | Authorization |
269+
|---|---|---|
270+
| CL-00 | Architecture authority, contracts, scenario catalogue, incident corpus | This PR |
271+
| CL-01 | Deterministic protocol-conformance runner and fixtures | Not started; requires CL-00 to be accepted |
272+
| CL-02 | Immutable JSONL ledger, artifacts and SQLite projection | Not started |
273+
| CL-03 | Bounded live-route probes | Not started |
274+
| CL-04 | Lab CLI and management read surfaces | Not started |
275+
| CL-05 | Compatibility Matrix UI | Not started |
276+
| CL-06 | Existing Routing Profile compatibility controls and Router Intelligence consumption | Not started |
277+
| CL-07 | Agent Fabric task-effectiveness ingestion | Not started |
278+
| CL-08 | Shadow/automatic/public evidence workflows | Not started |
279+
280+
Phase numbering after CL-01 is programme planning, not implementation
281+
authorization. A later accepted plan may split a phase while preserving these
282+
ownership boundaries.
283+
284+
## CL-00 acceptance criteria
285+
286+
CL-00 is accepted only when:
287+
288+
1. all three evidence layers have separate subjects, scenarios and verdicts;
289+
2. every canonical verdict is reproducible from immutable inputs;
290+
3. environmental blockers cannot poison compatibility conclusions;
291+
4. exact route identity prevents evidence reuse across behavior changes;
292+
5. scenario semantics and initial IDs are implementable without an LLM judge;
293+
6. representative historical incidents map to abstract regression scenarios;
294+
7. future compatibility policy extends existing Routing Profiles;
295+
8. probes and task execution are excluded from production request routing;
296+
9. privacy and sandbox ceilings are explicit;
297+
10. an independent review finds no unresolved Critical, High, or Medium issue.
298+
299+
## CL-00 hard stop
300+
301+
This phase does not implement a runner, mock upstream, persistence code, live
302+
probe, CLI, management endpoint, UI, profile field, routing behavior, shadow
303+
route, Fabric ingestion, automatic routing, or public publisher.
304+
305+
Acceptance of CL-00 authorizes discussion and planning of CL-01; it does not
306+
start CL-01 automatically.

0 commit comments

Comments
 (0)