|
| 1 | +# OpenCodex Compatibility Lab / EvalGrid |
| 2 | + |
| 3 | +Status: CL-00 architecture authority |
| 4 | +Authority baseline: `upstream/dev` at `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` |
| 5 | +Package/runtime at baseline: OpenCodex `2.10.2`, Bun `1.3.14` |
| 6 | + |
| 7 | +## Purpose |
| 8 | + |
| 9 | +Compatibility Lab turns compatibility claims into bounded, reproducible |
| 10 | +evidence. It tests OpenCodex protocol behavior, exact configured routes, and |
| 11 | +later execution-grounded task outcomes without becoming a provider registry, a |
| 12 | +user-policy system, or a production router. |
| 13 | + |
| 14 | +This directory is the programme authority. The contracts frozen by CL-00 are: |
| 15 | + |
| 16 | +- [Architecture and evidence](./010_architecture_and_evidence_contract.md) |
| 17 | +- [Scenario model and initial catalogue](./020_scenario_contract_and_catalogue.md) |
| 18 | +- [Protocol V1 manifest authority](./021_protocol_v1_manifest_authority.md) |
| 19 | +- [Protocol V1 canonical cases](./022_protocol_v1_cases.json) |
| 20 | +- [Historical incident corpus](./030_incident_corpus.md) |
| 21 | +- [Security and privacy](./040_security_and_privacy.md) |
| 22 | +- [CL-00 independent acceptance review](./050_cl00_acceptance_review.md) |
| 23 | +- [PR stack status](./001_pr_stack_status.md) |
| 24 | + |
| 25 | +Later phases may add implementation detail, but must amend these contracts |
| 26 | +explicitly rather than silently changing their meaning. |
| 27 | + |
| 28 | +## Live repository truth |
| 29 | + |
| 30 | +CL-00 audited the live `dev` tree before defining new authority. |
| 31 | + |
| 32 | +### Shipped and authoritative |
| 33 | + |
| 34 | +- Provider declarations and model metadata: |
| 35 | + `src/providers/registry.ts`, `src/providers/derive.ts`, `src/types.ts`; |
| 36 | + route-time claimed capability assembly in `src/routing/capability.ts` also |
| 37 | + consumes provider config, cached Codex catalog rows and native metadata. |
| 38 | + Generated fallback metadata lives in `src/generated/model-metadata.ts` and is |
| 39 | + sourced by `scripts/model-metadata.source.json`. |
| 40 | +- Routing Profile public/config types: `OcxRoutingProfileConfig` in |
| 41 | + `src/types.ts`; validation, normalization, revision hashing, persistence and |
| 42 | + resolution in `src/routing/profile.ts`. |
| 43 | +- Deterministic profile evaluation and route traces: |
| 44 | + `src/routing/evaluator.ts`, `src/routing/trace.ts`, and |
| 45 | + `src/router.ts`. |
| 46 | +- Profile management CRUD and dry-run: |
| 47 | + `src/server/management/routing-profile-routes.ts`. |
| 48 | +- Dashboard profile editor, dry-run and routing analytics: |
| 49 | + `gui/src/pages/RoutingProfiles.tsx`, mounted under Models -> Routing. |
| 50 | +- Canonical append-only request/usage evidence and rebuildable history |
| 51 | + projection: |
| 52 | + `src/usage/log.ts`, `src/routing/history/indexer.ts`, and |
| 53 | + `src/routing/analytics.ts`. |
| 54 | +- Why-this-route evidence: |
| 55 | + `RouteDecisionTraceV1`, request-history explain endpoints in |
| 56 | + `src/server/management/request-history-routes.ts`, and CLI explain support. |
| 57 | + The GUI Logs modal renders only a compact route summary through |
| 58 | + `gui/src/pages/log-route-decision.ts`; there is no GUI request-history browser |
| 59 | + or full trace + attempts + outcome view on this baseline. |
| 60 | +- Existing diagnostics are narrower than Compatibility Lab: |
| 61 | + the default `ocx doctor` path is observe-only environment/OAuth/runtime |
| 62 | + diagnosis; explicit `--fix-codex-runtime` may persist a repair. In contrast, |
| 63 | + `POST /api/providers/test` performs a bounded live `/models` connectivity |
| 64 | + check only when applicable; forward providers return configured status and |
| 65 | + static catalogues return not-applicable without network access. |
| 66 | +- Protocol behavior is already covered by many focused tests under `tests/`, |
| 67 | + but those tests are not a versioned scenario catalogue or evidence ledger. |
| 68 | + |
| 69 | +### Not shipped |
| 70 | + |
| 71 | +- No Compatibility Lab runner, scenario registry, evidence ledger, SQLite |
| 72 | + projection, CLI, management API, or UI exists. |
| 73 | +- Generated Cursor agent protobufs include task/grind/subagent message types, |
| 74 | + but OpenCodex has no native Agent Fabric task persistence, harness handoff, |
| 75 | + portable task-state model, or management API. Router Intelligence's own |
| 76 | + master plan explicitly excluded Agent Fabric. |
| 77 | +- No Routing Profile compatibility fields exist. |
| 78 | + |
| 79 | +Current routing nuances that later phases must preserve rather than |
| 80 | +over-describe: |
| 81 | + |
| 82 | +- selection traces and execution attempts are separate; the explain API merges |
| 83 | + trace + `attempts[]` + final outcome at read time; |
| 84 | +- `optimize.latency` is currently a declaration-priority share, while observed |
| 85 | + latency contributes through health evidence rather than an independent |
| 86 | + top-level score; |
| 87 | +- cost evidence is commonly unknown on the live pre-dispatch path because |
| 88 | + request usage is not yet available; |
| 89 | +- profile dry-run is evaluation-only and never dispatches upstream; |
| 90 | +- an unknown canonical `policy/<id>` currently falls through to ordinary model |
| 91 | + routing rather than failing closed. |
| 92 | + |
| 93 | +Consequently, CL-00 defines future contracts and integration seams only. It |
| 94 | +does not rename existing Router Intelligence concepts or describe speculative |
| 95 | +Agent Fabric endpoints as current behavior. |
| 96 | + |
| 97 | +## Architectural invariant |
| 98 | + |
| 99 | +```text |
| 100 | +Provider Registry |
| 101 | + ↓ |
| 102 | +Compatibility Lab |
| 103 | + ↓ |
| 104 | +Compatibility Graph / Verified Evidence |
| 105 | + ↓ |
| 106 | +Routing Profiles |
| 107 | + ↓ |
| 108 | +Router Intelligence |
| 109 | + ↓ |
| 110 | +Selected Model / Route |
| 111 | + ↓ |
| 112 | +Agent Fabric / Real Execution |
| 113 | + └──────────────→ execution-grounded outcomes back to Lab |
| 114 | +``` |
| 115 | + |
| 116 | +The arrows are data dependencies, not ownership transfers. |
| 117 | + |
| 118 | +### Provider Registry |
| 119 | + |
| 120 | +The Provider Registry declares what a provider/model is believed to support and |
| 121 | +supplies defaults used to construct an effective route. The shipped |
| 122 | +`candidateCapabilityEvidence()` also combines explicit provider config, cached |
| 123 | +catalog rows, adapter-level inference and native-model metadata. These local |
| 124 | +declarations are claims. They may seed `CLAIMED`; they cannot by themselves |
| 125 | +produce `PROBED`, `VERIFIED`, `DEGRADED`, or `UNSUPPORTED`. |
| 126 | + |
| 127 | +The Lab may snapshot a registry claim with its source revision for |
| 128 | +reproducibility. It must not create a parallel provider catalogue or write |
| 129 | +provider declarations back into the registry. |
| 130 | + |
| 131 | +Registry-owned runtime defaults such as model wire selection, discovery policy, |
| 132 | +upstream streaming, reasoning replay, service-tier support, and item-ID repair |
| 133 | +are intentionally not all persisted to `config.json`. Claim snapshots capture |
| 134 | +the effective sources; they do not freeze runtime defaults into user config. |
| 135 | + |
| 136 | +### Compatibility Lab |
| 137 | + |
| 138 | +The Lab owns versioned scenarios, immutable compatibility evidence, failure |
| 139 | +attribution, freshness, derived verdicts, and regression history. It may |
| 140 | +project evidence into a compatibility graph keyed by exact route subject and |
| 141 | +evidence layer. |
| 142 | + |
| 143 | +The Lab never chooses a production candidate, mutates a Routing Profile, |
| 144 | +changes provider metadata, or turns a probe result directly into a route. |
| 145 | + |
| 146 | +### Routing Profiles |
| 147 | + |
| 148 | +Routing Profiles remain the sole compatibility-policy surface. Future |
| 149 | +compatibility requirements extend `OcxRoutingProfileConfig`, its |
| 150 | +normalizer/revision, the existing evaluator, the existing management |
| 151 | +CRUD/dry-run endpoints, and the Models dashboard editor. Existing combo and |
| 152 | +account-pool controls retain their separate non-compatibility responsibilities. |
| 153 | +There will be no compatibility-specific profile store, evaluator, or editor. |
| 154 | + |
| 155 | +### Router Intelligence |
| 156 | + |
| 157 | +Router Intelligence combines the selected profile with current capability, |
| 158 | +compatibility, health, quota, cost, and latency evidence and makes the |
| 159 | +deterministic route decision. Its existing `RouteDecisionTraceV1` remains the |
| 160 | +authority for explaining that decision. Compatibility inputs will later add |
| 161 | +bounded evidence to that trace rather than introduce a second explanation |
| 162 | +record. |
| 163 | + |
| 164 | +### Agent Fabric |
| 165 | + |
| 166 | +Agent Fabric is a future producer of task-effectiveness observations. It owns |
| 167 | +real task execution and its sandbox. The Lab accepts only structured outcome |
| 168 | +data and sanitized content-addressed artifact references; it does not copy task |
| 169 | +repositories, prompts, worktrees, or hidden reasoning. |
| 170 | + |
| 171 | +Because a native Agent Fabric is not present on the CL-00 baseline, this |
| 172 | +programme freezes the consumer semantics, not a fictitious production API. A |
| 173 | +later producer contract must identify its schema version, task class, exact |
| 174 | +route subject, deterministic verifier results, timing, resource limits, |
| 175 | +outcome, and sanitized artifact references. Existing request-grounded evidence |
| 176 | +may be linked through `RouteDecisionTraceV1`, `PersistedUsageAttempt`, and the |
| 177 | +final request outcome; prompt-bearing `responses-state.json` and generated |
| 178 | +Cursor task protobufs are not Lab feeds. |
| 179 | + |
| 180 | +## Evidence-layer invariant |
| 181 | + |
| 182 | +Every scenario and observation has exactly one layer: |
| 183 | + |
| 184 | +1. `protocol_conformance`: whether OpenCodex translates and preserves a |
| 185 | + protocol contract correctly. |
| 186 | +2. `live_route_compatibility`: whether an exact |
| 187 | + provider/model/adapter/configuration route works now. |
| 188 | +3. `task_effectiveness`: whether that route produces verifier-confirmed |
| 189 | + outcomes for a versioned class of coding work. |
| 190 | + |
| 191 | +Verdicts are projected per `(subject, layer, suite)`. Evidence from one layer |
| 192 | +may be shown as a prerequisite or correlated signal, but cannot promote or |
| 193 | +degrade another layer's verdict. There is no universal compatibility score. |
| 194 | + |
| 195 | +## Persistence authority |
| 196 | + |
| 197 | +Future implementation uses the existing OpenCodex config root returned by |
| 198 | +`getConfigDir()` (`OPENCODEX_HOME`, default `~/.opencodex`) and owns: |
| 199 | + |
| 200 | +```text |
| 201 | +~/.opencodex/lab/ |
| 202 | + compatibility.jsonl |
| 203 | + compatibility.sqlite |
| 204 | + artifacts/ |
| 205 | +``` |
| 206 | + |
| 207 | +- `compatibility.jsonl` is the canonical append-only evidence/event ledger. |
| 208 | +- `compatibility.sqlite` is a disposable query projection rebuilt from JSONL. |
| 209 | +- `artifacts/` contains bounded, sanitized, content-addressed artifacts. |
| 210 | +- Scenario/suite manifests and synthetic fixture/source anchors are |
| 211 | + content-addressed contract artifacts retained with the observations that |
| 212 | + reference them. |
| 213 | +- Verdicts are derived projections, never mutable canonical booleans. |
| 214 | +- Corrections append invalidation events or a new claim snapshot with explicit |
| 215 | + `supersedes[]`; prior bytes are not edited. |
| 216 | +- The Lab does not copy `usage.jsonl` or routing-history rows. When useful, an |
| 217 | + observation references an existing request ID or route decision ID. |
| 218 | +- Agent Fabric supplies structured outcome data/references, never repositories |
| 219 | + or prompt transcripts. |
| 220 | + |
| 221 | +This location follows current repository state-root conventions. No filename |
| 222 | +or location change from the proposed architecture was justified by the audit. |
| 223 | + |
| 224 | +## Routing Profiles boundary for CL-06 |
| 225 | + |
| 226 | +CL-06 must add optional compatibility controls alongside existing capability, |
| 227 | +health, quota, cost, and latency policy: |
| 228 | + |
| 229 | +- required compatibility suites; |
| 230 | +- minimum compatibility status; |
| 231 | +- maximum evidence age; |
| 232 | +- unknown-evidence behavior; |
| 233 | +- degraded-evidence behavior. |
| 234 | + |
| 235 | +`minimum compatibility status` is not a total ordering across all verdicts. |
| 236 | +Only `PROBED` and `VERIFIED` are positive thresholds. `DEGRADED` is governed by |
| 237 | +its explicit behavior, `UNKNOWN`/`CLAIMED`/`BLOCKED` by unknown-evidence |
| 238 | +behavior, and `UNSUPPORTED` fails a required suite. |
| 239 | + |
| 240 | +The exact future flow is: |
| 241 | + |
| 242 | +```text |
| 243 | +Routing Profile |
| 244 | + ↓ |
| 245 | +Configured candidates |
| 246 | + ↓ |
| 247 | +Hard capability gates |
| 248 | + ↓ |
| 249 | +Compatibility requirements / penalties |
| 250 | + ↓ |
| 251 | +Eligible candidates |
| 252 | + ↓ |
| 253 | +Health / quota / cost / latency scoring |
| 254 | + ↓ |
| 255 | +Deterministic winner |
| 256 | +``` |
| 257 | + |
| 258 | +All compatibility fields are optional. Profiles that omit them retain their |
| 259 | +current validation, revision, eligibility and scoring behavior. A profile |
| 260 | +evaluation reads an existing projection only. No compatibility probe, network |
| 261 | +request, task, or projection rebuild may run synchronously on the production |
| 262 | +request path. |
| 263 | + |
| 264 | +## Programme phases |
| 265 | + |
| 266 | +Only CL-00 is authorized by this document at present. |
| 267 | + |
| 268 | +| Phase | Purpose | Authorization | |
| 269 | +|---|---|---| |
| 270 | +| CL-00 | Architecture authority, contracts, scenario catalogue, incident corpus | This PR | |
| 271 | +| CL-01 | Deterministic protocol-conformance runner and fixtures | Not started; requires CL-00 to be accepted | |
| 272 | +| CL-02 | Immutable JSONL ledger, artifacts and SQLite projection | Not started | |
| 273 | +| CL-03 | Bounded live-route probes | Not started | |
| 274 | +| CL-04 | Lab CLI and management read surfaces | Not started | |
| 275 | +| CL-05 | Compatibility Matrix UI | Not started | |
| 276 | +| CL-06 | Existing Routing Profile compatibility controls and Router Intelligence consumption | Not started | |
| 277 | +| CL-07 | Agent Fabric task-effectiveness ingestion | Not started | |
| 278 | +| CL-08 | Shadow/automatic/public evidence workflows | Not started | |
| 279 | + |
| 280 | +Phase numbering after CL-01 is programme planning, not implementation |
| 281 | +authorization. A later accepted plan may split a phase while preserving these |
| 282 | +ownership boundaries. |
| 283 | + |
| 284 | +## CL-00 acceptance criteria |
| 285 | + |
| 286 | +CL-00 is accepted only when: |
| 287 | + |
| 288 | +1. all three evidence layers have separate subjects, scenarios and verdicts; |
| 289 | +2. every canonical verdict is reproducible from immutable inputs; |
| 290 | +3. environmental blockers cannot poison compatibility conclusions; |
| 291 | +4. exact route identity prevents evidence reuse across behavior changes; |
| 292 | +5. scenario semantics and initial IDs are implementable without an LLM judge; |
| 293 | +6. representative historical incidents map to abstract regression scenarios; |
| 294 | +7. future compatibility policy extends existing Routing Profiles; |
| 295 | +8. probes and task execution are excluded from production request routing; |
| 296 | +9. privacy and sandbox ceilings are explicit; |
| 297 | +10. an independent review finds no unresolved Critical, High, or Medium issue. |
| 298 | + |
| 299 | +## CL-00 hard stop |
| 300 | + |
| 301 | +This phase does not implement a runner, mock upstream, persistence code, live |
| 302 | +probe, CLI, management endpoint, UI, profile field, routing behavior, shadow |
| 303 | +route, Fabric ingestion, automatic routing, or public publisher. |
| 304 | + |
| 305 | +Acceptance of CL-00 authorizes discussion and planning of CL-01; it does not |
| 306 | +start CL-01 automatically. |
0 commit comments