Repository navigation
[P1] [SPX-AI-006] Execute and analyze the existing 18-trial paired coding-agent pilot #105
Description
SPX-AI-006 — Execute and analyze the existing 18-trial paired coding-agent pilot
Priority: P1 | Workstream: evaluation | Import action: New implementation issue
Planning baseline: wavect/semaprax@ae25c6a49dc09ec4613c7a6f52a27daa69dcd3f3 (11 September 2026).
Implementation prerequisites: SPX-AI-005
Important: prerequisites mean the required implementation/evidence is accepted, not merely that its GitHub issue exists.
Outcome
Produce the first actual comparative observations: three tasks × two available Semaprax lanes × three repetitions. Report failures as data and rank the practical blockers. This is an execution/evaluation issue, not a request to build another framework.
Current evidence and distinction from existing work
The current manifest declares repetitions=3 and the available graph-operational and source-first lanes. The current contract still says comparative model trials have no observations. A green framework gate does not imply the trials ran.
Read these owners first
- docs/AGENT-TASK-COMPARISON-V1.md
- benchmarks/agent-task-comparison-v1/manifest.json
- benchmarks/agent-task-comparison-v1/tasks
- scripts/agent-task-comparison.py
Directory pointers are navigation roots. Resolve current files with rg --files; proposed new files must follow the current module ownership. The referenced snapshot may be older than the implementing branch.
Bounded scope and non-goals
Use the frozen existing tasks, rubric and isolation. One approved model configuration is enough for this first pilot. Do not wait for Agent v2 live integration, change task scope, delete failed runs, or infer superiority from eighteen small trials. Paid execution requires an operator-authorized budget; lacking it leaves execution blocked, not completed.
Implementation steps
- Freeze the exact repository revision, task hashes, model identifier, decoding configuration, tokenizer accounting method, runner revision and host. Confirm no other process is mutating the trial fixture or target directory.
- Generate and archive the complete available matrix. Determine a reproducible counterbalanced trial order outside the task contents so cache/temporal effects do not consistently favor one lane.
- Run all eighteen requested tuples with the same model and resource policy. Retain every failed/aborted attempt. Never silently substitute a different model after a provider error.
- Run independent acceptance, stale-preservation and ownership-sensitive checks on each candidate. Follow the existing blinded review protocol for active review time; record interventions separately.
- Derive and audit every observation. If one observation lacks mandatory measurements, report it as ineligible and explain the missing field rather than filling it with zero.
- Use the existing report command for complete available-lane aggregation. Produce paired per-task results for acceptance, usage, tool calls, context, recovery, validation and review. Show raw denominators and failed-run costs.
- Classify observed failures into language/profile limitations, incorrect model proposals, poor discovery, stale recovery, runtime defects, measurement/infrastructure problems and human intervention. Map each finding to an existing issue or a small follow-up with its reproduction.
- Publish only a descriptive pilot conclusion. Clearly separate framework correctness, compiler conformance and model productivity; do not claim general statistical significance or Zero parity.
Required tests and failure cases
- Every expected tuple is accounted for exactly once; missing, duplicate and extra tuples fail validation.
- Acceptance decisions are reproduced independently from archived candidate/evidence bytes.
- Token and time reports have authentic counters/timers, not lexical-unit estimates.
- Source-first and graph-operational runs use identical task/fixture/model configuration and isolated state.
- The summary retains failed and aborted trials and states the limits of a three-task pilot.
Starting commands
Run these from the implementing checkout. A command containing /absolute/path/... requires an explicit local output path. Use the current --help for option details and verify that test filters select actual tests. These are starting points, not a claim that this pack executed the compiler.
python3 scripts/agent-task-comparison.py audit --help
python3 scripts/agent-task-comparison.py report --helpDefinition of done
- Eighteen real-model trial records exist, or the unresolved execution blocker is explicitly open.
- All eligible observations pass the existing audit and the complete matrix can be reported.
- The prioritized failure taxonomy cites trial artifacts and drives implementation choices.
Guardrails for the implementing agent
- Read the current
AGENTS.md, affected RFC/specification, completion rows and architecture owners before implementation. This pack is pinned planning input, not permission to ignore newer code or decisions. - Use a private worktree and a private
CARGO_TARGET_DIRinside it. Do not usegit stash, overwrite sibling worktrees, or share compiler/test scratch output. Add tests to the existing owning harness, not a new compiler-linking top-level test binary unless repository rules require it. - Preserve source/graph/schema compatibility where contracted, left-to-right evaluation, lazy operands, ownership transfer, canonical cleanup order and sticky failure. Do not weaken tests, diagnostics, range checks or authority checks to get green.
- A proposed new shape/API below is a design target, not a claim that syntax or a command already exists. Run
--helpand discover exact test selectors. Zero selected tests, unavailable tools and skipped required rows are not execution evidence. - Keep model output as untrusted data; compiler facts and evidence never mint capabilities. Human support/publication decisions, paid model calls and real registry/Git writes require their own explicit authority.
- Reuse implemented Agent v2, migrations, iterators, closures, semantic candidates, persistent HIR cache and hosted PG grammar/metadata work. If the requested slice already landed, verify its gate and close only the residual gap rather than building a competitor implementation inside this repo.
Evidence to attach before closing
For the implementation handoff, include: exact starting and finishing commit; changed files and their owning invariants; the pre-fix failure or negative-control result; exact commands, exit status and nonzero selected test counts; target/toolchain/provider identity; artifact/log locations; and remaining limitations. Run the repository's complete required quality profile (scripts/quality.sh full on supported Unix, or the documented platform equivalent) before merge. A local focused pass is not hosted evidence and a green framework is not a completed model experiment. For architecture/ABI/ownership changes, obtain independent semantic review before promotion. Do not mark an execution issue complete using authored tests or documentation alone.
Activity
feat(benchmarks): connect an external coding-agent runner to the exis…
feat(semantic-discovery): expose compact capability discovery and rev…
style(semantic-discovery): apply cargo fmt to the new module
feat(project): add explicit untraced prepared execution
wsdt commentedon Sep 13, 2026
Pushed the dependency-priority implementation batch in
06bec1c29adc0cc9c28671ba64bb2e7975a3381f(starting85014337). The existing pilot now drives both lanes through a confined local MCP argv tool, with no model shell/native source tools. Source reads expose a preimage digest; writes refuse stale preimages. The frozen stale patch is applied after the recorded identifying inspection. Bounded process/output/log handling retains failure records, exact candidate source bytes/diffs, external MCP frames, provider exports, independent acceptance and monotonic validation duration. Owned-task review packages are retained.The existing runner also emits the exact authenticated 18-tuple matrix in a counterbalanced order: rotating task positions, alternating first lanes, honest 5/4 first-lane counts across nine pairs. No task or rubric bytes changed.
Local evidence: one schedule inventory/balance regression passed; the owning source+graph confined run/export/archive stub test passed (one test); actual OpenCode local MCP discovery connected for both lanes. No model observations are claimed by those checks. Root has frozen the new schedule and started the first free Muse tuple; no completed real trial is claimed yet.
Still open: eighteen real tuples and complete eligible observation/audit/report coverage. Capture is explicitly ineligible while exact presented context, stale recovery, intervention and blinded-review measurements are absent. Internal gateway counts are not mislabeled external tool traffic. No full/hosted gates repeated, per operator direction.
wsdt commentedon Sep 13, 2026
Two real free Muse setup attempts are preserved: pilot-06bec1c2-01 had no tools because the runner passed the gateway log path instead of executable; pilot-96f90af1-01 reached MCP but failed confined /dev/null access. Fixes landed through b8e7ccc, including actual confined gateway-command regression. A no-model preflight successfully ran the frozen compiler through MCP; new cohort pilot-b8e7ccca started. All setup failures remain ineligible, with authentic usage and failure artifacts. Missing presentation/review/stale metrics remain unavailable, never zero.
Working cohort first tuple made61 MCP calls,32 rejected, with unchanged candidate and failed acceptance. The model could inspect compiler semantics; discovery/proposal failures are retained as data. OpenCode export truncated at65536 bytes; supplemental stream derivation in
c2b25986retains all token components (including cached input/reasoning), reports421767 input and11758 output for23 unique finished steps. This is a configured-model CLI report, not independent provider authentication. Remaining17 tuples are running serially under the frozen policy to conserve resources.wsdt commentedon Sep 13, 2026
All 18 frozen real-model tuples are now retained exactly once, with independent candidate replay. Report and hash-bound index pushed in
e0482399:docs/AGENT-TASK-PILOT-2026-09-13.mdandbenchmarks/agent-task-comparison-v1/pilot-2026-09-13.json.Results: 0/18 full task acceptance; 4 process completions, 14 timeouts; 1,180 MCP calls including 757 failed calls. Every failure/abort and the two separate setup attempts remain archived. All 18 are ineligible: exact model-context presentation bytes, blinded active review time, complete typed stale/recovery metrics and intervention ledger are missing. No zeros or productivity/Zero-parity claims substituted for missing data. Completed-step token reports include cache/reasoning; unreported in-flight usage remains unknown.
Prioritized reproduced blockers: harness command/write-path discovery; incorrect persistent-ID edits in source owned trial2; no demonstrated stale recovery; truncated OpenCode exports and missing eligibility measurements. Raw local artifacts live under
benchmarks/agent-task-comparison-v1/evidence/pilot-b8e7ccca-01through-18; committed index binds original record hashes. Source replay retains exact frozen compiler/hash and never rewrites records.After the cohort ended,
8da8ef38fixed lane-specific--helpand explicit.pilot/artifact guidance. No-model two-lane help regression passed; two export-counter regressions passed. No rerun or completed eligible aggregation is claimed. #105 stays OPEN for missing measurement/review and a separately frozen follow-up. No full or hosted gates ran per operator instruction.fix(pilot): stop install_gateway eating real binaries; compute trial …
wsdt commentedon Sep 17, 2026
Progress —
ad4a863e— not closingA destructive bug found on the way, fixed first
install_gatewayRENAMES itscompilerargument tosemaprax-realand writes the recorded wrapper over that exact path. Two tests inscripts/test-opencode-agent-task-pilot.pyhanded it the livesys.executable. Running the pilot test suite therefore renamed the host's real Python 3.14 interpreter and replaced it with the wrapper, whose#!/usr/bin/env python3shebang then resolved back to itself — so everypython3invocation on that machine died withenv: python3: Argument list too long(exit 126).It happened twice on the development machine, to two separate installs (2026-09-13 and 2026-09-17). Fixed at both levels: the callers now pass
provision_semaprax()'s disposable copy, andinstall_gatewayitself now refuses anycompileroutside the state directory rather than trusting its callers.provision_semapraxalways returns<state>/bin/semaprax, so no legitimate caller is affected.Eligibility is now computed, not hardcoded
ineligibility_reason()returned a fixed string andreplay.pyhardcodedeligible_observation: False, so no trial could ever have been eligible on any evidence. The four measurements this cohort was missing are now genuinely measured, in a newscripts/opencode_agent_task_pilot/eligibility.py:tools/callresponse byte, summed with repeats, counted off the pilot's own transport — never estimated from tokensrecord-reviewCLI, exclusive-createreview.json) binding reviewer id, blinding attestation and the exactcandidate.diffSHA-256;active_msbounded by the elapsed intervalinterventions.jsonl, closedkindenum, monotonic sequence, created empty at trial start so zero interventions is auditable rather than indistinguishable from unmeasuredcompute_eligibilityreplaces both hardcodes: eligible only when all four are observed; anything missing or malformed is fail-closed with the specific reason named; it never raises and never fabricates a value.Evidence:
/usr/bin/python3 scripts/test-opencode-agent-task-pilot.py→ 51 tests, 46 passed, all 33 new tests among them. The 5 failures are pre-existingSubprocessBoundaryTests/TupleTransportTestscases and every one fails withenv: python3: Argument list too long— they exercise the wrapper's shebang against the corrupted interpreter described above. They are expected to pass once it is restored; that is not claimed as verified.Residual scope
Nothing was backfilled.
benchmarks/was not touched, no measurement was inferred or defaulted to zero, and no model trial was re-run or paid for. The 2026-09-13 cohort therefore remains ineligible on its real evidence —review.jsonandinterventions.jsonlnever existed for those 18 trials.Before a new authorized cohort run: (1) restore the corrupted interpreters from their
semaprax-realsiblings; (2) wire an actual blinded-reviewer workflow around the newrecord-reviewCLI, which today is only a recorded-input slot with no reviewer UI; (3) decide an operator convention for logging interventions mid-run — theinterveneCLI exists but nothing calls it automatically.wsdt commentedon Sep 18, 2026
Closure audit @
e9ab9242— NOT CLOSEABLE, environment-blocked.The 18-tuple cohort ran, but 0/18 were eligible. Closing requires a new paid/credentialed cohort plus a reviewer workflow — neither available in an agent session. This is a budget-and-people task, not an engineering one.
feat(cli): measure the persistent semantic cache across real processes
instrument(SPX-AI-031/AI-032): quantify FrontendPass::lookup's clone …
fix(agent-pilot): classify stale recovery exactly (Refs #105)
wsdt commentedon Sep 20, 2026
Pushed d120aae to main. The no-spend pilot instrumentation now counts stale recovery only for an exact, canonical pilot-write-source of src/core.spx; reads and other-path writes cannot create a false recovery, malformed writes fail closed, and rejected_stale_write requires the gateway exact stale-precondition return code and stderr. Focused eligibility suite: 38 passed; stale subset: 12 passed; py_compile and diff checks passed. No historical evidence was backfilled and no paid/model calls ran. #105 remains open because a new authorized cohort and reviewer workflow are still required.
feat(benchmark): bind blinded review and cohort audit (Refs #105)
fix(evaluation): serialize intervention evidence
wsdt commentedon Sep 20, 2026
Progress pushed to main in be8e0d6 (Refs #105). The no-spend intervention ledger now uses exclusive no-follow creation and holds a POSIX advisory lock across bounded read, validation, sequence allocation, and append, preventing concurrent operators from emitting duplicate sequence IDs. Ledger parsing is shared between reader and writer. Hostile lock-contention and preexisting-symlink regressions were added. Local evidence: full pilot script tests 71 passed; focused InterventionLedgerTests 10 passed; py_compile, ledger smoke, and diff check passed. One focused adversarial review found no remaining defect in this tranche. This does not close #105: a fresh authorized real-model cohort and blinded human review remain required.
feat(evaluation): bind presented context evidence
wsdt commentedon Sep 20, 2026
A second no-spend instrumentation tranche is on main in 0f35dde (Refs #105). Fresh trials now write canonical, exclusive/no-follow presentation.json evidence binding frozen prompt bytes/digest, raw mcp-wire bytes/digest, and exact tool-response totals. Eligibility and replay re-derive it from primary evidence and fail closed for missing, noncanonical, linked, substituted-prompt, or substituted-wire records; legacy evidence is not backfilled. Runner failure tuples are retained with the presentation error. Local evidence: 72/72 pilot Python tests, 10/10 focused context/eligibility tests, py_compile, and diff check. One adversarial pass removed an unsafe cleanup unlink so partial evidence remains fail-closed. No model calls. Not closing: authorized cohort execution and blinded review remain.
feat(evaluation): bind active review timing
wsdt commentedon Sep 20, 2026
A third no-spend instrumentation tranche is on main in 5240caf (Refs #105). Eligible active-review timing now uses start-review/finish-review: an exclusive label-free packet and immutable canonical session bind the candidate while host-clock start/end derive duration. Eligibility requires timed_packet_bound; legacy record-review remains compatible but noneligible. Stale packets/sessions, manual duration, tampering, duplicate and partial artifacts fail closed. Focused tests 29/29 and full pilot Python suite 75/75 passed; py_compile/diff checks passed. One adversarial review fixed the partial re-start artifact hazard. No model calls. Remaining closure still requires an authorized cohort and real blinded review.
feat(evaluation): bind stale recovery evidence
wsdt commentedon Sep 20, 2026
The final identified no-spend eligibility instrumentation gap is pushed in 1cca807 (Refs #105). A canonical successful pilot-drift receipt now binds trigger and before/after SHA-256 values; eligibility verifies source-read drift against the recorded raw preimage and only credits success/rejection writes when expected digests and exact gateway stdout/stderr agree. A primary gateway end-to-end regression and hostile receipt/write cases passed: 20 focused tests plus py_compile/diff checks. One adversarial review found no remaining instrumentation defect. #105 now remains open for the authorized cohort execution and blinded human review only; no model calls were made.