Repository navigation
[P2] [SPX-AI-007] Add a held-out application and repair benchmark beyond signature-migration tasks #106
Description
SPX-AI-007 — Add a held-out application and repair benchmark beyond signature-migration tasks
Priority: P2 | Workstream: evaluation | Import action: New implementation issue
Planning baseline: wavect/semaprax@ae25c6a49dc09ec4613c7a6f52a27daa69dcd3f3 (11 September 2026).
Implementation prerequisites: SPX-AI-006
Important: prerequisites mean the required implementation/evidence is accepted, not merely that its GitHub issue exists.
Outcome
Broaden evaluation to new-feature work, data/text processing, dependency use and realistic repair, while retaining the original pilot unchanged. Make language and workflow improvements measurable on tasks not used to design them.
Current evidence and distinction from existing work
The existing three-task corpus focuses on signature migration and stale/owned variants of that workflow. It is useful but does not measure general application construction. This issue is a proposed evaluation extension, not a claim that broader results already exist.
Read these owners first
- docs/AGENT-TASK-COMPARISON-V1.md
- benchmarks/agent-task-comparison-v1/tasks
- scripts/agent-task-comparison.py
- docs/AGENT-QUICK-REFERENCE.md
Directory pointers are navigation roots. Resolve current files with rg --files; proposed new files must follow the current module ownership. The referenced snapshot may be older than the implementing branch.
Bounded scope and non-goals
Create a separately versioned corpus or an additive supported manifest route as required by the existing validator. Include explicit supported and unsupported scenarios. Hidden oracle data must live outside agent-visible roots. No benchmark task may require access to real customer data or production accounts.
Implementation steps
- Choose at least eight small tasks covering feature addition, imported library use, structured input errors, owned-data transformation, stale edit preservation, backend discrepancy diagnosis, realistic repair and clean-install use.
- For every task define exact inputs, permitted tools, acceptance outputs, failure cases, candidate artifact boundary and resource ceilings before running agents. Include independent oracles and at least one negative control.
- Partition tasks into development, validation and held-out evaluation families, not merely random copies of the same fixture. Keep problem templates or near-duplicates from crossing splits.
- Keep existing pilot tasks and observations frozen. Review any new schema needed for broader outcome categories instead of reusing v1 bytes with new semantics.
- Run oracle self-tests using intentionally wrong and correct candidates. Verify that the candidate cannot alter the oracle, scoring code or hidden test inputs.
- Record unsupported-language outcomes in the denominator of cross-system reports rather than omitting them. Make a separate supported-subset report only when clearly labeled.
- Execute a bounded repeat of the extended corpus with the approved model policy, retaining raw observations. Compare with the pilot failure taxonomy and avoid tuning on the held-out set.
Required tests and failure cases
- An implementation that merely prints an expected sample fails hidden input variants.
- A candidate editing its own visible tests cannot change independent acceptance.
- Task-family leakage, mismatched prompts and altered baselines are detected before aggregation.
- Unsupported, failed, aborted and ineligible outcomes remain separately visible.
- Original v1 tasks and validator known answers remain unchanged.
Test discovery and integration
Use rg --files src tests scripts and the owning harness command cargo test --locked -p semaprax --test <existing-harness> -- --list or its existing library selector. Replace <existing-harness> with the owner established from the references. Register new focused tests in that harness, prove they execute, then run the documented full profile. Do not copy a guessed selector into CI and accept a zero-test success.
Definition of done
- A versioned broader corpus and independent oracle self-tests exist.
- A documented split protects held-out evaluation from development and later training.
- Any reported improvement is tied to actual trial observations and a stated denominator.
Guardrails for the implementing agent
- Read the current
AGENTS.md, affected RFC/specification, completion rows and architecture owners before implementation. This pack is pinned planning input, not permission to ignore newer code or decisions. - Use a private worktree and a private
CARGO_TARGET_DIRinside it. Do not usegit stash, overwrite sibling worktrees, or share compiler/test scratch output. Add tests to the existing owning harness, not a new compiler-linking top-level test binary unless repository rules require it. - Preserve source/graph/schema compatibility where contracted, left-to-right evaluation, lazy operands, ownership transfer, canonical cleanup order and sticky failure. Do not weaken tests, diagnostics, range checks or authority checks to get green.
- A proposed new shape/API below is a design target, not a claim that syntax or a command already exists. Run
--helpand discover exact test selectors. Zero selected tests, unavailable tools and skipped required rows are not execution evidence. - Keep model output as untrusted data; compiler facts and evidence never mint capabilities. Human support/publication decisions, paid model calls and real registry/Git writes require their own explicit authority.
- Reuse implemented Agent v2, migrations, iterators, closures, semantic candidates, persistent HIR cache and hosted PG grammar/metadata work. If the requested slice already landed, verify its gate and close only the residual gap rather than building a competitor implementation inside this repo.
Evidence to attach before closing
For the implementation handoff, include: exact starting and finishing commit; changed files and their owning invariants; the pre-fix failure or negative-control result; exact commands, exit status and nonzero selected test counts; target/toolchain/provider identity; artifact/log locations; and remaining limitations. Run the repository's complete required quality profile (scripts/quality.sh full on supported Unix, or the documented platform equivalent) before merge. A local focused pass is not hosted evidence and a green framework is not a completed model experiment. For architecture/ABI/ownership changes, obtain independent semantic review before promotion. Do not mark an execution issue complete using authored tests or documentation alone.
Activity
feat(benchmarks): add a held-out repair task and tighten the Zero guard
Add independent structured-envelope benchmark validation
wsdt commentedon Sep 13, 2026
Progress pushed to main in 51651a7: a third, separate validation task covers structured-envelope error precedence. Rust, TypeScript, and the additive SEMAPRAX Project adapter keep hidden assertions separate while importing the copied public candidate. The frozen pilot and existing single-file adapter remain unchanged.
Focused local evidence: all 14 cross-language harness tests passed. The new negative control passes visible tests but fails hidden tests for a wrong precedence implementation, and still fails hidden acceptance after visible assertions are deleted. Real Rust 1.98.0, TypeScript 5.8.3, and SEMAPRAX Project runs each passed public and hidden phases with leak_check=ok (3 ok, 0 blocked/failed). Raw local receipt:
.agent-logs/0913-structured-input-results.json; its revision metadata accurately records the tested dirty candidate before commit.This supplies three task families across development/validation/held-out splits, not the requested eight-family corpus or a coding-agent experiment. Remaining families and bounded model trials keep #106 open. No performance or cross-system superiority claim is made.
Add multi-module invoice benchmark with three-port negative control (…
wsdt commentedon Sep 13, 2026
Pushed 110c5a1 to main: a fourth distinct multi-module invoice task family, with Rust, TypeScript, and SEMAPRAX Project ports. The calculation imports a whole-subtotal tax helper; hidden overlays replace only assertion entry modules and preserve the public candidate and helper.
Validation:
cargo test --locked -p semaprax --test documentation cross_language_benchmark_suite -- --nocapturepassed 15 tests, including the new three-port control: per-item rounding mutants pass public vectors, fail hidden vectors, and retainleak_check=ok. Separate real-runner controls confirm the correct implementations pass public and hidden phases in all three ports. Local macOS arm64 / Rust 1.98 / Node 24.3 / TypeScript 5.8.3. Logs:.agent-logs/0913-benchmark-docs.logand.agent-logs/module-import-controls-0913/summary.jsonin the private integration checkout.This is corpus progress only. The larger family inventory and coding-agent trials remain unfinished; the frozen earlier pilot is unchanged. No hosted CI claim.
Add half-open booking benchmark with three-port negative controls (Refs
wsdt commentedon Sep 13, 2026
Pushed fifth held-out task family in
70ea6f02c62e405e8d143b9fead12327596ad78d:booking-window-conflict-v1checks half-open booking overlap with shared candidate modules in Rust, TypeScript and SEMAPRAX Project ports.Local evidence: the focused
documentationharness selected 17 cross-language tests, all passed. Correct candidates pass public and hidden phases in all three ports; an inclusive-end comparison mutation passes public but fails hidden adjacency cases in all three, with leak checks passing. Direct runner positive report also contains three successful task/language pairs.Artifacts in the private integration checkout:
.agent-logs/0913-benchmark-five.log,.agent-logs/0913-booking-positive.json. Toolchain: macOS arm64, Rust 1.98, Node 24.3, TypeScript 5.8.3. This is corpus implementation and local execution evidence, not hosted CI or a coding-agent trial. The issue remains open for the remaining corpus breadth and real held-out trials; original pilot inputs remain unchanged.Validate calling descriptors and extend held-out task families (Refs #…
wsdt commentedon Sep 13, 2026
Pushed
5ac1331d: sixth and seventh independent families add sensor conjunction and stable three-job ordering. Both have equivalent Rust/TypeScript/SEMAPRAX Project ports, copied candidate modules preserved by hidden entry-point overlays, correct public+hidden controls, and plausible wrong candidates that pass public but fail hidden in every port.Actual suite selected 21 tests: 20 initially passed; the cold-chain negative SPX fixture was refused as noncanonical. Corrected the fixture through the canonical formatter, then both cold-chain tests passed on rerun. Thus all 21 cases have passing local evidence; no zero-selection/skip is counted. Independent cheaper-model review checked port equivalence and hidden overlay boundaries. Original paired pilot remains unchanged.
Seven task families do not complete #106. The requested owned-data transformation, stale-edit preservation, backend diagnosis, clean-install families and bounded actual coding-agent trials remain. An attempted eighth cross-module own Bytes → Bytes candidate was rejected by the real Project checker (SPX-G172); the unsupported draft was removed, not counted as execution support. No measured speed/model improvement is claimed.
wsdt commentedon Sep 13, 2026
Pushed 6b6d691 to main: the eighth corpus family, owned-byte-sentinel-balance-v1, performs a real owned byte mapping followed by a positional checksum in Rust, TypeScript and SEMAPRAX Project.
The SPX application consumes owned Bytes, fills an owned output buffer through the admitted bytes_set replacement chain and reads the transformed result. Rust and TypeScript use their own copied buffers. The public borrowed wrapper stays within the admitted cross-module ABI; no unsupported owned export or backend relaxation was added.
Three focused documentation-harness tests passed: correct candidates pass public+hidden in all three real ports; a zero-mapping defect passes public but fails hidden in every port; deleting visible assertions still cannot change hidden rejection. All leak checks pass. Independent direct runner controls also executed the actual ports. Logs: .agent-logs/0913-owned-benchmark-tests.log. No full quality/hosted gate or coding-agent experiment.
The frozen original pilot is unchanged. Eight families alone do not finish #106: stale-edit preservation, backend discrepancy diagnosis, clean-install coverage and bounded actual coding-agent trials remain; no measured language/model improvement is claimed.
test(benchmark): add the stale-edit-preservation held-out task
wsdt commentedon Sep 19, 2026
Progress in
b21a4eb2, plus an accurate count of what the corpus already had.What already existed (five prior pushes,
git log --oneline -- benchmarks/cross-language-v1): eight task families inbenchmarks/cross-language-v1/tasks.json, a documented split protocol and hidden-overlay leak check inbenchmarks/cross-language-v1/docs/METHODOLOGY.md, and real-toolchain coverage intests/documentation/cross_language_benchmark_suite.rsdriving actualrustc/tsc/node/semaprax. Eight families is not eight categories, though: three of this issue's required categories were missing while three others were doubled up.Added:
stale-edit-preservation-v1, the missing category whose absence mattered most. A repair that is correct in itself but clobbers unrelated, already-finished code on its way past is exactly the failure mode no existing task could catch.The task asks for a corrupted-percentage discount to be floored at zero. The candidate module also carries a
stale_notehelper from a notional prior session that the task never mentions. The public tests never call it; the hidden tests call it directly. So a solver that fixes the real bug but rewrites or deletes the untouched helper passes every visible test and fails the hidden phase.Two permanent negative controls, each mutating the committed candidate in all three ports and asserting the exact wrong-candidate shape (
public.passed == true,hidden.passed == false,leak_check == "ok",status == "failed"):use function … stale_note …import no longer resolves.Plus a positive control confirming the committed candidate passes both phases in all three ports.
The suite's existing generic guards (
every_committed_task_declares_a_split_and_the_held_out_task_is_not_development,every_committed_task_language_directory_exists_and_hidden_adds_something) validated the new entry with no change to them.The frozen v1 pilot is untouched —
git statusreports nothing modified underbenchmarks/agent-task-comparison-v1/,scripts/agent-task-comparison.py, ordocs/AGENT-TASK-COMPARISON-V1.md.Still open, stated so this is not read as finished:
benchmarks/cross-language-v1/agent/driver seam remains inert; it has never called a real model.Verified:
--test documentation cross_language_benchmark_suite27 passed, 0 failed against real toolchains. All four new.spxfixtures confirmed canonical withsemaprax fmt --check. No hosted run has covered this commit yet.wsdt commentedon Sep 19, 2026
6e99e278adds the last two categories. Ten task families now cover every category this issue's implementation-step list names — and the more interesting of the two is the one I did not build as asked.telemetry-overflow-diagnosis-v1(diagnosis,held_out). The category asked for is "backend discrepancy diagnosis". A literal interpreter / native-C11 / Core-Wasm result divergence cannot be authored honestly here. AGENTS.md's first non-negotiable invariant is that a safe program has equivalent checked behaviour on every backend that claims to implement the feature — so a real divergence between SEMAPRAX's three backends would itself be a compiler defect (issue #21's class), not a legitimate axis to design a task around. Building one would mean either relying on an unadmitted feature, which is not a discrepancy a real user would hit, or fabricating a bug in this repository's own backends to make a fixture work.EQUIVALENCE.mdsays exactly that rather than shipping a fake split, and it was checked rather than assumed: an overflow probe run through the real interpreter raises a definedsemaprax.arithmetic.v1fault, it does not diverge.What the task encodes instead is the diagnosis skill, portably. One root cause — fixed-width addition not saturated before the sum can leave its declared range — invisible on every public example, surfacing as a different mechanism in each real runtime:
attempt to add with overflowpanic, exit 101{"schema":"semaprax.status.v1","domain_id":"semaprax.arithmetic.v1",…}The correct fix is the same in all three regardless of which symptom the solver happened to observe. That is the skill the category was after.
clean-install-calculator-v1(onboarding,held_out) measures extending a scaffold's own conventions rather than importing a habit from a neighbouring task in this same corpus — specifically, flooring a negative difference at zero, which is correct for the bounded counter here and wrong for ordinary subtraction. ItsAGENTS.md,README.md,semaprax.tomlandsrc/tests.spxare the real output ofsemaprax new --template calculator. I verified that rather than taking it on report: regenerating the scaffold against this checkout's own binary and diffing gives four byte-identical files.Each task ships a positive control and a permanent negative control asserting the exact wrong-candidate shape in all three ports —
public.passed == true,hidden.passed == false,leak_check == "ok",status == "failed".The frozen v1 pilot is untouched (
git statusreports nothing modified underbenchmarks/agent-task-comparison-v1/,scripts/agent-task-comparison.py,docs/AGENT-TASK-COMPARISON-V1.md).One thing remains, and it is the only thing: step 7 — a bounded repeat under an approved model policy. No model was called, no trial was run, and no aggregate or improvement claim is made anywhere in this work. The
benchmarks/cross-language-v1/agent/seam stays inert. Separately,574996ffput that seam's own 23 Python tests into a hosted job for the first time, so its plumbing is now gated even though nothing has ever driven it against a real provider.Verified:
--test documentation65 passed, 0 failed (31 of them the benchmark suite, up from 27). All eight new.spxfixtures confirmed canonical.