bench: record native tool-selection reference-token capture - #14

No linked issue in this repository. Companion evidence for aictrl-dev/cli#246; this PR does not close that issue.

Intent

Make the unreleased CLI tool selector's actual same-binary capture reviewable: five fresh default runs and five selected runs completed the same scripted three-Read task. Cumulative reference input tokens were 15,150 versus 9,390 (38.02% lower); output was 59 in both arms.

Expected Impact on Users

No direct CLI behavior changes. Developers evaluating the tool selector can inspect raw requests, actual native read events, reference-token accounting and source/build provenance for five default and five selected runs. This makes the 38.02% input-reference-token reduction on the scripted short task auditable instead of relying on a headline. The captured binary is not distributed, and the result does not establish coding quality, provider charges or general workload savings.

Expected Outcomes

• The exact nine-tool default and four-tool selected catalogs, four main calls and three actual Reads per run are inspectable.
• Bound manifests identify the tested native bytes and source; original capture and historical benchmark files retain their hashes.

Implementation

Adds the selector capture adapter, offline counter wrapper, 59 actual positive artifacts and concise protocol/provenance/QA notes. Source correspondence points to CLI commit 2db98b3, based on guidance PR252. The binary was compiled from pinned reviewed bytes before that commit; the source owner verified product hashes after normal hooks.

Scope Caveat

Scripted non-GPT fixture, permissive permissions, 50ms stub latency, tiktoken 0.12.0/o200k_base. This is offline reference-token evidence, not coding quality, provider billing/cache savings, speed, general task equivalence or release readiness. Native binary, local base/image, source/dependency snapshot and private build driver are not distributed. Original local provenance paths remain in manifests. Host networking does not provide an OS egress sandbox. Separate local fault-only campaigns are summarized, not included among positive samples.

Test Plan

• Ten actual native executions: 40 raw main request bodies, 30 distinct Read executions, exact catalogs, all exits 0/Done, no failed/excluded positive samples.
• Offline token accounting and range checks: all five measures per arm equal their medians; acceptance passed.
• Real native wrong-catalog, failed-Read and unchanged 180-second deadline controls rejected; independent final exact-label Docker queries were empty.
• Scoped strict TypeScript/noEmit and Python parsing, input/hash/build-cleanup/output guards passed.
• Public packaging audit: all 89 historical benchmark files and all 59 capture artifact bytes unchanged; text and decompressed raw bodies checked for credential patterns.

Verification

Native SHA256: c60d37a2d91de46abbbddccf10fdb0bb0c4940df9c2295ce42ba715344986874.

Campaign SHA256: 0c10185781dfbc4a21e5720d55641a35b31ff97243e78060a9dc1fc8c998c404.

Reference accounting SHA256: 5867bac1e4a5824fb06358eb148c21e7581a8bcfdc34b06beb8cd34ce768f96c.

CodeQL Analyze (javascript-typescript), Analyze (python) and aggregate CodeQL checks passed on exact head cc47b1e5f5277de19c222f7216081110571c5504 (formal poll 21 October 2026 02:51:21 UTC).

git diff --cached --check passed before commit. This PR retains existing results and changes no historical runtime files. No linked milestone; no merge or release is requested.