Skip to content

Laptop evidence: ultracode-subagent E2E of 13 token/memory/RAG tools; vLLM 0.30.0 embedder switch - #316

Merged
seathatflowsinourveins merged 1 commit into
mainfrom
claude/laptop-e2e-evidence-20260926
Sep 26, 2026
Merged

seathatflowsinourveins merged 1 commit into
mainfrom
claude/laptop-e2e-evidence-20260926

Conversation

@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator

Scope

SOTA sources

Evidence-class table

Claim Class
13 tools ran inside real ultracode workflow children (29 children: 15 Sonnet 5 and 14 Opus 5.5, effort max); each answer was checked deterministically against a redirect-only baseline, and an independent verifier ran per tool native_proven on this host
Verifier verdicts. rtk: refuted (git branch -a '+' worktree marker; the bare git show exclusion misses -C). Serena: check failed (missed cross-file references in .mjs). Repomix: check failed (--compress dropped a def). The other 10: confirmed native_proven
Exact o200k comparisons. Only baseline_kind: counterfactual_raw pairs are read as reductions; check_ground_truth pairs are not savings local_integration
Upstream lifetime counters before and after (rtk, jCodeMunch, Headroom, Context Mode) upstream_estimate: host-wide over the window, not isolated
vLLM 0.30.0 embedder switch. Gates a, c and e pass. b and d keep top-10 set identity but fail strict order in 3 of 4 trials each, with production against itself showing comparable swap totals. The post-switch gate is 30 texts, min cosine 0.9999999999999998 native_proven

Review. An independent evidence-reviewer ran three rounds:

  1. 14 findings: privacy, vLLM overstatement, left-out runs and counter sources. All were fixed.
  2. 10 findings. All were fixed.
  3. A diff-only check, which approved after the swap count was corrected from 20 to 23, as it required.

Checks at the pushed head:

  • validate.py: passed (5997 files, 154 receipts).
  • evidence_manifest --check: passed.
  • tests.test_host_receipts, test_evidence_manifest and test_validate: 232 OK.
  • validate.py --scan-file on the new files: passed.
  • Guarded gitleaks on origin/main..HEAD: no leaks.

The rtk recipe fix these findings led to is in a separate PR.

🤖 Generated with Claude Code

… vLLM 0.30.0 embedder switch

Mirrors the workstation's #296 method on wsl-authoring-20260923 (13 of #296's 16 tasks; 29
children, 15 Sonnet 5 + 14 Opus 5.5 at effort max). Publishes each tool's check result and
independent verifier verdict, including the refuted rtk claim (branch -a '+' marker; the
bare git show exclusion missing the -C form), Serena's missed cross-file references and
repomix --compress dropping a def. Adds a baseline_kind field so only counterfactual_raw
pairs read as reductions. Adds the laptop's vLLM 0.25.0 -> 0.30.0 qualification and switch
receipt, reporting set identity, strict-order failures and the production-vs-production
control totals.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@seathatflowsinourveins seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Sep 26, 2026
@seathatflowsinourveins
seathatflowsinourveins enabled auto-merge (squash) September 26, 2026 02:45
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@seathatflowsinourveins
seathatflowsinourveins merged commit 5d8bd61 into main Sep 26, 2026
24 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the claude/laptop-e2e-evidence-20260926 branch September 26, 2026 02:58
seathatflowsinourveins added a commit that referenced this pull request Sep 26, 2026
…ion form and git branch

The laptop's ultracode E2E verifier (#316) found two gaps on the pinned rtk 0.50.0:
"^git show [^ ]*:" misses `git -C DIR show REV:path` (the blob is windowed), and
filter_branch_output (src/cmds/git/git_cmd.rs 3185-3244) keeps git's '+ ' worktree marker, so
`git branch -a` misreports local branches as remote-only when a same-name remote exists. The
recipe now adds two subcommand-anchored patterns whose global-option group mirrors rtk's
GIT_GLOBAL_OPT (-C, -c, --git-dir, --work-tree, --flags), tells hosts to replace the existing
exclude_commands line, and bootstrap-linux.sh reminds unless the key appears exactly once with
all four entries. Retained raw hook-check on the pinned binary; dated receipt addendum.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins added a commit that referenced this pull request Sep 26, 2026
…ion form and git branch (#318)

The laptop's ultracode E2E verifier (#316) found two gaps on the pinned rtk 0.50.0:
"^git show [^ ]*:" misses `git -C DIR show REV:path` (the blob is windowed), and
filter_branch_output (src/cmds/git/git_cmd.rs 3185-3244) keeps git's '+ ' worktree marker, so
`git branch -a` misreports local branches as remote-only when a same-name remote exists. The
recipe now adds two subcommand-anchored patterns whose global-option group mirrors rtk's
GIT_GLOBAL_OPT (-C, -c, --git-dir, --work-tree, --flags), tells hosts to replace the existing
exclude_commands line, and bootstrap-linux.sh reminds unless the key appears exactly once with
all four entries. Retained raw hook-check on the pinned binary; dated receipt addendum.

Co-authored-by: seathatflowsinourveins <[email protected]>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins added a commit that referenced this pull request Sep 27, 2026
…ta from the 2026-09-27 verdict wave (#401)

* E1/E2 token E2E receipts: dated adjudications, errata and reproduced answer oracles

Applies the 2026-09-27 cross-family GPT-6 review of the E1 (#296) and E2 (#316)
receipts, append-only: every recorded field keeps its value, and each receipt keeps
its own serialization.

- Every tools[] row gains adjudication {as_of, task_acceptance, bad_input_control_recorded,
  basis, evidence}, using the review's status table and vocabulary (partial,
  retracted_later, untested, unsupported).
- E1: QMD retracted_later (wrong_document, corrections_to_296); Repomix retracted_later
  (count=47 PASS on a 48-function file; repomix v1.18.1 PythonParseStrategy.ts L81-86
  tests only a def's first line); ai-memory untested (base_empty=True is a vacuous pass);
  the other thirteen partial with no recorded failing control. A dated errata block
  names the pre-errata sha256 the preregistration records as a historical identity.
- E2: context-mode, jcodemunch, qmd and ast-grep stay partial with no recorded failing
  control; SocratiCode's comparison counts an abridged transcription; RTK retracted by
  its verifier; Repomix's FAIL accurate.
- READMEs carry dated corrections; the review is retained, sanitized, beside E1.
- tests/test_token_e2e_receipt_checks.py: receipt contracts (red before the errata) and
  answer oracles rebuilt from pinned Git blobs, hash-matched to the recorded baselines,
  each with failing controls (red under a weaker historical-style check).

Sources: docs/acceptance-evidence-policy.md (Discriminating controls);
https://github.com/yamadashy/repomix/blob/v1.18.1/src/core/treeSitter/parseStrategies/PythonParseStrategy.ts#L81
evidence/artifacts/token-e2e-codex-20260926/receipt.json#/corrections_to_296.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Token layer: routing rules and recipe limits for known upstream defects

Closes the documentation, recipe and routing gaps from the 2026-09-27 per-tool
verdict wave for the token-efficiency layer. Every rule cites the pinned upstream
source or a committed receipt; no pin changes.

recipes/README.md (token-tool rows and sections):
- agentsview: v0.44.0 unqualified; worker populations need --include-children,
  --include-automated and --include-one-shot, and --fts reads message bodies only
  (v0.43.0 docs/session-api.md, docs/commands.md); archive answers are observation.
- ast-grep: every 0.45.3 command registers customLanguages from any sgconfig.yml in
  the working directory or a parent (lib.rs L107-139, config.rs L98-131/L280-299);
  PR #2960's opt-in is unreleased; outline (#2957) and YAML rule (#2963) limits.
- ccusage: token-only while any model is unpriced (v20.0.24 json-output.md
  Unpriced Models; config-files.md pricingOverrides).
- codebase-memory: trace_path excludes tests and evidence by default, search_code
  hits can be mentions (v0.11.0 src/mcp/mcp.c schemas); caller-set procedure.
- context-hub: check metadata.versions (FastAPI guide 0.136.3 vs 0.141.1).
- context-mode: npm tarball digest and plugin commits are separate identities.
- headroom: v0.39.1 changes only the proxy limiter, so the omission-count defect stands.
- markitdown: HTML as .txt passes through; -x html / -m text/html; selective extras.
- mcporter: --name does not select the server; a positional token becomes the
  selector (v0.14.1 call-arguments.ts L164-192, ephemeral-flags.ts L97-105); the
  jCodeMunch example gains --server.
- qmd: MCP query expands and reranks by default (v2.8.3 README, server.ts); typed
  lex + rerank:false + bounded get; catalog scope; refresh guard for #989/#991
  (src/store.ts reindexCollection).
- repomix: multi-line Python signatures are dropped (v1.18.1 PythonParseStrategy.ts
  L81-86); exact-definition tasks go to an uncompressed pack or the original.
- toon: BOM root-string round trip (#339); no record-count threshold upstream;
  nested-uniform columns are tabular (tabular.ts L61-78).

docs/token-session-handbook.md: a "Known upstream limits behind the lanes" section
outside the carrier text (every carrier line unchanged), and row pointers.
docs/token-practice.md: counts, comparisons and acceptance rules (invocation counts
are not success rates; comparisons need task acceptance; count recovery reads;
gpt-tokenizer's special-token contract; receipts carry adjudications).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Apply the GPT-6 review of this branch: exact inventory oracle, TOON eligibility wording

One cross-family review round (GPT-6 through codex-omniroute, read-only) returned
DEFECTS FOUND with two findings; both are fixed:

- P2 tests/test_token_e2e_receipt_checks.py: inventory_oracle checked only the claimed
  count plus subset membership, so a correct count with missing, duplicated or no names
  passed. It now requires the exact name set with one entry per function. New controls keep
  the correct count while dropping, duplicating or inventing a name, or listing none; they
  failed against the old oracle first (red) and pass now.
- P3 docs/token-session-handbook.md (and the matching recipes/README.md toon row): "accepts
  any non-empty uniform array" overstated TOON 4.1.1's tabular eligibility. The encoder
  needs non-empty objects sharing one key set, with columns of primitives or, recursively,
  of non-empty objects sharing one key set; an empty object or an array-valued column falls
  back to list form (packages/toon/src/encode/tabular.ts L6-78 at v4.1.1; confirmed with the
  installed 4.1.1 CLI on [{}], [{"a":[]}], [{"a":{}}] and [{"a":{"b":1}}]).

Also: the ast-grep oracle keys counts by path under scripts/ rather than file stem, and E1's
Context Mode adjudication states that attempt 1's marker-grep FAIL on a refused call is not
shown to be a run of attempt 2's answer check (the receipt does not record that command).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Apply the independent verifier's findings: TOON source line, citation anchors, E2 Serena erratum

The independent verifier of this branch reported five defects; all are fixed:

- docs/token-session-handbook.md, the TOON Sources line under the carrier (on main
  before this branch) still said "The encoder accepts a non-empty uniform array",
  the wording GPT-6 finding P3 refuted. It now states the encoder's rule from
  packages/toon/src/encode/tabular.ts L6-78 at v4.1.1 and links the handbook's
  "Known upstream limits behind the lanes" section. It is not a "- " carrier line,
  so tests/test_token_lanes_subagent_start.py's verbatim-line check is unaffected.
- Citation anchors: SocratiCode's INCLUDE_DOT_FILES default is documented under
  README.md "Indexing Behaviour" (v1.14.0 L1574-1579), not "Ignore Rules", so the
  handbook links #indexing-behaviour. codebase-memory-mcp v0.11.0 search_code gets
  its own link to src/mcp/mcp.c#L632-L659 in the handbook, and the recipe gives
  index_repository (L466-481), trace_path (L532-560) and search_code (L632-659)
  one link each. Both upstream files matched the retained copies by sha256 on
  2026-09-27.
- E2 receipt errata.items[4] and the E2 README erratum now name Serena's accurate
  reference FAIL (find_referencing_symbols missed 6 of 8 call sites; the definition
  check passed) and give each group of partial rows its own reason. Only that one
  finding string changed in receipt.json (same serialization, 1 line); written_at
  is unchanged because this refines the unmerged 2026-09-27 erratum on the same day.
- tests/test_token_e2e_receipt_checks.py: the require_commit docstring now says the
  guard is modelled on RetainedEvidenceTests' Git check and adds the pinned-commit
  probe (tests/test_adoption_status.py L1721-1724 checks only git and the checkout).

Sources: toon-format/toon v4.1.1 packages/toon/src/encode/tabular.ts;
giancarloerra/SocratiCode v1.14.0 README.md; DeusData/codebase-memory-mcp v0.11.0
src/mcp/mcp.c; the E2 receipt's own serena row (tools[5]) and README.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

---------

Co-authored-by: Scout <scout@local>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant