Skip to content

Gate A U1: command-position measurement layer on the pinned tree-sitter-bash parser - #516

Merged
seathatflowsinourveins merged 84 commits into
mainfrom
claude/pra-u1-cmdpos-2d-20260928
Sep 29, 2026
Merged

seathatflowsinourveins merged 84 commits into
mainfrom
claude/pra-u1-cmdpos-2d-20260928

Conversation

@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator

Scope

  • What this PR changes, in one or two sentences: moves the command-position layer of the measurement kernel (commandInvocations, cli_lanes, the rtk proxy reading and the Codex adapter) from hand-written shell scanners to the upstream tree-sitter-bash parser (pinned, hash-verified and fail-closed), repairs the shell shapes the parser reads wrongly (each gated by real bash), and adds the oracle, the differential evidence and the decision record. It does not change the M4 fetch detection.
  • Base commit: ed293987
  • Lane: lane:foundation (Gate A unit U1; owner native-agent-stack-2d)
  • Owned paths touched: examples/claude-native/workflows/{child-usage.mjs,test-child-usage.mjs,shell-parser.pin.json,README.md,SHA256SUMS}, tools/skill-usage/{skill_usage.py,README.md}, tests/test_token_measurement.py, tests/test_skill_usage.py, tests/test_child_usage_suite.py, tests/test_command_position_oracle.py (new), evidence/artifacts/pra-u1-differential-20260929/ (new), docs/decisions/2026-09-29-shell-command-parser.md (new), manifests/evidence.json (hot file, last commit)

SOTA sources

  • openai/codex rust-v0.157.1 (36650394c5b38c2990ccf2a3457165ca3e9d9726): codex-rs/shell-command/src/bash.rs:13,124,140,162 (Codex itself parses shell text with tree-sitter-bash and rejects a tree with an error whole), codex-rs/Cargo.toml:522 (tree-sitter-bash = "0.25"), codex-rs/core/src/tools/context.rs:389,535 (exit-code header).
  • tree-sitter/tree-sitter-bash v0.25.1 (tag a06c2e4415e9, npm gitHead 801326684a26): grammar.js:528-540, src/node-types.json; tree-sitter/tree-sitter v0.27.0 (6070dbfefd32) and web-tree-sitter 0.27.0: docs/src/using-parsers/4-walking-trees.md, lib/src/node.c, lib/binding_web/src/tree_cursor.ts. Pin, integrity values and the six file digests: examples/claude-native/workflows/shell-parser.pin.json (npm registry and GitHub API lookups of 2026-09-29).
  • openclaw/mcporter v0.14.1 (93e0916cafe2): src/cli.ts:137-145, src/cli/flag-utils.ts, src/cli/help-output.ts, src/cli/cli-factory.ts, src/cli/call-arguments.ts, src/cli/ephemeral-flags.ts. rtk-ai/rtk 1d87b8e7: src/main.rs:68-90,3008-3066, src/core/utils.rs:615-632 (rtk proxy is an execvp, not a shell). mksglu/context-mode v1.0.169: hooks/core/routing.mjs:788-795 (the routing detector the M4 reading mirrors).
  • POSIX.1-2024 XCU 2.6.3, 2.7.4, 2.9.1-2.9.4; bash(1) 5.2; OpenSSH ssh(1) 9.6p1. Alternatives compared and rejected (sh-syntax, mvdan-sh, bash-parser, shell-quote, continued scanner patching): docs/decisions/2026-09-29-shell-command-parser.md, table "Alternatives".

Evidence-class table

Claim Evidence class Command / receipt
The tree reading agrees with real GNU bash 5.2.21 on 163 of 163 hand-built probes, 28 of 28 recovery probes, 456 of 456 generated and 463 of 463 extended commands (a stub PATH records what a shell runs) native_proven (real shell execution) tests/test_command_position_oracle.py
75 of 75 M4 fixtures agree with real bash, dash, the documented curl count and the kernel; a control whose stub logs nothing makes 67 of 75 disagree and exits 1 native_proven evidence/artifacts/pra-u1-differential-20260929/m4-real-shells.py
The final kernel reads 217,428 inputs (136,361 real commands, 40,000 soup, 20,000 heredoc, 20,000 lane, 1,067 covering) exactly like the stage-a kernel, and the differential against the scanner reading classifies every difference by mechanism local_integration (private host commands reduced to counts, no text) counts.json, README.md of the differential directory
First overturn condition of the decision (a grammar-limit shape in 0.1 percent or more of lane-bearing commands): NOT met, an upper bound of 5 of 6,527 (0.077 percent) against a threshold of 7 commands, by 2 commands local_integration counts.json shapes.overturn_1, bound to the record by a test
Kernel and adapter suites pass with the parser installed local_integration see commands below
The pinned parser loads only when the six file digests and both npm integrity values match; a flipped byte gives hash_mismatch; no parser gives parser_unavailable and no lane is counted local_integration (mutation controls) examples/claude-native/workflows/test-child-usage.mjs
Independent review: Opus evidence-reviewer 1 medium and 8 low; independent command re-run: stack-verifier, 28 verdicts: 26 pass; the 2 non-passes are a whitespace defect in a file that came from main (#409) and is not in this PR's range after the merge, and the documented ten-digit exit-code limit (the comments were corrected) source_review Gate A workflow record
GPT-6 cross-family review NOT run (native and pooled GPT-6 capacity exhausted); recorded as a missing round -

Local commands run

$ python3 -B -m unittest tests.test_token_measurement tests.test_child_usage_suite tests.test_skill_usage tests.test_command_position_oracle
Ran 209 tests in 151s / OK (0 skipped, parser installed)
$ node test-child-usage.mjs (examples/claude-native/workflows)
SUMMARY passed=174 failed=0 total=174   (test-contract-mutations 74/74, test-usage-receipts 18/18, test-envelope 254/254, test-codex-envelope 0 FAIL)
$ CHILD_USAGE_SHELL_PARSER=<nonexistent dir> python3 -B -m unittest <the four modules>
Ran 209 tests / OK (skipped=40: the lane tests skip, the fail-closed parser tests run)   <- what CI proves until the provisioning follow-up lands
$ sha256sum --check --strict SHA256SUMS
14 of 14 OK
$ python3 scripts/validate.py
{"status": "passed", ...}
$ git diff --check ed293987..HEAD
(no output, exit 0)

Residuals (bounded review loop: one review, one repair round; each is documented in the decision record with counts from this host): (1) CI does not install the parser yet, so there the lane tests skip and only the fail-closed gate is proved; the provisioning step is a separate PR (U1d, stacked next) with its own hosted run; (2) a run of unclosed a=( or << operators is quadratic inside tree-sitter-bash's own parse (about 12 s at 64,000 characters; this host's largest command has 49,096 characters and no real command reaches it); a parse budget is a follow-up; (3) bash -c "$(cat <<'EOF' ... EOF)" is a documented blind spot (5 real commands); (4) ten-digit exit codes read unknown, never succeeded (the brief fixed nine digits; no Windows Codex was available); (5) 209 host-hook refusal rows still read as failed calls; (6) the Python M4 mirror keeps the pre-N1 reading and disagrees with the kernel on the documented shapes; (7) the node timing checks are ratio-based with retries and could flake on a loaded runner; (8) the pin records the tag commit a06c2e44 while the npm package was published from 80132668 (decision 6 of the record); (9) the first independent command re-run agent finished without calling StructuredOutput and counted as failed; the repair stage re-ran the acceptance list and an automatic independent re-verify followed; (10) the stage-b commits (b907cc17 to 2bad7320) came from a run that returned no handoff and were accepted through a later read-only verification; one early subject is 75 characters (history was not rewritten; the PR is squash-merged).

Decision record

docs/decisions/2026-09-29-shell-command-parser.md (alternatives, the decision, its limits with counts and the overturn conditions; the first was evaluated in the repair round and is not met).

Host evidence

Not applicable (no files under evidence/hosts/).

Checklist

  • New/changed GitHub Actions are pinned to a full commit SHA with a version comment (no workflow changes in this PR).
  • New/changed workflows declare top-level permissions: contents: read (no workflow changes).
  • No secrets are printed, logged or committed; the differential publishes counts only and its scripts read host commands privately.
  • No new paid hosting, subscription or billing surface was introduced.
  • Peer-owned untracked files and worktrees were preserved.

🤖 Generated with Claude Code

Scout and others added 30 commits September 28, 2026 23:42
N1 is the #432 fixup3 residual: openers() does not track "$(" inside
double quotes, and a quoted string that a shell runs is appended
without its own heredoc resolution, so heredoc data reads as executed.

Adds three tests through the existing measure/exports bridges on all
three shell carriers plus fetchKind:
- p1 (git commit -m "$(cat <<'EOF' ...)" with a line-start curl), its
  body variants with ', " and ( ), p2 (gh api in the body), p3
  (bash -c 'cat <<EOF > x.sh ...') and a double-quoted p3 variant;
- must-stay controls: an executed curl in "$( )", a shell heredoc in
  "$( )", an escaped substitution in a double-quoted run string, the
  command after a heredoc in "$( )" (guards against resolving one
  heredoc twice), the outer-shell heredoc in a double-quoted run
  string, unquoted $( ) and the "Subject (scope)" body;
- the nested-quote case from design 1.5.

At cf3fb72 the new tests fail 23 subtests (p1, the ' variant, p3 and
the double-quoted p3: shell fetch 1 != 0 and fetchKind 'fetch'; p2:
unclassifiable 1 != 0; nested quotes: 0 != 1 and fetchKind None); the
must-stay test passes. Source: POSIX.1-2024 XCU 2.6.3 Command
Substitution, checked with GNU bash 5.2.21 and dash.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The N1 fix analyzes a string that a shell runs (RUN_QUOTED) as that
shell's input in both modes, instead of appending it verbatim unless
inlineHttp is set. Two consequences get their own tests first:
- in a double-quoted run string the outer shell removes the backslash
  before " (POSIX.1-2024 XCU 2.2.3), so `bash -c "echo \"a; curl
  ...\""` runs only echo: base reads a confirmed shell fetch;
- a string the inner shell runs in turn is analyzed too, so the curl
  in `ssh host 'bash -c "curl ..."'` is confirmed: base counts it in
  neither the confirmed nor the unconfirmed fetches, so the gate's
  lower bound could not see it.

Against the cf3fb72 kernel the test fails 7 subtests (1 != 0 and
0 != 1 on each carrier, fetchKind ['fetch', None]). GNU bash 5.2.21
and dash print `a; echo RAN` for the escaped form.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Fixes the #432 fixup3 residual N1 in child-usage.mjs. POSIX.1-2024 XCU
2.6.3: inside double quotes the text from "$(" to the matching ")" is
tokenized recursively; 2.2.3: in a double-quoted word the outer shell
removes the backslash before $ ` " \ and newline. GNU bash 5.2.21 and
dash agree on every shape below.

- step() is one frame machine (', ", `, and $( with depth) shared by
  phase 1 openers() and the phase 2 closeQuote()/matchParen(), so the
  phases agree on where a quote or substitution ends. With no frame
  open it reads exactly as the old openers(); only a double-quoted
  word gains $( and backquote frames, so a << inside "$( )" is found
  and its body resolved like any other heredoc.
- executedTrace splits into resolveHeredocs (phase 1) and scanQuotes
  (phase 2). The naive end-of-quote scan becomes closeQuote; a "$( )"
  body in double-quoted data is spliced as "$(" + scanQuotes(body) +
  ")" with exact offsets (no second phase 1 over it).
- A string a shell runs (RUN_QUOTED) is analyzed with the full
  executedTrace in both modes, a double-quoted one after unquoted()
  removes the outer shell's escapes at its own level. A shared set of
  resolved operator offsets keeps a heredoc the outer shell resolved
  in "$( )" from reading later lines as its body again.
- NESTING_LIMIT (32) bounds recursion: deeper analyses read as data.
  Without it '"$(' x 3000 and a 3000-deep shell heredoc chain throw
  RangeError, which would abort a sweep.
- The paren-free DOUBLE_QUOTED_DATA regex is removed; the scanners are
  linear character scanners (no new regex; CodeQL js/redos).

The failing-first tests from 1d206a12 and 54299c44 now pass; the
must-stay controls are unchanged. SHA256SUMS is regenerated in a
later stage.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The workflows README replaces the N1 limit paragraph with the rule the
kernel now follows: "$( )" inside double quotes is read recursively
(POSIX.1-2024 XCU 2.6.3), a string a shell runs is analyzed as that
shell's input after the outer escapes are removed (XCU 2.2.3), and
reading text as data moves a raw detector match to the unconfirmed
count instead of dropping it. It states the remaining limits:
backquoted spans inside double quotes, case patterns inside "$( )",
the 32-level nesting bound, and text bash rejects as incomplete.

tools/skill-usage/README.md notes that the legacy Python executed_text
rule predates this reading and can disagree on these shapes.

test-child-usage.mjs adds fetchKind and exact executedText cases for
the N1 shapes and a nesting witness. Against the cf3fb72 kernel the
three new cases fail (95/98); with NESTING_LIMIT removed the witness
fails (97/98, RangeError); with the fix all 98 pass.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The previous wording said reading text as data leaves the denominator
of routed_share_lower_bound unchanged. That holds for raw detector
matches only. Measured against the cf3fb72 kernel:
- `echo "$(echo "a" && curl http://127.0.0.1:9/)"` moves from
  unconfirmed to confirmed loopback, which leaves the denominator;
- `echo "$(\curl ...)"` gains a confirmed fetch no raw detector sees;
- `echo "$(echo " ; \curl ...")"` loses a confirmation the old
  reading created (bash prints it as data), so the bound can rise.
The README now states those three cases. It also adds a limit: a
backquoted span outside quotes is still read as plain command text,
so a quote it leaves open runs past its closing backquote.

test-envelope (254/254), test-usage-receipts (18/18) and
test-contract-mutations (74/74) pass; they read this README as
routing_doc and contract_doc.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Add the U1 design section 7 fixtures before the kernel change. The
Python integration cases feed measureTranscript, aggregateMeasurements
and the sweep CLI: the stack.json lane commands (:1003, :797, :579, :762,
:519, :629, :280, :407), wrappers, compound commands and substitutions,
runners, every mcporter call form, negatives (data, lookups,
registrations, near names, gcm and serena-hooks), version and help,
remote and unresolved programs, the call states, the rtk proxy
population by command position with prefix_rule_calls, M3 and M4
by_carrier, aggregation with actors_with_success, and ID-free sweep
output. The node suite adds commandInvocations unit cases.

Sources: #381 AA-PLAN PR-A item 3 and M6/M14; POSIX.1-2024 XCU 2.9.1-2.9.4
and 2.6.3; rtk-ai/rtk@1d87b8e7 src/main.rs:68-90,708-713,3008-3042 and
the installed rtk 0.50.0 (proxy -- and --skip-env bind, -v after proxy
is the program); openclaw/mcporter@93e0916c src/cli.ts:115-364,
src/cli/command-inference.ts:10-95, call-arguments.ts:79-233 and
call-command.ts:114-173,309-348 (a first positional with = is the
selector). Before the fix: 10 Python tests fail (102 subtests) and the
8 new node cases fail on the missing export.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Export commandInvocations: every simple command of the executed text
(POSIX.1-2024 XCU 2.9.1-2.9.4, 2.6.3) is resolved past reserved words,
assignments, wrappers (timeout, env, nice, stdbuf, command, exec, time,
nohup, sudo 1.9.15p5, xargs), package runners and rtk proxy to a lane
executable by exact basename. mcporter operations and a call's
downstream server follow openclaw/mcporter@93e0916c (cli.ts,
command-inference.ts, call-arguments.ts, call-command.ts); HTTP and
stdio selectors read (http) and (stdio), and no host is emitted. rtk
proxy follows rtk-ai/rtk@1d87b8e7 src/main.rs:68-90,708-713,3008-3042
and the installed rtk 0.50.0 (no shell runs the proxied program).

One memoized reading per call drives carrierOf (M3 and M4 by_carrier),
measurement.proxy (adds invocations, nested, in_ctx_code and
prefix_rule_calls) and the new measurement.cli_lanes with per-call
states (AA-PLAN "Successful" and M14; not_executed from observed
transcript markers). aggregateMeasurements sums the fields and adds
actors_with_success; the sweep limits gain the static limits.
executedTrace takes an optional marks collector for ssh-run text and
data-heredoc substitutions; its output is unchanged (0 differences in
executedText and fetchKind over 24,489 inputs).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Replace the character-class regexes in httpUrl and httpToolSelector,
the path-prefix alternation for an ad-hoc stdio selector and the
blank split of an inline npx server with character checks and the
existing shellWords splitter. The brief asks for linear character
scanners rather than regexes in this kernel (CodeQL js/redos flagged it
before). Behaviour is unchanged: the rules still follow
openclaw/mcporter@93e0916c src/cli/http-utils.ts:1-69,
call-argument-values.ts:68-83 and ephemeral-target.ts:134-158, and the
node suite passes unchanged (107/107).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Add a CLI-lanes subsection to the PR-A measurement fields: the lane
executables with the gcm and serena-hooks exclusions, the resolver
(wrappers, runners, rtk proxy), each cli_lanes field, the mcporter
operation and downstream-server precedence of mcporter v0.14.1, the
call states with the observed not-executed markers and their M14
mapping, the static limits, and the changed meanings of proxy.calls,
M3 by_carrier.rtk_proxy and m4.by_carrier with prefix_rule_calls kept
for #369-era comparison. Record the decision that proxy.in_ctx_code
stays outside M6 (Gate A plan M6 and rtk rows) and state that
rtk_parts, proxy_parts and model_typed keep their prefix rules. The
skill-usage README points Codex readers to the same fields and states
that declined items and legacy exec headers still read succeeded until
the adapter maps them.

Sources: rtk-ai/rtk v0.50.0 src/main.rs:68-90,3008-3042; openclaw/mcporter
v0.14.1 src/cli and docs/call-heuristic.md; mksglu/context-mode v1.0.169
src/exit-classify.ts:15-33; POSIX.1-2024 XCU 2.9.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
U1 design 2.2 reads only the POSIX exec form and marks any word that
starts with '-' after exec as unresolved, because bash's exec options
are not pinned. The shells disagree on `exec -- cmd`: GNU bash 5.2.21
runs cmd, while dash tries to execute "--" (probe on this host). The
kernel currently takes `--` as the end of options and reads qmd.
Observed before the fix: "exec -- qmd mcp" => ["qmd/qmd"].

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Follow U1 design 2.2: only the POSIX exec form is read, so a word that
starts with '-' after exec, `--` included, leaves the program
unresolved. bash 5.2.21 runs the command after `exec --` while dash
executes "--" itself, so neither reading is safe to assume. The failing
case from 1fcc9f23 now passes (node suite 107/107).

Also cover the inline npx server of an mcporter call
(openclaw/mcporter@93e0916c src/cli/ephemeral-target.ts:134-158): a
--server value that is an npx command line is an ad-hoc stdio server,
and another blank-holding name reads (other). These cases were added
after the implementation, so they are coverage, not failing-first.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
U1 design 6 fixtures (i)-(vii) run rollout records through
measure_codex_records, the Node bridge and the kernel's cli_lanes and
proxy. Record shapes follow openai/codex rust-v0.157.1 (36650394):
CommandExecutionItem and its snake_case status (protocol/src/items.rs
:199-285), exec end states (core/src/tools/events.rs:529-573: a
rejection is declined with exit -1) and the unified exec response
header (core/src/tools/context.rs:524-575). A rollout output never
carries the success flag (protocol/src/models.rs:2173-2182).

Before the adapter change 6 of 8 subtests fail:
- (iii) code-mode item declined: qmd succeeded 1, expected failed 1
  and not_executed 1; the same for a direct call's declined item;
- (iv) legacy exec_command with "Process exited with code 1":
  rtk_proxy succeeded 1, expected failed 1;
- (v) "Process running with session ID 3": qmd succeeded 1, expected
  unknown 1;
- (vi) no header and no item: qmd succeeded 1, expected unknown 1;
- (vii) local_shell_call running stack.json:280:
  codebase-memory-mcp succeeded 1 and downstream codebase-memory
  succeeded 1, expected unknown 1 in both.
(i) nested rtk proxy (proxy.calls 1, proxy.nested 1, succeeded 1) and
(ii) a failed item already pass. Must-stay controls pass: an exit 0
header, an exit line inside the output, a completed item that beats
a running header, a failed item with its header, and the aggregate
passing cli_lanes and proxy through with actors_with_success.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
U1 design 6: the Codex adapter now hands the kernel's callState what a
rollout shows about a shell call's outcome, so cli_lanes stops reading
declined or failed legacy commands as succeeded.

- A declined CommandExecution (openai/codex rust-v0.157.1 36650394
  core/src/tools/events.rs:562-573, exit -1) sets is_error and
  native_state "declined" on both result paths: the call's own
  function_call_output and the item's aggregated output. The kernel
  counts it failed and not_executed.
- A Bash-mapped call (exec_command, shell_command, shell or
  local_shell_call, chosen by call kind, never by output text) with no
  persisted item state is read by the unified exec header of
  core/src/tools/context.rs:524-548, and only by the lines before
  "Output:", in a linear line scan with a strict ASCII exit code.
  Exit 0 is success, another exit is_error. A running line (it wins
  over an exit line, unified_exec/process_manager.rs:1066-1071), a
  header with neither line or text without the header sets
  native_state "unknown": a rollout output never carries success
  (protocol/src/models.rs:2173-2182), and legacy history mode keeps no
  item (rollout/src/policy.rs:94-112). No shell, shell_command or
  local-shell handler exists at that revision, so those outputs read
  unknown unless an item state decides; apply_patch's "Exit code: N"
  text is not read.
- codex_call_name shares the MCP namespace rule between the call loop
  and the shell-call set, so an MCP tool named exec_command is not a
  shell call.

The failing-first fixtures from 4a7aa75a now pass (the three Python
suites: 160 tests OK). The header unit test
(test_exec_header_state_reads_only_the_pinned_header) was written
after the parser and is coverage, not failing-first. The lanes CLI
still writes nothing to stderr, and the Codex aggregate passes
cli_lanes and proxy through unchanged.
tools/skill-usage/README.md replaces the stage B sentence that said
these calls still read succeeded.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The N1 fix and the CLI-lane reading changed child-usage.mjs and
test-child-usage.mjs; stage C leaves the kernel byte-identical to
0c421c66 (sha256 acc7bb51...). Regenerated with the documented
command, `sha256sum -- *.mjs *.js *.json > SHA256SUMS` in
examples/claude-native/workflows, and `sha256sum --check --strict
SHA256SUMS` passes for all 13 files. Only those two digests change.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The hot-file protocol (docs/lanes.md:94-103) puts the shared manifest
edit in the branch's last commit and re-registers every changed file
that the manifest already lists, with host_receipts.register_file.
scripts/validate.py checks SHA-256 and byte counts for each of them
and failed before this commit ("SHA-256 mismatch", "byte count
mismatch" for the workflows README, child-usage.mjs,
test-child-usage.mjs, tests/test_token_measurement.py and
tools/skill-usage/README.md). The eight re-registered entries are the
two READMEs, SHA256SUMS, child-usage.mjs, test-child-usage.mjs,
skill_usage.py, test_skill_usage.py and test_token_measurement.py.
Only their sha256 and bytes change, and validate.py now passes
(status passed, 7986 hashed files).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Tests only, run against 7091a30 before any fix (U1 pivot brief D1, D6-D9,
D11; GPT-6 findings #8, #9, #11, #12, #13; Claude review R1, R2, R3, R9).

- D1 loader (test-child-usage.mjs, test_skill_usage.py): pin file contents,
  loadShellParser statuses (installed, not_installed, hash_mismatch),
  directory order, --shell-parser, commandInvocations null and cli_lanes
  parser_unavailable with proxy prefix_fallback, aggregate statuses,
  withShellTree freeing its tree, and the Node bridge awaiting the loader.
- D6: shell_script quotes argv with shlex.join, and -c is a script only for
  a shell; exec_header_state bounds the exit code to 9 digits (GPT-6 #11,
  #12 inputs verbatim).
- D7: outer-shell substitutions in a double-quoted run string are executed
  text (GPT-6 #8 verbatim), with single-quote, comment, backquote, sh, eval
  and ssh variants and must-stay controls that count once, not once per view.
- D8: n = 8000..64000 doubling for unclosed ((, $((, "$( ((, and ( << E
  runs, at most 2.5x per doubling and under 150 ms at 64000.
- D9: the stress check lets an exception fail it, includes an unquoted
  '$('.repeat(3000) input (NESTING_LIMIT), and a mutation control (a reader
  that throws for long input) shows the check now fails.
- R1, R3: a heredoc after a closed "$( )" keeps its reader; an escaped blank,
  semicolon or newline before a # is not a comment start.

Every expected value of the D7, R1 and R3 commands was run under GNU bash
5.2.21 and dash with a stub curl (and a stub ssh that runs sh on its stdin,
the remote login shell): both shells agreed on every count.

At 7091a30: node suite 115 passed, 31 failed (5 linear, 26 parser); D7 9
commands, R1 5, R3 3 fail with 0 != 1 on all three carriers; exec_header_state
raises ValueError on 5000 digits; shell_script flattens argv. The old D9
assertion passes when 7 of 7 stress inputs throw.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
U1 pivot brief D6 (GPT-6 findings #11 and #12).

shell_script joined an argv array with spaces, so a metacharacter inside one
element became shell syntax that was never executed: ["echo", "qmd; rtk proxy
qmd status"] counted one qmd call and one rtk proxy call. It now joins with
shlex.join (Python 3 shlex.join, the inverse of shlex.split), so each element
is one quoted word. The [shell, -lc, script] form (openai/codex rust-v0.157.1
codex-rs/core/src/shell.rs) stays the script itself, but only when the program
is a shell: -c of `echo -c ...` is one of echo's arguments.

exec_header_state passed the exit code text to int() whenever it was ASCII
digits, and CPython raises ValueError above 4,300 digits (int max str digits,
docs.python.org/3/library/stdtypes.html#int-max-str-digits), which stopped the
scan on one malformed header. The code is an i32 written in decimal (at most 10
digits with its sign, codex-rs/core/src/tools/context.rs:534-540); a code of
more than 9 digits is no native header and reads (False, "unknown").

The D6 tests of the previous commit pass: exec_header_state on 5000 digits,
the shell_script cases, and the three Codex bridge cases (local shell call,
code-mode item, `echo -c`). The existing Codex state tests still pass.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
U1 pivot brief D8 (GPT-6 finding #9; Claude review R9). Every unclosed "((" or
"$((" made step() look ahead to the end of its line (indexOf the newline, then
count parentheses), so n = 8000, 16000 and 32000 of "((" took 246, 1218 and
4888 ms at 7091a30 (four times per doubling), and openers() filtered its whole
cut list once per heredoc operator.

parenClose(s) now matches every "(" of a text with its ")" in one stack pass per
line (bash(1) ARITHMETIC EVALUATION: (( )) and $(( )) count every ( and ) up to
the end of the line, which is classic parenthesis matching), and step() reads
its answer from the table. The last eight texts stay memoized so the scans that
alternate between a text and its slices do not rebuild it; executedTrace clears
the memo when a top-level analysis ends. openers() finds each operator's bounds
in the ascending cut list by binary search instead of filter and find.

Output is unchanged: executedText (both modes) and fetchKind of the base
kernel and this one agree on all 150,000 checks (the 243 covering commands, the
10,000 heredoc fuzz strings, and two seeded 20,000-string token corpora with
and without heredocs; 0 differences, 0 throws). Timings at n = 8000, 16000,
32000, 64000: "((" 3/6/13/27 ms, "$((" 4/9/19/40 ms, unclosed (( inside "$( "
5/8/18/38 ms, "(<<E" runs 8/14/30/64 ms, each within 2.5x per doubling and
under 150 ms at 64000. The D8 test allows 5 ms of timer noise on the 2.5x
bound (a linear scan showed 7.6 -> 19.4 ms once) and takes the best of five.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude review R3 of 3cb7c4f6. In a "$( )" frame step() treated a # as a comment
start whenever the previous character was a word break, ignoring whether that
break was escaped. `x="$(echo a\ #b)"; curl ...` therefore read `#b)"; curl ...`
as a comment: closeQuote and matchParen ran to the end of the command and the
`; curl` fell inside double-quoted data (shell_fetch 0, fetchKind null, one
unconfirmed mention). bash(1) COMMENTS: a word beginning with # starts a
comment, and an escaped blank, `;`, `(` ... or newline (a line continuation)
does not end a word (QUOTING: a backslash escapes the next character).

escapedAt(s, k) counts the backslashes before s[k]; an odd count means it is
escaped. It runs only for a # that follows a word break, and reads one run of
backslashes per such #, so the scan stays linear.

Verified with GNU bash 5.2.21 and dash and a stub curl: for each of the six
R3 inputs (escaped blank, escaped semicolon, backslash-newline, an even
backslash count before a real comment, a real comment inside "$( )", and a
top-level escaped blank) both shells ran the stub curl once, so each is one
executed fetch.

R3 test passes on all three shell carriers. Differential of the previous
commit's kernel and this one over 90,243 inputs (243 covering commands,
10,000 heredoc fuzz strings, and two seeded 40,000-string token corpora, with
and without heredocs; executedText in both modes and fetchKind): 228 differing
inputs, every one containing an escaped word break before a # (0 unexplained).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Tests only. Extends the R1 cases of 62cb693 with the shapes the fix has to
cover beyond the reviewer's repro, each run under GNU bash 5.2.21 and dash with
a stub curl (both shells agreed):

- a quote inside the substitution: FOO="$(echo "a b")" bash <<'EOF' and the
  ssh variant, where the naive word split cuts the quoted word at the inner
  blank (curl ran once; with cat as the reader it ran 0 times);
- a string that runs over lines before the operator: x="a\nb" bash <<'EOF' and
  the single-quoted variant, and a "$( )" that closes on a later line with text
  after its ")" (curl ran once each);
- two closed "$( )" words, and controls that already read correctly at
  f83f836 (a separator before the operator, a backquote span, a nested
  "$( )" with its own heredoc, cat as the reader).

At f83f836 the 11 executed shapes each fail on all three shell carriers (33
subtests: shell_fetch 0 != 1), the four controls pass. Also: an explicit
--shell-parser that names a directory with no install exits 2 with the reason
and no report (the --rtk-db and --exceptions convention), never a silent
fallback to the default.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude review R1 of 3cb7c4f6. openers() kept one cut list for every nesting
level, so the ( and ) that a double-quoted "$( )" adds also bounded the heredoc
operators outside it: in FOO="$(pwd)" bash <<'EOF' the operator's command
started at the closing quote, its reader resolved to the empty string, and a
body that bash reads as a script became data (shell_fetch 0, one unconfirmed
mention, fetchKind null; real bash and dash run the stub curl once).

step() now reports each cut with its frame (on.cut(i, frame): the "$(" frame
that a ( opens and a ) closes, else the frame the unit sits in, undefined for
the top level) and openers() keeps one ascending cut list per level; an
operator is bounded by the cuts of the level on top of the stack, found by
binary search. The cuts of a substitution bound the commands inside it and no
longer the command that contains it. commandWords() reads a "$( )" or a
backquoted span inside double quotes as one unit of its word (POSIX.1-2024 XCU
2.6.3: the substitution's text is tokenized on its own), so a quote or blank
inside it does not split the word: FOO="$(echo "a b")" bash <<'EOF' names bash.

R1 tests (62cb693, 58502c6) pass on all three shell carriers, and the lane
reading of FOO="$(pwd)" bash <<'EOF' with qmd in the body counts one qmd call.
The shapes below are documented limits, pinned by their own test: a command that
a quote or "$( )" carries over lines (x="$(cat <<'A' ... \n)" bash <<'B'
and x="a\nb" bash <<'EOF') has its head on an earlier line, so this line alone
gives the operator no reader and the body reads as data (fetch_mentions_unconfirmed
1, status incomplete: a possible fetch, never a lost one). A first attempt cut
the line at the closing quote's word end; measured on the fuzz corpus it named
the wrong owner as often as the right one (ssh host cat "a\nb" bash <<EOF), so
it was dropped.

Differential of f83f836 and this commit over 90,243 inputs (243 covering
commands, 10,000 heredoc fuzz strings, two seeded 40,000-string token corpora,
without and with heredocs; executedText in both modes, fetchKind and the M4
confirmed and unconfirmed counts): 17 inputs differ, all with a quote,
backquote or "$( )" before a heredoc operator (0 unexplained); no input lost
mass (confirmed plus unconfirmed); 16 keep both counts, one moves a fetch from
confirmed to unconfirmed. That one, bash -s -- "$(pwd)"bash <<EOF with an
unquoted-delimiter body, now names bash as the reader (correct) and reads the
body as source, where a substitution inside single quotes is data: the
"$( )"-free twin reads C=0 U=2 at 7091a30 as well (real bash runs curl once:
the outer shell expands an unquoted heredoc body before the reader sees it, a
limit that predates R1 and is unchanged).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
U1 pivot D7, GPT-6 finding #8. N1 read a double-quoted string that a shell runs
only as that shell's input, so a substitution the OUTER shell runs while it
expands the word was lost whenever the inner shell reads it as data: a quoted
heredoc, single quotes, a comment. bash -c "cat <<'EOF'\n$(curl https://example.org)\nEOF"
went from fetchKind 'fetch' (cf3fb72) to null with shell_fetch 0. bash 5.2.21 and
dash run the stub curl once: the expanding shell runs each unescaped "$( )" and
backquote before the shell it starts reads anything (POSIX.1-2024 XCU 2.2.3
Double-Quotes, 2.6.3 Command Substitution). Two views now: outer bodies first,
each as commands of their own, then the inner shell's read of the string with
each outer span replaced by a placeholder word (its output is unknown), so a
substitution both views could see counts once. A backslash-escaped \$( ) or
backquote is no outer span and stays the inner shell's (bash -c "echo \$(curl u)"
runs curl once, in the inner shell). Single-quoted run strings have no outer
expansion and are unchanged.

outerSpans() is the one scanner of a double-quoted string's substitutions (escape
pairs skipped, "$((" arithmetic not a span but the substitutions inside found,
NESTING_LIMIT and unterminated spans as before); quotedData() now consumes it, so
the data view and the run view cannot disagree. The ssh remote ranges (marks)
cover the literal text between spans, not the local substitutions.

D7 tests (62cb693) pass on all three carriers: 13 executed, 5 inner and 2
twice-run commands, 2 data commands; the qmd lane test counts one call.

Evidence, all against GNU bash 5.2.21 and dash with stub curl/wget/ssh (ssh runs
its arguments with sh -c, or reads stdin as the remote login shell), counting
stub calls against the kernel's confirmed count (shell_fetch + loopback):
- compositional oracle, 360 commands (4 double-quoted run carriers x 10 contexts
  x 4 atoms, two-atom strings, single-quoted carriers, heredoc readers x
  delimiters x bodies): base 7091a30 301 exact, 59 under, 0 over; 43b9d94 329,
  31, 0; this commit 353, 7, 0. The 7 left are one class, a shell-read heredoc
  with an unquoted delimiter whose body has a substitution inside single quotes
  (the outer shell expands the body first), next commit.
- random valid-bash fuzz, 40,000 candidates, 7,883 accepted by bash -n and dash -n
  with equal stub-call counts under both (361 with a fetch): base, f83f836,
  43b9d94 and this commit agree on the same 7,789 (98.81%) and the same 94
  disagreements (12 over: word glue and a trailing backslash; 82 under: remote
  ssh commands and glued words), all present at 7091a30.
- differential of 43b9d94 and this commit over the covering commands and three
  fuzz corpora (90,243 inputs): 13 inputs lose confirmed mass, every one invalid
  bash in at least one shell; of the 24 differing inputs without HTTP-library or
  gh tokens, 21 are invalid, 2 valid inputs move closer to real bash and 1
  (xcat &&bash -c "'$(curl u)'") counts a substitution that a failing && never
  reaches, which no static reading models.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Tests only. D7 for a heredoc body (U1 pivot D7, same defect class as GPT-6 #8):
with an unquoted delimiter the shell that reads the command line expands the
body before the shell that reads the heredoc sees it, so each unescaped
substitution runs in the outer shell whatever its quotes or a comment say
(bash(1) Here Documents; POSIX.1-2024 XCU 2.7.4), quotes are literal, and a
backslash escapes only $ ` \ and a newline. 20 commands, each run under GNU bash
5.2.21 and dash with stub curl and ssh: curl ran once for 17 (a single-quoted
substitution, a comment, nested quotes, a substitution over lines, a <<- body
with tab indents, arithmetic, an ssh reader, a run string in the body, and
controls that already read correctly), twice for one (an outer and an inner
substitution), and never for two (a quoted delimiter passes the body as is).

At 67a4acb 9 of the 17 fail on all three shell carriers (shell_fetch 0 != 1,
one unconfirmed mention: bash <<EOF with echo '$(curl u)', the FOO="$(pwd)" and
ssh readers, a comment, nested quotes, a multi-line substitution, <<-, and
arithmetic); the rest and the data commands pass.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The heredoc form of D7 (GPT-6 #8: the outer shell runs what it expands before
the inner shell sees the text). With an unquoted delimiter the shell that reads
the command line expands the body first: each unescaped $( ) and backquote runs
there whatever its quotes or a comment say, quotes are literal in a body, and a
backslash escapes only $ ` \ and a newline (bash(1) Here Documents; POSIX.1-2024
XCU 2.7.4). resolveHeredocs() analyzed a body that a shell (or in inlineHttp
mode an interpreter) reads only as that reader's source, so bash <<EOF with
echo '$(curl u)' read as data although real bash and dash run curl once, and
so did a substitution in a comment, nested quotes inside it, one over lines,
arithmetic, and a <<- body: the same 9 shapes as the failing-first commit
b6f3b45. R1 made the FOO="$(pwd)" bash <<EOF form read this way too, so this
also removes the accident that had hidden the limit there.

The body now has two views, sharing D7's outerSpans/withoutSpans/outerBody: the
outer spans first, as commands of their own (heredocs inside them resolved
there), then the reader's source with each span replaced by the placeholder word
and its own escapes applied (heredocText). A quoted delimiter is unchanged
(nothing is expanded), and so is a data heredoc such as cat <<EOF, which keeps
the substitutions its regular expression finds. The ssh remote ranges cover the
body's literal text, not its local substitutions.

The heredoc test of b6f3b45 passes on all three carriers (17 once, 1 twice, 2
data commands); tests.test_token_measurement passes (59). Evidence, real bash
5.2.21 and dash with stub curl/wget/ssh, kernel confirmed count against stub calls:
- compositional oracle, 441 commands (D7's 360 plus escaped, comment and
  nested-quote bodies): base 7091a30 368 exact, 73 under; 67a4acb 420, 21;
  this commit 441 of 441 exact, 0 under, 0 over.
- grammar fuzz, 6,000 valid nested commands (statements, sequences, substitutions,
  backquotes, single and double quotes, run strings, heredocs with unique
  delimiters) that bash and dash ran with equal stub-call counts (3,049 with a
  fetch, 1,597 with a run string, 1,471 with a heredoc): base 5,788 exact
  (96.47%), 212 under, 0 over; 67a4acb 5,970 (99.50%), 30 under, 0 over; this
  commit 5,975 (99.58%), 25 under, 0 over. All 25 are one older gap, a run string
  right after a backquote (echo `bash -c "curl u"`), whose text is invisible to
  the raw detector as well; fixed next.
- random valid-bash fuzz, two seeds of 40,000 candidates (7,883 and 7,838 valid,
  361 and 374 with a fetch): base, 67a4acb and this commit agree on every
  input.
- differential of 67a4acb and this commit over 90,243 inputs: 924 differ, 20 with
  a lower confirmed count, of which 19 are invalid bash or valid only with a
  heredoc-at-end-of-file warning, and the one clean input (a backquote span that
  holds bash <<EOF) moves from 3 to 0 confirmed where real bash and dash run
  curl 0 times.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Tests only. RUN_QUOTED anchors a shell, eval or ssh word at the text start, a
blank, ; & | or (, but not at a backquote, so bash -c "curl u" inside `...` read
as double-quoted data. The fetch was lost, and the raw detector, which anchors
curl itself and not the quote, counted no possible fetch either (C=0 U=0). Found
by the grammar fuzz of 2a33ef9: all 25 commands left under-counted there were
this one shape.

8 commands run under GNU bash 5.2.21 and dash with stub curl and wget: the stub
ran once for each (bash -c and sh -c with double or single quotes, eval, an ssh
string, a backquote inside a single-quoted run string, an escaped backquote in a
double-quoted one, and two controls that already read correctly: a blank after the
backquote and $( ) in its place) and never for echo `echo "curl u"` (data).

At 2a33ef9 the 6 backquote-adjacent shapes fail on all three shell carriers
(shell_fetch 0 != 1, no unconfirmed mention: 19 failures with fetchKind).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
RUN_QUOTED and RUN_HTTP_CODE anchored a shell, eval, ssh or interpreter word at the
text start, a blank, ; & | or (, but not at the backquote that opens a `...`
substitution, so bash -c "curl u" inside backquotes read as double-quoted data: the
fetch was lost and no possible fetch was counted either (C=0 U=0). bash(1)
Command Substitution: a `...` body is shell text with its own command positions,
as a $( ) body is; the anchor class now holds the backquote (one character in
each regular expression; no new alternative, so no new backtracking).

The 6 backquote shapes of the failing-first commit (6655064) pass on all three
carriers; tests.test_token_measurement passes (60).

Evidence, real bash 5.2.21 and dash with stub curl/wget/ssh, kernel confirmed count
against stub calls: the 441-command oracle stays 441 exact; the grammar fuzz that
found the gap is now exact on every input, 6,000 of 6,000 (3,049 with a fetch,
1,597 with a run string, 1,471 with a heredoc; 2a33ef9 5,975 exact, 7091a30
5,788, 96.47%) and, on a second seed, 11,999 of 11,999 (base 11,608, 96.74%, 391
under; 0 over throughout). Differential of 2a33ef9 and this commit over 90,243
inputs: 216 differ, 1 with a lower confirmed count, an unterminated string (invalid
bash).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
U1 pivot D1. The CLI-lane reading is to run on tree-sitter-bash, the grammar
OpenAI Codex uses for the same job (openai/codex rust-v0.157.1 36650394
codex-rs/shell-command/src/bash.rs; tree-sitter-bash = "0.25" at
codex-rs/Cargo.toml:522, both read at the tag). This commit adds the seam and the
gate; the AST reading itself is the next stage.

shell-parser.pin.json pins the install: web-tree-sitter 0.27.0 and
tree-sitter-bash 0.25.1 with their npm integrity values (checked against the
registry's dist.integrity), upstream tags and commits (v0.27.0 6070dbfe =
npm gitHead; v0.25.1 names a06c2e44 while the package was published from
80132668, five minutes earlier, recorded as a note), the sha256 of the six
files the loader reads, and the install command (npm install --prefix <dir>
--ignore-scripts --no-audit --no-fund --save-exact). loadShellParser(dir)
resolves the directory as the argument (the new --shell-parser flag), then
CHILD_USAGE_SHELL_PARSER, then the ecosystem tools directory under HOME, reads
each pinned file once, compares its sha256 and the lockfile's version and
integrity values with the pin, and executes only those bytes: the runtime module
is imported from a private 0600 copy of the hashed bytes and the two wasm files
are handed over as bytes, so the code that was hashed is the code that runs.
Results carry no path: { ok, versions, wasm_sha256 } or { ok: false, reason }
with not_installed (directory, a pinned file or the lockfile missing),
hash_mismatch, load_error (an unreadable file, the pin file, init) and
not_loaded (this process never awaited the loader, so a forgotten await is not
mistaken for a missing install). A failed load leaves no parser, whatever an
earlier call did; a successful one reuses the initialized runtime. withShellTree()
frees every tree in a finally.

Fail closed: commandInvocations() returns null without a parser and measurement
reports cli_lanes { status: parser_unavailable, reason }, keeps the prefix rule for
the rtk proxy carrier (proxy.rule prefix_fallback, invocations and in_ctx_code
null) and never falls back to the text scanners for lane counting; M4 stays text
based. A measured cli_lanes carries status measured and the parser record (versions
and the two wasm sha256 values); aggregates report measured, incomplete (mixed
actors, with the counts), parser_unavailable or not_measured, and sum proxy
counters null-aware. The CLI awaits the loader; a --shell-parser that cannot be
honored exits 2 with the reason (like --rtk-db), any other missing install only
leaves cli_lanes unavailable. The Codex bridge awaits the loader too.

Interim, until the AST reading lands: with a verified parser loaded,
commandInvocations() still runs the scanner-based lane layer of 3cb7c4f6, so every
existing lane fixture reads as before; the gate, the status fields and the seam
(withShellTree) are final.

Tests (62cb693, 58502c6 first, run failing at 7091a30): the pin's contents;
not_installed for an absent directory, a missing pinned file and a missing
lockfile; hash_mismatch for one flipped byte in the bash wasm, a modified
web-tree-sitter.js that is never imported, a modified package.json, another
integrity value and another version in the lockfile; load_error for an unreadable
file; directory order; --shell-parser; a good install loading again after
failures; unavailable and mixed aggregates; the bridge in both states. The lane
fixtures load the parser first and skip with a message on a host without an install.
tests: node suite 149 passed; tests.test_token_measurement, test_skill_usage and
test_child_usage_suite 175 passed. D9's stress check reports the TypeError the
lane checks threw before the parser was loaded (threw: TypeError x8), which the
old assertion would have passed.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Tests only. D8 asks the M4 text scanners for linear time; the unclosed "((" input
of GPT-6 #9 is fixed (5e81433), but profiling a realistic 346 KB script (4,000
lines of echo "step N: $(date)" >> log; qmd search "term N" | head) at e5238f3
showed the same quadratic shape from another term: 51% of the time in the
run-string regular expression and 33% in scanQuotes, both from one read of the
whole text built so far per quote (the regex, and the string flattening that
indexing a concatenation rope forces). A 42 KB script takes 47 ms, 84 KB 108 ms,
172 KB 467 ms, 346 KB 2,078 ms and 694 KB 8,236 ms a scan, and the kernel scans
each shell call several times.

Five doubling checks (n = 8000..64000, best of five, at most 2.5x per doubling
plus 5 ms of noise, 64,000 under 1.5 s): a run of quoted words, a run of
double-quoted substitutions in inlineHttp mode, a run of `bash -c "x"` strings,
a run of words with a # inside, and a long script of echo, substitution and pipe
lines through fetchKind. At e5238f3: 61, 230, 902, 5,861 ms; 168, 651, 3,728 ms;
991, 4,486 ms; 9, 32, 120, 459 ms; 89, 392, 1,930 ms (the series end where a run
passes 1.5 s). All five fail.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
scanQuotes() tested RUN_QUOTED (and in inlineHttp mode RUN_HTTP_CODE) against the
whole text built so far at every quote. Reading out.s for that flattens the string
concatenation and scans the full prefix each time, so a script with n quoted
words cost n squared. CPU profile of a 346 KB script (4,000 lines of echo "step N:
$(date)" >> log; qmd search "term N" | head) at e5238f3: 51% in the run-string
regular expression, 33% in scanQuotes itself (the flattening), 10% in GC. The
five timing tests of 9da5741 failed on it.

scanQuotes now keeps the end of its output in `tail`, cut back to 512 characters
whenever it passes 1,024, updated at every write (put and putTrace; quotedData
builds into its own buffer first), and reads only that: the # test uses the last
character of the tail, the emptiness test out.p.length. Once a cut has been made
the two detectors run without their start-of-text alternative (a cut is no start
of text). Limit, stated in the code: a run keyword more than 512 characters
before its quote, after a text that has grown past 1,024, is missed and the
string reads as data. Only a single ssh option word that long can put it there
(ssh -oProxyCommand=<1500 characters> host "curl u": fetch before, null after;
100, 400, 520 and 700 characters read the same), since the words between a
shell, eval or ssh and its string are short options and one host.

Timings (best of five, ms), n = 8000, 16000, 32000, 64000: quoted words 61, 230,
902, 5,861 -> 8, 15, 31, 69; double-quoted substitutions (inlineHttp) 168, 651,
3,728 -> 15, 31, 67, 140; run strings 991, 4,486 -> 38, 83, 173, 358; words with
# 9, 32, 120, 459 -> 3, 6, 14, 37; a long echo/substitution/pipe script through
fetchKind 89, 392, 1,930 -> 18, 34, 71, 154. Realistic scripts, executedText: 84
KB 108 -> 37 ms, 346 KB 2,078 -> 100 ms, 694 KB 8,236 -> 205 ms, 1.4 MB 393 ms
(2x per doubling); the node suite runs in 12 s instead of 41 s.

Output is unchanged: executedText (both modes) and fetchKind of e5238f3 and this
commit agree on all 1,084 shell texts of this repository (822 .sh files and fenced
shell blocks of .md files, 1.66 MB, largest 82 KB, 83 over 4,000 characters), on
the 243 covering commands and the three fuzz corpora (90,000 inputs), and on 600
synthetic long scripts (0 differences, 0 throws); the 441-command oracle stays 441
exact, and the 6,000-command grammar fuzz stays 6,000 exact (real bash 5.2.21 and
dash with stub curl).

Also fixed while rewriting the call: the ssh remote offset test listed the
separators before RUN_QUOTED's keyword but not the backquote that 296c4f1 added,
so in echo `ssh host "qmd get a"` the qmd call read as local; it now reads
qmd/qmd remote (fixture added; the same input read local at e5238f3).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The loader tests changed the bash grammar wasm by reading it as UTF-8 and
appending a NUL, which rewrites the whole binary: the test passed because the
file differed, not because one byte did. It now reads a Buffer and flips bit 0 of
the middle byte in place (same length, so only the hash can tell), and does the
same for each of the six pinned files in turn: every one gives hash_mismatch,
leaves no parser (commandInvocations returns null, the scanners do not step in) and
runs no unverified code (the modified web-tree-sitter.js sets a global if it is
ever executed; it is not). The missing-file and missing-lockfile cases now also
assert that no parser stays loaded. Node suite: 160 passed (149 before).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Scout and others added 23 commits September 29, 2026 11:43
D12 and the closing evidence of the U1 pivot. Every number in the docs is
rendered from evidence/artifacts/pra-u1-differential-20260929/counts.json
(kernel sha256 prefix a9126a77fd7f, commit 6b3958b).

- workflows README: how the parser is provisioned and pinned, that it
  fails closed, the tree-sitter-bash and Codex reference
  (openai/codex rust-v0.157.1 bash.rs, Cargo.toml:522), what the reading
  covers and where it repairs the grammar, what stays text-based in M4 and
  why (the routing detector it mirrors) with the divergences that were
  checked under bash and dash, mcporter's help and version rule with its
  upstream citations (R6, rebutted: code and test already matched), the
  states with the observed not-executed shapes and `interrupted` (R7), the
  static and grammar limits with counts from this host, the local checks.
- tools/skill-usage README: the Python mirror of M4 disagrees with the
  kernel in both directions (R8), the Codex argv rule (D6), the exit-code
  bound.
- docs/decisions/2026-09-29-shell-command-parser.md: alternatives with
  their dated disqualifiers (recorded by the pivot brief, not re-measured),
  the choice, what would overturn it, limits and residuals, SOTA sources.
- evidence/artifacts/pra-u1-differential-20260929/: the scripts, a
  counts-only README and counts.json. The differential compares the
  scanner reading of 0c421c66 with the walker on the covering commands,
  three seeded corpora (bash -n as the validity oracle, heredocs included)
  and this host's real commands; each difference is reduced to a minimal
  witness and classified (a lane difference in a runnable corpus is judged
  by running the input under real bash with stub executables; unresolved
  and remote counts by mechanism; real commands by argument, never run).
  Unexplained: 0 in every corpus; the classes that stay heuristic labels
  or unisolated mechanisms are counted apart. M4 (executedText, fetchKind,
  m4) is identical between the kernel before the walker and this one on
  151,221 inputs, and the executedText/fetchKind claim of the earlier
  commit 9a4e9f97 (24,489 or 24,495 inputs, harness not retained) is
  re-run on the same corpora with 0 differences.

The kernel comments no longer quote counts; the README dates its figures.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The pivot changed child-usage.mjs and test-child-usage.mjs and added
shell-parser.pin.json to examples/claude-native/workflows. Regenerated
with the documented command, `sha256sum -- *.mjs *.js *.json >
SHA256SUMS` in that directory; `sha256sum --check --strict SHA256SUMS`
passes for all 14 files. The child-usage.mjs digest (a9126a77fd7f...) is
the one counts.json records; the pin file enters through the *.json glob.
Only these three lines change.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Hot-file protocol of docs/lanes.md, in the branch's last commit: the
manifest is the copy of b2767df (the main this branch merged; the shared
origin/main ref has since advanced by six commits, none touching the U1
paths but this manifest, so integration re-runs this step), and the
branch's files are re-registered with register_file:

- the eight files that main's manifest already lists (workflows README,
  SHA256SUMS, child-usage.mjs, test-child-usage.mjs, tests/test_skill_usage.py,
  tests/test_token_measurement.py, tools/skill-usage/README.md,
  tools/skill-usage/skill_usage.py): new digests;
- the 14 new files under evidence/artifacts/pra-u1-differential-20260929/,
  which scripts/validate.py requires to be hash-listed;
- shell-parser.pin.json and tests/test_command_position_oracle.py, new
  files the measurement kernel and its oracle load, registered like the
  eight above. The decision record is not registered: it is not under
  evidence/, main does not list it, and the protocol does not require it.

component_matrix.py --write and new_host_grand_list.py --write ran and
changed no file. scripts/validate.py exits 0 (8,096 hashed files).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The independent review of head 33dcfd2 found that the decision record's
first overturn condition (a grammar limit in 0.1% or more of lane-bearing
commands) was presented as checked but never evaluated. counts.json holds 247
tree errors and 6,527 lane invocations and no count that combines them, and
the scan counts lane-bearing commands only from lanes the reading found, so a
lane hidden under a skipped ERROR subtree is never lane-bearing.

Three tests that fail at 33dcfd2:
- shape-counts.mjs counts lane words without the parser and bounds what a
  grammar limit can lose or invent (a synthetic corpus of 8 texts, the
  tree-sitter-bash 0.25.1 reading of each named beside it; --scanner adds a
  second reading's opinion).
- real-commands.mjs --until rebuilds the corpus of an earlier run as of a
  timestamp (synthetic transcript rows).
- the decision record quotes counts.json shapes.overturn_1.

One check that passes at 33dcfd2 by design, since the review could not read
it at the pin: mcporter v0.14.1 reads a flag written with = (--config=x,
--server=x, --http-url=x) as a named argument or a command word, never as a
global, server or ad-hoc server flag. Sources: openclaw/mcporter@93e0916c
src/cli/cli-factory.ts:19, flag-utils.ts:12, call-arguments.ts:63,115,120-121,
333, ephemeral-flags.ts:30, and the installed 0.14.1 binary run in a scratch
HOME.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
shape-counts.mjs now counts lane-bearing commands without the parser (a lane
word, anywhere in the text) and bounds what a grammar limit can lose or
invent: a limit command counts when a lane word lies inside an ERROR node, it
is a heredoc ended early with a lane word, its only error is in a script the
reading read again and it holds a lane word, or the reading reads a lane in a
tree with an error (a lane that could be invented). overturn_1 compares the
bound with 0.1% of the commands in which the reading reads a lane (the smaller
denominator, so the stricter one). --scanner adds the scanner reading of
0c421c66 as a second opinion. The 18 keys of the earlier run are unchanged.

real-commands.mjs --until keeps only the rows stamped at or before an instant.
The corpus of counts.json was extracted without a recorded cutoff: the
instant 2026-09-29T07:08:07Z RECONSTRUCTS it (136,361 distinct texts, 15,128
with a heredoc operator, and all 18 recorded shape counts equal on it, run
with the committed script). run-differential.py records the corpus size and
cutoff, and passes the scanner kernel to the shape scan.

Failing-first tests: 017f920.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The review could not tell where the unread lane words of the grammar-limit
commands stand, and the scaling of commandInvocations was never measured.
Measured at 33dcfd2: '(('.repeat(n) + 'qmd' took 467, 1853 and 7369 ms at n =
8000, 16000 and 32000 (four times per doubling), so GPT-6 #9 was fixed for the
M4 scanners only. The parse is linear; node.child(i) and fieldNameForChild(i)
cost O(i) per call on a wide node in web-tree-sitter 0.27.0, and a flat run of
unclosed constructs or of comments is one wide node.

- test-child-usage.mjs: doubling checks of commandInvocations for nine
  shapes ((( $( "$( $(( backquotes { if case comments). At 33dcfd2 all nine
  fail: [461, 1831], [118, 462, 1840], [208, 820, 3274], [68, 264, 1040,
  4133], [462, 1838], [119, 464, 1836], [58, 220, 857, 3464], [72, 278, 1093,
  4381] and [29, 111, 434, 1728] ms at n = 8000, 16000, 32000, 64000 (a run over
  1.5 s ends the doubling).
- shape-counts.mjs: an unread lane word of a limit command is classified as no
  command slot (it follows plain words: an argument whatever the tree makes of
  it), a slot in data (comment, heredoc body, string, assignment value), a slot
  elsewhere (an ERROR node included), or unplaced (the error is only in a script
  read again); a scanner that throws is counted.
- m4-identity.mjs --fields lanes: whole commandInvocations and cli_lanes
  records of two kernels, an exception in either counted.

The identity check of a kernel with itself passes at 33dcfd2 (a sanity check,
not a failing-first test).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
GPT-6 #9 (commandInvocations on '(('.repeat(n) + 'qmd') was fixed for the M4
scanners only: the kernel walked each node's children with node.child(i) and
node.fieldNameForChild(i), which cost O(i) per call in web-tree-sitter 0.27.0,
and read node.parent (O(depth)) in the hot loop. A flat run of unclosed
constructs or of comments is one wide node, so the reading took four times as
long for twice the text (7.4 s at n = 32000). childrenOf() reads a node of
more than 16 children with one cursor (a step is O(1); a narrower node stays
on the index path, which is cheaper) and the walk carries the parent's type.
At n = 8000, 16000, 32000 and 64000 the nine doubling checks now take [12,
23, 47, 97] ms for '((' and 3 to 97 ms for the others, and 30,000 real
commands read 1.03 times as long as before. Two shapes stay quadratic inside
tree-sitter-bash's own parse: a run of unclosed a=( and of << (documented).

Behaviour is unchanged: commandInvocations and cli_lanes records of the
33dcfd2 kernel and this one are identical on 217,428 inputs (136,361 real
commands, the three seeded corpora whole, both covering lists) and the m4
object on the real commands. A wide-node check runs the cursor path for a
command's words, assignments, a list, case branches, comments and heredoc
operator arguments; forcing the cursor's field name to null fails it.

shape-counts.mjs classifies each unread lane word of a limit command (no
command slot / a slot in data / a slot elsewhere / unplaced) so the exclusion
of the 26 of 31 is a committed count, and m4-identity.mjs --fields lanes
compares whole lane records of two kernels. Failing-first tests: e08e466.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The linear-time claim for the lane reading and the exclusion of the unread
lane words of the grammar-limit commands need a committed producer, not
numbers typed into a README.

- scaling.mjs prints, per shape (the nine of the doubling checks, controls and
  the two parse-bound shapes), the text length and the best time at each size
  and the largest growth between sizes.
- update-round2.py re-runs only the shape scan, the whole-record identity check
  against a baseline kernel and the scaling measurement, and writes them with
  the corpus cutoff into counts.json; every other section is left alone (the
  driver of round 1 would also re-capture the covering tests, which now hold
  new fixtures, and recount the live transcript store).

Both fail at the previous commit: the scripts do not exist (exit 1 with a
missing module; exit 2 with no such file).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
scaling.mjs times commandInvocations by input size for 21 shapes: the nine
that the doubling checks cover, ordinary long texts as controls, and the two
whose parse is quadratic inside tree-sitter-bash (a run of unclosed a=( and of
<<). update-round2.py splices the shape scan, the whole-record identity check
against a baseline kernel and the scaling measurement into counts.json and
leaves every other section as round 1 measured it, on the kernel it names.

Failing-first tests: 3743f29.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The claim that a run of unclosed a=( or of << is quadratic inside
tree-sitter-bash's parse, not in the reading, needs a committed measurement:
scaling.mjs should time kernel.withShellTree (the parse and nothing else) at
each size next to the reading. Fails at 0c9489a with KeyError: 'parse_ms'.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
When the kernel exports withShellTree, each shape also gets parse_ms and
parse_max_ratio: the tree is built and freed and nothing is read. A shape whose
parse takes the reading's time is slow in tree-sitter-bash's parser (a run of
unclosed a=( or of <<), not in the reading, and no walk of the tree changes it.

Failing-first test: the commit before.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The reviewers could not tell whether the differential counts of round 1 were
reproducible. update-round2.py should re-run differential.mjs (no reduction)
on any corpus (NAME=INPUTS[:VALID]) and record its totals next to a flag that
says whether they equal round 1's record of the same corpus (distinct_witnesses
needs a reduction and is not compared). Fails at bdb53b8: the script rejects
--differential (exit 2, unrecognized arguments).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
--differential NAME=INPUTS[:VALID] runs differential.mjs (no reduction) of the
scanner reading against the kernel on a corpus and records the totals in a new
differential_rerun section with equal_to_round1: whether they equal round 1's
record of the same corpus (distinct_witnesses needs a reduction and is not
compared; null when round 1 has no record). A re-run on the reconstructed real
corpus gave 136,248 the same and 113 different, as recorded in round 1.

Failing-first test: the commit before.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The two shapes whose parse is quadratic (a run of here-document operators and
of unclosed a=( ) are documented as a limit, not fixed: no real command comes
near them. That claim needs committed maxima: the longest command, the most
heredoc operators (a here-string is not one) and the most =( in one command.
Fails at 257bd4c: the keys are missing (None, None, None).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
longest_command_chars, most_heredoc_operators_in_a_command (a here-string <<<
is not one) and most_array_openers_in_a_command (=( ) are maxima over the
corpus, by linear scans. They bound the two shapes whose parse is quadratic
inside tree-sitter-bash (scaling.mjs parse_ms): a run of here-document
operators and of unclosed array openers.

Failing-first test: the commit before.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The decision record stated its first overturn condition as checked; it was
never evaluated. It is now, from counts.json (shapes.overturn_1): an upper bound
of 5 of 6,527 lane-bearing commands (0.077%) against a threshold of 7 commands,
not met, by 2. The lane-word screen alone is 31 of the 247 grammar-limit
commands (0.475%) and is reported as such: it counts data. Of the 31, 4 read a
lane, 13 follow plain words, 13 stand in data and 1 is inside an ERROR node.
The repaired shapes are excluded by their repair, not their frequency (37 and 1
with a lane word would flip the verdict). The Limitations bullet no longer says
that all 247 hide a lane.

counts.json gains corpus (the cutoff is reconstructed: 2026-09-29T07:08:07Z
reproduces 136,361 texts and all 18 recorded shape counts), shapes (extended),
lanes_identity (217,428 inputs, whole commandInvocations and cli_lanes records
of the 33dcfd2 kernel and this one: 0 different, 0 throws), scaling (21 shapes,
the reading and the parse alone) and differential_rerun (the totals of the six
corpora equal round 1's). Every other section is round 1's, on kernel
a9126a77fd7f, and the evidence README says which sections were run again.

The record and the workflows README now state what the parser costs: the walk
was quadratic in the width of a node (GPT-6 #9 was fixed for the M4 scanners
only) and is linear; a run of unclosed a=( or of << is quadratic inside the
parse itself, this host's largest command has 49,096 characters, at most 71
heredoc operators and 12 array openers, and a parse budget is a follow-up.
tools/skill-usage/README.md records what a ten-digit Windows exit code reads.
SOTA sources added: tree-sitter v0.27.0 node.c, walking-trees.md and
tree_cursor.ts; mcporter v0.14.1 cli-factory.ts, flag-utils.ts,
call-arguments.ts, ephemeral-flags.ts; codex rust-v0.157.1 context.rs.

Failing-first test: 017f920 (the record now quotes counts.json; the test
compares text with line wrapping removed, and three mutations of the record
fail it).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
child-usage.mjs (the tree is read with a cursor for a wide node) and
test-child-usage.mjs (nine doubling checks of commandInvocations and a
wide-node check) changed; the two lines are recomputed and
sha256sum --check --strict passes.

The evidence README records how far the absolute timing bounds can slow down
before they fail, measured on one core shared with busy loops: the 150 ms bound
on executedText of 64,000 unclosed (( holds to a slowdown of about 3.9 (38.5 ms
alone, 158 ms at about four times), the 1.5 s bounds on the lane reading to about
ten. A continuous-integration runner was not measured.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The evidence README lists the 75 fixtures of the M4 decisions (D7, R1, R3,
heredoc expansion, backquote anchor) as run again under real bash and dash
with a stub curl, but the run was an ad hoc probe and nothing in the
repository reproduces it. m4-real-shells.py should run every fixture list of
tests/test_token_measurement.py in both shells and compare the stub's runs
with the documented count and with the kernel's confirmed-fetch reading, print
counts only and exit 1 on any disagreement; a control with a stub that logs
nothing must make the 67 fixtures that should run curl disagree.

Both tests fail at 13079c3: the script does not exist (exit 2, no such file).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Every command of the fixture lists of tests/test_token_measurement.py (D7, R1,
R3, heredoc expansion, backquote anchor: 75 commands in 13 lists) runs under
real bash 5.2 and dash with stub curl, wget and ssh; the stub must run as often
as the list documents in both shells, and the kernel's fetchKind must read a
fetch exactly when it ran (R1_MULTILINE is the documented limit). Counts only,
exit 1 on any disagreement. Result: 75 of 75 agree. Control: with a stub that
logs nothing, 67 of 75 disagree (the 8 fixtures that run no curl still agree)
and the exit is 1.

Failing-first tests: the commit before.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The evidence README listed the 75 M4 fixtures as run again under real bash and
dash without a producer; it now names m4-real-shells.py (75 of 75 agree, the
control fails 67 of 75) and says that the parser pins were re-read by hand with
the commands of the verified block of shell-parser.pin.json, which is not a
committed script. The decision record says that the one heredoc ended early
with a lane word is among the five commands of the bound (the four that read a
lane and the one slot elsewhere are disjoint, and their union is five).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Hot-file protocol of docs/lanes.md, in the branch's last commit. The manifest
stays the copy this branch merged (b2767df plus the registrations of
33dcfd2): the shared origin/main ref has advanced since (ed29398, eleven
commits past the merge base) and none of its newer entries has a file in this
tree, so integration re-runs this step.

- the twelve files that the manifest already lists, with new digests (the
  workflows README, SHA256SUMS, child-usage.mjs, test-child-usage.mjs,
  tests/test_child_usage_suite.py, tools/skill-usage/README.md and the
  README, counts.json, m4-identity.mjs, real-commands.mjs, run-differential.py
  and shape-counts.mjs of the evidence directory);
- the three new files of that directory, which scripts/validate.py requires to
  be hash-listed: scaling.mjs, update-round2.py and m4-real-shells.py. The
  decision record is not registered: it is not under evidence/, main does not
  list it, and the protocol does not require it.

component_matrix.py --write and new_host_grand_list.py --write ran and changed
no file. scripts/validate.py exits 0 (8,099 hashed files).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…the files (hot file)

Merge origin/main (ed29398). The two comments on the nine-digit exit-code bound contradicted themselves (an i32 has up to ten digits): they now say that the brief's bound reads a valid ten-digit code as unknown, never succeeded. The manifest is main's copy with the U1 files re-registered.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
@seathatflowsinourveins seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Sep 29, 2026
Scout and others added 2 commits September 29, 2026 15:58
The hosted validate-macos run of #516 (36618373211) failed on one R3 fixture of m4-real-shells.py: macOS runners put bash 3.2 or a newer Homebrew bash first on PATH, while the curl counts are documented for bash 5.2 (Ubuntu 24.04 ships 5.2.21). The two real-shell tests now skip with that reason unless bash reports 5.2; nothing else changes and the kernel is untouched.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant