PR-CI: discriminating token-tool tests for context-mode, Serena, ai-memory, context-hub, agentsview and an RTK exactness fix by seathatflowsinourveins · Pull Request #375 · native-agent-stack/native-agent-stack

seathatflowsinourveins · 2026-09-27T00:40:35Z

Summary

Adds hosted-CI token E2E coverage for the six tools native-token-e2e.yml/Harden the last three workflows, pin exact release comments, and give native-token-e2e discriminating fixtures #370 did not reach:
RTK's hook-check and exactness probes, context-mode (npm 1.0.169), Serena (commit
c6fbd1c5932df2494ffa0020af5a9fbe80b82143), ai-memory (2.4.1), context-hub (0.1.4) and
agentsview (0.43.0), each with run.check(...) assertions proving a correct response passes
and a passthrough/unchanged/wrong response fails, per docs/acceptance-evidence-policy.md's
discriminating-controls rule.
9 new tests in tests/test_native_token_ci.py, one per fixture/helper group
(rtk_hook_check_fixture + the §4.1 negative controls, rtk_exactness_checks/
rtk_exact_rows/rtk_exactness_fixture, context_mode_fixture, serena_fixture,
ai_memory_fixture, context_hub_fixture, agentsview_fixture), following the file's own
established pattern (test_headroom_fixture_requires_a_real_saving_and_exact_recovery,
test_ast_grep_outcome_is_frozen_and_structural): mock boundary is run.command
(label-keyed) or ci.mcp_stdio_session (Serena, ai-memory).
Writing the RTK-exactness test's empty-output arm exposed a real bug: rtk_exactness_checks
indexed lines(case, "rtk")[-1] unconditionally, so an empty RTK response raised IndexError
instead of failing its named check. Fixed with a one-line bool(lines(...)) guard; confirmed
red before (IndexError, errors=3) and green after (OK).
Added one workflow step the code already assumed existed: context_hub_fixture's own docstring
says "the workflow lifts" Ubuntu 24.04's AppArmor restriction on unprivileged user namespaces
before its unshare -rn call, but native-token-e2e.yml had no such step. GitHub's hosted
ubuntu-24.04 runner image keeps that restriction by default
(actions/runner-images#10443), so this
would otherwise fail "Operation not permitted" on every hosted run of the new fixture — not a
flake. One run: step (sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0) after
harden-runner, which stays first.
Repair round (GPT-6 cross-family review, findings below): Run.clean() — the sanitizer
every fixture's retained command/observation output already routes through — did not redact
two categories scripts/validate.py's own PRIVATE_CONTENT scanner defines: bare UUIDs and
personal home paths. The real hosted run's artifact (linked in the PR comment) showed both,
from this PR's two new instances (ai-memory's own generated page/workspace path components;
an upstream example path in agentsview's --help text) plus a pre-existing instance from
markitdown's ONNX dependency that this fix also now redacts. Run.clean() now runs those two
PRIVATE_CONTENT patterns (cited by name, not re-derived) as a regex pass after its existing
literal substitutions, with new tests proving the redaction, the /home/example exemption,
and no regression to the existing literal substitutions.
Integrator follow-up, found by verifying the repair round against the real hosted artifact
rather than trusting the second green run alone: Run.observe() — used by six of this PR's
fixtures to record state-directory contents — built its returned path list directly, never
routing it through clean(), so ai-memory's UUID-shaped page paths still leaked in the second
hosted run's artifact even after the fix above. Same one-line, same-source fix
(self.clean(...) around the per-path expression), one more failing-first test (confirmed
failing by temporarily reverting the fix, not by inspection). Applied directly, not through a
second GPT-6 build round — the task's review/repair cycle is capped at one round, already used
above; this is a same-source extension of that already-reviewed fix to the one call site it
missed.

This PR's diff against origin/main is scripts/native_token_ci.py and
tests/test_native_token_ci.py in full (git diff 34c56340 <head> --numstat) — it is not
independently separable into "pre-existing" and "added" parts once pushed, since the branch's
two local commits (a WIP checkpoint plus this round) were squashed into one commit by this
repository's hot-file rebase protocol before push. The bullets above describe what changed and
why; they do not claim the whole diff is one line.

Verification

$ python3 -m py_compile scripts/native_token_ci.py tests/test_native_token_ci.py
exit 0, no stdout/stderr
$ python3 -m unittest tests.test_native_token_ci -v
Ran 41 tests in 2.354s
OK
$ bash rec/validate.sh <ci-venv python>   # this repo's aggregate acceptance check
FAILS=0
$ python3 -m unittest discover -s tests -q   # whole-repository suite, before push
Ran 6356 tests in 576.640s
OK (skipped=665)

Evidence class: local_integration (each fixture runs the pinned tool's own CLI/MCP surface; this
repository's checks assert on that output — see Run.report["limits"] in
scripts/native_token_ci.py). The new unit tests themselves are synthetic (fixture-level):
mocked subprocess/MCP boundaries proving the harness's own discriminators reject a
passthrough/wrong response.

Hosted native-token-e2e.yml run
36283215966
(triggered automatically by this PR's paths filter): success, all steps green. Its
receipt.json — downloaded and read directly, not inferred from the green checkmark — reports
status: "passed", failures: [], 82/82 checks true, including all 32 new checks and,
critically, serena-initialize-byte-parity-with-the-pinned-install /
serena-tools-list-byte-parity-with-the-pinned-install = true against the real pinned commit's
installed output. Full table and the repair round's own hosted result are in the PR comment.

Residuals

exactness.sh named in the plan's PR-CI row is a private scratch script from the 2026-09-26 RTK
coverage study (full-save/rtk/exactness.sh), never committed; its checks are already ported
into rtk_exactness_checks (comment already in the file cites this). No new shell script was
added; adding one would duplicate committed logic with a second, divergent copy.
context-hub and agentsview have no entry in adoption/pins-linux-x86_64.json by design
(adoption_pin_mismatches's docstring: "Tools the adoption pins do not list are not compared");
this PR does not add one, since that file's ownership is a separate lane.
The repair round's Run.clean() fix redacts UUIDs and personal home paths for every fixture,
including the pre-existing markitdown/ONNX instance — but it does not add a validate.py --scan-file gate to the workflow itself, which would be a wider, cross-tool concern over all
17 tools' artifacts (closer to the plan's separate evidence-sanitization lane) than this PR's
six tools. Flagging it as a follow-up, not fixing it here.

SOTA sources

RTK: rtk-ai/rtk v0.50.0, pinned in manifests/stack.json and
adoption/pins-linux-x86_64.json. rtk hook check --agent claude is its own documented decision
path; the five exclude_commands entries are fixtures/rtk-hook-exclusions.toml
(recipes/README.md's four entries plus jq, per rtk 0.50.0 hook recipe: exclude git show REV:path in every global-option form and git branch #318/PR-R's widening).
context-mode: mksglu/context-mode; this fixture
installs the npm [email protected] release (a distinct source revision from the native
plugin pin, per recipes/README.md's own caveat) and calls its documented ctx_doctor and
ctx_execute MCP tools through MCPorter's stack.json command form.
Serena: oraios/serena, pinned commit
c6fbd1c5932df2494ffa0020af5a9fbe80b82143 (adoption/pins-linux-x86_64.json,
recipes/README.md's serena row, adoption/bootstrap.md step 4a). Its MCP initialize/
tools/list behavior for the claude-code/codex contexts is
src/serena/resources/config/contexts/{claude-code,codex}.yml at that commit.
ai-memory: akitaonrails/ai-memory v2.4.1
(adoption/pins-linux-x86_64.json), its documented init/serve --transport stdio MCP surface
and memory_write_page/memory_query/memory_read_page tools.
context-hub: aisuite/chub npm package v0.1.4;
its own telemetry/feedback opt-outs in $CHUB_DIR/config.yaml (config.js/telemetry.js at
that version, cited in the fixture's docstring) and CHUB_TELEMETRY/CHUB_FEEDBACK environment
overrides.
agentsview: kenn-io/agentsview v0.43.0
(recipes/README.md's row), --version/--help and AGENTSVIEW_TELEMETRY_ENABLED.
Test pattern precedent: the existing test_headroom_fixture_requires_a_real_saving_and_exact_recovery,
test_ast_grep_outcome_is_frozen_and_structural and
test_rtk_fixture_runs_the_upstream_inline_filter_tests in tests/test_native_token_ci.py
(unchanged by this PR) define the mock-boundary and assertRaisesRegex idiom this PR's new tests
follow.
Discriminating-controls requirement: docs/acceptance-evidence-policy.md ("Discriminating
controls" — a passing check counts as evidence only after the same check has been run with its
condition absent and failed).
Sanitization patterns (repair round): scripts/validate.py's PRIVATE_CONTENT tuple and
scan_file_for_private_content (its docstring already anticipates exactly this situation: "a
generated artifact ... is invisible to that scan ... callers that publish a built artifact must
scan it explicitly ... before attesting/uploading it").
Full-save plan: full-save/PLAN.md §3.1 PR-CI row, §4.1 (RTK negative-control table), §8
finding 3 (scratchpad; not part of this repository).

🤖 Generated with Claude Code

GPT-6's cross-family review of #375 found that Run.clean() -- the sanitizer every fixture's retained command/observation output already routes through -- did not redact two categories scripts/validate.py's own PRIVATE_CONTENT scanner already defines: bare UUIDs and personal home paths. Confirmed directly against the real hosted-CI artifact (run 36283215966's receipt.json): ai-memory 2.4.1 generates its own UUID-shaped page/workspace path components (new in this PR's ai_memory_fixture), and the pinned agentsview v0.43.0 binary's own --help text has an upstream-authored example path under /home/me that matches the scanner's pattern (only /home/example is exempted). A third, pre-existing instance from markitdown's ONNX runtime dependency (a UUID-shaped PCI bus id in stderr warnings) will also now be redacted, as a side effect of fixing the shared clean() method rather than patching each fixture. Run.clean() now applies the two PRIVATE_CONTENT regexes (copied verbatim, cited by source and kept literal-first: the existing substitutions still run before these) as a redaction pass after its existing literal substitutions. Five failing-first tests: redaction of a fabricated UUID and a fabricated home path; preservation of the /home/example and /Users/example exemptions and of ordinary non-matching text; no regression to the existing literal substitutions; and an explicit ordering test that fails under a regex-before-literal mutation. Not done here, and flagged as a residual instead: a validate.py --scan-file gate on the workflow's own artifact upload, which would be a wider, all-17- tools concern closer to the plan's separate evidence-sanitization lane than this PR's six tools. Co-Authored-By: Claude Sonnet 5 <[email protected]>

seathatflowsinourveins · 2026-09-27T01:17:24Z

GPT-6 cross-family review (`gpt-6-astra`, max effort, live search, read-only) — triage and resolution

Verdict: DEFECTS FOUND (2 findings, both triaged at source with independent evidence, both held, both resolved in one repair round; commit aa40bf70).

#	Finding (severity)	Verdict	Evidence checked	Resolution
1	`Run.clean()` doesn't redact UUID-shaped or personal-home-path text; the repo's own `scripts/validate.py` scanner would flag it (medium)	Holds	Downloaded the hosted run's own `native-token-e2e-results` artifact (run 36283215966) and read `receipt.json` directly: `state_observations["ai-memory-store"]` lists real ai-memory-generated paths with UUID components (new, from this PR's `ai_memory_fixture`); `agentsview --help`'s own text contains an upstream example under `/home/me` (new, from `agentsview_fixture`); a third, pre-existing instance from markitdown's ONNX dependency (a PCI-bus-id warning, one of the 11 tools #370 already covered) confirmed the gap predates this PR. Confirmed `scripts/validate.py`'s `PRIVATE_CONTENT` tuple's "local session identifier" and "personal home path" patterns match all three verbatim, and its `scan_file_for_private_content` docstring already anticipates exactly this case for generated/uploaded artifacts.	Fixed in two steps, each verified against a real hosted artifact rather than trusting the green checkmark alone: (a) commit `aa40bf70` added both `PRIVATE_CONTENT` patterns to `Run.clean()` as a redaction pass after its existing literal substitutions, with 5 failing-first tests including a mutation test on the literal-first/regex-second ordering; (b) re-downloading the next hosted run's artifact (commit `aa40bf70`, run `36284521807`) still showed two raw UUIDs in `state_observations["ai-memory-store"]`, because `Run.observe()` never routed its path list through `clean()` at all — fixed in commit `656faf02` (same one-line pattern, same source, one more failing-first test, confirmed failing by reverting the line and restoring it). Not done (flagged as a residual instead): a `validate.py --scan-file` gate on the workflow's artifact-upload step — that would cover all 17 tools' artifacts (the pre-existing markitdown instance included), which is a wider concern than this PR's six tools and closer to the plan's separate evidence-sanitization lane.
2	The PR/scope description of the diff size understated it ("one-line fix", "+651/-0") against the real base-relative diff (low)	Holds	Verified independently: `git diff 34c56340 41072a72 --numstat` → `scripts/native_token_ci.py` `779 29`, `tests/test_native_token_ci.py` `670 6` — matches the review's own cited numbers exactly. The smaller figures described only this integrator's second local commit relative to its first, a distinction real in local git history but invisible in the PR itself, since the repository's hot-file rebase protocol squashes both local commits into the one commit the PR contains.	Fixed: edited the PR body (before this push, per this repository's own body-edit-before-push rule) to state the full base-relative diff and explain why it isn't independently separable into "pre-existing" and "added" parts once pushed; corrected `scope.md` the same way for the record.

Hosted `native-token-e2e.yml` results

First run, 36283215966 (commit 41072a72, before the repair round): success, all steps green. receipt.json: status: "passed", failures: [], 82/82 checks true.

Tool	New checks	Result
RTK hook-check	4	all true
RTK exactness	8	all true
context-mode	5	all true
Serena	4	all true — resolves the one residual noted in `scope.md`: `serena-initialize-byte-parity-with-the-pinned-install` / `serena-tools-list-byte-parity-with-the-pinned-install` both true against the real pinned commit's installed output, not just the local synthetic test
ai-memory	4	all true
context-hub	4	all true (the AppArmor/`unshare` step added in this PR was exercised for real and worked)
agentsview	3	all true

Second run, 36284521807 (commit aa40bf70, the Run.clean() fix): success, all steps green, 82/82 checks true again. receipt.json now contains the <UUID>/<HOME-EXAMPLE> placeholders — but a direct re-scan of the downloaded artifact still found two raw UUIDs, in state_observations["ai-memory-store"] specifically, which is Run.observe()'s output, a call site Run.clean()'s fix didn't reach — see finding 1's resolution column.

Third run, 36284916855 (commit 656faf02, the Run.observe() fix): success, all steps green, 82/82 checks true. This time a full re-scan of the downloaded artifact for both PRIVATE_CONTENT patterns found zero remaining raw UUID-shaped strings and zero remaining raw personal-home-path strings anywhere in receipt.json, with both redaction placeholders present — the finding is now fully closed against real, hosted, native execution, not just the local synthetic tests.

Token tools used, and why

ToolSearch once, to load the deferred token-lane tools before first use.
mcp__plugin_context-mode_context-mode__ctx_batch_execute / ctx_execute, repeatedly, to explore native_token_ci.py's structure, review both diffs, and filter the downloaded receipt.json for UUID/home-path patterns — each call printed only the derived answer, never the raw file.
WebSearch once, to check whether GitHub's hosted ubuntu-24.04 runner already lifts Ubuntu 24.04's AppArmor unprivileged-userns restriction, since that was a live, checkable factual claim central to a correctness fix, not something to assume either way.
Read/Edit for the small number of known-path, bounded reads this session judged the content of directly (source functions cited by name/line, scope.md, this comment).
gh/git natively for PR, run, and artifact operations that need exact bytes or structured JSON (git diff --numstat, receipt.json), matching this repository's own guidance not to run those through a lossy rewrite.

Two GPT-6 (gpt-6-astra, max reasoning effort, live search) rounds did the actual code/test writing in a read/write sandbox with no git access: the initial build (9 tests + one IndexError fix) and this repair round (the Run.clean() fix + 5 tests). The review itself was a third, separate, read-only GPT-6 session.

…emory, context-hub, agentsview and an RTK exactness fix Replayed onto main c71d66d (after #364) through the hot-file protocol: the five PR files are unchanged; manifests/evidence.json is re-registered and the generated matrix and grand list are regenerated. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

Replayed onto main 881164e (after #364 and #375) with a 3-way squash: main had changed adoption/bootstrap.md, which merged cleanly; manifests/evidence.json was re-registered and the matrix and grand list regenerated. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

…e and practice docs (fix plan steps 7-8) Replayed onto main 881164e (after #364 and #375) with a 3-way squash. Main had changed three of this PR's files: - The settings template and the macOS page merged cleanly; the template keeps #364's OTEL_METRICS_INCLUDE_SESSION_ID and this PR's 11 Read(**/...) twins. - adoption/bootstrap.md conflicted on one note: it keeps this PR's extended settings-template sentence, followed by #364's writer-identity paragraph. manifests/evidence.json was re-registered, and the matrix and grand list were regenerated. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

Replayed onto main 881164e (after #364 and #375) with a 3-way squash: main had changed adoption/bootstrap.md, which merged cleanly; manifests/evidence.json was re-registered and the matrix and grand list regenerated. Co-authored-by: Scout <scout@local> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>

…e and practice docs (fix plan steps 7-8) Replayed onto main 881164e (after #364 and #375) with a 3-way squash. Main had changed three of this PR's files: - The settings template and the macOS page merged cleanly; the template keeps #364's OTEL_METRICS_INCLUDE_SESSION_ID and this PR's 11 Read(**/...) twins. - adoption/bootstrap.md conflicted on one note: it keeps this PR's extended settings-template sentence, followed by #364's writer-identity paragraph. manifests/evidence.json was re-registered, and the matrix and grand list were regenerated. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

…e and practice docs (fix plan steps 7-8) (#374) Replayed onto main 881164e (after #364 and #375) with a 3-way squash. Main had changed three of this PR's files: - The settings template and the macOS page merged cleanly; the template keeps #364's OTEL_METRICS_INCLUDE_SESSION_ID and this PR's 11 Read(**/...) twins. - adoption/bootstrap.md conflicted on one note: it keeps this PR's extended settings-template sentence, followed by #364's writer-identity paragraph. manifests/evidence.json was re-registered, and the matrix and grand list were regenerated. Co-authored-by: Scout <scout@local> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>

seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Sep 27, 2026

seathatflowsinourveins force-pushed the claude/native-token-ci-more-tools-20260926 branch from 656faf0 to fbc3a88 Compare September 27, 2026 02:47

seathatflowsinourveins merged commit 881164e into main Sep 27, 2026
25 checks passed

seathatflowsinourveins deleted the claude/native-token-ci-more-tools-20260926 branch September 27, 2026 03:14

Navigation Menu

Repository navigation

PR-CI: discriminating token-tool tests for context-mode, Serena, ai-memory, context-hub, agentsview and an RTK exactness fix - #375

PR-CI: discriminating token-tool tests for context-mode, Serena, ai-memory, context-hub, agentsview and an RTK exactness fix#375
seathatflowsinourveins merged 1 commit into
mainfrom
claude/native-token-ci-more-tools-20260926

seathatflowsinourveins commented Sep 27, 2026 •

edited

Loading

Uh oh!

seathatflowsinourveins commented Sep 27, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

seathatflowsinourveins commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Verification

Residuals

SOTA sources

Uh oh!

seathatflowsinourveins commented Sep 27, 2026

GPT-6 cross-family review (gpt-6-astra, max effort, live search, read-only) — triage and resolution

Hosted native-token-e2e.yml results

Token tools used, and why

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

seathatflowsinourveins commented Sep 27, 2026 •

edited

Loading

GPT-6 cross-family review (`gpt-6-astra`, max effort, live search, read-only) — triage and resolution

Hosted `native-token-e2e.yml` results