Repository navigation
fix(evidence): verify scheduled memory reports and scope digest exceptions - #400
Merged
Merged
Conversation
seathatflowsinourveins
enabled auto-merge (squash)
September 27, 2026 13:19
seathatflowsinourveins
force-pushed
the
codex/memory-daily-20260927
branch
from
September 28, 2026 13:16
9ddaa79 to
01c5dc0
Compare
seathatflowsinourveins
pushed a commit
that referenced
this pull request
Sep 28, 2026
The branch was rebased onto 9f8db58 (#400) with every patch unchanged (git range-diff shows each pair as =). review-444/README.md says 0779e91a, the reviewed head, was never pushed and was 7e5eddcd plus a registration commit, and gives the rebased SHAs of 9e036e7d and 7e5eddcd; the decision record points there. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
5 tasks done
seathatflowsinourveins
pushed a commit
that referenced
this pull request
Sep 28, 2026
The branch was rebased onto 9f8db58 (#400) with every patch unchanged (git range-diff shows each pair as =). review-444/README.md says 0779e91a, the reviewed head, was never pushed and was 7e5eddcd plus a registration commit, and gives the rebased SHAs of 9e036e7d and 7e5eddcd; the decision record points there. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins
pushed a commit
that referenced
this pull request
Sep 28, 2026
The branch was rebased onto 9f8db58 (#400) with every patch unchanged (git range-diff shows each pair as =). review-444/README.md says 0779e91a, the reviewed head, was never pushed and was 7e5eddcd plus a registration commit, and gives the rebased SHAs of 9e036e7d and 7e5eddcd; the decision record points there. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins
added a commit
that referenced
this pull request
Sep 28, 2026
…CLAUDE.md, 4 of 4 blind judgments; dashboard manifest bound (#444) * AGENTS.md: scope role dispatch to new or ad-hoc stages; current-state holdout wording Two items of the 2026-09-27 prompt audit, resolved by cross-family convergence (docs/decisions/2026-09-27-prompt-audit-resolution.md, lane:foundation PR): - Role dispatch applies to each new or ad-hoc workflow agent() stage; the saved scripts vendored in examples/claude-native/workflows keep their reviewed routing, byte-identical to agent-lab, as that README states in its opening paragraph and the Workflow contract's Dispatch by role bullet (F1). - The inspected 2021 control segment: "has been inspected, so no experiment may present it as a fresh untouched holdout" (F4). o200k (gpt-tokenizer 3.4.0): AGENTS.md 2,374 -> 2,401 tokens (+27). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * X9 second round: blind two-family re-judgment splits again; CLAUDE.md unchanged After round 1 split 2 and 2 on X9 (CLAUDE.md:3-4), a second round added: - the installed client's own refusals of /context and /mcp through the Skill tool; - an executed comparison of the three texts, all headless claude -p runs through promptfoo 0.123.1: K1-K3 as 27 preregistered runs on Opus 5.5, and K4 as 9 runs on Sonnet 5; - 38 dated sources, plus 5 recorded searches that found nothing. Attempt 1 was void, because its judges could read this record and the coordinator's work directory. Attempt 2 was final. Its judges read a plain export of ba1700a and judged in both orders: blind-adjudicator agents and the packaged GPT-6 runner. A void audit, hashed before any return was read, found no voiding hit. Both GPT-6 judgments chose the GPT-6 lane's text, and both Claude judgments chose the Claude lane's text, so under the unanimity rule nothing is applied. All eight judgments across both rounds reject the current line. The addendum records both positions and the comparison that would decide it. Review: - One round, by GPT-6 and Claude reviewers, then one repair round: - disclose that K4 ran on Sonnet 5 while the packets said Opus 5.5; - make check_orders.py exit 1 on a difference, and add retally.py; - correct the counts; - add receipts (run windows, settings file names per run, the K4 skill-list sentences, the isolation receipt, a source supplement); - fix the wording. - A separate re-check found all 13 items fixed and two new defects: an overstated "keeps only" claim and an overstated account of K4's origin. Both are fixed. The recovered first check_orders.py is kept to show its flaw. - A pre-handoff review found that the published packets, prompts and K4 files quoted answer heads containing client configuration (an enabled-plugin entry, status-line wiring, plugin install records). Every published copy now withholds them. attempt2/sent-sha256.json keeps the hashes of the files as sent. This commit squashes the earlier ones, so no commit on the branch carries that text. docs/harness-defaults.md gains three dated anti-pattern rows: - a model scope taken from the preregistration instead of each run's recorded model; - a check that reported a mismatch and exited 0; - judge inputs published with answer text that quotes client configuration. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Prompt audit round 3: X9 applied to CLAUDE.md; lane A's S1 and X5b in AGENTS.md; AN-13 receipt - CLAUDE.md lines 3-8 (X9): round 3 compared the two texts on the measure the second round named (native /mcp and /context refusal in print mode, with a promptfoo-graded, preregistered comparison) and the blind adjudication chose the GPT-6 lane's text in all four judgments. - AGENTS.md:3 (X5b): the operator's 2026-09-28 top-rule paragraph, byte for byte; AGENTS.md:37 (S1): build a PR description from the pull request template. Both converged in lane A (agree / agree). X5a and X5c land in #458. - AN-13: the headless /doctor prompt-audit run the 2026-09-28 community sweep left open. Facts of both runs in evidence/artifacts/prompt-audit-20260927/an13/ (the report text is not published); no edit made from it; each finding's disposition is in its README and in the decision record. - Decision record and x9-round3/ evidence (preregistration, isolation receipt, fixtures, judges, usage). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Prompt audit: print-mode anti-pattern row (AN-13), line-7 co-change with X5b, round-3 token receipts - docs/harness-defaults.md: a dated row for running a headless skill that starts background agents under print mode's default 600 s wait (AN-13's first run: four Explore agents stopped by the client, an interim note instead of the report); the rerun with CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS=0 returned the report (code.claude.com/docs/en/env-vars). - docs/harness-defaults.md:7 restates AGENTS.md's new top-rule heading (X5b), disclosed in the decision record. - controls/token-counts-r3-{f508ffb,base,after}.txt: the o200k counts the record cites for CLAUDE.md (30 -> 96 at f508ffb; 96 -> 162 at fb14ded) and AGENTS.md (2,374 -> 2,484 at fb14ded). - The record cites validate.yml's heading rule at both revisions and notes the Codex-lane host step done on this workstation after #458. - x9-round3/README.md: the scan note names every kind of match (names, not values), including one judgment's quote of the untested cases. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Grand dashboard: the evidence manifest gets its own read bound This pull request's registrations take manifests/evidence.json past the 2,000,000-byte bound progress.py applied to every source (1,987,653 bytes on main at fb14ded; 2,017,827 with this branch), so the snapshot refused it and eight dashboard tests failed. The snapshot reads only the manifest's receipt count: SOURCE_MAX_BYTES gives the manifest 8,000,000 bytes and every other source keeps MAX_BYTES. read(root, relative) keeps its signature (the tests patch it). The new test fails on the unchanged reader (controls/dashboard-bound-before.txt) and passes with 16 others (dashboard-bound-after.txt). Overturn: when the manifest nears the new bound, count receipts from a smaller source instead of raising it again. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Prompt audit round 3: review repairs, the audit's control, the Codex lane host receipt One review round on 0779e91a (GPT-6 codex exec and a Claude evidence-reviewer, both changes-needed); every finding was in the record or the evidence text: - the record: --max-turns 10 with one 11-turn result; every builder change after the freeze; the applied X9 text ties with the line it replaced on the preregistered metrics; the runs used ba1700a's files; X5b's wording and the tests that read AGENTS.md; the advisor calls; M4 is project-side; a review subsection; - x9-round3: audit_control_r3.py and judges/audit-control.json (clean copies reproduce the published verdicts; one planted access per judgment voids all four); the README lists every builder change, the judge-action flag and the tested configuration; - codex-worker-lane-host-20260928: the 12:14-12:15Z dry run, apply and proof, sanitized as #406's were, and a recheck with exit codes and times (dry run: block in place; proof 7 of 7); - review-444: both prompts, returns, usage and each finding's disposition. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Prompt audit: map the reviewed commits to the rebased ones The branch was rebased onto 9f8db58 (#400) with every patch unchanged (git range-diff shows each pair as =). review-444/README.md says 0779e91a, the reviewed head, was never pushed and was 7e5eddcd plus a registration commit, and gives the rebased SHAs of 9e036e7d and 7e5eddcd; the decision record points there. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Prompt audit: #446 carried the dashboard bound; map reviewed commits by subject After the rebase onto 3058b23, the dashboard commit keeps only its record subsection and the two receipts: #446 landed the same progress.py and test hunks (two-file patch-id 8f693abd4c601190). The review README now names the reviewed commits by subject, since each rebase changes their SHAs. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * AN-13: point M4 and M5 to their 2026-09-28 second-family lanes M4: both lanes reject the rewrite (no edit; the preload stays with the 2026-10-25 comparison). M5: the lanes amended with different texts and the blind adjudication split (unchanged). Both are recorded in docs/decisions/2026-09-28-an13-m4-m5.md, a separate lane:foundation PR. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * AN-13 M4 pointer: the trial's removal rule is open (#460) #460's record now says the M4 lanes did not test the skills trial's rule that removes a trial skill whose instructions conflict with CLAUDE.md or AGENTS.md, and leaves that question open. Say so where this record points to it. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Register the instruction files, the decision record, the anti-pattern rows and the prompt-audit evidence Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> --------- Co-authored-by: Scout <scout@local> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
Correct the memory/RAG handbook's outdated settings and retain the actual September 27–28 scheduled observations. Repair PR400's seven Gitleaks false positives with one separate exact-path/seven-value exception; unrelated keys, paths, credentials and the earlier exception remain covered.
fb14dedflane:foundation.gitleaks.toml, its regression tests and evidence hashes.SOTA sources
regexTarget = "line"; this extends the existing repository exception/test pattern without a blanket path exemption.62e73148fa4712540dab82d7fadf2ebf09a42cec:crates/ai-memory-store/src/auto_improve.rs:1368-1396constructs SHA-256 rejection fingerprints;reader.rs:8701-8727returns them as report keys;crates/ai-memory-consolidate/src/auto_improve_telemetry.rs:179-191copies those values into findings. Native output and redacted CI artifact locations agree.config.rs,embedding.rs, and MCPserver.rs. #859 merged as7552d4fd390c1370a09f7452dfbd76bdd707db47into release/2.5; #861 merged as21fc32800c2c9549692372241310e675a5b25240. Stock2.4.1 lacks the installed prefixes, while its query-dispatch fix is already present locally.observability/memory-scheduled-20260923.jsonate833e3ef. Native QMD search/get and SocratiCode retrieval were checked against original source.Evidence-class table
Both native reports return 83 runs /19 approved proposals; report bodies match except reporting-window timestamps. Hourly learning stays paused. On September28, native session-end capture and the store's matching generation31006 completed at01:26:43Z agree. Completion does not prove provider success, absence of heuristic fallback, quality or token savings. Native default session selection sorts by start time; explicit latest-end retrieval resolves the apparent difference.
Local commands run
The original GitHub secret scan reported seven findings, all retained in the actual returned report. New test_e3 reproduced those seven findings before the fix. The final native scan includes the committed config/test changes; this is bounded branch coverage, not a local whole-history scan. GitHub still performs its own required checks.
Decision record and review
observability/memory-scheduled-20260927.jsonandobservability/memory-scheduled-20260928.jsonretain source decisions, failures, command results and limits. Full sanitized September28 results are retained privately by hash; unchanged report content reuses the published September27 result.Independent review confirmed the seven values against upstream construction, exactly one added exception, preserved old restrictions, adverse controls, current monitoring and native session/store agreement. No remaining supported finding.
No install, restart, model/account/permission change, dedicated model qualification, manual learning, new monitor, broker operation or paid resource. Required semantic retrieval uses embedding inference. Exact savings and restart persistence remain unknown.