Skip to content

fix(evidence): verify scheduled memory reports and scope digest exceptions - #400

Merged
seathatflowsinourveins merged 6 commits into
mainfrom
codex/memory-daily-20260927
Sep 28, 2026
Merged

seathatflowsinourveins merged 6 commits into
mainfrom
codex/memory-daily-20260927

Conversation

@seathatflowsinourveins

@seathatflowsinourveins seathatflowsinourveins commented Sep 27, 2026 •

Copy link
Copy Markdown
Collaborator

Scope

Correct the memory/RAG handbook's outdated settings and retain the actual September 27–28 scheduled observations. Repair PR400's seven Gitleaks false positives with one separate exact-path/seven-value exception; unrelated keys, paths, credentials and the earlier exception remain covered.

  • Base commit: fb14dedf
  • Lane: lane:foundation
  • Owned paths: lifecycle handbooks, two correction-log rows, dated observation/results files, .gitleaks.toml, its regression tests and evidence hashes.

SOTA sources

  • Gitleaks v8.30.1 configuration: rule-specific AND criteria and regexTarget = "line"; this extends the existing repository exception/test pattern without a blanket path exemption.
  • Installed ai-memory source 62e73148fa4712540dab82d7fadf2ebf09a42cec: crates/ai-memory-store/src/auto_improve.rs:1368-1396 constructs SHA-256 rejection fingerprints; reader.rs:8701-8727 returns them as report keys; crates/ai-memory-consolidate/src/auto_improve_telemetry.rs:179-191 copies those values into findings. Native output and redacted CI artifact locations agree.
  • ai-memory v2.4.1: config.rs, embedding.rs, and MCP server.rs. #859 merged as 7552d4fd390c1370a09f7452dfbd76bdd707db47 into release/2.5; #861 merged as 21fc32800c2c9549692372241310e675a5b25240. Stock2.4.1 lacks the installed prefixes, while its query-dispatch fix is already present locally.
  • Existing observation format: observability/memory-scheduled-20260923.json at e833e3ef. Native QMD search/get and SocratiCode retrieval were checked against original source.

Evidence-class table

Claim Evidence class Command / receipt
Actual scheduler envelopes, native reports, scoped retrieval and completed-session readback native_proven September27 returned results and September28 receipt
Existing services/collectors healthy, 9/9 targets up; swap warning cleared by September28 observation native_proven Independent observer's retained native service/API results
Prefix compatibility and digest construction source_review Pinned files above and both dated receipts
Narrow exception accepts reviewed digests while adverse controls remain detected synthetic 23 project regression tests using guarded Gitleaks8.30.1
Changed committed history passes native scanner native_proven Guarded 6-commit scan, returned output below
Publication integrity and handbook integration local_integration Project checks below

Both native reports return 83 runs /19 approved proposals; report bodies match except reporting-window timestamps. Hourly learning stays paused. On September28, native session-end capture and the store's matching generation31006 completed at01:26:43Z agree. Completion does not prove provider success, absence of heuristic fallback, quality or token savings. Native default session selection sorts by start time; explicit latest-end retrieval resolves the apparent difference.

Local commands run

ecosystem-bounded-run python3 scripts/validate.py
passed:69 components,4 profiles,161 receipts,7661 hashed files
ecosystem-bounded-run python3 scripts/validate_catalogs.py
passed
ecosystem-bounded-run python3 -m unittest tests.test_grand_dashboard tests.test_ecosystem_manifest
72 tests passed after current-main rebase
ecosystem-bounded-run python3 -m unittest tests.test_gitleaks_config.GitleaksConfigContextRestrictionTests
23 tests passed
gitleaks version
8.30.1
gitleaks git . --config .gitleaks.toml --log-opts=origin/main..HEAD --redact --no-banner
6 commits scanned;~54501 bytes;no leaks found;exit0
python3 scripts/validate.py --scan-file observability/memory-scheduled-20260928.json
passed
git diff --check
passed

The original GitHub secret scan reported seven findings, all retained in the actual returned report. New test_e3 reproduced those seven findings before the fix. The final native scan includes the committed config/test changes; this is bounded branch coverage, not a local whole-history scan. GitHub still performs its own required checks.

Decision record and review

observability/memory-scheduled-20260927.json and observability/memory-scheduled-20260928.json retain source decisions, failures, command results and limits. Full sanitized September28 results are retained privately by hash; unchanged report content reuses the published September27 result.

Independent review confirmed the seven values against upstream construction, exactly one added exception, preserved old restrictions, adverse controls, current monitoring and native session/store agreement. No remaining supported finding.

No install, restart, model/account/permission change, dedicated model qualification, manual learning, new monitor, broker operation or paid resource. Required semantic retrieval uses embedding inference. Exact savings and restart persistence remain unknown.

@seathatflowsinourveins seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Sep 27, 2026
@seathatflowsinourveins
seathatflowsinourveins enabled auto-merge (squash) September 27, 2026 13:19
@seathatflowsinourveins seathatflowsinourveins changed the title docs(memory): refresh scheduled lifecycle evidence and current settings fix(evidence): verify scheduled memory reports and scope digest exceptions Sep 28, 2026
@seathatflowsinourveins
seathatflowsinourveins merged commit 9f8db58 into main Sep 28, 2026
26 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the codex/memory-daily-20260927 branch September 28, 2026 13:35
seathatflowsinourveins pushed a commit that referenced this pull request Sep 28, 2026
The branch was rebased onto 9f8db58 (#400) with every patch unchanged (git range-diff
shows each pair as =). review-444/README.md says 0779e91a, the reviewed head, was never
pushed and was 7e5eddcd plus a registration commit, and gives the rebased SHAs of
9e036e7d and 7e5eddcd; the decision record points there.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins pushed a commit that referenced this pull request Sep 28, 2026
The branch was rebased onto 9f8db58 (#400) with every patch unchanged (git range-diff
shows each pair as =). review-444/README.md says 0779e91a, the reviewed head, was never
pushed and was 7e5eddcd plus a registration commit, and gives the rebased SHAs of
9e036e7d and 7e5eddcd; the decision record points there.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins pushed a commit that referenced this pull request Sep 28, 2026
The branch was rebased onto 9f8db58 (#400) with every patch unchanged (git range-diff
shows each pair as =). review-444/README.md says 0779e91a, the reviewed head, was never
pushed and was 7e5eddcd plus a registration commit, and gives the rebased SHAs of
9e036e7d and 7e5eddcd; the decision record points there.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins added a commit that referenced this pull request Sep 28, 2026
…CLAUDE.md, 4 of 4 blind judgments; dashboard manifest bound (#444)

* AGENTS.md: scope role dispatch to new or ad-hoc stages; current-state holdout wording

Two items of the 2026-09-27 prompt audit, resolved by cross-family convergence
(docs/decisions/2026-09-27-prompt-audit-resolution.md, lane:foundation PR):

- Role dispatch applies to each new or ad-hoc workflow agent() stage; the saved
  scripts vendored in examples/claude-native/workflows keep their reviewed routing,
  byte-identical to agent-lab, as that README states in its opening paragraph and the
  Workflow contract's Dispatch by role bullet (F1).
- The inspected 2021 control segment: "has been inspected, so no experiment may
  present it as a fresh untouched holdout" (F4).

o200k (gpt-tokenizer 3.4.0): AGENTS.md 2,374 -> 2,401 tokens (+27).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* X9 second round: blind two-family re-judgment splits again; CLAUDE.md unchanged

After round 1 split 2 and 2 on X9 (CLAUDE.md:3-4), a second round added:
- the installed client's own refusals of /context and /mcp through the Skill
  tool;
- an executed comparison of the three texts, all headless claude -p runs
  through promptfoo 0.123.1: K1-K3 as 27 preregistered runs on Opus 5.5, and
  K4 as 9 runs on Sonnet 5;
- 38 dated sources, plus 5 recorded searches that found nothing.

Attempt 1 was void, because its judges could read this record and the
coordinator's work directory. Attempt 2 was final. Its judges read a plain
export of ba1700a and judged in both orders: blind-adjudicator agents and the
packaged GPT-6 runner. A void audit, hashed before any return was read, found
no voiding hit.

Both GPT-6 judgments chose the GPT-6 lane's text, and both Claude judgments
chose the Claude lane's text, so under the unanimity rule nothing is applied.
All eight judgments across both rounds reject the current line. The addendum
records both positions and the comparison that would decide it.

Review:
- One round, by GPT-6 and Claude reviewers, then one repair round:
  - disclose that K4 ran on Sonnet 5 while the packets said Opus 5.5;
  - make check_orders.py exit 1 on a difference, and add retally.py;
  - correct the counts;
  - add receipts (run windows, settings file names per run, the K4
    skill-list sentences, the isolation receipt, a source supplement);
  - fix the wording.
- A separate re-check found all 13 items fixed and two new defects: an
  overstated "keeps only" claim and an overstated account of K4's origin.
  Both are fixed. The recovered first check_orders.py is kept to show its
  flaw.
- A pre-handoff review found that the published packets, prompts and K4 files
  quoted answer heads containing client configuration (an enabled-plugin
  entry, status-line wiring, plugin install records). Every published copy
  now withholds them. attempt2/sent-sha256.json keeps the hashes of the files
  as sent. This commit squashes the earlier ones, so no commit on the branch
  carries that text.

docs/harness-defaults.md gains three dated anti-pattern rows:
- a model scope taken from the preregistration instead of each run's
  recorded model;
- a check that reported a mismatch and exited 0;
- judge inputs published with answer text that quotes client configuration.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Prompt audit round 3: X9 applied to CLAUDE.md; lane A's S1 and X5b in AGENTS.md; AN-13 receipt

- CLAUDE.md lines 3-8 (X9): round 3 compared the two texts on the
  measure the second round named (native /mcp and /context refusal in
  print mode, with a promptfoo-graded, preregistered comparison) and the
  blind adjudication chose the GPT-6 lane's text in all four judgments.
- AGENTS.md:3 (X5b): the operator's 2026-09-28 top-rule paragraph, byte
  for byte; AGENTS.md:37 (S1): build a PR description from the pull
  request template. Both converged in lane A (agree / agree). X5a and X5c
  land in #458.
- AN-13: the headless /doctor prompt-audit run the 2026-09-28 community
  sweep left open. Facts of both runs in
  evidence/artifacts/prompt-audit-20260927/an13/ (the report text is not
  published); no edit made from it; each finding's disposition is in its
  README and in the decision record.
- Decision record and x9-round3/ evidence (preregistration, isolation
  receipt, fixtures, judges, usage).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Prompt audit: print-mode anti-pattern row (AN-13), line-7 co-change with X5b, round-3 token receipts

- docs/harness-defaults.md: a dated row for running a headless skill
  that starts background agents under print mode's default 600 s wait
  (AN-13's first run: four Explore agents stopped by the client, an
  interim note instead of the report); the rerun with
  CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS=0 returned the report
  (code.claude.com/docs/en/env-vars).
- docs/harness-defaults.md:7 restates AGENTS.md's new top-rule heading
  (X5b), disclosed in the decision record.
- controls/token-counts-r3-{f508ffb,base,after}.txt: the o200k counts
  the record cites for CLAUDE.md (30 -> 96 at f508ffb; 96 -> 162 at
  fb14ded) and AGENTS.md (2,374 -> 2,484 at fb14ded).
- The record cites validate.yml's heading rule at both revisions and
  notes the Codex-lane host step done on this workstation after #458.
- x9-round3/README.md: the scan note names every kind of match (names,
  not values), including one judgment's quote of the untested cases.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Grand dashboard: the evidence manifest gets its own read bound

This pull request's registrations take manifests/evidence.json past the
2,000,000-byte bound progress.py applied to every source (1,987,653 bytes
on main at fb14ded; 2,017,827 with this branch), so the snapshot refused
it and eight dashboard tests failed. The snapshot reads only the
manifest's receipt count: SOURCE_MAX_BYTES gives the manifest 8,000,000
bytes and every other source keeps MAX_BYTES. read(root, relative) keeps
its signature (the tests patch it). The new test fails on the unchanged
reader (controls/dashboard-bound-before.txt) and passes with 16 others
(dashboard-bound-after.txt). Overturn: when the manifest nears the new
bound, count receipts from a smaller source instead of raising it again.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Prompt audit round 3: review repairs, the audit's control, the Codex lane host receipt

One review round on 0779e91a (GPT-6 codex exec and a Claude evidence-reviewer, both
changes-needed); every finding was in the record or the evidence text:
- the record: --max-turns 10 with one 11-turn result; every builder change after the
  freeze; the applied X9 text ties with the line it replaced on the preregistered
  metrics; the runs used ba1700a's files; X5b's wording and the tests that read
  AGENTS.md; the advisor calls; M4 is project-side; a review subsection;
- x9-round3: audit_control_r3.py and judges/audit-control.json (clean copies reproduce
  the published verdicts; one planted access per judgment voids all four); the README
  lists every builder change, the judge-action flag and the tested configuration;
- codex-worker-lane-host-20260928: the 12:14-12:15Z dry run, apply and proof, sanitized
  as #406's were, and a recheck with exit codes and times (dry run: block in place;
  proof 7 of 7);
- review-444: both prompts, returns, usage and each finding's disposition.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Prompt audit: map the reviewed commits to the rebased ones

The branch was rebased onto 9f8db58 (#400) with every patch unchanged (git range-diff
shows each pair as =). review-444/README.md says 0779e91a, the reviewed head, was never
pushed and was 7e5eddcd plus a registration commit, and gives the rebased SHAs of
9e036e7d and 7e5eddcd; the decision record points there.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Prompt audit: #446 carried the dashboard bound; map reviewed commits by subject

After the rebase onto 3058b23, the dashboard commit keeps only its
record subsection and the two receipts: #446 landed the same
progress.py and test hunks (two-file patch-id 8f693abd4c601190). The
review README now names the reviewed commits by subject, since each
rebase changes their SHAs.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* AN-13: point M4 and M5 to their 2026-09-28 second-family lanes

M4: both lanes reject the rewrite (no edit; the preload stays with the
2026-10-25 comparison). M5: the lanes amended with different texts and
the blind adjudication split (unchanged). Both are recorded in
docs/decisions/2026-09-28-an13-m4-m5.md, a separate lane:foundation PR.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* AN-13 M4 pointer: the trial's removal rule is open (#460)

#460's record now says the M4 lanes did not test the skills trial's rule that
removes a trial skill whose instructions conflict with CLAUDE.md or AGENTS.md,
and leaves that question open. Say so where this record points to it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Register the instruction files, the decision record, the anti-pattern rows and the prompt-audit evidence

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

---------

Co-authored-by: Scout <scout@local>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant