Skip to content

Shared fail-closed credential guard for both paper loaders (market_research gap closed) - #219

Merged
seathatflowsinourveins merged 11 commits into
mainfrom
claude/market-research-credential-guard-20260924
Sep 25, 2026
Merged

seathatflowsinourveins merged 11 commits into
mainfrom
claude/market-research-credential-guard-20260924

Conversation

@seathatflowsinourveins

@seathatflowsinourveins seathatflowsinourveins commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

What

This closes a documented paper-lane credential gap. market_research.py read the Alpaca paper key file through its own loader, which had no owner, mode or outside-worktree checks. Both paper loaders, runner.credentials() and market_research.credentials(), now go through one shared, stdlib-only guard, blueprints/us-equities/adaptive-paper/credential_guard.py.

The guard

Opening the file.

  • The walk descends component by component with dir_fd from /. Every ancestor is opened with O_DIRECTORY|O_NOFOLLOW, and the file with O_RDONLY|O_NOFOLLOW|O_NONBLOCK|O_NOCTTY.
  • Every check runs on the fd that was actually opened.
  • A symlink swapped in during the walk fails closed.
  • In no-follow mode, .. is refused outright.

Ancestor and parent rules.

  • Every ancestor follows OpenSSH's secure_filename rule: it must be owned by root or by getuid(). It may be group- or world-writable only if it is sticky and the next component down is owned by getuid() or root.
  • Every opened ancestor is checked for a .git entry at the moment it is inspected.
  • The immediate parent must be owned by exactly getuid() and must not be group- or world-writable. So /tmp/file and /file are refused, while private directories under /tmp or $HOME load.

The file itself.

  • It must be owned by getuid(), have mode exactly 0600, have st_nlink == 1, and be a regular file.
  • At most MAX_CREDENTIAL_BYTES + 1 bytes are read.
  • The contents must pass isascii() before any decode, so no UnicodeDecodeError can carry file bytes.

Errors and descriptors.

  • Every refusal raises one fixed credential_file_permissions:<code>: missing, not_regular, symlink, owner, mode, hardlink, worktree, ancestor, parent_owner, parent_mode, encoding or size.
  • No refusal carries a path, a basename or file content. It is raised outside any except block, so __context__ and __cause__ stay empty.
  • The fd is closed exactly once on every failure path: FileIO is built first; if that fails, the fd is closed directly, and if the later wrap fails, the FileIO is closed.

Stated limits (decision doc and gate note):

  • A worktree with no marker (GIT_DIR, GIT_WORK_TREE or core.worktree with no .git ancestor) cannot be detected by a check local to the file.
  • A bind-mounted repository subdirectory is also undetectable.
  • The .git check is a per-ancestor snapshot taken when each ancestor is inspected, not an atomic location guarantee.
  • Runner's symlink-follow mode accepting a compliant replacement file remains documented.

runner.credentials() keeps its accepted configurations and its credential_file_permissions prefix.

Review

Six rounds, with two model families on each round.

Codex, gpt-6-astra (read-only security review), found:

  • a parent-directory race;
  • a hard-link bypass;
  • ancestor relocation by another UID;
  • exception chains carrying a path or the credential bytes;
  • a descriptor leak and a double close;
  • a mutation driver reporting false kills.

Claude reviewers ran real attacks and real mutations:

  • Attacks: 312 attack cases across both loaders and both guard modes, plus a live RENAME_EXCHANGE race of about 660k swaps per 6 s run. Every case failed closed, and none produced an attacker read in the worktree or no-follow cases.
  • Mutations: 46 of 47 were killed by real credential-test failures. The last survivor was then killed in round 6.
  • Driver checks: the builder's subprocess mutation driver needs a pristine baseline and counts only a named-test FAIL. Skip, error and teardown outcomes are reported as inconclusive. It produced no false kills across 22 probe scenarios.

Final verdicts: Claude said ready at 68ad970. Round 6 closed the last test and driver items from both reviewers. The most important was the /tmp test, which would have failed under macOS's private TMPDIR.

Checks

  • Adaptive suite: 866 tests OK (1 skip) on the pinned runtime, both with TMPDIR unset and set to a private directory.
  • Other suites: tests.test_native_faults_min and tests.test_credential_tools give 45 OK.
  • Validators: all six pass (5320 hashed files).
  • Mutation driver: 17 of 17 declared mutations killed. The driver's own self-tests pass 6 of 6.
  • Evidence class: local integration (synthetic fixtures and real filesystem races on this host). No credential file was read.

Later rounds (after the PR opened)

  • Secret-scan false positive. File paths containing "credential" made gitleaks' generic-api-key rule flag the sha256 values in adaptive-paper/source-hashes.json. The fix:

    • a dedicated rule-scoped allowlist for that exact file only. Keys must be real repo paths under adaptive-paper/ or tests/, with no . or .. segments, and values must be 64 lowercase hex;
    • a new stdlib-only CI test (tests/test_adaptive_paper_source_hashes_manifest.py) that fails unless every entry is the real sha256 of an existing file, so a secret cannot sit in that manifest;
    • negative gitleaks config tests (25 OK).
  • macOS portability. On macOS, /tmp and /var are OS symlinks, which no-follow mode refuses, so the test fixtures now use realpath temporary roots, and fd counting is portable. validate-macos passes. The docs state that no-follow mode refuses OS symlinks.

  • Review limitation. The Codex CLI reached its usage limit, which resets 2026-09-30 23:50, so round 7 and the dot-segment fix had no cross-family review. The independent Claude reviewer covered them instead:

    • no secret got past the allowlist and manifest test in any smuggling probe;
    • the macOS-layout simulation passed 112/112 tests;
    • 47/47 mutations were killed;
    • the guard code is unchanged since the security rounds apart from docstrings.

    The one Low it found (dot segments) is fixed in 04c867b.

🤖 Generated with Claude Code

Scout and others added 8 commits September 24, 2026 15:51
…closed rules

market_research.py's own credentials(path) opened its env file with only
O_NOFOLLOW, a regular-file check, and a size cap -- unlike runner.credentials(),
it never checked owner uid, mode 0600, or outside-any-Git-worktree location
(documented as an open gap in gates-20260922.json's credential-handling note
and docs/decisions/2026-09-22-broker-credential-handling.md).

Add credential_guard.py (stdlib-only: os, stat, pathlib) with open_verified(),
a TOCTOU-safe (fstat re-checked against a pre-open lstat by device/inode, the
pattern tools/credentials/alpaca_rate_limit_probe.py:read_env_file already
uses) fail-closed check for owner uid, mode 0600, and outside-any-worktree
location, parameterized by follow_symlinks so each loader keeps its own prior
symlink behavior: runner.credentials() still resolves a symlink to its target
(follow_symlinks=True), market_research.credentials() still refuses a symlink
outright (follow_symlinks=False, its prior O_NOFOLLOW rule). A FIFO is refused
by the pre-open lstat regular-file check before open() is ever reached, so it
can never hang either loader. No raised error includes file content.

Both loaders now import the same open_verified(), so the rules cannot drift.
runner.credentials()'s asserted "credential_file_permissions" error-string
substring is unchanged and its existing tests pass unmodified.

Tests: new tests.test_adaptive_market_research.MarketResearchCredentialFilePermissions
(wrong mode, wrong owner, symlink, inside a Git worktree, missing file, a FIFO
that must not hang, valid file), new symlink/FIFO cases added to
tests.test_adaptive_paper_runner.CredentialFilePermissions, and a new
SharedCredentialGuardParity test asserting both loaders import the identical
open_verified and reject the same violation the same way.

Update source-hashes.json and manifests/evidence.json (sha256/bytes only, for
the changed registered files) and close the gap in gates-20260922.json's
credential-handling note and the 2026-09-22 decision doc, naming the tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
runner.py and market_research.py import it, so it is an engine source like every
other adaptive-paper module.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…e hard-link/worktree/encoding gaps

Independent Codex security review of 9b02cc8 returned "not ready" with 7
items; this commit addresses each.

1. MEDIUM parent-directory race: the previous fix still resolved a pathname
   once, checked that fd, then separately re-resolved the path (by string)
   for the Git-worktree walk -- a swap of an ancestor directory between
   those two steps was not caught. Rewritten so every check is bound to the
   same dir_fd-chained traversal that produces the returned descriptor: from
   "/", each ancestor opens with O_DIRECTORY|O_NOFOLLOW via dir_fd, the
   final component with O_RDONLY|O_NOFOLLOW|O_NONBLOCK|O_NOCTTY via dir_fd,
   and the Git-worktree check runs on each already-opened ancestor fd
   (os.stat(".git", dir_fd=fd, follow_symlinks=False)) as it is opened, not
   on a re-resolved pathname afterward. A symlink substituted at any
   position after the one-time initial resolution now fails closed with
   ELOOP/ENOTDIR. The immediate parent directory must be owned by the caller
   and not group/other-writable. gates-20260922.json's note now states
   exactly what this does and does not guarantee.
2. MEDIUM hard links: st_nlink != 1 is now refused (a second name for the
   same inode could be compliant at one path and worktree-tracked at
   another). Regression tests added to both loaders.
3. MEDIUM worktrees without a `.git` marker: narrowed the claim in
   credential_guard.py's docstring, the decision doc, and the gate note --
   detection only recognizes an ancestor `.git` entry; a tree configured
   purely through GIT_DIR/GIT_WORK_TREE or core.worktree is not detectable
   by a file-local check and is explicitly called out as out of scope.
4. MEDIUM race protections were untested: added
   tests/test_adaptive_paper_credential_race.py. credential_guard._hook is a
   no-op in production and lets tests deterministically inject a filesystem
   mutation at an exact traversal stage (ancestor-directory symlink swap,
   final-component symlink/FIFO swap, metadata change between open and
   fstat) instead of relying on real concurrency. A naive resolve-then-open
   reference implementation demonstrates the same ancestor-swap silently
   succeeds against it while open_verified() rejects it. Every "does not
   hang" assertion runs under a signal.alarm deadline. A mutation-kill
   report disables one guard protection at a time (hard-link check, worktree
   check, parent-directory check, O_NOFOLLOW, O_NONBLOCK) and asserts a
   specific named test now fails to reject the case that protection existed
   for.
5. LOW encoding: both loaders now require and document ASCII explicitly
   (matching what every writer of this file emits) instead of silently
   accepting UTF-8; a non-ASCII byte is refused before any line is parsed.
   Regression tests added.
6. LOW error content: every rejection now raises a single fixed,
   interpolation-free message (credential_guard.GUARD_DENIED) -- no path,
   basename, or upstream OSError text is ever included. Regression tests
   assert this explicitly.
7. Documentation: "effective user" corrected to os.getuid() (the real uid,
   not an "effective user" concept); the decision doc corrected to say
   market_research.credentials() raises ResearchError, not SafetyError;
   market_research.credentials()'s size cap is now enforced by reading at
   most MAX_CREDENTIAL_BYTES + 1 bytes from the opened fd and rejecting a
   longer result, not by trusting an earlier fstat-reported size that a
   concurrent writer could grow past.

Runner's accepted configurations are unchanged: a normal 0600 file you own,
in a 0700 or 0755 directory you own, outside any worktree, still loads
(explicit regression test added). runner.credentials()'s asserted
"credential_file_permissions" error-string substring is unchanged.

source-hashes.json and manifests/evidence.json updated surgically (sha256/
bytes only) for every changed or newly registered file.

Tests: full tests/test_adaptive_*.py discovery, 826 tests, OK (skipped=1);
tests.test_native_faults_min and tests.test_credential_tools, OK. All six
validators (validate.py, validate_catalogs.py, landscape.py,
evidence_manifest.py --check, verdict_review_gate.py, trading_gates.py
--check) pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…real mutation-kill evidence

Two independent reviews (Codex security review r2; Claude real-mutation and
attack run, 260 attack cases plus a live RENAME_EXCHANGE race, all refused)
both returned "not ready" on 2e1ef68. This commit addresses every item.

1. MEDIUM (Codex) another UID can relocate an already-checked ancestor:
   only the immediate parent's owner/mode were checked, so an
   attacker-writable /shared containing a victim-owned 0700 /shared/private
   could have that private directory renamed under a worktree root after it
   was opened and checked. Every ancestor from "/" down through the
   immediate parent is now checked with the OpenSSH safe_path/
   secure_filename rule: owned by root or getuid(), and group/other-writable
   only if sticky -- a writable, non-sticky ancestor is refused the moment
   it is opened, independent of what gets renamed into it afterward. A
   private directory under /tmp (sticky) still passes; an attacker-writable
   /shared (not sticky) never does, regardless of what it contains.
   Regression tests simulate ownership via per-fd fstat/open faking (never a
   process-wide os.getuid() patch), per the fix-round's explicit test
   requirement.
2. LOW (both) a symlink loop leaks the path: Path.resolve(strict=True)
   raises RuntimeError (not OSError) with the path on Python 3.12, and an
   embedded NUL raises ValueError. Both are now caught alongside OSError.
   Every CredentialGuardError is built and raised from code that has
   already left the except block that caught the original failure, so
   Python never attaches it as __context__ (from None alone does not
   achieve this -- the interpreter re-populates __context__ at the raise
   statement itself if one is still executing inside a handler). Tests
   assert __context__ and __cause__ are None, not only the message text.
3. LOW (both) ".." was normalized lexically for market_research's
   non-follow mode, which could select a different file through an
   intermediate symlink without ever refusing it. Any ".." component is now
   refused outright.
4. LOW (Codex) the fd leaked if fdopen() itself failed after every other
   check passed. Closed on any such failure now; tested by injecting a
   failure into fdopen.
5. LOW (Codex) GUARD_DENIED was one generic message. Replaced with fixed,
   path-free credential_file_permissions:<code> reason codes (missing,
   not_regular, symlink, owner, mode, hardlink, worktree, ancestor,
   parent_owner, parent_mode, encoding, size). Runner's asserted
   credential_file_permissions prefix is unchanged.
6. runner's read was unbounded; it now reads at most MAX_CREDENTIAL_BYTES +
   1 bytes from the fd, the same shared cap market_research uses.
7. MEDIUM (both) surviving mutations and test gaps, each closed with a test
   that specifically fails when its protection is removed:
   (a) file-owner check was masked by wrong-owner tests patching
       os.getuid() process-wide, so the parent-owner check always fired
       first. New test simulates a foreign file owner via per-fd fstat
       faking with the parent genuinely left intact.
   (b) a by-name worktree check computed once, before any traversal, would
       miss a worktree marker added mid-walk; new test injects one via
       _hook exactly when a still-open ancestor is about to be opened.
   (c) path-in-error was only checked for a subset of failure modes; new
       tests cover missing, symlink-loop, NUL-byte, and permission-error
       paths, all asserting no path/basename and clean __context__/__cause__.
   (d) a static oversized fixture rejects identically whether the read is
       bounded or unbounded, so it never established boundedness; new tests
       assert the exact read() size argument each loader's call uses.
   (e) removing O_NONBLOCK would hang the suite instead of failing it; both
       FIFO tests (test_adaptive_market_research.py, test_adaptive_paper_runner.py)
       now run under a signal.alarm deadline.
   (f) fstat(fd) vs a pathname re-stat was not established; new test makes
       os.stat() lie about a file's mode and confirms rejection still uses
       the fd's own (correct) metadata.
   (g) added root-marker (root's own fd is genuinely walked, not skipped)
       and fd-leak tests (open fd count unchanged across success and every
       refusal path, via /proc/<pid>/fd).
   (h) replaced the in-process MUTATION_KILLS meta-test (which asserted a
       hardcoded string-to-test-name mapping without ever running those
       tests) with tests/_credential_mutation_driver.py: copies the real
       source tree to a private temp directory, applies one real
       exact-count textual mutation, and runs the actually-named test(s)
       against that copy in a subprocess. All twelve declared mutations are
       genuinely killed (subprocess exits non-zero); a baseline run with no
       mutation applied is verified to pass first.
8. Docs: the decision doc and the gate note now name the real test file
   (tests/test_adaptive_paper_credential_race.py, not the nonexistent
   tests/test_credential_guard_race.py an earlier round named), describe
   only what the tests actually do (no overclaimed deadline/mutation-run
   coverage), and state the markerless-worktree and bind-mount limits
   explicitly. Runner's symlink-follow acceptance of a compliant swapped-in
   file stays documented, unchanged.

Accepted configurations are unchanged: a 0600 file in a 0700 or 0755
directory you own, under /home or /Users paths owned by root or by you, and
a private directory under /tmp -- each has an explicit regression test.

source-hashes.json and manifests/evidence.json updated surgically (sha256/
bytes only) for every changed file, including the driver's own self-hash.

Tests: full tests/test_adaptive_*.py discovery, 849 tests, OK (skipped=1);
tests.test_native_faults_min and tests.test_credential_tools, OK (45 tests).
All six validators (validate.py, validate_catalogs.py, landscape.py,
evidence_manifest.py --check, verdict_review_gate.py, trading_gates.py
--check) pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…lose, strict parent rule

Both reviewers again found no access-rule bypass (268 attack cases, a live
race with ~600k swaps); 31 of 33 real mutations were killed. This commit
addresses the remaining items.

1. MEDIUM (both) ASCII refusal carried the credential bytes: runner.py and
   market_research.py raised inside a UnicodeDecodeError handler, whose
   `.object` attribute holds the entire input. Both now check
   `raw.isascii()` on the bytes before ever decoding, so no
   UnicodeDecodeError is constructed at all, and the refusal is raised from
   clean control flow (never inside an except block, so `__context__` stays
   None -- `from None` alone does not clear it). Both non-ASCII tests now
   assert `__context__`/`__cause__` are None and that no exception attribute
   contains a file byte.
2. MEDIUM (both) the mutation driver reported false kills: it counted any
   non-zero subprocess exit as a kill, so disable_worktree_check reported
   KILLED against completely unmutated code (the copy had no .git). The
   driver now git-inits every copy, runs the named test id(s) on the
   pristine copy first (must report a plain "ok"), and counts a kill only
   when a named test transitions to a genuine unittest "FAIL" (never an
   "ERROR", never merely a non-zero exit). New MutationDriverSelfTests
   proves this: an empty mutation, an unrelated-import-breaking mutation,
   and a mistyped test id must each report "not killed". Re-run with the
   corrected driver: all fourteen declared mutations are genuinely killed
   (true table below).
3. LOW fdopen cleanup could double-close: if `os.fdopen`'s internal
   `io.FileIO` construction itself failed after accepting fd ownership,
   CPython already closes that fd, so a second `os.close()` either raised
   EBADF or closed an unrelated, since-reused fd. Now builds
   `io.FileIO(file_fd, "rb", closefd=True)` first and, on a later failure
   wrapping it in a `BufferedReader`, closes only that already-constructed
   object, never the bare integer. Tested by making the wrapping step fail
   and confirming no descriptor leak.
4. Two mutations added and killed: skipping root's own ownership/mode check
   (a simulated foreign-owned, world-writable "/", via per-fd fstat faking)
   is now rejected; removing runner's size check is now rejected too (a
   fixture with real KEY/SECRET lines followed by a huge trailing comment
   line demonstrates the size check itself is what fires, independent of
   what a truncated read would otherwise parse).
5. Policy: tightened the immediate parent back from round 3's relaxation.
   The immediate parent must be owned by exactly getuid() (root does not
   excuse it) and never group/other-writable, with no sticky exception --
   this rejects /tmp/file and /file while a private directory under /tmp
   still passes (checked one level down by this same rule). Ancestors above
   the immediate parent keep the OpenSSH safe_path/secure_filename rule
   with its sticky exception, unchanged.
6. LOW reason codes unified: market_research's oversized-file rejection now
   raises credential_file_permissions:size, the same code runner already
   used, instead of the older invalid_credential_file.
7. Docs: the decision doc and gate note no longer claim an atomic location
   guarantee for the worktree marker check -- it is a snapshot at each
   ancestor's own inspection time; another uid permitted to write into a
   sticky ancestor (e.g. /tmp) can still create a new .git entry there
   afterward, which does not bypass any file's own checks or relocate an
   already-verified descriptor. A symlinked directory component (as opposed
   to the final file) is now reported as :symlink, not the generic
   :missing -- Linux reports ENOTDIR, not ELOOP, for O_DIRECTORY|O_NOFOLLOW
   against a symlink, so one extra failure-path-only lstat distinguishes
   the two cheaply.

source-hashes.json and manifests/evidence.json updated surgically (sha256/
bytes only) for every changed file.

Tests: full tests/test_adaptive_*.py discovery, 855 tests, OK (skipped=1);
tests.test_native_faults_min and tests.test_credential_tools, OK (45 tests).
All six validators (validate.py, validate_catalogs.py, landscape.py,
evidence_manifest.py --check, verdict_review_gate.py, trading_gates.py
--check) pass.

True mutation table (subprocess-executed, pristine-baseline-verified, FAIL-
not-ERROR-required -- all 14 killed):
  allow_ancestor_writable_without_sticky: KILLED
  disable_hardlink_check: KILLED
  disable_o_nofollow: KILLED
  disable_o_nonblock: KILLED
  disable_parent_directory_check: KILLED
  disable_root_ancestor_check: KILLED
  disable_runner_size_check: KILLED
  disable_worktree_check: KILLED
  leak_fd_on_fdopen_failure: KILLED
  normalize_dotdot_lexically: KILLED
  read_whole_file_market_research: KILLED
  skip_fstat_owner_identity: KILLED
  skip_fstat_use_prestat: KILLED
  worktree_final_path_by_name: KILLED

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…O leak, protect the untested rules

Both reviewers again found no access-rule bypass (288 attack cases, a live
race with ~670k swaps); all 33 earlier mutations plus round 4's 14 were
killed, and the driver produced no false kills across 21 scenarios. This
commit addresses the remaining small items.

1. _is_symlink_component()'s failure-path lstat caught only OSError, so a
   NUL byte in an ancestor directory component raised a bare ValueError,
   and a surrogate raised UnicodeEncodeError (whose .object attribute
   retains the ancestor's own name). Now catches (OSError, ValueError,
   UnicodeError), so the already-sanitized REASON_MISSING code applies
   instead. Two new tests cover both cases, asserting no path in the
   message, .object, __context__, or __cause__.
2. The double-close fix's own comment was wrong, and it was actually
   leaking a fd in one case: io.FileIO(fd, "rb", closefd=True) does NOT
   take ownership of fd until construction succeeds (verified directly
   against CPython: a failed construction on a directory fd leaves it
   open), so open_verified() was never closing file_fd when the FileIO
   constructor itself is what fails. Fixed: that specific failure now
   closes file_fd directly; only the already-constructed raw object (never
   the bare integer) is closed if the later BufferedReader wrap fails.
   Two new tests: one confirms no leak when FileIO's constructor fails;
   the other asserts raw.closed is True after a wrap failure -- the
   deterministic way to distinguish a proper raw.close() from a bypassing
   os.close(file_fd), since io.FileIO.close() is implemented in C and
   never calls Python's os.close() (so counting os.close() calls cannot
   observe this at all).
3. Four new tests protect the stricter immediate-parent rule (round 4),
   which reverting entirely previously still passed the suite: a real
   0600 file directly in the real /tmp; a simulated root-owned 0755
   parent; a simulated sticky 1777 parent; and a bare /file. Each is
   chosen so only the strict parent rule, not the generic sticky-excused
   ancestor rule, can be what rejects it.
4. A new test protects the double-close fix specifically (distinct from
   the existing leak test, which only proves no leak, not no double-close).
5. A new test directly asserts a symlinked directory component reports
   :symlink, not :missing.
6. The mutation driver's "skipped" result fell through to "survived",
   indistinguishable from the mutated code genuinely being exercised and
   passing. Now reported as "inconclusive" and treated as not-killed, so
   the gate fails on it the same as a real survival. Docstring corrected:
   a "FAIL" can come from setUp(), not only the named test method's own
   body. New MutationDriverSelfTests case injects a real
   @unittest.skip(...) and confirms the driver reports "inconclusive".

Also removed one round-4 mutation (leak_fd_on_fdopen_failure) whose target
text no longer matched after item 2's restructuring and whose protection is
now covered more precisely by the new double_close_on_bufferedreader_failure
mutation.

source-hashes.json and manifests/evidence.json updated surgically (sha256/
bytes only) for every changed file.

Tests: full tests/test_adaptive_*.py discovery, 865 tests, OK (skipped=1);
tests.test_native_faults_min and tests.test_credential_tools, OK (45 tests).
All six validators (validate.py, validate_catalogs.py, landscape.py,
evidence_manifest.py --check, verdict_review_gate.py, trading_gates.py
--check) pass.

True mutation table (subprocess-executed, pristine-baseline-verified, FAIL-
not-ERROR-required -- all 15 killed):
  allow_ancestor_writable_without_sticky: KILLED
  disable_hardlink_check: KILLED
  disable_o_nofollow: KILLED
  disable_o_nonblock: KILLED
  disable_parent_directory_check: KILLED
  disable_root_ancestor_check: KILLED
  disable_runner_size_check: KILLED
  disable_worktree_check: KILLED
  double_close_on_bufferedreader_failure: KILLED
  normalize_dotdot_lexically: KILLED
  read_whole_file_market_research: KILLED
  revert_strict_parent_rule: KILLED
  skip_fstat_owner_identity: KILLED
  skip_fstat_use_prestat: KILLED
  worktree_final_path_by_name: KILLED

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…k, real high surrogate, driver exit code

Claude verified the round-5 head as ready (312 attack cases, no bypass; 46 of
47 mutations killed) and Codex confirmed its round-4 findings fixed. Four
small test/driver fixes remained.

1. test_file_directly_in_real_tmp_is_refused used tempfile.gettempdir(),
   which honors TMPDIR -- private and per-user by default on macOS, so this
   test would silently exercise a different, already-private parent there
   and never reach the code path it means to test (breaking CI
   validate-macos). Now creates the file exclusively in the literal /tmp
   via tempfile.mkstemp(dir="/tmp"), skipping with a clear reason if /tmp
   does not exist or is not sticky and world-writable.
2. The double-close test asserted only raw.closed, which a subtler
   surviving mutant (raw.close() followed by a redundant os.close(fd))
   still satisfies -- the extra close's EBADF is itself an OSError, which a
   bare assertRaises(OSError) also silently accepted, masking that the
   wrong exception propagated. Now asserts the propagated exception is
   exactly the simulated one by identity, and separately forces the OS to
   reuse the exact freed fd number (opening files in a bounded loop until
   one receives it, rather than hoping the next unrelated open() does) and
   confirms that file is still open and readable afterward. A new
   extra_close_after_raw_close mutation is killed by this test.
3. The surrogate test used U+DCFF, which is in surrogateescape's low-
   surrogate range and round-trips straight back to a plain byte on
   encode -- os.fsencode() never raised there at all, so the test exercised
   only a plain OSError and passed even with the pre-fix, OSError-only
   exception handler restored. Now uses U+D800, a lone high surrogate with
   no such round-trip, which always raises UnicodeEncodeError on encode
   (verified directly against the reverted handler). The test also now
   catches broadly and calls self.fail() explicitly on the wrong exception
   type, since assertRaises(CredentialGuardError) would otherwise report an
   uncaught UnicodeEncodeError as an "ERROR", which the real mutation
   driver never counts as a kill. A new
   narrow_symlink_component_exception_handler mutation is killed by this
   test.
4. The mutation driver ignored the mutated subprocess's exit code once
   every named test id reported "ok": a tearDownClass() (or other class-
   scoped fixture) failure is reported by unittest as a separate pseudo-
   test-id that never matches anything in test_ids, so it never appeared in
   the collected results at all, and the driver reported "survived" even
   though the run did not cleanly succeed. Now requires a clean process
   exit for "survived"; a non-zero exit unexplained by any named test id's
   own FAIL/ERROR/skip is "inconclusive" instead. New
   MutationDriverSelfTests.test_teardownclass_error_is_inconclusive_not_survived
   injects a real, always-raising tearDownClass and confirms this.

source-hashes.json and manifests/evidence.json updated surgically (sha256/
bytes only) for every changed file.

Tests: full tests/test_adaptive_*.py discovery, 866 tests, OK (skipped=1),
run once with TMPDIR unset and once with TMPDIR set to a private directory;
tests.test_native_faults_min and tests.test_credential_tools, OK (45 tests).
All six validators pass. The driver's own self-tests (6, including the two
new ones) pass, and all 17 declared mutations are genuinely killed.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…h-credential-guard-20260924

# Conflicts:
#	manifests/evidence.json
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

Scout and others added 3 commits September 24, 2026 19:09
…digests

blueprints/us-equities/adaptive-paper/source-hashes.json maps engine file paths
to their sha256. Paths that contain "credential" (credential_guard.py,
tests/_credential_mutation_driver.py, tests/test_adaptive_paper_credential_race.py)
made gitleaks' generic-api-key rule flag the digest on the same line.

The manifest is added to the existing rule-scoped, line-anchored allowlist for
reviewed checksum manifests. That allowlist's regex requires a filename-shaped
key (path separator, extension or leading dot) and a 40- or 64-hex value, so a
credential-shaped key name in the same file is still detected.
tests.test_gitleaks_config passes with the pinned gitleaks 8.30.1 (19 OK).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…OS symlink portability

Codex reviewed 1bd4e2f (which added source-hashes.json to the shared
"filename-shaped keys in reviewed checksum-manifest files" allowlist) and
returned "not ready": that shared regex also matches a credential-shaped key
such as "credential.key", nothing enforced that every entry in
source-hashes.json is a real file digest, and no gitleaks config test
covered the new path. Then CI validate-macos failed on #219 with a
confirmed root cause: market_research.credentials() (no-follow mode)
refuses every macOS temp fixture, since /tmp and /var are themselves
root-owned symlinks to /private/tmp and /private/var there.

1. Reverted 1bd4e2f's addition to the shared allowlist. Added a dedicated,
   rule-scoped [[rules.allowlists]] under generic-api-key for exactly
   blueprints/us-equities/adaptive-paper/source-hashes.json:
   condition = "AND", regexTarget = "line", and a stricter line-anchored
   regex that requires the ENTIRE key to be a real repo path under
   blueprints/us-equities/adaptive-paper/ or tests/, ending in .py, .json,
   .sh, or .txt (the manifest's own actual extension set), mapped to
   exactly 64 lowercase hex -- a credential-shaped key such as
   "credential.key" or "api_key" in this same file is not a path and still
   fails the regex.
2. Added tests/test_adaptive_paper_source_hashes_manifest.py: stdlib-only
   (hashlib, json, re, unittest, pathlib -- no nautilus_trader, no alpaca),
   so it runs in CI's system-python "Test validation failure modes" step
   (validate.yml's bare `python3 -m unittest`). For every key in
   source-hashes.json: the file must exist, its current sha256 must equal
   exactly the recorded value, no duplicate keys, and every key must match
   the same path shape the dedicated allowlist trusts (kept in sync
   deliberately) -- so a secret can never sit in this manifest without
   failing CI's validate job, independent of the allowlist.
3. Added 6 gitleaks config tests (tests/test_gitleaks_config.py, following
   its existing test_a/test_b/... pattern): a path-keyed 64-hex digest in
   the exact path is not detected; a credential-shaped key
   ("credential.key", "api_key") with 64 hex in that same path is still
   detected; a non-lowercase-hex value under a path-shaped key is still
   detected; a second secret (api_key) sharing one physical line with an
   allowlisted path digest is still detected; the identical path-keyed line
   in a different file is still detected.

macOS portability (item confirmed against a full CI failure log):
4. Added tests/adaptive_paper_hermetic.py's real_tmp_root(), a shared
   helper both test_adaptive_market_research.py and
   test_adaptive_paper_credential_race.py now use for every fixture root a
   follow_symlinks=False test builds: os.path.realpath() resolves the
   macOS-only /var (hence /var/folders/..., tempfile's default root there)
   and /tmp symlinks before the guard ever sees the path, so no-follow mode
   reaches the actual scenario a test means to exercise instead of refusing
   the OS-provided ancestor symlink first. The literal-/tmp test now uses
   os.path.realpath("/tmp") explicitly as its own parent for the same
   reason (still refused as parent_owner/parent_mode on the resolved
   target, never symlink). _open_fd_count() now checks /proc/<pid>/fd or
   /dev/fd, whichever exists, instead of the Linux-only /proc path (macOS
   has no /proc at all). The NUL-byte and surrogate tests reach the actual
   bad component correctly once their fixture root is resolved; no
   platform-specific skip was needed. credential_guard.py's module
   docstring and the decision doc now state explicitly that no-follow mode
   refuses every symlink component, including OS-provided ones -- callers
   must pass an already symlink-free path, which the documented credential
   store under $HOME always is. The guard itself is unchanged: this is a
   fixture fix, not a weakening of the rule. Linux behavior is unchanged
   (os.path.realpath is a no-op there in practice).

Verified locally on Linux with TMPDIR unset, and additionally with TMPDIR
pointing at a symlink to a separate real directory (simulating the macOS
/var/folders layout) -- both runs pass identically, including the real
mutation-kill driver's own pristine-baseline runs (which exercise these
same fixtures via its own copied-tree subprocess).

source-hashes.json and manifests/evidence.json updated surgically (sha256/
bytes only) for every changed registered file.

Tests: full tests/test_adaptive_*.py discovery, 871 tests, OK (skipped=1),
run both with TMPDIR unset and with TMPDIR set to a symlinked directory;
tests.test_native_faults_min and tests.test_credential_tools, OK (45
tests); the new source-hashes-manifest enforcement test, OK (5 tests); all
six validators pass; tests.test_gitleaks_config with
GITLEAKS_TESTS_REQUIRED=1 and the pinned gitleaks 8.30.1 on PATH, OK (24
tests); gitleaks git (--log-opts=HEAD, 511 commits, ~1.12 GB) and gitleaks
dir scans both report 0 findings; all 17 real mutations still killed.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The dedicated gitleaks allowlist and the manifest enforcement test both
accepted keys such as tests/../scripts/validate.py, which contradicts the
under adaptive-paper/ or tests/ scope. Each path segment must now start with
[A-Za-z0-9_-]. Added a key-shape unit test and a gitleaks config test that a
dot-segment key holding a digest is still detected (25 config tests OK).

Also corrected the allowlist comment: the per-entry check is
tests/test_adaptive_paper_source_hashes_manifest.py; scripts/validate.py only
pins the manifest file's own hash.

Found by the independent Claude review of round 7. Codex was at its usage
limit, so that review was the independent check for this round.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@seathatflowsinourveins
seathatflowsinourveins merged commit b7c6da4 into main Sep 25, 2026
22 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the claude/market-research-credential-guard-20260924 branch September 25, 2026 00:21
seathatflowsinourveins pushed a commit that referenced this pull request Sep 25, 2026
…ction guard branch

- runner.py: kept both import lines (corporate_actions, credential_guard); the
  two changes touch different regions.
- source-hashes.json: took the union of both sides' keys and recomputed every
  digest from the merged tree. The #219 manifest test passes.
- evidence.json: rebuilt from main's version. Re-registered this branch's
  native-check record, registered corporate_actions.py and its tests, and
  refreshed the six stale digests surgically.

Checks, with the network blocked: 986 adaptive tests OK; native-faults,
order-contract and credential-tools 56 OK; order-throughput, trading-gates,
evidence-manifest and validate 163 OK; all six validators pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins added a commit that referenced this pull request Sep 25, 2026
…-09-24/25 merges (#225)

Updates four checkpoint entries: the paper lane, the simulation lane, alpaca-paper-operational and corporate-action-data. They now reflect #198, #199, #212, #218, #219 and #222. tests.test_grand_dashboard passes, and all state strings satisfy progress.py's public-token rule.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins pushed a commit that referenced this pull request Sep 25, 2026
…e changes job, NUL-safe diff

Independent review found that needs: changes plus a bare if: (no status function) applies
an implicit success() to bootstrap-linux/bootstrap-macos/bootstrap-macos-brew, so once
changes itself is skipped (it only runs on pull_request), all three jobs were skipped too
on every push, schedule and workflow_dispatch run -- confirmed live by the coordinator's
dispatch run 36085789483 (changes skipped, all three bootstrap-* jobs completed as
skipped, validate-macos ran normally). Fix: if: ${{ !cancelled() && (github.event_name !=
'pull_request' || needs.changes.outputs.bootstrap != 'false') }}, following the
security-scan.yml precedent; != 'false' also makes a failed/cancelled changes job run
everything (fail safe). The changes job itself now fails safe on a missing payload SHA or
a failed git diff (echoes bootstrap=true and exits 0 instead of exiting 1 and letting
needs: skip the dependents), and switches to git diff -z with a NUL-delimited read so a
quoted non-ASCII filename still matches a PATTERNS glob. Also: SC2254-annotated case glob
comment corrected to describe over-matching (not exact paths: semantics), stale workflow
comments and decision-record claims corrected (PR #219 did run and fail validate-macos,
just wasn't required; "fourth job" -> "fifth"; measured cost 8.2-13.4 min, median 10.9 over
36 runs; decision record restructured to flat Evidence/Alternatives/Decision/Cost/Overturn/
Measured; added a rollout note about already-open PRs needing a new push after the ruleset
is applied), and tests/test_workflow_hardening.py strengthened to require a status function
and != 'false' (proven to fail against the pre-fix condition text) plus fail-safe/NUL-safety
assertions on the changes job's script.

Verified: actionlint 1.7.12 (with shellcheck 0.11.0) and zizmor --offline --no-config
--no-ignores --persona regular --strict-collection both report 0 findings on
.github/workflows. tests.test_workflow_hardening, test_adoption_bootstrap_macos,
test_adoption_bootstrap, test_adoption_docs_consistency, test_workflow_security_coverage,
test_workflow_security and the full python3 -m unittest suite (4720 tests) all pass. All
six validators (validate.py, validate_catalogs.py, landscape.py, evidence_manifest.py
--check, verdict_review_gate.py --base origin/main, trading_gates.py --check) pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins pushed a commit that referenced this pull request Sep 25, 2026
…s in the record

- shell: bash runs as bash -eo pipefail, so the script now runs set +e and
  guards mktemp with the fail-safe bootstrap=true write. The test asserts both.
- Workflow comment and decision record: only push is path-filtered; schedule
  and workflow_dispatch run every job.
- #219: validate-macos failed on earlier heads (72a15b8, 1bd4e2f) and passed
  at the merged head 04c867b. It was not a red run beside the merge.

From the independent delta review of 8ec88c1/c19cb31d (verdict ready).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins added a commit that referenced this pull request Sep 25, 2026
Adds validate-macos (the full test suite on macOS) to the main ruleset target, and makes it report a status on every pull request.
- adoption-bootstrap.yml drops the pull_request path filter. A fail-safe changes job, which diffs base and head with git, keeps bootstrap-linux, bootstrap-macos and bootstrap-macos-brew path-gated on PRs through if: !cancelled() && (event != pull_request || bootstrap != 'false').
- Push keeps its paths filter. Schedule and dispatch run every job.
- Measured: before the fix, dispatch run 36085789483 skipped the bootstrap jobs. After it, dispatch run 36086815267 ran them all and they succeeded.
- strict_required_status_checks_policy stays false, because its recorded overturn condition has not occurred.
- Evidence for requiring macOS: #219's macOS-only failures (72a15b8, 1bd4e2f).

Reviewed by Claude (Codex at its usage limit until 2026-09-30).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant