Skip to content

Scan large CI secret targets with an explicit explorer exclusion - #964

Merged
seathatflowsinourveins merged 5 commits into
mainfrom
codex/ns2604-ci-secret-scan-size-20261010
Oct 10, 2026
Merged

seathatflowsinourveins merged 5 commits into
mainfrom
codex/ns2604-ci-secret-scan-size-20261010

Conversation

@seathatflowsinourveins

@seathatflowsinourveins seathatflowsinourveins commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

Scope

CI secret scans silently skipped large tracked files under --max-target-megabytes 2. Remove that flag from all four gitleaks/betterleaks git and directory commands and replace it with the exact generated-path exclusion ^docs/ecosystem/index\.html$. Large manifests and catalogs receive coverage; neighboring and prefixed paths remain covered.

  • Landing rebase base: a7c15bd051088d0493d58bd84a3c8c35dcde6624, fetched for the additional CC-cued rebase after the 31917b5 landing micro ACK. Previous landing base: 03a986ed637349403658f23da694c1833527819d; historical first published landing base: 0947b01289dcaa9157aa38392a94cf00f592b86d; original branch point: c8a29c3ee9b4337564165d9e588b3fe81e8ba817.
  • Main fetched immediately before the push: a7c15bd051088d0493d58bd84a3c8c35dcde6624; native merge-tree returns rc 0. Main's registrations and non-file fields are preserved. Local gates below ran on this rebased head; fresh hosted acceptance remains separate.
  • Head: fa8a0050f51cc7ef4ddf1beb04e16b2cc385149c.
  • Lane: lane:foundation; Class A security-fix capacity exception.
  • Owned paths: .github/workflows/validate.yml, .gitleaks.toml, automation guide and decision record, scanner/timer/metrics/workflow/grants regressions, scripts/secret_scan_metrics.py, its single repository inventory grant, local measurement artifact and manifests/evidence.json bindings.

Regressions consume the production scanner arguments and require canary detection at 2,000,001 and 3,000,000 bytes in gitleaks git, gitleaks dir and betterleaks dir. GNU time measurement is selected only after verifying GNU time on Linux; other platforms invoke the scanner directly and retain their actual host label. All four production scans retain elapsed seconds, peak RSS, original scanner status and exact revision/event scope. The hosted regression selection includes ScannerTimerTests. The native policy tooling measures 93 repository grants in a genuine worktree; the expected count is set manually and the sole missing script grant is registered.

The latest forward commit adds the requested documentation contracts and corrects the stale base label. The callback control binds the pinned Git Path/ShouldSkip explanation and distinguishes the archive-blob File callback from ordinary text-diff scanning. Two header controls require the metrics-writer failure condition and bind the historical figures to their named run, attempt, event, ref, checkout and HEAD-ancestry scope. Actual text from the previous revision fails these controls before the corrected text passes; existing prefilter/function-locator controls remain. Committed tests read current-tree files; historical Git text is confined to the retained one-time reproduction.

The additional CC-cued rebase preserves all five reviewed source patches: source-only range-diff is five equals, and the full range-diff changes only regenerated registrations. The policy inherits main's #984 changes automatically; its sole addition over current main remains the exact metrics-script repository grant. The hot-file protocol takes main's manifest and re-registers the owned files through the native helper; both official generators leave their reports unchanged. Every non-manifest file at this head matches Git's automatic merge preview of the reviewed head and the captured landing base. All main rows outside the eleven-path ownership set and all non-file fields are retained, and all eleven registrations match the actual resolved files. The policy inventory is independently measured at 93 and set-equal to its grants. The co-op's 31917b5 landing micro is ACK; this landing head awaits the next range-diff micro and immediate CC landing.

SOTA sources

Evidence-class table

Claim Evidence class Command / receipt
Silent large-target omission and repaired detection local_integration Actual workflow arguments/native binaries: 8 failing-before assertions without harness errors; repaired git/dir detections at both boundary sizes, with exact-path exclusion controls
Betterleaks cap boundary local_integration Detects 1,999,999/2,000,000 bytes and skips 2,000,001/2,999,999/3,000,000 with cap 2
Timer portability and resource/status contracts local_integration GNU Linux and mocked BSD/Darwin fallback controls; four actual step scripts and twelve status combinations preserve scan status and receipt scope
Repository inventory and hosted timer selection local_integration Native 93-row proposal; complete grants module fails first and passes after the grant/count correction; selection regression rejects omitted ScannerTimerTests
Corrected callback/header/base documentation source_review/local_integration FIX4 controls: previous callback 1 assertion failure, previous headers 2, previous base 1; all 0 errors/skips. Current documentation class: 6 passed
Complete touched modules at this landing head local_integration Five named modules: 228 tests, 227 passed, one existing empty hash-frozen-set skip, rc 0, 63.232 s; all three timer controls and six documentation contracts execute
FULL integrity at this landing head local_integration scripts/validate.py: rc 0, 11,427 bindings, 239 receipts, 70 components, 4 profiles, 56.110 s
Publication privacy local_integration Silent guards: zero home-path matches in each replayed commit's additions, commit messages and the complete PR body; main's inherited records retained
Publication scan local_integration Uncapped, redacted PR range a7c15bd0..HEAD, 5 commits, rc 0, no leaks
Prior complete changed modules local_integration At parent 7ed91bf: five named modules, 219 tests, rc 0, two existing skips; grant count 93
Historical full hosted acceptance native_proven Prior reviewed revision 7ed91bf: FULL run 38067781434 completed SUCCESS, all eight shards, aggregate and both scanner jobs successful. New-head hosted acceptance is separate

The earlier complete local discovery ran 12,810 tests, rc 1 with one known cross-device F27c fixture failure, no errors and 1,231 skips. The unmodified test passes with temporary storage on the repository's filesystem, and the co-op independently reproduced base=head for the environment boundary. This historical discovery is not a claimed green local run. The latest commit uses the CC's complete-touched-modules-plus-FULL-validator policy.

Real production measurements come from hosted run 38053675250, attempt 1, pull_request, refs/pull/964/merge, checkout f2f10412c68a859800b2bcad034c8ab639fd6f2e. Git scans cover HEAD ancestry. The measurement identifies historical execution and is not a dependency or fixture pin; fixture timing remains separate.

Production scan Seconds Peak RSS KiB Scanner status Findings
gitleaks 8.30.1 git 37.42 745128 0 0
gitleaks 8.30.1 dir 14.52 126772 0 No JSON report; native gating status 0
betterleaks 1.8.1 git 19.01 1074456 0 1687, report-only
betterleaks 1.8.1 dir 4.16 147360 0 1685, report-only

Full job durations include checkout, installation, binary verification, regressions, both scans and post steps. The gitleaks job completed successfully in 116/600 s, leaving 484 s; betterleaks completed successfully in 64/1200 s, leaving 1136 s. Artifact IDs are 11670637536 and 11670032999. Betterleaks remains report-only; its counts do not claim a gating clean scan.

Local commands run

python3 -m unittest tests.test_workflow_hardening tests.test_local_pages_policy_grants tests.test_secret_scan_metrics tests.test_secret_scan_size tests.test_gitleaks_config -v --durations 50
# Genuine rebased worktree: 228 tests, 227 passed, one existing skip, rc 0.
# BETTERLEAKS_TEST_BINARY selects the verified 1.8.1 binary; Python 3.13.16.
python3 scripts/validate.py
# FULL, rebased head: rc 0; 11,427 bindings / 239 receipts / 70 components / 4 profiles.
python3 scripts/evidence_manifest.py --check
# rc 0; changed bindings registered through scripts.host_receipts.register_file.
gitleaks git . --config .gitleaks.toml --redact=100 --no-banner --log-opts="a7c15bd051088d0493d58bd84a3c8c35dcde6624..HEAD"
# Committed PR range only: 5 commits, uncapped, rc 0, no leaks.
git diff --check
# rc 0.

Prior parent-head validation also passed all five changed modules, CI-pinned actionlint 1.17.0 and regular offline zizmor 1.30.1. The latest forward commit changes only the documentation test module, decision text and their evidence bindings; production workflow, scanner configuration and metrics writer are byte-identical to its parent.

Decision record

docs/decisions/2026-10-10-secret-scan-large-targets.md records pinned mechanisms, corrections, the exact exclusion, alternatives and separate local/hosted resource evidence. It now distinguishes the original branch point from the first published landing base. Review receipts and failing-before reproductions are retained in the lane's ci-secret-scan-size-20261010 work directory. Fresh micro and CC landing remain pending at this head.

Host evidence

Local scanner fixtures are registered under evidence/artifacts/; this PR does not change host receipts.

Checklist

  • Actions retain full commit pins and reviewed job permissions.
  • Exact generated-HTML exclusion is visible in configuration and the automation guide.
  • Large-target and corrected-documentation regressions fail before the fixes.
  • Scanner findings are redacted; no credential is printed or committed.
  • No new required secret, paid hosting or billing surface is introduced.
  • Local git scans use PR ranges only; peer worktrees and held heads are preserved.

@seathatflowsinourveins seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Oct 10, 2026
@socket-security

socket-security Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Dependency limit exceeded — report not shown.

This pull request scan exceeded the 10,000-dependency limit applied to this scan, so the results are incomplete and may be inaccurate. To avoid reporting false positives, Socket has not posted a report.

Upgrade your plan to raise the dependency limit and get complete reports, or view the partial scan in the dashboard.

Socket is always free for open source. If this is a non-commercial open source project, contact us to request a free Team account.

@seathatflowsinourveins
seathatflowsinourveins force-pushed the codex/ns2604-ci-secret-scan-size-20261010 branch 2 times, most recently from 64d1a61 to 31917b5 Compare October 10, 2026 19:46
@seathatflowsinourveins
seathatflowsinourveins force-pushed the codex/ns2604-ci-secret-scan-size-20261010 branch from 31917b5 to fa8a005 Compare October 10, 2026 21:28
@seathatflowsinourveins
seathatflowsinourveins marked this pull request as ready for review October 10, 2026 21:54
@seathatflowsinourveins
seathatflowsinourveins merged commit eb31d23 into main Oct 10, 2026
38 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the codex/ns2604-ci-secret-scan-size-20261010 branch October 10, 2026 22:07
seathatflowsinourveins added a commit that referenced this pull request Oct 10, 2026
The three removed workflows were in the local-pages repository grants. At the
landing base, main eb31d23 (93 grants after #964), the test's own derivation
(grants.inventory_paths over git ls-files) gives 90 for this branch, equal to
the policy's repository grants, and the test pins 90. At the earlier base
9d4c00c the same derivation gave 89.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
seathatflowsinourveins added a commit that referenced this pull request Oct 11, 2026
…n_pin (W4 PR 2)

scripts/final_stack.py joins the landscape verdict rows, the planned layers,
the committed component evidence matrix, the completed saturation-ledger sweeps
and report, research state, the verdict-wave registry, the newest seeds file and
the run-pin sources (the trading uv.lock, the adoption pin files,
manifests/stack.json and runtime-target.json#/engine) into
catalogs/landscape/final-stack.json: deterministic, no clock, every read inside
the repository root. Status is the first match of unswept, no-winner, reopened,
converged, current; v1 is 30 reopened, 2 no-winner, 2 unswept, 0 current,
0 converged. Free-text pins parse only in four exact forms and are otherwise
flagged unpinned or unparsed. --check fails on byte drift, a changed source
hash, a layer without exactly one landscape, research-state or matrix row, a
planned-layer collision, an inconsistent index or private content; run-pin and
drift flags and --today next_review flags exit 0 in v1.

catalogs/landscape/planned-layers.json adds catalysts-sentiment and
judgment-features from the approved north-star plan's ten-layer table. CI runs
the check and a report-only freshness step after the new-host grand list; the
adoption-bootstrap path filters list the script. The two new landscape JSON
files are granted as Architecture supplements and the script as an inventory
grant (count 93 -> 94, as #964 did), so the Architecture refresh keeps reading
every landscape file.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
seathatflowsinourveins added a commit that referenced this pull request Oct 11, 2026
…n_pin (W4 PR 2)

scripts/final_stack.py joins the landscape verdict rows, the planned layers,
the committed component evidence matrix, the completed saturation-ledger sweeps
and report, research state, the verdict-wave registry, the newest seeds file and
the run-pin sources (the trading uv.lock, the adoption pin files,
manifests/stack.json and runtime-target.json#/engine) into
catalogs/landscape/final-stack.json: deterministic, no clock, every read inside
the repository root. Status is the first match of unswept, no-winner, reopened,
converged, current; v1 is 30 reopened, 2 no-winner, 2 unswept, 0 current,
0 converged. Free-text pins parse only in four exact forms and are otherwise
flagged unpinned or unparsed. --check fails on byte drift, a changed source
hash, a layer without exactly one landscape, research-state or matrix row, a
planned-layer collision, an inconsistent index or private content; run-pin and
drift flags and --today next_review flags exit 0 in v1.

catalogs/landscape/planned-layers.json adds catalysts-sentiment and
judgment-features from the approved north-star plan's ten-layer table. CI runs
the check and a report-only freshness step after the new-host grand list; the
adoption-bootstrap path filters list the script. The two new landscape JSON
files are granted as Architecture supplements and the script as an inventory
grant (count 93 -> 94, as #964 did), so the Architecture refresh keeps reading
every landscape file.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
seathatflowsinourveins added a commit that referenced this pull request Oct 11, 2026
…n_pin (W4 PR 2)

scripts/final_stack.py joins the landscape verdict rows, the planned layers,
the committed component evidence matrix, the completed saturation-ledger sweeps
and report, research state, the verdict-wave registry, the newest seeds file and
the run-pin sources (the trading uv.lock, the adoption pin files,
manifests/stack.json and runtime-target.json#/engine) into
catalogs/landscape/final-stack.json: deterministic, no clock, every read inside
the repository root. Status is the first match of unswept, no-winner, reopened,
converged, current; v1 is 30 reopened, 2 no-winner, 2 unswept, 0 current,
0 converged. Free-text pins parse only in four exact forms and are otherwise
flagged unpinned or unparsed. --check fails on byte drift, a changed source
hash, a layer without exactly one landscape, research-state or matrix row, a
planned-layer collision, an inconsistent index or private content; run-pin and
drift flags and --today next_review flags exit 0 in v1.

catalogs/landscape/planned-layers.json adds catalysts-sentiment and
judgment-features from the approved north-star plan's ten-layer table. CI runs
the check and a report-only freshness step after the new-host grand list; the
adoption-bootstrap path filters list the script. The two new landscape JSON
files are granted as Architecture supplements and the script as an inventory
grant (count 93 -> 94, as #964 did), so the Architecture refresh keeps reading
every landscape file.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
seathatflowsinourveins added a commit that referenced this pull request Oct 11, 2026
…n_pin (W4 PR 2)

scripts/final_stack.py joins the landscape verdict rows, the planned layers,
the committed component evidence matrix, the completed saturation-ledger sweeps
and report, research state, the verdict-wave registry, the newest seeds file and
the run-pin sources (the trading uv.lock, the adoption pin files,
manifests/stack.json and runtime-target.json#/engine) into
catalogs/landscape/final-stack.json: deterministic, no clock, every read inside
the repository root. Status is the first match of unswept, no-winner, reopened,
converged, current; v1 is 30 reopened, 2 no-winner, 2 unswept, 0 current,
0 converged. Free-text pins parse only in four exact forms and are otherwise
flagged unpinned or unparsed. --check fails on byte drift, a changed source
hash, a layer without exactly one landscape, research-state or matrix row, a
planned-layer collision, an inconsistent index or private content; run-pin and
drift flags and --today next_review flags exit 0 in v1.

catalogs/landscape/planned-layers.json adds catalysts-sentiment and
judgment-features from the approved north-star plan's ten-layer table. CI runs
the check and a report-only freshness step after the new-host grand list; the
adoption-bootstrap path filters list the script. The two new landscape JSON
files are granted as Architecture supplements and the script as an inventory
grant (count 93 -> 94, as #964 did), so the Architecture refresh keeps reading
every landscape file.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
seathatflowsinourveins added a commit that referenced this pull request Oct 11, 2026
…n_pin (W4 PR 2)

scripts/final_stack.py joins the landscape verdict rows, the planned layers,
the committed component evidence matrix, the completed saturation-ledger sweeps
and report, research state, the verdict-wave registry, the newest seeds file and
the run-pin sources (the trading uv.lock, the adoption pin files,
manifests/stack.json and runtime-target.json#/engine) into
catalogs/landscape/final-stack.json: deterministic, no clock, every read inside
the repository root. Status is the first match of unswept, no-winner, reopened,
converged, current; v1 is 30 reopened, 2 no-winner, 2 unswept, 0 current,
0 converged. Free-text pins parse only in four exact forms and are otherwise
flagged unpinned or unparsed. --check fails on byte drift, a changed source
hash, a layer without exactly one landscape, research-state or matrix row, a
planned-layer collision, an inconsistent index or private content; run-pin and
drift flags and --today next_review flags exit 0 in v1.

catalogs/landscape/planned-layers.json adds catalysts-sentiment and
judgment-features from the approved north-star plan's ten-layer table. CI runs
the check and a report-only freshness step after the new-host grand list; the
adoption-bootstrap path filters list the script. The two new landscape JSON
files are granted as Architecture supplements and the script as an inventory
grant (count 93 -> 94, as #964 did), so the Architecture refresh keeps reading
every landscape file.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
seathatflowsinourveins added a commit that referenced this pull request Oct 11, 2026
* Generate the per-layer final-stack manifest with is_definitive and run_pin (W4 PR 2)

scripts/final_stack.py joins the landscape verdict rows, the planned layers,
the committed component evidence matrix, the completed saturation-ledger sweeps
and report, research state, the verdict-wave registry, the newest seeds file and
the run-pin sources (the trading uv.lock, the adoption pin files,
manifests/stack.json and runtime-target.json#/engine) into
catalogs/landscape/final-stack.json: deterministic, no clock, every read inside
the repository root. Status is the first match of unswept, no-winner, reopened,
converged, current; v1 is 30 reopened, 2 no-winner, 2 unswept, 0 current,
0 converged. Free-text pins parse only in four exact forms and are otherwise
flagged unpinned or unparsed. --check fails on byte drift, a changed source
hash, a layer without exactly one landscape, research-state or matrix row, a
planned-layer collision, an inconsistent index or private content; run-pin and
drift flags and --today next_review flags exit 0 in v1.

catalogs/landscape/planned-layers.json adds catalysts-sentiment and
judgment-features from the approved north-star plan's ten-layer table. CI runs
the check and a report-only freshness step after the new-host grand list; the
adoption-bootstrap path filters list the script. The two new landscape JSON
files are granted as Architecture supplements and the script as an inventory
grant (count 93 -> 94, as #964 did), so the Architecture refresh keeps reading
every landscape file.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(catalog): close final-stack review boundaries

Fold both review families with failing-first privacy and reuse checks, ranked run precedence, exact uv root selection, explicit query and report semantics, and derived Architecture exclusions. Preserve pure data construction and reuse shared native publication mechanisms.

* fix(catalog): scan publication with native decoder rules

Fold CC, Astra and Codex micro findings with failing-first native account and email plants, deterministic check mode, preserved publication failure state, covered planned layers and corrected dependent citations. Retain pinned captures and adjacent caller evidence for review.

* fix(final-stack): retain complete publication tokens and source bindings

* fix(final-stack): scan publication entries through native stdin

* fix(final-stack): bound native scans to publication records and retain source pins

* fix(final-stack): retain classified temp reads and accurate publication claims

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant