Skip to content

Split the OSV SARIF upload into a tool-free job; fix the blind comparison's confirmed defects and file its evidence - #121

Merged
seathatflowsinourveins merged 4 commits into
mainfrom
claude/automation-closure-polish
Sep 23, 2026
Merged

seathatflowsinourveins merged 4 commits into
mainfrom
claude/automation-closure-polish

Conversation

@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator

This is the preregistered graft after the blind comparison of the two catalog automation implementations. Closure (tools/compare/closure.mjs) retained the merged implementation, and the publication gate passed: 3 of 3 blind refuters did not refute, with 0 leaks.

Security change

  • osv-scanner now runs with only contents: read.
  • A new osv-sarif-upload job (needs: osv-scanner, !cancelled()) holds security-events: write and runs no shell step or installed tool. This matches the zizmor split, so the job running the installed scanner no longer holds a token that can dismiss the CodeQL alerts the code_scanning rule gates on.
  • Artifact names now include run_attempt, so a re-run is safe.

Tests

  • Failure-path assertions cover the OSV artifact step and the upload job (!cancelled()) and the zizmor artifact step (no forced run).
  • The step and if: parsers now also catch a leading - if:, folded block scalars and plain-scalar continuation lines.
  • Every new assertion was mutation-checked: each break fails the suite.

Docs and evidence

  • Corrected: the harden-runner count (21 of 25), the Scorecard layout citation (the ossf/scorecard example at d13ba3f that the README links) and the observed-upload analysis ids, the dependabot header, and the upload wording.
  • Decision-record sections 3 to 6 now have complete evidence, alternatives, decision and overturn entries.
  • The lane-sourced catalog text is kept verbatim, with its staleness recorded.
  • The comparison evidence is filed in evidence/artifacts/blind-comparison-20260923/: preregistration with amendment 1, protocol, packet, closure, battery, harness, and a path-redacted judgments file with the original's sha256.

Checks

  • validate.py, evidence_manifest --check, build_verdicts --check, build_ecosystem --check and the full unittest suite pass.
  • actionlint and zizmor --offline regular report 0 findings.
  • The comparison battery on this branch passes 13/13.
  • Guarded gitleaks finds no leaks.
  • Independent Opus review: the split semantics were confirmed, and all findings are resolved.
  • The push run after merge will exercise osv-sarif-upload on GitHub for the first time.

🤖 Generated with Claude Code

…ind-comparison evidence

Correct the stale harden-runner ubuntu-job count (18 of 22 -> 20 of 24,
measured at HEAD with the exact counting method), fix dependabot.yml's
"github-actions only" header and every doc claim that Dependabot owns only
GitHub Actions references now that a deliberately vulnerable pip fixture
entry exists, and fix the decision record: the stale "only the osv-scanner
job holds security-events: write" claim (zizmor-sarif-upload also holds it),
the "separate upload job scans twice" rationale that conflicts with the
chosen design, and add the missing Evidence/Alternatives/Decision/Overturn
elements to the zizmor-online, Scorecard SARIF and dependency-review
sections. Add mutation-checked assertions covering the OSV/zizmor SARIF
upload failure-path semantics to tests/test_workflow_hardening.py.

Record the "Blind comparison (2026-09-23)" section in the decision record
and copy its comparison artifacts into
evidence/artifacts/blind-comparison-20260923/ (with the judgments file's
scratch/home paths redacted and a README listing every file's sha256),
registered in manifests/evidence.json.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Fix the review's confirmed high/medium findings:
- security-scan.yml section 3 never had only one job holding
  security-events: write (zizmor-online held it from the founding commit
  a9e5180); say so, with the verification command that shows the count.
- Restate the OSV separate-upload-job rejection honestly: it would not
  scan twice, and the real trade-off is the write scope sitting on the
  job that runs a curl-installed binary; mark it keep-but-compare.
- Reword the dependabot.yml header and the two docs/github-automation.md
  passages: ignore: urllib3 suppresses security-update PRs for the
  fixture pin; open-pull-requests-limit: 0 only stops version updates;
  alerts still appear via the dependency graph regardless.
- Restore the sealed 2026-09-22 lane:claude Dependabot why_not_default
  text verbatim in catalogs/landscape/foundation.json, and move the
  pip-fixture correction into a labelled coordinator annotation in the
  layer's limitations list instead of overwriting lane output;
  regenerate layer-verdicts-20260922.json and the handbook with
  build_verdicts.py --write.
- Section 5: cite the observed hosted upload (the first push run on
  4970ba0, dispatch run 35824151483) and scorecard-action's documented
  example workflow at its pinned SHA, not just the source diff.
- Section 4: name zizmor's actual online-only audit classes instead of
  claiming offline analysis covers no secrets class (its secrets audits
  are static). Section 6: replace the #97/#98 "nothing to flag"
  inference with what those PRs actually show (gate latency, not
  threshold behavior).

Fix the low findings:
- tests/test_workflow_hardening.py: the failure-path test read a fixed
  line offset after each step/job marker, so a reordered key or a
  folded if: block scalar could pass vacuously. Add step_block()/
  block_if() helpers that search the whole block and resolve folded
  scalars, and assert zizmor-online's analyzer step carries no
  continue-on-error (a crash there could otherwise still upload a
  partial SARIF). Mutation-verified: reordered uses:/if: with the
  cancelled() guard dropped, added continue-on-error to the analyzer
  step, and folded the upload job's if: with a hidden always() --
  each mutation failed the test before being reverted.

Re-registered all seven changed files in manifests/evidence.json with
scripts/host_receipts.register_file.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…eview findings

The osv-scanner job held security-events: write while running the installed
scanner on every event; it is now contents: read and a needs/!cancelled()
osv-sarif-upload job (no shell step or installed tool) uploads, matching the
zizmor split. Record sections 3 and 5 are rewritten to match (Scorecard upload
now cited from the code-scanning analyses list). The lane-sourced catalog text is
restored verbatim and its staleness recorded instead; the redacted judgments copy
now replaces only path tokens; the folded-if bypass in the artifact-step test is
closed. Mutation-checked: a folded always(), a lost !cancelled() and a write scope
back on the scan job each fail the suite.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…roof step checks, corrected citations

Artifact names now include run_attempt so a re-run cannot hit a same-name
conflict. step_block() spans the whole list item and block_if() collects
plain-scalar continuations, so a leading '- if:' or an always() on a continuation
line is caught (mutation-checked). Docs: harden-runner 21 of 25 ubuntu jobs after
the split; Scorecard layout cited to the example the README links
(ossf/scorecard scorecard-analysis.yml @ d13ba3f) and the upload to paginated
analysis ids; the unmerged-branch commit dropped from the grep list; the
redaction and upload-job wording corrected.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@seathatflowsinourveins
seathatflowsinourveins enabled auto-merge (squash) September 23, 2026 14:58
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-23T15:10:31.716734Z d4c9c5f PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@seathatflowsinourveins
seathatflowsinourveins merged commit 6e47651 into main Sep 23, 2026
17 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the claude/automation-closure-polish branch September 23, 2026 15:01

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d4c9c5f7bc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +8 to +9
`<scratch>/`, keeping each file name (2 occurrences; every other byte is unchanged). `SEALED-KEY.json`, arm checkouts, `*.diff` files and battery JSON
files are deliberately not copied here.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Retain the omitted comparison outputs and inputs

The newly added evidence explicitly omits the battery JSON outputs, so an auditor cannot verify the recorded per-check details or command results from actual returned output; the packet contains only normalized booleans and exit codes. This is not recoverable by rerunning the committed harness either, because battery_catalog.py references the absent ../gh-settings/after input, causing B13 to become non-applicable rather than reproducing the reported 13/13 result. Retain sanitized battery outputs and immutable input/revision references with the closure evidence.

AGENTS.md reference: AGENTS.md:L10-L15

Useful? React with 👍 / 👎.

seathatflowsinourveins added a commit that referenced this pull request Oct 16, 2026
…rivate horizon repository (#1194)

* tools(landing): add horizon_land.sh, the local-first lander for the private horizon repository

horizon_land.sh lands pull requests of seathatflowsinourveins/horizon, which has no hosted CI,
no scripts/validate.py, no evidence manifest and no Codex connector. Its main ruleset requires
the commit statuses lander/local-ci, lander/verdict-claude and lander/verdict-gpt (strict), so
the lander is the only merge path: it checks the local evidence, posts the three statuses at the
exact head and squash-merges pinned with --match-head-commit.

Checks, in order: the arguments and inputs (HORIZON_CLAUDE_VERDICT, HORIZON_GPT_VERDICT,
HORIZON_MUSE_VERDICT as a TRIAL, LOCAL_CI_RECEIPTS), with the other landers' inputs refused
before any call; its own horizon.lock; the pull request's head must equal the read head (no
head-following); origin/main must be an ancestor of the head and GitHub must report neither
BEHIND nor DIRTY (the branch is never updated here); the verdicts (nas_land.sh's acceptance plus
a last Gate/Verdict line naming the full 40-hex head, the right Family line, two different
files); the receipts (CI: exit <code> at <head>: <command>, the last line per command counts,
sh tests/hooks_test.sh required, every command exit 0); main's CODEOWNERS owned by the merging
account alone, and GitHub's codeowners/errors at that main reporting none; then a draft is
marked ready, after the local refusals.

Correction #121 is carried as nas_land.sh carries it (the CC's ruling of 2026-10-15): a verdict
of any family whose text states a landing gate (nas's four phrases, any letter case) needs
GATES_DONE, a receipt that names the full 40-hex head (the shared block's lg_receipt_ok given the
full head, so the 8-digit prefix alone is refused), or exit 2 before any change on GitHub; it is
read again at the final gate (exit 7 when its bytes changed) and recorded as Lander-gates-done.

Statuses: each success carries "<file name> sha256 <digest>" (local-ci: one digest over the
receipt files). An EXIT trap, with HUP, INT and TERM trapped, posts failure for all three unless
the merge succeeded. The merge state is polled until CLEAN. The squash body ends in a trailer
paragraph (Lander-head, Lander-local-ci with one Lander-local-ci-receipt line per file,
Lander-verdict-claude, Lander-verdict-gpt, Lander-verdict-muse when set, Lander-gates-done when
a verdict states a landing gate, Lander-codeowners). final_gate re-reads the head, the verdicts,
GATES_DONE and the receipts and their digests, the combined status
(exact descriptions, contexts compared case-insensitively), CODEOWNERS, origin/main and the
merge state, with nothing but the merge after it.

Exit codes: 2 inputs, verdicts or a stated landing gate without GATES_DONE, 3 head moved, 5
receipts or statuses, 6 merge state not settled, 7 ready, body or merge failed, or a verdict,
receipt or GATES_DONE changed after its post, 8 BLOCKED or
CODEOWNERS, 11, 12 and 14 as in the other landers, 13 not up to date or main unreadable or
moved, 15 (new) a status POST failed, 129, 130 and 143 a signal.

The shared land-gate-fns block is nas_land.sh's byte for byte (now a three-way test);
nas_land.sh and uet_land.sh are unchanged. Tests: HorizonEndToEndTests with its own scratch
repository, the fake gh's status POST and codeowners/errors branches, and horizon joining the
existing script-text checks. README, its digest row and the manifest rows updated.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Fold the #1194 reviews: names-only statuses and trailers, base binding, PROPOSAL labels

The GPT read (CHANGES_REQUESTED at 34add43) and the CC's read (PASS with items), each finding
and its fix:

- GPT P1, private digests published: the statuses and the trailers named each private file's
  sha256. Now the three descriptions are file names (local-ci: the receipt names, one space
  apart) and every Lander-* trailer names files only; CODEOWNERS keeps its sha256, its preimage
  being main's own file. The digests stay local: compared at the final gate and kept in a
  landing receipt, $LANDING_COORD/horizon-landings/<pr>-<head12>.json, written once after the
  merge with mode 0400 (exclusive create; its place checked before ready and the posts, exit 7;
  a failed write after the merge, new exit 16).
- GPT P2, base not bound: the head and baseRefName are read together; the base must be main
  before ready and the posts and again at the final gate (exit 13, the name withheld). Cites
  github/docs about-code-owners.md:44 and cli v2.102.0 pkg/cmd/pr/merge/http.go:47-50, :77-80.
- GPT P3, terminator: the full head must be a whole word (no letter, digit or _ after it), in a
  verdict and in GATES_DONE; head+A is refused.
- GPT P3, post-merge report: the merge commit is read with its exit checked and printed in its
  typed shape only; a failed read is exit 12 (the receipt then records no merge commit).
- GPT P3, body guard: #1194's base, main at the first reads and its first head registered in
  PUBLIC_COMMITS with their URLs (each read on GitHub), and the first head's horizon_land.sh
  digest in PUBLIC_DIGESTS.
- CC P2, "the only merge path": labelled PROPOSAL, pointing to #1195's decision record, with the
  residual stated: GitHub checks only the contexts' states and anyone with write access can set
  one (available-rules-for-rulesets.md:176, :178), so a web-UI merge after the posts is not
  refused.
- CC P2, ruleset mechanics: strict cited at available-rules-for-rulesets.md:188-193 and the
  checks at :172-178; what those lines do not say is labelled PROPOSAL.
- CC P3, the merge-state lag: "measured on us-equities-trading #58, not documented by GitHub".
- CC P3, UNSTABLE: MergeStateStatus cited at schema.docs.graphql:26511-26554 (:26551-26553).
- CC P3, a second signal during revocation: trap '' HUP INT TERM is hz_revoke's first statement.

Tests, each failing at 34add43 (9 tests, 34 failures) and against single-rule mutants: no
digest or 7-digit prefix of a private file in a description or the body, the receipt keeping
them; the receipt's place and its write; a base other than main, at the start and as a late
retarget flip; head+A, G, 0 or _; the post-merge read with a stdout sentinel; a TERM during the
first revocation POST. The shared block stays nas_land.sh's byte for byte; nas_land.sh and
uet_land.sh are unchanged. README, its digest row and the manifest rows refreshed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Fold the #1194 delta reads: required local workflow wording, signal-safe merge finalisation, fail-closed CODEOWNERS and receipt lists

The command center's delta read at b4489b1 (PASS with micro items), the GPT delta read
(ACK with four P3s) and the local review (three P2s), in one commit:
- Wording matches #1195's Decision 4: the lander is "the required local workflow" (GitHub
  enforces three contexts and freshness; the web-UI residual), and the self-audit is "the
  PROPOSAL self-audit", never "planned" (CC P2-1).
- The merge is finalised whole: HUP, INT and TERM are ignored from just before the merge
  call until the landing receipt is written and the merge reported, so a signal neither
  revokes a merged head's statuses nor loses the receipt (CC P3-3, local review P2-2);
  when exits 12 and 16 both apply, 12 is the exit (documented and tested).
- CODEOWNERS fails closed: a git ls-tree or read that fails is exit 8, and the next
  location is tried only after a successful lookup with no entry (local review P2-1).
- LOCAL_CI_RECEIPTS with a leading, trailing or doubled ':' is exit 2 before any call;
  read -a dropped a trailing empty field (local review P2-3).
- uet#58's lag is worded as measured there, not documented by GitHub (CC P3-1); the
  noclobber and trap '' texts cite the installed bash 5.3 bash(1) (CC P3-2).
- Comments no longer describe a posted digest (GPT P3-4).
- The README digest row and the published-digest list follow the new script bytes;
  b4489b1's script digest is registered with its URL.

Fail-first: the four new tests fail 7 times against b4489b1. Follow-ups, not taken here:
GPT P3-1 (the shared author check, in the block that stays byte-identical to nas), P3-2
(a late local symlink swap after the receipt check) and P3-3 (a documentation assertion
for the recovery limits).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Reconcile a failed merge call before revoking; cite bash for the receipt split and pipefail

The command center stopped the auto-land at 4fad273 on the local review's P2, and the
GPT micro read there was an ACK with five P3s; one commit:
- A failed merge call is reconciled before anything is revoked (local review P2): GitHub
  can commit the merge and the reply be lost, so the state is read three times, two
  seconds apart. MERGED at the read head continues as a landing (HZ_MERGED, the receipt,
  the report); OPEN or CLOSED there on all three reads is exit 7 and the EXIT trap
  revokes; anything else is the new exit 17: the outcome is unknown, the statuses are kept
  (HZ_KEEP), nothing is revoked, and the pull request is checked by hand. README exit
  codes and step list updated; the old merge-failure test now serves an OPEN pull
  request. Fail-first: the new test fails against 4fad273.
- GPT P3-4: the empty-field comment cites bash(1) EXPANSION, Word Splitting ("field
  terminators"; adjacent delimiters "produce a null field"), and the pipefail comment
  quotes SHELL GRAMMAR, Pipelines (the rightmost non-zero status), both from the
  installed bash 5.3 (5.3-2ubuntu1).
- GPT P3-5 is the PR body's wording (aligned there); P3-1, P3-2 and P3-3 stay Follow-ups.
- The README digest row and the published-digest list follow the new bytes; 4fad273 and
  its script digest are registered with their URLs.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant