Skip to content

Saturation tracking loop: hash-chained per-layer ledger, weekly model-free report, one tracking issue - #166

Merged
seathatflowsinourveins merged 5 commits into
mainfrom
claude/saturation-tracking-20260924
Sep 24, 2026
Merged

seathatflowsinourveins merged 5 commits into
mainfrom
claude/saturation-tracking-20260924

Conversation

@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator

Adds a saturation tracking loop for the SOTA landscape. It implements the stopping rule in docs/landscape-continuation.md:89-101, which defines saturation as a named candidate set for frozen requirements and a date. It needs no unattended model calls: catalogs/landscape/research-state.json keeps automatic_model_calls: false.

Parts

  • catalogs/saturation/ledger.json, its schema and README. The ledger is append-only and hash-chained (prev_sha256 plus head_sha256). Each sweep records, per layer:
    • candidates that were proposed, known, new, survived and refuted, with both refuter votes;
    • reopen triggers;
    • the requirement and platform-profile hashes.
  • Counting. A layer becomes a saturation candidate after 3 clean completed sweeps at least 7 days apart with an unchanged requirement. A stopped run neither counts nor resets. Any survivor, reopen trigger, changed requirement or platform, or pin_moved/stale receipt resets the count. A candidate is only an input to closure: the landscape owners still close layers in research-state.json closure_refs.
  • scripts/saturation_ledger.py, with --check [--base REF], --report, --append RESULT.json and --derive-seed. It refuses to rewrite history. The writer can only write inside catalogs/saturation/.
  • .github/workflows/saturation-tracking.yml, weekly on Monday at 07:53 plus workflow_dispatch dry_run:
    • it calls no model and has only contents: read;
    • harden-runner is the first step of each job;
    • the report combines the ledger, receipt_staleness.py --json and the latest catalog-freshness artifact;
    • one final job with only issues: write keeps one issue labelled saturation-tracking up to date.
  • recipes/saturation-sweep.md. Sweeps are scoped to the layers that are due. Each sweep runs the existing landscape lane in a coordinator session, retains usage, stopped runs and votes, then appends.
  • Seed. The seed covers the 2026-09-23 landscape sweep: the completed run wf_38aa6d5d-d6c and the stopped run wf_28d47bbc-1b5, which is kept with lower-bound usage. Every value is derived from retained files. No seed layer counts as clean, because the per-vote returns were not retained. It covers 32 layers and 11 survivors, each with a source review.

Separation. No verdict-review-gate trust path or data binding is touched. verdict_review_gate.py --base origin/main reports "no verdict rows changed". The change leaves research-state.json, catalog-freshness.yml, freshness_propose.py, receipt_staleness.py and adoption/* untouched.

Verification (on b0c65c5, merged with main at aa6fc79)

  • 99 targeted tests OK (2 skipped: jsonschema and PyYAML are absent). The 42 ledger tests cover:
    • counting fixtures;
    • hash-chain tampering and --base rewrite detection;
    • binding every survivor to its manifest row and source review;
    • vote recomputation;
    • usage registration;
    • write confinement, enforced with an audit hook;
    • workflow permissions.
  • validate.py, evidence_manifest.py --check, landscape.py --root . and host_receipts.py validate all pass.
  • zizmor 1.30.1 finds nothing. actionlint 1.7.12, checked against its pinned sha256, exits 0.
  • The full suite has one failure, the host-only test_effort_default_guard.test_shipped_guard_is_verbatim (a stale ~/.claude hook). It also fails on plain main.

Review. Two independent reviewers looked at gate separation and workflow security, and at ledger correctness and evidence honesty. They raised 10 findings, one of them high (a byte-for-byte seed test that would break after the first real append). The fix round a768186 fixed 8 and rejected 2 with stated reasons: current-state triggers stay stateless, and the stopped record keeps manifest_ref.

Not yet done. No live dry_run or real dispatch has run, and the first appended sweep, which is the loop's end-to-end acceptance, is still to come. Both happen after merge.

🤖 Generated with Claude Code

Scout and others added 3 commits September 24, 2026 02:28
An append-only, hash-chained ledger (catalogs/saturation/ledger.json) records each
landscape sweep per layer: proposals, survivors and refuted candidates with both
refuter votes, reopen triggers, and the registered usage. scripts/saturation_ledger.py
checks the chain and every binding (survivor to manifest lane row and source review,
both votes with survival recomputed, completed usage registered and complete, stopped
usage marked lower_bound), derives consecutive clean-sweep counts and saturation
candidates, and appends new records without rewriting history. It never writes
research-state.json or any verdict path.

The seed is the 2026-09-23 landscape sweep: stopped attempt wf_28d47bbc-1b5 and
completed run wf_38aa6d5d-d6c, derived from retained files (--derive-seed, re-derived
byte for byte by the tests). Per-vote returns and prompts were not retained, so every
seed layer is votes: not_retained / not_returned and none counts as clean.

.github/workflows/saturation-tracking.yml runs weekly (53 7 * * 1) and on dispatch with
dry_run: it checks the ledger, builds the report from the ledger, research-state,
receipt staleness and the latest catalog-freshness manifest, and a final job with only
issues: write upserts one issue labelled saturation-tracking. No model or CLI-agent
calls and no secret beyond the job token. recipes/saturation-sweep.md describes the
person-started sweep and the append.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…returns

- Append-only across commits: tests/test_saturation_ledger.py compares the
  ledger with the one at the merge base of HEAD and origin/$GITHUB_BASE_REF,
  so validate.yml's full unittest run catches a rewrite that recomputes
  every hash. check_append_only now rejects an unknown base ref and treats
  only a valid commit without the ledger file as "no ledger yet".
- The seed test compares the rebuilt seed records without the append-time
  requirement/platform-profile hashes, and it recomputes those hashes from
  research-state.json and adoption/manifest.json at 9eac1f9, 0465141 and
  4a4c8a2. A landscape edit or a later --append no longer fails the suite.
  The weekly workflow no longer runs the unit tests before its report.
- votes: retained now needs the sweep's registered returns_ref and a
  per-layer discovery_ref whose proposed list equals the layer's. Every
  vote and discovery ref needs a JSON pointer. Retained votes must resolve
  in returns_ref to the same role and repository. Lens votes must resolve
  to that lens in this layer's lane row for the same repository. A
  retained layer without a discovery return is never clean.
- Sweep dates must be calendar dates.
- The stopped seed attempt records its 8 in-flight children as
  lost_workers, which are bound to the usage output. Its notes now say
  that its date and manifest_ref come from the superseding run.
- Receipt-flag and freshness triggers are documented as current-state
  holds. The recipe records them as reopen entries so the reset stays
  durable.
- The weekly workflow reads only scheduled catalog-freshness runs and
  marks the freshness input partial on upstream errors. A dry run uses an
  issues: read plan job, and the issues: write job is skipped.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-24T07:03:56.595956Z b0c65c5 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b0c65c541e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/saturation_ledger.py Resolved
Comment thread scripts/saturation_ledger.py Resolved
Comment thread scripts/saturation_ledger.py Resolved
Comment thread scripts/saturation_ledger.py Resolved
Comment thread scripts/saturation_ledger.py Resolved
Comment thread scripts/saturation_ledger.py Resolved
Comment thread scripts/saturation_ledger.py Resolved
Comment thread scripts/saturation_ledger.py Resolved
A relative $id made jsonschema resolve #/$defs refs against a doubled path
(catalogs/saturation/catalogs/saturation/...). CI has jsonschema installed,
so test_the_schema_matches_when_jsonschema_is_available failed there while
skipping locally. Verified with jsonschema and PyYAML present: 99 tests OK.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Review follow-ups on the saturation ledger checker:

- Record the scope frozen before the run: a retained discovery return
  carries requirement_sha256 and platform_profiles_sha256 (from the new
  --scope), --append records those, and --check requires them to match,
  so a scope change during a sweep stands as a current trigger.
- Date a completed sweep by its manifest's checked_at.
- Reject replayed evidence: no two records share a workflow_run, usage
  output or returns file (path or sha256), no two completed sweeps share
  a manifest lane, and workflow_run must be the run the usage measured.
- Require a completed discover:<layer> child per layer and both
  refute-facts/refute-fit children per layer with proposals.
- lost_workers must equal the usage output's incomplete children.
- A survivor's source review must name the layer in a layers list.
- Report a selection removed from (or moved out of) a layer in the
  rebuilt freshness manifest as selection_changed.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@seathatflowsinourveins
seathatflowsinourveins merged commit 15ddf0c into main Sep 24, 2026
22 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the claude/saturation-tracking-20260924 branch September 24, 2026 08:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant