Skip to content

Landscape-sweep templates: OpenSSF Scorecard maintenance rule; licenses never a refutation reason - #385

Merged
seathatflowsinourveins merged 5 commits into
mainfrom
claude/sweep-templates-maintained-license-20260927
Sep 27, 2026
Merged

Landscape-sweep templates: OpenSSF Scorecard maintenance rule; licenses never a refutation reason#385
seathatflowsinourveins merged 5 commits into
mainfrom
claude/sweep-templates-maintained-license-20260927

Conversation

@seathatflowsinourveins

@seathatflowsinourveins seathatflowsinourveins commented Sep 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

What

This PR fixes the landscape-sweep harness's frozen prompt templates (tools/sota-convergence/landscape-sweep/templates.json) so that the next per-layer sweep applies the operator's rules.

  1. Maintenance rule, derived from the OpenSSF Scorecard Maintained check (operator, 2026-09-27: "never use stale repos").
    • The common principle "maintained upstream with 2025-2026 activity" becomes a rule derived from Scorecard's check. Scorecard gives an archived repository its lowest score, uses a 90-day window, and gives its top score to at least one commit per week, with partial credit for maintainers' issue activity.
    • Stale, and so excluded whatever its stars: archived, or no commit on the default branch in the last 90 days.
    • How to check: establish commit activity with gh api repos/<o>/<r>/commits?since=<90 days ago>&per_page=1, or the same api.github.com REST URL. Don't use pushed_at, which counts any branch.
    • Scope: the rule excludes only the zero-activity end. Lower activity is recorded as a maintenance fact.
    • Too new: a repository created less than 90 days ago starts its demonstrated_gap with "TOO NEW TO ASSESS (<90 days):".
    • Old tags: an old release tag on an actively committed branch is not stale; pin the commit instead.
    • Fit refuter: it now counts staleness as a refutation reason.
  2. Licenses are information only (operator, 2026-09-26: "license are not applicable for our env, as we are private experimental env not company").
    • common no longer has "OSI or clearly usable license" as a selection principle. Its merit criteria read "maintenance and supply-chain health, and platform fit (license is information only)"; before, they read "… license and platform fit".
    • The fit refuter's instruction to refute a candidate whose "license is non-commercial, custom-restrictive or unclear" is removed.
    • The facts refuter still checks that license claims are accurate.
  3. docs/harness-defaults.md: the anti-pattern row "Gating candidates on license" now names this change and its test, instead of saying it is pending.

Reviews

  • Independent review 1 (headless Sonnet 5, effort max, at 01766f33): needs_changes, 1 major and 4 minor findings. All five were fixed in 65cb1e83. The major one was the common merit clause, which still judged licenses.
  • Independent re-check (headless Opus 5.5, effort max, at 65cb1e83): the major finding and 3 of the 4 minors resolved. It raised two minors: the Scorecard wording ("only"), and this description. Both are fixed at the current head.
  • GPT-6 cross-family review: running now, at the current head.

Tests

  • The template-hash change detector is PROMPTS_SHA256_CURRENT = a9722fec…. Per the test's own procedure, an intended template edit updates it. PROMPTS_SHA256_20260926 (3adfbed7…) stays as that run's recorded value, which the fixtures use.
  • test_templates_never_refute_on_license_and_name_the_maintenance_rule scans every template for license-as-criterion phrases and checks that the maintenance rule's command, pushed_at exclusion and too-new prefix are present.
  • python3 -m unittest tests.test_landscape_sweep_harness tests.test_saturation_ledger: 153 run, OK (3 skipped).
  • scripts/validate.py passed.
  • git diff origin/main -- templates.json changes only the common and fit strings; the file's 1-space JSON formatting is unchanged.

SOTA sources

🤖 Generated with Claude Code

https://claude.ai/code/session_012qwXNtFyUhY5x2dtrm7tG5

Scout @claude
Landscape-sweep templates: OpenSSF Scorecard maintenance rule; licens…
01766f3 
…es never a refutation reason

The sweep's templates change in two ways:
- "Maintained upstream with 2025-2026 activity" becomes the OpenSSF Scorecard
  Maintained check. A stale repository (archived, or without a default-branch
  commit in the last 90 days) is excluded whatever its stars. A repository
  under 90 days old is flagged as too new to assess. An old tag on an active
  branch is not stale; pin the commit instead.
- Licenses are recorded as information only and are never a reason to exclude,
  refute or rank down a candidate. This follows the operator's 2026-09-26
  decision; the fit refuter was still told to refute non-commercial or unclear
  licenses.
The template-hash change detector moves to PROMPTS_SHA256_CURRENT (dc5ffcb6...).
The 2026-09-26 value stays as that run's recorded hash. A new test checks both
rules are in the templates.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_012qwXNtFyUhY5x2dtrm7tG5
@seathatflowsinourveins seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Sep 27, 2026
Scout and others added 3 commits September 27, 2026 01:38
Scout @claude
#385 review repair: drop license from the shared merit criteria; stat…
65cb1e8 
…e the maintenance rule precisely

Independent review (headless Sonnet 5, effort max): needs_changes, 1 major and 4 minor findings.
- major: the shared common template, which reaches every role including the
  fit refuter, still judged "license and platform fit". It now reads "and
  platform fit (license is information only)".
- minor: the rule is now "derived from" the Scorecard Maintained check and
  states Scorecard's actual criteria (archived is lowest; a 90-day window;
  top score needs a commit per week). This rule excludes only the
  zero-activity end.
- minor: the commit fact is established with
  gh api .../commits?since=<90 days ago>, not pushed_at (any branch).
- minor: a repository under 90 days old says "TOO NEW TO ASSESS (<90 days)"
  at the start of its demonstrated_gap.
- minor: docs/harness-defaults.md's anti-pattern row no longer calls the
  template change pending.
The regression test now scans every template for license-as-criterion phrases.
PROMPTS_SHA256_CURRENT is updated to 0e443585.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_012qwXNtFyUhY5x2dtrm7tG5
Scout @claude
#385 re-check residuals: Scorecard's weekly commit is sufficient, not…
b87a656 
… necessary; REST URL alternative

The independent Opus re-check agreed the major and 3 of 4 minor findings were
resolved, and raised two minors.
- The shared template said Scorecard "gives its top score only for at least one
  commit per week". It now gives the top score to at least one commit per week
  and partial credit for maintainers' issue activity (the Scorecard doc's
  sufficient condition).
- The commit check also names the same https://api.github.com REST URL, for
  lanes without the gh CLI.
The test comment says "derived from", and the anti-pattern log names the test.
PROMPTS_SHA256_CURRENT is updated to a9722fec.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_012qwXNtFyUhY5x2dtrm7tG5
Scout @claude
#385 GPT-6 review: unknown maintenance is never a refutation (web-onl…
99837f2 
…y lanes cannot reach the commit evidence)

GPT-6 cross-family review (gpt-6-astra, max) of b87a656 returned needs_changes,
with 1 medium finding. The web-only GPT-6 lane cannot run gh (make_prompt.py
TAIL), and its web tools failed to fetch the commits API URL. Since the fit
refuter defaults to refuted, maintained repositories could be refuted for
unreachable evidence.
- common: a lane that cannot reach either source records the commit fact as
  unknown and never excludes or refutes on unknown maintenance, because a
  missing result is pending, never refuted.
- fit: staleness must be established from evidence; unknown maintenance is
  never a reason to refute.
The test asserts both. PROMPTS_SHA256_CURRENT is updated to f64eec22.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_012qwXNtFyUhY5x2dtrm7tG5
@seathatflowsinourveins

seathatflowsinourveins commented Sep 27, 2026

Copy link
Copy Markdown
Collaborator Author

Reviews complete at 99837f2e. Ready to merge.

Round Reviewer Head Verdict Outcome
1 Independent headless Sonnet 5, effort max 01766f33 needs_changes: 1 major, 4 minor All fixed in 65cb1e83
2 Independent headless Opus 5.5, effort max 65cb1e83 Major and 3 of the 4 minors resolved; 2 new minor Fixed in b87a6563
3 GPT-6 cross-family (gpt-6-astra, max) b87a6563 needs_changes: 1 medium (the web-only GPT-6 lane can't reach the commit evidence, so default-refute could reject maintained repositories) Fixed in 99837f2e: unknown maintenance is never a refutation
4 GPT-6 re-check 99837f2e accept, no new findings —

At 99837f2e:

  • python3 -m unittest tests.test_landscape_sweep_harness tests.test_saturation_ledger: 153 run, OK (3 skipped).
  • scripts/validate.py: passed.

@seathatflowsinourveins seathatflowsinourveins mentioned this pull request Sep 27, 2026
Merged
Scout
Merge remote-tracking branch 'origin/main' into claude/sweep-template…
4b3c3c7 
…s-maintained-license-20260927

# Conflicts:
#	manifests/evidence.json
@seathatflowsinourveins seathatflowsinourveins mentioned this pull request Sep 27, 2026
Merged
@seathatflowsinourveins
seathatflowsinourveins merged commit 1d8f2f7 into main Sep 27, 2026
24 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the claude/sweep-templates-maintained-license-20260927 branch September 27, 2026 06:40
@seathatflowsinourveins seathatflowsinourveins mentioned this pull request Sep 28, 2026
Merged
5 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@seathatflowsinourveins