Skip to content

Verify regression fixtures for the secret and workflow-lint gates #14

Closed

Description

@egdev6

Goal

Verify the secret and workflow-lint regression fixtures with positive and negative proof, and confirm the remote aggregate gate rejects non-success outcomes.

Observations

  • Positive and negative fixtures are already implemented locally and proven.
  • The remote aggregate test gate must reject security failure, cancellation, and skipped, not read them as green.
  • Fixture values must be clearly fake and redacted in output.

Decision gate

Approve the fixture set and the aggregate gate failure conditions before wiring remote checks.

Acceptance criteria

  • Fixtures prove a detected violation (non-zero) and a clean pass (zero) locally.
  • The aggregate check never reports success for a failed, cancelled, or skipped security job.
  • Fake fixture values are redacted in output.

Verification

  • Local fixture run -> positive case non-zero, negative case exit 0.
  • Controlled checks against the published candidate -> failed or skipped security jobs cannot yield a successful aggregate check; cancellation cannot appear green.

Release checkpoint

Remaining risk: remote aggregate behavior is unverified until the candidate is published.

Tracking

Roadmap item: B01-02. All acceptance criteria remain pending; publication does not grant approval or release authority.

Dependencies

Activity

  1. egdev6 commented on Oct 10, 2026

    @egdev6
    OwnerAuthor

    ODD reconciliation: regression fixtures and fail-closed aggregate

    Acceptance and current evidence

    Original criterion retained Evidence
    Detected violations return non-zero; clean fixtures return zero Gitleaks fixture record and actionlint fixture record; main CI runs 4 cases for each, without failures or skips
    Failed, skipped or cancelled security cannot produce a green aggregate Controlled scheduler cases below and the current aggregate implementation, which requires both matrix and security success
    Clearly fake fixture values are redacted The Gitleaks fixture checks history-only detection/redaction; CI scans use --redact=100. This is fixture/scanner evidence, not a claim of universal CLI redaction
    Controlled run Security outcome Aggregate test outcome
    37848737265 failure failure
    37848922091 skipped failure
    37848992748 cancelled failure

    These are positive proofs of the negative fail-closed cases. The final run's overall conclusion is cancelled, while its aggregate job is failure. The probe branch is evidence only and must not be merged or automatically cleaned up.

    Remaining ODD

    Review fixture/redaction coverage and the controlled gate cases against every retained criterion, then record human acceptance or precise gaps. Do not repeat probes merely because their intended outcomes were failures. No workflow dispatch/rerun, issue closure, label/body edit or release is authorized by this comment.

    State: acceptance reconciliation, with #13 still requiring its own acceptance record. The shared roadmap will link this verified comment.

  2. egdev6 commented on Oct 11, 2026

    @egdev6
    OwnerAuthor

    Acceptance recorded: regression fixtures and fail-closed aggregate

    The maintainer has accepted all three original criteria for #14 following a read-only audit. Its dependency #13 is accepted and completed (acceptance record). Original scope and criteria are retained.

    Current evidence: immutable main 65927d8 and CI 38097323552.

    Original criterion Verified result
    Detected violations return non-zero; clean fixtures return zero Current Gitleaks 4/4 and actionlint 4/4 passed with zero skips, failures or cancellations. Assertions distinguish detected violations (1) from clean cases (0).
    Failed, cancelled or skipped security cannot produce a successful aggregate The current shipped gate's fixture covers all 16 result combinations. The three existing scheduler cases below each have aggregate test failure. Their immutable test-gate blocks match current main exactly.
    Fake fixture values are redacted The passed Gitleaks fixture checks absence of its synthetic value in the report and stdout/stderr, and the presence of REDACTED. The invalid actionlint fixture also rejects private source-marker output.
    Existing run, attempt 1 Security Aggregate test Overall run
    37848737265 failure failure failure
    37848922091 skipped failure failure
    37848992748 cancelled failure cancelled

    These are intentional negative scheduler proofs, not unresolved product failures. No probe was repeated. The probe branch remains evidence only, never to be merged or automatically cleaned up. Current main retains the strict required test context.

    Redaction evidence is bounded to fixtures/scanners, not universal CLI output or complete production-governance proof. Existing macOS 2 and Windows 3 suite skips remain B03 work; security fixtures have zero skips. This acceptance does not complete the remaining package/runtime/migration work or authorize live sandbox operations, release/version changes, broader credentials or cleanup. The next release remains deferred until the agreed roadmap is accepted as complete.

    With the maintainer's explicit approval, #14 can now be marked completed. Original body, labels, assignees, milestone assignment and previous evidence are preserved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions