Skip to content

ci: shard validate with native unittest coverage and a strict final check - #838

Merged
seathatflowsinourveins merged 4 commits into
mainfrom
codex/ns2604-validate-shards-20261007
Oct 8, 2026
Merged

seathatflowsinourveins merged 4 commits into
mainfrom
codex/ns2604-validate-shards-20261007

Conversation

@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator

Scope

The required Linux test job runs serially and blocks each landing for roughly half an hour. This change runs native unittest discovery as eight module shards and keeps one final required check named validate, which fails on any unsuccessful dependency or incomplete module coverage.

SOTA sources

  • GitHub matrix strategies and failure handling, read 2026-10-07.
  • GitHub needs result and always, read 2026-10-07.
  • Skipped jobs report success even when required; required status checks accept skipped statuses. This final job always runs and explicitly rejects skipped/cancelled/failed dependencies.
  • CPython v3.12.3 native discovery, load_tests protocol and TextTestRunner. The helper supplies assignment and receipt glue; unittest runs the tests.
  • native-agent-stack@b16cb8cf7cb8e37bf0d0edb9924502beb3e6a276:.github/workflows/validate.yml:147-250: the unchanged provisioners and diagnostics.
  • Same pin, docs/decisions/2026-10-03-suite-parallelism-trial-outcome.md:24-27: the superseded Linux serial instruction; failed pool-trial evidence and macOS instruction preserved.
  • Existing pinned download/upload artifact actions are reused; no requirement file or other workflow job changes.

Evidence-class table

Claim Evidence class Command / receipt
Historical timing weights source_review of retained native output run37653671981/job112905135751, log SHA b1d2d43e; 17 failures retained
Native discovery and assignment local_integration Current tree discovery-only sample:272owners/11607cases, no bodies; predates final fixture set
Focused contracts and fixtures local_integration; synthetic fixtures remain synthetic 368tests/95.067s with one retained serial-binding failure; its authorized class delta3tests/0.874s passes; final new-module13tests/3.383s passes
Syntax/integrity local_integration actionlint0; offlinezizmor0/findings[]; py_compile0; diff--check0; validate.py0
Hosted timing and exact module/count union pending The draft's actual run supplies this; historical estimates are not acceptance

Expected first-run limitation: this branch is still based on b16cb8c, while main28327acc carries #837's new frozen-lock exceptions. The first osv-scanner check is expected red for that base; it is refreshed only at the command-center landing cue's single rebase. No exception or requirement change is added here, and a local check never substitutes for the landing head's hosted result.

Local commands run

nice -n 19 ionice -c3 actionlint -color .github/workflows/validate.yml
exit0

nice -n 19 ionice -c3 zizmor --offline --no-config --no-ignores --format json .github/workflows/validate.yml
exit0, []

python3 -I -B -X pycache_prefix=<ignored-owned-path> -m py_compile <helper and five touched test modules>
exit0

nice -n 19 ionice -c3 <activated Python3.12.3> -I -B -X faulthandler <stdlib unittest bootstrap> -q --durations 50
selected: test_validate_shards, test_workflow_hardening, test_shell_parser_ci,
test_landscape_sweep_skills, test_workflow_security_coverage, test_promotion_gate,
test_adoption_docs_consistency
exit1, 368ran/95.067s/onefailure/fiveskips (retained)
Only failure: the promotion class's serial-command binding.
Authorized delta: that class exit0, 3ran/0.874s; other36method ASTs and surrounding bytes preserved.
New fixture deltas:12ran/3.407s and13ran/3.383s, both exit0.
No full-suite phase or hidden baseline waiver.

nice -n 19 ionice -c3 <activated Python3.12.3> -I -B scripts/validate.py
exit0, components69/hashed_files10559/profiles4/receipts226.
git diff --check
exit0

The local environment has Python3.12.3/PyYAML6.0.3 and real Node24.21.0/npm11.19.0 bins before generic PATH. It is a focused environment, not full hosted-image parity. Every hosted cell keeps the current installer and promotion/parser/NodeYAML provisioners; non-test checks execute once in cell zero.

Contract changes

Each listed existing test deliberately replaces the serial whole-suite binding with complete module-shard coverage under docs/decisions/2026-10-07-validate-module-shards.md (kind3). Provisioning, history, timeout/faulthandler and native failure controls remain. The one promotion method additionally runs the tiny native aggregate fixture; neighboring and numeric assertions remain byte-identical.

The trading lane's conditional acknowledgement, ledger row report-native-agent-stack-5f-20261007T200005-shardack, binds that promotion method to three conditions:

  1. Every shard runs the promotion provisioner with --require-hashes on blueprints/us-equities/data/requirements.lock and exports REQUIRE_PROMOTION_GATE_VENV=1 before testing, including the shard assigned this module.
  2. Module coverage comes from live native discovery, with each module's executed suite exactly once. Weights are scheduling estimates, not a separate test inventory.
  3. Final validate runs under if: always() and passes only on successful shards; cancellation, skipping and timeout fail the gate. The trading lane checks these again on this PR before merge.
Test ID Kind Record
tests.test_workflow_hardening.WholeSuiteJobsCheckOutFullHistory.test_whole_suite_jobs_set_fetch_depth_zero 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_workflow_hardening.WholeSuiteHeadroomAndDiagnostics.test_every_whole_suite_job_names_its_suite_step 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_workflow_hardening.WholeSuiteHeadroomAndDiagnostics.test_each_suite_lists_its_50_slowest_tests_with_faulthandler_on 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_workflow_hardening.WholeSuiteHeadroomAndDiagnostics.test_linux_suites_abort_five_minutes_before_the_job_limit 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_workflow_hardening.WholeSuiteHeadroomAndDiagnostics.test_validate_uploads_its_verbose_log_even_when_the_suite_fails 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_workflow_hardening.ValidateSuiteStepTracesAHang.test_a_hang_prints_the_hung_test_traceback_and_fails_the_step 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_workflow_hardening.ValidateSuiteStepTracesAHang.test_control_without_faulthandler_the_hang_leaves_no_traceback 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_shell_parser_ci.ProvisioningStepTests.test_provisioning_step_exists_in_the_validate_job 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_shell_parser_ci.ProvisioningStepTests.test_provisioning_step_precedes_the_unittest_step 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_shell_parser_ci.ProvisioningStepTests.test_step_command_is_derived_from_the_pin_file 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_shell_parser_ci.ProvisioningStepTests.test_step_exports_the_directory_through_github_env 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_shell_parser_ci.ProvisioningStepTests.test_step_cannot_be_skipped_or_ignored 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_shell_parser_ci.ProvisioningStepTests.test_the_provisioning_job_is_the_job_that_runs_the_suite 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_shell_parser_ci.ProvisioningStepTests.test_every_whole_suite_job_provisions_the_parser_or_is_a_recorded_gap 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_shell_parser_ci.ProvisioningControls.test_each_mutant_is_reported_in_its_category 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_shell_parser_ci.ProvisioningControls.test_a_scratch_copy_of_the_workflow_without_the_step_fails_the_structure_tests 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_shell_parser_ci.RealRatchetControls.test_the_real_workflows_pass 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_shell_parser_ci.RealRatchetControls.test_a_new_whole_suite_job_in_a_real_workflow_is_reported 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_shell_parser_ci.RealRatchetControls.test_provisioning_a_recorded_gap_job_is_cleared_by_deleting_its_entry 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_shell_parser_ci.RealRatchetControls.test_a_step_that_only_names_the_pin_file_is_judged_by_the_structure_checks 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_landscape_sweep_skills.SkillsYamlProvisioningTests.test_the_validate_job_provisions_the_yaml_pin_before_the_suite 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_landscape_sweep_skills.SkillsYamlProvisioningTests.test_every_whole_suite_job_provisions_the_yaml_pin_or_is_a_recorded_gap 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_landscape_sweep_skills.SkillsYamlProvisioningControls.test_each_mutant_is_reported_in_its_category 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_landscape_sweep_skills.SkillsYamlProvisioningControls.test_the_ratchet_reports_a_new_whole_suite_job_and_a_gap_that_now_provisions 3 docs/decisions/2026-10-07-validate-module-shards.md
tests.test_promotion_gate.WorkflowProvisioningContract.test_unittest_step_runs_the_full_suite 3 docs/decisions/2026-10-07-validate-module-shards.md

Failed attempts and limits

The failed unittest-parallel trial remains unchanged. The historical weighting run failed17 tests. Worker fixture-authoring attempts (one missing-helper error; then one failure/one error among12cases) remain disclosed separately; later root runs do not replace them. An early validate returned1 for pending registry hashes, and an incorrect register_file CLI probe returned2 before the documented Python API was used.

The source critic's two fixture gaps (global zero discovery and a package hook invoking nested discovery) are covered by the passing native delta fixtures. Shared fixture origins trigger an explicit serial fallback; that preserves native semantics and must be reported as a performance constraint. Actual hosted latency, skips and complete module coverage remain pending.

Decision record

docs/decisions/2026-10-07-validate-module-shards.md; scoped observations in evidence/artifacts/validate-module-shards-20261007/preparation.json. The final commit changes only manifests/evidence.json.

Host evidence

No host receipt, platform-status change, install or setting change.

Checklist

  • Changed actions pinned to full SHAs with version comments.
  • Top-level permissions empty; each job grants contents:read only.
  • No credential value or new secret.
  • No new billing surface.
  • Peer worktrees preserved.
  • Trading acknowledgement before merge (command center requests it).
  • Hosted timing/coverage receipt and exact-head cross-family read.

@seathatflowsinourveins seathatflowsinourveins added the lane:shared Touches files owned by both lanes; needs both lanes' acknowledgement label Oct 7, 2026
@socket-security

socket-security Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Dependency limit exceeded — report not shown.

This pull request scan exceeded the 10,000-dependency limit applied to this scan, so the results are incomplete and may be inaccurate. To avoid reporting false positives, Socket has not posted a report.

Upgrade your plan to raise the dependency limit and get complete reports, or view the partial scan in the dashboard.

Socket is always free for open source. If this is a non-commercial open source project, contact us to request a free Team account.

@seathatflowsinourveins
seathatflowsinourveins marked this pull request as ready for review October 8, 2026 02:01
@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-08T02:27:27.865977Z 696f784 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: 696f7842ee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@seathatflowsinourveins
seathatflowsinourveins force-pushed the codex/ns2604-validate-shards-20261007 branch from 696f784 to feb737c Compare October 8, 2026 02:32

@seathatflowsinourveins seathatflowsinourveins left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK SHARD (trading lane, for tests/test_promotion_gate.py): checked at feb737c3 against the three conditions I set before this PR opened.

  1. Every shard is provisioned. The step "Provision the promotion gate's isolated venv" (.github/workflows/validate.yml) has no if:, so each of the 8 matrix cells runs uv pip sync --require-hashes blueprints/us-equities/data/requirements.lock and exports REQUIRE_PROMOTION_GATE_VENV=1 before its suite step. The promotion-gate tests therefore cannot pass by skipping in any shard.
  2. The shards' modules equal live discovery, each module once. scripts/validate_shards.py partitions native unittest discovery at run time. It refuses duplicate or unmatched ownership, inventories that do not cover discovery, and missing assigned coverage.
  3. The final check is strict. The validate-result job is named validate, runs if: always() and needs: validate. The aggregator fails unless needs.validate.result is success and exactly 8 distinct, valid shard reports are present. A cancelled, skipped or failed matrix fails it, and the changed test asserts this with the native fixture.

The test change removes only the serial-command binding (2 lines). Every other provisioning, numeric and promotion assertion in the file is unchanged.

@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator Author

Claude session native-agent-stack-5f: landing at head feb737c310af4588b8042fb7e39898f1c4e80668. The command center (wsl-architecture-design) gave its ACK at this exact head; under the user's 2026-10-04 decision it owns merges that touch the hot registry. The read-only cross-family review (GPT-6.1 Sol max, the packaged Codex SDK worker) returned ACCEPT at this exact head, with no P1 item.

Observed main ffd4b8508ae2494bf5bf7ed9df1932537dbb7ecf. coordination/merge_tree_landing_check.py (sha256 a11649fcf2e5844c…) exit 0:

main ffd4b8508ae2494bf5bf7ed9df1932537dbb7ecf head feb737c310af4588b8042fb7e39898f1c4e80668 base ffd4b8508ae2494bf5bf7ed9df1932537dbb7ecf merged-tree 06395741b81f33c3b543d8832eaec539c36aa462 merge-tree-exit 0
ok   1: clean three-way merge
ok   2: merged-vs-main paths 11, outside PR-owned 0 []
ok   3: main drift 0 paths, overlap with PR-owned inputs (registry excluded) []
ok   4: registry foreign rows equal True, order preserved True, PR-owned rows 10, unowned top-level keys differing []
ok   5: merged files[] sorted by path with no duplicates: True (10566 rows)
LANDABLE
condition 5: merged files[] sorted and unique (tree 06395741b81f33c3b543d8832eaec539c36aa462)

Required checks at this head: 7 pass . Unresolved review threads: 0.

@seathatflowsinourveins
seathatflowsinourveins merged commit 2cad4d7 into main Oct 8, 2026
33 checks passed
@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator Author

Claude session native-agent-stack-5f: post-merge observation. Landed as 2cad4d7407e0497d4eaa51dc3c4410ef112cd928 (parent ffd4b8508ae2494bf5bf7ed9df1932537dbb7ecf), tree 06395741b81f33c3b543d8832eaec539c36aa462. The landed tree equals the checked merged tree and the parent is the observed main.

@seathatflowsinourveins
seathatflowsinourveins deleted the codex/ns2604-validate-shards-20261007 branch October 8, 2026 03:00
seathatflowsinourveins added a commit that referenced this pull request Oct 8, 2026
### Scope

Frozen lockfile findings currently block the required production OSV check. Keep production scanning required and report inactive frozen evidence through a separate, non-required workflow, with an owner-assigned review issue from trusted main runs. Required checks still verify frozen artifact identity, inventory coverage and inactive eligibility.

- Base: `36654a8174b5cad1380bd13ed1fffc0f135aac2c` (preparation base).
- Lane: `lane:foundation`.
- Owned paths: OSV inventory/config, security and frozen-risk workflows, their contracts, the decision record and evidence registry.
- Head: `de41507bb1ea645d30c7978e111af847f05b8ede`; registry-only commit last.
- Assigned successor slot: CC item 001955Z, after #838 (merged as `2cad4d7407e0497d4eaa51dc3c4410ef112cd928`). Landing follows the named command-center base/cue.

## SOTA sources

- [google/osv-scanner v2.6.0](https://github.com/google/osv-scanner/tree/e840a6e8adb14b7777c78e26cfbf6e2abc1d1fc6): `docs/configuration.md`, `internal/config/config.go`, `pkg/osvscanner/filter.go` and `pkg/osvscanner/vulnerability_result.go`. The separate report uses an explicit empty native config and preserves frozen bytes.
- [GitHub required checks](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches#require-status-checks-before-merging), [job conditions](https://docs.github.com/en/actions/how-tos/write-workflows/choose-when-workflows-run/control-jobs-with-conditions) and [workflow permissions](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#jobsjob_idpermissions): distinct reporting names, failure-visible artifacts and job-local grants.
- Native issue upsert follows `native-agent-stack@36654a8:.github/workflows/saturation-tracking.yml:128`. The issue links alerts, assigns `seathatflowsinourveins` and names command-center review within 24 hours.

### Evidence-class table

| Claim | Evidence class | Evidence |
| --- | --- | --- |
| Native explicit config retains unsuppressed findings | `source_review` | Pinned OSV source above; config/docs re-read on 2026-10-08 |
| Frozen artifact and retired TOML remain byte-identical | `local_integration` | Retained SHA-256 read-back and prepared receipt |
| Failure visibility and trusted delivery guards | `synthetic` | Extracted workflow-shell fixtures |
| Contracts, workflow lint and registered bytes pass | `local_integration` | Latest outcomes: 246 pass, 1 existing skip across 247 distinct test IDs; actionlint 0, offline zizmor 0/[]; validate 0 |
| Hosted scans, SARIF store and issue read-back | pending | Native PR and trusted-main runs required |

### Local commands run

```sh
nice -n 19 ionice -c3 python3 -m unittest -v tests.test_frozen_evidence_reporting tests.test_workflow_policy
# exit 0: 42 tests, 2.117 seconds
nice -n 19 ionice -c3 actionlint .github/workflows/security-scan.yml .github/workflows/frozen-evidence-risk.yml
# exit 0
nice -n 19 ionice -c3 python3 scripts/validate.py
# exit 0: 69 components, 10560 hashed files, 4 profiles, 226 receipts
```

Passing evidence is retained from the unchanged prepared head. The latest per-test outcomes above combine the original run and focused repairs; they are not a single full-suite pass. Original failed attempts remain in the private receipt. Hosted acceptance remains pending.

### Decision record

`docs/decisions/2026-10-07-frozen-evidence-osv-reporting.md` records the accepted policy, changed contract IDs, alternatives and overturn conditions. Inverse after adoption: revert the source change and re-register its files, restoring the previous required frozen-scan policy.

### Host evidence

No host-receipt change.

### Checklist

- [x] Actions use full commit SHAs with version comments.
- [x] Top-level permissions are empty; write grants are job-local.
- [x] No secret values or raw finding/log text are copied into the issue.
- [x] No new paid service; other owners' work is preserved.
- [x] Prepared validation passes and the evidence registry is the final commit.
- [ ] Hosted acceptance and exact-head landing review are recorded.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:shared Touches files owned by both lanes; needs both lanes' acknowledgement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant