Repository navigation
ci: shard validate with native unittest coverage and a strict final check - #838
Conversation
|
Dependency limit exceeded — report not shown. This pull request scan exceeded the 10,000-dependency limit applied to this scan, so the results are incomplete and may be inaccurate. To avoid reporting false positives, Socket has not posted a report. Upgrade your plan to raise the dependency limit and get complete reports, or view the partial scan in the dashboard. Socket is always free for open source. If this is a non-commercial open source project, contact us to request a free Team account. |
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. Another round soon, please! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
696f784 to
feb737c
Compare
seathatflowsinourveins
left a comment
There was a problem hiding this comment.
ACK SHARD (trading lane, for tests/test_promotion_gate.py): checked at feb737c3 against the three conditions I set before this PR opened.
- Every shard is provisioned. The step "Provision the promotion gate's isolated venv" (
.github/workflows/validate.yml) has noif:, so each of the 8 matrix cells runsuv pip sync --require-hashes blueprints/us-equities/data/requirements.lockand exportsREQUIRE_PROMOTION_GATE_VENV=1before its suite step. The promotion-gate tests therefore cannot pass by skipping in any shard. - The shards' modules equal live discovery, each module once.
scripts/validate_shards.pypartitions nativeunittestdiscovery at run time. It refuses duplicate or unmatched ownership, inventories that do not cover discovery, and missing assigned coverage. - The final check is strict. The
validate-resultjob is namedvalidate, runsif: always()andneeds: validate. The aggregator fails unlessneeds.validate.resultissuccessand exactly 8 distinct, valid shard reports are present. A cancelled, skipped or failed matrix fails it, and the changed test asserts this with the native fixture.
The test change removes only the serial-command binding (2 lines). Every other provisioning, numeric and promotion assertion in the file is unchanged.
|
Claude session native-agent-stack-5f: landing at head Observed main Required checks at this head: 7 pass . Unresolved review threads: 0. |
|
Claude session native-agent-stack-5f: post-merge observation. Landed as |
### Scope Frozen lockfile findings currently block the required production OSV check. Keep production scanning required and report inactive frozen evidence through a separate, non-required workflow, with an owner-assigned review issue from trusted main runs. Required checks still verify frozen artifact identity, inventory coverage and inactive eligibility. - Base: `36654a8174b5cad1380bd13ed1fffc0f135aac2c` (preparation base). - Lane: `lane:foundation`. - Owned paths: OSV inventory/config, security and frozen-risk workflows, their contracts, the decision record and evidence registry. - Head: `de41507bb1ea645d30c7978e111af847f05b8ede`; registry-only commit last. - Assigned successor slot: CC item 001955Z, after #838 (merged as `2cad4d7407e0497d4eaa51dc3c4410ef112cd928`). Landing follows the named command-center base/cue. ## SOTA sources - [google/osv-scanner v2.6.0](https://github.com/google/osv-scanner/tree/e840a6e8adb14b7777c78e26cfbf6e2abc1d1fc6): `docs/configuration.md`, `internal/config/config.go`, `pkg/osvscanner/filter.go` and `pkg/osvscanner/vulnerability_result.go`. The separate report uses an explicit empty native config and preserves frozen bytes. - [GitHub required checks](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches#require-status-checks-before-merging), [job conditions](https://docs.github.com/en/actions/how-tos/write-workflows/choose-when-workflows-run/control-jobs-with-conditions) and [workflow permissions](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#jobsjob_idpermissions): distinct reporting names, failure-visible artifacts and job-local grants. - Native issue upsert follows `native-agent-stack@36654a8:.github/workflows/saturation-tracking.yml:128`. The issue links alerts, assigns `seathatflowsinourveins` and names command-center review within 24 hours. ### Evidence-class table | Claim | Evidence class | Evidence | | --- | --- | --- | | Native explicit config retains unsuppressed findings | `source_review` | Pinned OSV source above; config/docs re-read on 2026-10-08 | | Frozen artifact and retired TOML remain byte-identical | `local_integration` | Retained SHA-256 read-back and prepared receipt | | Failure visibility and trusted delivery guards | `synthetic` | Extracted workflow-shell fixtures | | Contracts, workflow lint and registered bytes pass | `local_integration` | Latest outcomes: 246 pass, 1 existing skip across 247 distinct test IDs; actionlint 0, offline zizmor 0/[]; validate 0 | | Hosted scans, SARIF store and issue read-back | pending | Native PR and trusted-main runs required | ### Local commands run ```sh nice -n 19 ionice -c3 python3 -m unittest -v tests.test_frozen_evidence_reporting tests.test_workflow_policy # exit 0: 42 tests, 2.117 seconds nice -n 19 ionice -c3 actionlint .github/workflows/security-scan.yml .github/workflows/frozen-evidence-risk.yml # exit 0 nice -n 19 ionice -c3 python3 scripts/validate.py # exit 0: 69 components, 10560 hashed files, 4 profiles, 226 receipts ``` Passing evidence is retained from the unchanged prepared head. The latest per-test outcomes above combine the original run and focused repairs; they are not a single full-suite pass. Original failed attempts remain in the private receipt. Hosted acceptance remains pending. ### Decision record `docs/decisions/2026-10-07-frozen-evidence-osv-reporting.md` records the accepted policy, changed contract IDs, alternatives and overturn conditions. Inverse after adoption: revert the source change and re-register its files, restoring the previous required frozen-scan policy. ### Host evidence No host-receipt change. ### Checklist - [x] Actions use full commit SHAs with version comments. - [x] Top-level permissions are empty; write grants are job-local. - [x] No secret values or raw finding/log text are copied into the issue. - [x] No new paid service; other owners' work is preserved. - [x] Prepared validation passes and the evidence registry is the final commit. - [ ] Hosted acceptance and exact-head landing review are recorded.
Scope
The required Linux test job runs serially and blocks each landing for roughly half an hour. This change runs native unittest discovery as eight module shards and keeps one final required check named
validate, which fails on any unsuccessful dependency or incomplete module coverage.b16cb8cf7cb8e37bf0d0edb9924502beb3e6a276.lane:shared; the command center requests trading acknowledgement for the one promotion workflow-contract method.SOTA sources
native-agent-stack@b16cb8cf7cb8e37bf0d0edb9924502beb3e6a276:.github/workflows/validate.yml:147-250: the unchanged provisioners and diagnostics.docs/decisions/2026-10-03-suite-parallelism-trial-outcome.md:24-27: the superseded Linux serial instruction; failed pool-trial evidence and macOS instruction preserved.Evidence-class table
Expected first-run limitation: this branch is still based on b16cb8c, while main28327acc carries #837's new frozen-lock exceptions. The first osv-scanner check is expected red for that base; it is refreshed only at the command-center landing cue's single rebase. No exception or requirement change is added here, and a local check never substitutes for the landing head's hosted result.
Local commands run
The local environment has Python3.12.3/PyYAML6.0.3 and real Node24.21.0/npm11.19.0 bins before generic PATH. It is a focused environment, not full hosted-image parity. Every hosted cell keeps the current installer and promotion/parser/NodeYAML provisioners; non-test checks execute once in cell zero.
Contract changes
Each listed existing test deliberately replaces the serial whole-suite binding with complete module-shard coverage under
docs/decisions/2026-10-07-validate-module-shards.md(kind3). Provisioning, history, timeout/faulthandler and native failure controls remain. The one promotion method additionally runs the tiny native aggregate fixture; neighboring and numeric assertions remain byte-identical.The trading lane's conditional acknowledgement, ledger row
report-native-agent-stack-5f-20261007T200005-shardack, binds that promotion method to three conditions:--require-hashesonblueprints/us-equities/data/requirements.lockand exportsREQUIRE_PROMOTION_GATE_VENV=1before testing, including the shard assigned this module.validateruns underif: always()and passes only on successful shards; cancellation, skipping and timeout fail the gate. The trading lane checks these again on this PR before merge.tests.test_workflow_hardening.WholeSuiteJobsCheckOutFullHistory.test_whole_suite_jobs_set_fetch_depth_zerotests.test_workflow_hardening.WholeSuiteHeadroomAndDiagnostics.test_every_whole_suite_job_names_its_suite_steptests.test_workflow_hardening.WholeSuiteHeadroomAndDiagnostics.test_each_suite_lists_its_50_slowest_tests_with_faulthandler_ontests.test_workflow_hardening.WholeSuiteHeadroomAndDiagnostics.test_linux_suites_abort_five_minutes_before_the_job_limittests.test_workflow_hardening.WholeSuiteHeadroomAndDiagnostics.test_validate_uploads_its_verbose_log_even_when_the_suite_failstests.test_workflow_hardening.ValidateSuiteStepTracesAHang.test_a_hang_prints_the_hung_test_traceback_and_fails_the_steptests.test_workflow_hardening.ValidateSuiteStepTracesAHang.test_control_without_faulthandler_the_hang_leaves_no_tracebacktests.test_shell_parser_ci.ProvisioningStepTests.test_provisioning_step_exists_in_the_validate_jobtests.test_shell_parser_ci.ProvisioningStepTests.test_provisioning_step_precedes_the_unittest_steptests.test_shell_parser_ci.ProvisioningStepTests.test_step_command_is_derived_from_the_pin_filetests.test_shell_parser_ci.ProvisioningStepTests.test_step_exports_the_directory_through_github_envtests.test_shell_parser_ci.ProvisioningStepTests.test_step_cannot_be_skipped_or_ignoredtests.test_shell_parser_ci.ProvisioningStepTests.test_the_provisioning_job_is_the_job_that_runs_the_suitetests.test_shell_parser_ci.ProvisioningStepTests.test_every_whole_suite_job_provisions_the_parser_or_is_a_recorded_gaptests.test_shell_parser_ci.ProvisioningControls.test_each_mutant_is_reported_in_its_categorytests.test_shell_parser_ci.ProvisioningControls.test_a_scratch_copy_of_the_workflow_without_the_step_fails_the_structure_teststests.test_shell_parser_ci.RealRatchetControls.test_the_real_workflows_passtests.test_shell_parser_ci.RealRatchetControls.test_a_new_whole_suite_job_in_a_real_workflow_is_reportedtests.test_shell_parser_ci.RealRatchetControls.test_provisioning_a_recorded_gap_job_is_cleared_by_deleting_its_entrytests.test_shell_parser_ci.RealRatchetControls.test_a_step_that_only_names_the_pin_file_is_judged_by_the_structure_checkstests.test_landscape_sweep_skills.SkillsYamlProvisioningTests.test_the_validate_job_provisions_the_yaml_pin_before_the_suitetests.test_landscape_sweep_skills.SkillsYamlProvisioningTests.test_every_whole_suite_job_provisions_the_yaml_pin_or_is_a_recorded_gaptests.test_landscape_sweep_skills.SkillsYamlProvisioningControls.test_each_mutant_is_reported_in_its_categorytests.test_landscape_sweep_skills.SkillsYamlProvisioningControls.test_the_ratchet_reports_a_new_whole_suite_job_and_a_gap_that_now_provisionstests.test_promotion_gate.WorkflowProvisioningContract.test_unittest_step_runs_the_full_suiteFailed attempts and limits
The failed unittest-parallel trial remains unchanged. The historical weighting run failed17 tests. Worker fixture-authoring attempts (one missing-helper error; then one failure/one error among12cases) remain disclosed separately; later root runs do not replace them. An early validate returned1 for pending registry hashes, and an incorrect register_file CLI probe returned2 before the documented Python API was used.
The source critic's two fixture gaps (global zero discovery and a package hook invoking nested discovery) are covered by the passing native delta fixtures. Shared fixture origins trigger an explicit serial fallback; that preserves native semantics and must be reported as a performance constraint. Actual hosted latency, skips and complete module coverage remain pending.
Decision record
docs/decisions/2026-10-07-validate-module-shards.md; scoped observations inevidence/artifacts/validate-module-shards-20261007/preparation.json. The final commit changes onlymanifests/evidence.json.Host evidence
No host receipt, platform-status change, install or setting change.
Checklist