Skip to content

ci: separate frozen evidence OSV risk reporting - #845

Merged
seathatflowsinourveins merged 3 commits into
mainfrom
codex/ns2604-frozen-osv-report-20261007
Oct 8, 2026
Merged

seathatflowsinourveins merged 3 commits into
mainfrom
codex/ns2604-frozen-osv-report-20261007

Conversation

@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator

Scope

Frozen lockfile findings currently block the required production OSV check. Keep production scanning required and report inactive frozen evidence through a separate, non-required workflow, with an owner-assigned review issue from trusted main runs. Required checks still verify frozen artifact identity, inventory coverage and inactive eligibility.

  • Base: 36654a8174b5cad1380bd13ed1fffc0f135aac2c (preparation base).
  • Lane: lane:foundation.
  • Owned paths: OSV inventory/config, security and frozen-risk workflows, their contracts, the decision record and evidence registry.
  • Head: de41507bb1ea645d30c7978e111af847f05b8ede; registry-only commit last.
  • Assigned successor slot: CC item 001955Z, after ci: shard validate with native unittest coverage and a strict final check #838 (merged as 2cad4d7407e0497d4eaa51dc3c4410ef112cd928). Landing follows the named command-center base/cue.

SOTA sources

  • google/osv-scanner v2.6.0: docs/configuration.md, internal/config/config.go, pkg/osvscanner/filter.go and pkg/osvscanner/vulnerability_result.go. The separate report uses an explicit empty native config and preserves frozen bytes.
  • GitHub required checks, job conditions and workflow permissions: distinct reporting names, failure-visible artifacts and job-local grants.
  • Native issue upsert follows native-agent-stack@36654a8174b5cad1380bd13ed1fffc0f135aac2c:.github/workflows/saturation-tracking.yml:128. The issue links alerts, assigns seathatflowsinourveins and names command-center review within 24 hours.

Evidence-class table

Claim Evidence class Evidence
Native explicit config retains unsuppressed findings source_review Pinned OSV source above; config/docs re-read on 2026-10-08
Frozen artifact and retired TOML remain byte-identical local_integration Retained SHA-256 read-back and prepared receipt
Failure visibility and trusted delivery guards synthetic Extracted workflow-shell fixtures
Contracts, workflow lint and registered bytes pass local_integration Latest outcomes: 246 pass, 1 existing skip across 247 distinct test IDs; actionlint 0, offline zizmor 0/[]; validate 0
Hosted scans, SARIF store and issue read-back pending Native PR and trusted-main runs required

Local commands run

nice -n 19 ionice -c3 python3 -m unittest -v tests.test_frozen_evidence_reporting tests.test_workflow_policy
# exit 0: 42 tests, 2.117 seconds
nice -n 19 ionice -c3 actionlint .github/workflows/security-scan.yml .github/workflows/frozen-evidence-risk.yml
# exit 0
nice -n 19 ionice -c3 python3 scripts/validate.py
# exit 0: 69 components, 10560 hashed files, 4 profiles, 226 receipts

Passing evidence is retained from the unchanged prepared head. The latest per-test outcomes above combine the original run and focused repairs; they are not a single full-suite pass. Original failed attempts remain in the private receipt. Hosted acceptance remains pending.

Decision record

docs/decisions/2026-10-07-frozen-evidence-osv-reporting.md records the accepted policy, changed contract IDs, alternatives and overturn conditions. Inverse after adoption: revert the source change and re-register its files, restoring the previous required frozen-scan policy.

Host evidence

No host-receipt change.

Checklist

  • Actions use full commit SHAs with version comments.
  • Top-level permissions are empty; write grants are job-local.
  • No secret values or raw finding/log text are copied into the issue.
  • No new paid service; other owners' work is preserved.
  • Prepared validation passes and the evidence registry is the final commit.
  • Hosted acceptance and exact-head landing review are recorded.

@seathatflowsinourveins seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Oct 8, 2026
@seathatflowsinourveins
seathatflowsinourveins force-pushed the codex/ns2604-frozen-osv-report-20261007 branch from de41507 to d87f6c9 Compare October 8, 2026 07:11
@seathatflowsinourveins
seathatflowsinourveins marked this pull request as ready for review October 8, 2026 07:12
@socket-security

socket-security Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Dependency limit exceeded — report not shown.

This pull request scan exceeded the 10,000-dependency limit applied to this scan, so the results are incomplete and may be inaccurate. To avoid reporting false positives, Socket has not posted a report.

Upgrade your plan to raise the dependency limit and get complete reports, or view the partial scan in the dashboard.

Socket is always free for open source. If this is a non-commercial open source project, contact us to request a free Team account.

@seathatflowsinourveins
seathatflowsinourveins merged commit 3462d9b into main Oct 8, 2026
35 of 36 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the codex/ns2604-frozen-osv-report-20261007 branch October 8, 2026 10:04
seathatflowsinourveins added a commit that referenced this pull request Oct 8, 2026
### Scope

Carry the owner's directly approved fourth rule into every live instruction surface. Requests become a starting point for evidence-based related improvements, with timing that preserves the current focus and protected windows. The exact approved text replaces the previous fourth bullet across six instruction files and the maintained top-rule decision quotations.

- Base: `eb5fee4db9a494729deb29bdf38dbabac23ee643`.
- Head: `efd6b717f066f20c77fe604a37932c3adee6d8e4`.
- Lane: `lane:shared`; the shared core applies to both lanes, with the other lane's acknowledgement pending before any landing.
- Authority: owner-selected "Tightened" option, CC `task-ns2604-coop-20261008T134651Z`.
- Owned changes: six rule surfaces, the top-rule decision record, two pinned test modules and the evidence registry. The final commit contains both shared hot files, `AGENTS.md` and `manifests/evidence.json`.
- Exactly one PR; landing and the host F9/fresh-session steps remain on the command center's cue through 5f.

## SOTA sources

- [ECC search-first@`2b6e839771e53096d8451a213d40dc64ec8acac0`](https://github.com/affaan-m/ECC/blob/2b6e839771e53096d8451a213d40dc64ec8acac0/skills/search-first/SKILL.md), Workflow and Decision Matrix; reuse existing carriers, renderer and tests.
- [Claude Code best practices](https://code.claude.com/docs/en/best-practices#write-an-effective-claude-md) and [memory/rule organization](https://code.claude.com/docs/en/memory#organize-rules-with-clauderules), fetched October 8: broadly applicable rules load every session and stay concise. [Failure patterns](https://code.claude.com/docs/en/best-practices#avoid-common-failure-patterns) support preserving focus.
- [Official OpenAI documentation: Codex AGENTS.md loading](https://developers.openai.com/codex/guides/agents-md), fetched October 8; installed `codex-cli 0.161.0` and reviewed [openai/codex rust-v0.161.0 loader](https://github.com/openai/codex/blob/rust-v0.161.0/codex-rs/core/src/agents_md.rs).
- Existing supported integration at preparation base: `tools/adoption/new_wsl_client_config.py` F9, `scripts/host_receipts.py:register_file(root,relative_path)`, and the existing test helpers `template_segments()`/`startup_files()`.
- Historical evidence references: `docs/decisions/2026-10-07-instruction-core.md` and `docs/decisions/2026-09-25-model-fallback-guard.md:105–115,166–175`. The October 7 record cites the September StructuredOutput comparison; it is not a new October 7 run or an experiment on this bullet.

### Evidence-class table

| Claim | Evidence class | Observation |
| --- | --- | --- |
| Exact approved wording on all six live instruction files | `local_integration` | Native text/pin/budget tests; independent source/diff review |
| Full search census and untouched historical artifacts | `source_review` + `local_integration` | All 11,538 tracked files searched: 33 matching files / 93 lines; all 25 historical files byte-equal to the base |
| New core behavior matches owner direction | Owner policy decision | Exact tightened text and four dated owner excerpts in the amendment |
| Historical 5/30 vs0/30 schema errors | Historical reported native result | Sonnet 5 / Claude Code 2.1.282; primary untracked captures were not independently re-read here |
| New bullet improves adherence or is loaded in fresh sessions | Pending | No new model trial, host render or fresh-session check in this PR |

### Complete surface census

The maintained files are `AGENTS.md`, `adoption/scaffold/AGENTS.md`, `adoption/templates/codex.AGENTS.template.md`, `adoption/new-wsl/codex-user-instructions.md`, `adoption/new-wsl/claude-user-instructions.md`, `examples/claude-native/CLAUDE.md`, and `docs/decisions/2026-09-28-top-rule-templates.md`. Thus `surfaces=7`: six instruction files plus the decision record. The searched test literal is `tests/test_install_claude_profile.py`; `tests/test_codex_worker_lane.py` changes its hash/budget-related expectation.

The [amendment's full census](docs/decisions/2026-09-28-top-rule-templates.md#complete-search-census-and-frozen-files) names every matching file. These 25 historical evidence files were searched and are **untouched**:

- `evidence/artifacts/prompt-audit-20260927/lane-a/adjudication/attempt2/adjudication-inputs/x5.AB.json`
- `evidence/artifacts/prompt-audit-20260927/lane-a/adjudication/attempt2/adjudication-inputs/x5.BA.json`
- `evidence/artifacts/prompt-audit-20260927/lane-a/adjudication/attempt2/judge-actions.json`
- `evidence/artifacts/prompt-audit-20260927/lane-a/adjudication/attempt2/packets/x5.json`
- `evidence/artifacts/prompt-audit-20260927/lane-a/adjudication/attempt2/prompts/judge-AB.txt`
- `evidence/artifacts/prompt-audit-20260927/lane-a/adjudication/attempt2/prompts/judge-BA.txt`
- `evidence/artifacts/prompt-audit-20260927/lane-a/final/adjudication/adjudication-inputs/x5c.AB.json`
- `evidence/artifacts/prompt-audit-20260927/lane-a/final/adjudication/adjudication-inputs/x5c.BA.json`
- `evidence/artifacts/prompt-audit-20260927/lane-a/final/adjudication/amendment.diff`
- `evidence/artifacts/prompt-audit-20260927/lane-a/final/adjudication/gpt6.BA.json`
- `evidence/artifacts/prompt-audit-20260927/lane-a/final/adjudication/packets/x5c.json`
- `evidence/artifacts/prompt-audit-20260927/lane-a/final/adjudication/prompts/judge-AB.txt`
- `evidence/artifacts/prompt-audit-20260927/lane-a/final/adjudication/prompts/judge-BA.txt`
- `evidence/artifacts/prompt-audit-20260927/lane-a/final/claude.json`
- `evidence/artifacts/prompt-audit-20260927/lane-a/final/packets/packet.md`
- `evidence/artifacts/prompt-audit-20260927/lane-a/final/prompts/gpt6-prompt.txt`
- `evidence/artifacts/prompt-audit-20260927/lane-a/final/proposal.diff`
- `evidence/artifacts/prompt-audit-20260927/lane-a/final/tally-final.json`
- `evidence/artifacts/prompt-audit-20260927/lane-a/round1/claude-brief.txt`
- `evidence/artifacts/prompt-audit-20260927/lane-a/round1/claude.json`
- `evidence/artifacts/prompt-audit-20260927/lane-a/round1/gpt6-prompt.txt`
- `evidence/artifacts/prompt-audit-20260927/lane-a/round1/gpt6.json`
- `evidence/artifacts/prompt-audit-20260927/lane-a/round1/packet.md`
- `evidence/artifacts/prompt-audit-20260927/lane-a/round1/proposals.json`
- `evidence/artifacts/prompt-audit-20260927/lane-a/round1/user-level-top-rule.txt`

The three decision quotations are labelled amended projections; the original executed bytes, measurements and adjudications remain in those frozen files.

### Changed expectations, test by test

The [dated declaration](docs/decisions/2026-09-28-top-rule-templates.md#changed-expectations-test-by-test) records exact test IDs, old/new values and classification, following the CC's #845 rule.

| Test | Change |
| --- | --- |
| `StandingRuleSurfacesTests.test_the_portable_block_is_exactly_the_core_and_the_evidence_backed_line` | `CORE` takes the exact new bullet; 1,262 → 1,486 bytes |
| `StandingRuleSurfacesTests.test_every_layer_carries_the_same_core_and_no_dropped_rule` | Same exact core across all six layers; once-only/exclusion conditions retained |
| `StandingRuleSurfacesTests.test_the_repository_file_keeps_its_core_and_the_trading_prerequisite` | Same required core prefix; trading prerequisite retained |
| `TemplateTests.test_top_rule_is_pinned_and_rendered_rtk_is_the_unchanged_pinned_source` | `TOP_RULE_SHA256`:2618406a…c91fb62→bc9f3138…bd2aae7; RTK bytes/pin retained |
| `PortableTopRuleTests.test_rendered_startup_files_fit_each_clients_fixed_byte_budget` | Claude 4,140 → 4,610; Codex 4,246 → 4,716 bytes, same 105% procedure |
| `PortableTopRuleTests.test_growth_in_any_loaded_file_crosses_the_fixed_budget` | Same ceilings; negative growth condition retained |
| `TemplateTests.test_agents_template_is_one_managed_block` | Comment-only measured size 2,105 → 2,329; 8,192-byte ceiling retained |

No fixed word-count assertion exists in these native tests. Measured template words 286 → 323 and top-segment words 186 → 223 are recorded rather than inventing one.

### Local commands run

```sh
python3 -m unittest tests.test_install_claude_profile.StandingRuleSurfacesTests tests.test_codex_worker_lane.TemplateTests tests.test_install_claude_profile.PortableTopRuleTests
# unchanged main expectations after text edits: exit 1; 32 tests, 12 expected failing subtests

python3 -m unittest tests.test_install_claude_profile tests.test_codex_worker_lane tests.test_new_wsl_client_config tests.test_scaffold_repo tests.test_managed_block
# initial integrated run: 521 tests; 506 passed, 13 skipped, 2 failed
# private-log shell mistakenly gave fixtures umask 077; the two unchanged RoleStepTests expected 0750 but got 0700

umask 022
python3 -m unittest tests.test_codex_worker_lane.RoleStepTests.test_an_existing_agents_directory_and_identical_roles_are_not_the_runs_to_remove tests.test_codex_worker_lane.RoleStepTests.test_rollback_keeps_an_agents_directory_that_existed_before_the_run
# focused corrected-environment run: 2 tests, exit 0
# same two failures reproduced with base surfaces under 077; no assertion or implementation was weakened

python3 scripts/validate.py
# exit 0: 69 components, 10,831 hashed files, 4 profiles, 228 receipts
git diff --check
# exit 0
```

Final per-test outcomes are **508 pass / 13 skip across 521 distinct test IDs**, combining the initial run with the two focused environment corrections. This is not a claimed single full-suite green run. Original failed attempts remain retained. The corrected umask affects execution only; it introduces no repository/test exception. Command-center/5f main-version tests against this exact head remain separate.

### Decision record

`docs/decisions/2026-09-28-top-rule-templates.md`, dated 2026-10-08 amendment: owner approval, repeated October 4/5/8 words, exact text, evidence correction, complete census, expectation declarations, alternatives, inverse and overturn conditions. Completeness review corrected an unsupported CLI spelling of `register_file` to the actual Python helper.

### Host evidence

Pending after landing: CC runs F9 for the two user-level files, then fresh native sessions in both clients that visibly load the exact bullet. Optional promptfoo adherence A/B remains a separate api-credit-sota action. No host receipt was changed.

### Checklist

- [x] Exact owner wording and all maintained copies agree.
- [x] Historical evidence is untouched and the whole matching set is named.
- [x] Every changed expectation is declared; the shared hot files/registry are the final commit.
- [x] Native existing tests and required validation are retained with their actual outcomes.
- [x] Credential values and other owners' work are preserved; no new paid service.
- [ ] Main-version tests, other-lane acknowledgement and landing are on the command center's cue.
- [ ] F9 render and fresh-session load checks occur after landing.
seathatflowsinourveins added a commit that referenced this pull request Oct 8, 2026
<!-- Scope and base commit -->

### Scope

- When the frozen OSV scan completes with findings (native exit 1), its advisory check currently fails on unrelated PRs. This change reports those findings successfully while keeping scanner and report errors failing.
- Base commit: `89e89852161d8157e464f4bd07c2b5c747ad4706`
- Lane: `lane:foundation`
- Owned paths touched: frozen report workflow, its existing recording-double test, dated decision, measurement receipt and their evidence registrations.

### SOTA sources

- [google/osv-scanner v2.6.0](https://github.com/google/osv-scanner/blob/e840a6e8adb14b7777c78e26cfbf6e2abc1d1fc6/cmd/osv-scanner/internal/cmd/run.go): native findings return 1; operational failures return 127–130. The workflow preserves both raw codes and normalizes only completed-report findings.
- [GitHub shell and step semantics](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#jobsjob_idstepsrun) and [job conditions](https://docs.github.com/en/actions/how-tos/write-workflows/choose-when-workflows-run/control-jobs-with-conditions): explicit exit handling and unchanged always/trusted-event guards.
- [PR #845](#845), its dated frozen-report design and [native baseline run 37821865072](https://github.com/seathatflowsinourveins/native-agent-stack/actions/runs/37821865072): nine retained advisories, JSON/SARIF exits 1/1, no operational error.

### Evidence-class table

| Claim | Evidence class | Command / receipt |
| --- | --- | --- |
| Hosted before fails only on frozen findings | `native_proven` | Retained GitHub artifact digest `07cbf215…`, native run/job and input binding in the receipt |
| Finding-only report exits become 0; raw data/codes/logs stay identical | `synthetic` | Eight extracted-shell before/after cases; additional vendor 128/129 error cases |
| Coverage, error and privilege controls pass | `local_integration` | 254 focused tests (one existing skip); final expanded method separately passes; actionlint 1.17.0 |
| Inverse restores base bytes and forward patch restores candidate | `local_integration` | Native inverse round trip, both owned file digests verified |
| Hosted after | Unobserved | Await this exact PR's native report check and explicit CC workflow landing cue |

### Local commands run

```
python3 -m unittest tests.test_frozen_evidence_reporting tests.test_osv_lockfile_coverage tests.test_frozen_macos_variant_no_use tests.test_workflow_hardening tests.test_workflow_policy tests.test_workflow_security_coverage -q
# 254 tests, rc 0, one existing skip; before the final vendor-code test extension
python3 -m unittest tests.test_frozen_evidence_reporting.FrozenReportWorkflowTests.test_native_findings_errors_and_both_logs_are_retained -v
# final 128/129 extension: rc 0
actionlint .github/workflows/frozen-evidence-risk.yml
# rc 0
python3 scripts/validate.py
# rc 0: 70 components, 10878 hashed files, 4 profiles, 235 receipts
git apply --check <owned inverse patch>
# inverse and forward round trip passed; base/candidate file hashes matched
```

### Decision record

`docs/decisions/2026-10-08-frozen-risk-report-result.md` and
`evidence/artifacts/github-ci-opt6-report-status-20261008.json`.

Intentional kind 3 contract change: only
`FrozenReportWorkflowTests.test_native_findings_errors_and_both_logs_are_retained`.
Running all 13 unchanged base reporting methods on this candidate produces
exactly its two declared finding-only subtest failures, JSON1/SARIF0 and
JSON0/SARIF1; zero errors. Its original invocation/code/log assertions remain.

### Host evidence

No new host receipt or status flip. The actual hosted before artifact is retained
separately from synthetic after controls. The archived input, empty report config,
ordinary required scan, raw serialization, artifact retention and trusted write
guards keep their existing bytes/contracts.

### Checklist

- [x] Action pins and version comments preserved.
- [x] Empty default permissions and existing least-privilege job grants preserved.
- [x] No credential values, new required secret, provider or billing surface.
- [x] Peer work preserved; isolated owned worktree.
- [x] Inverse tested and receipt retained.
- [ ] Native hosted after recorded.
- [ ] Explicit CC workflow landing cue through 5f.

Revert this change through a reviewed inverse PR; findings stay visible in native
JSON/SARIF/logs and review alerts. Scanner errors and unavailable/malformed reports
remain failures. This PR remains draft while exact-head review and hosted checks finish.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant