Record blinded skills validator trials and stale source rule by seathatflowsinourveins · Pull Request #918 · native-agent-stack/native-agent-stack

seathatflowsinourveins · 2026-10-09T06:11:18Z

Scope

Records one preregistered, blinded instructions/skills validation trial, isolated vendor install/check/inverse stages, and a default-branch maintenance recheck. agnix remains a trial, SkillEvaluator remains keep-but-compare, and openai/skills is stale-not-maintenance-qualified; neither candidate is ADOPT-NOW.

Base commit: 958a6d1ab770c807c5ee59748475a5c14df7ba52.
Reviewed rebased head: 6facffaa82f70782835eb7b3343739d13f02d523.
Lane: lane:foundation.
Owned paths: evidence/artifacts/skills-trials-20261009/, docs/decisions/2026-10-09-skills-blinded-trials.md, and manifests/evidence.json.
The tested September 30 catalog snapshot and existing regression-test literals match the merge base byte-for-byte. The October 9 recheck is a separate dated record.
The retained checks/landscape-skills-tests.log is explicitly labelled pre-restoration in checks/README.md. The fresh module run below validates the final rebased head.

SOTA sources

agent-sh/agnix v0.57.0 @ 2c0e4efede3181599eca37619e8067ae2d941c00: website/docs/installation.md, native tests/eval.yaml, .github/workflows/ci.yml, crates/agnix-cli/src/json.rs, and skills/agnix/SKILL.md. Vendor archive and installed binary hashes are retained.
NVIDIA/SkillEvaluator v0.5.0 @ 7304d76cde371287b67ea99653409d014b6b9c85: installation/developer/Tier1 guides, native schema and quality validators, reporting/finding-tag model, and three upstream test modules. The supported scoped uv install binds the exact Git revision.
Agent Skills specification: frozen independent format labels; missing local resource integrity is a separate frozen check.
openai/skills @ 49f948faa9258a0c61caceaf225e179651397431 and anthropics/skills @ 683bc88e56f3e09ba94f7055977f3d3aa499f202: unmodified reference quick validators are controls. Explicit REST default-branch commits establish the OpenAI maintenance result.
Repository manifest helper at main 958a6d1ab770c807c5ee59748475a5c14df7ba52: register_file regenerates registrations from actual file bytes; its supported invocation is documented in tools/sota-convergence/landscape-sweep/README.md:640–646. Regeneration preserves all 10,939 main registrations and the original 225 lane registrations, with one additional provenance README.

Evidence-class table

Claim	Evidence class	Command / receipt
Protocol, corpus and private-map hashes preceded execution; independent judge received only anonymous evidence	native_proven	`PREREGISTRATION.md`, `preregistration-lock.json`, `judge-execution.json`, redacted events and blinding audit
Matched static reports: agnix F1 1.000; NVIDIA schema plus quality 0.143; OpenAI/Anthropic controls 0.824/0.875	synthetic	Same 12 frozen fixtures; anonymous packet, native reports, 56 sealed labels, mapping reveal and `RESULT.json`
Pinned vendor installs, checks and verified inverses	native_proven	Retained `install-and-inverse.json` and vendor/upstream test logs
Historical unnamed session reaches the built-in validator; per-session PSS is unmeasured	native_proven	Retained `fresh-session.json`; original events privately retained with redaction mapping
openai/skills fails the 90-day default-branch maintenance criterion	source_review	Explicit `sha=main&since=2026-07-11T05:20:15Z&per_page=1` REST result in `openai-maintenance.json`
Landscape module, validator and pre-cue pass at the exact rebased head	local_integration	Exact-head logs and rebase publication receipt; commands below

Local commands run

Pinned vendor outcomes remain retained: agnix native eval 61/61 cases; NVIDIA targeted schema, quality and CLI modules 51, 144 and 20 passed respectively; skills-manifest module 33 passed. These outcomes were not rerun during the rebase.

At 6facffaa82f70782835eb7b3343739d13f02d523:

PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=. nice -n 10 ionice -c2 -n7 timeout 600 python3 -B -m unittest tests.test_landscape_sweep_skills
rc 0; 111 tests, 109 passed and 2 skipped; 22.483 seconds

nice -n 10 ionice -c2 -n7 timeout 600 python3 -B scripts/validate.py
rc 0; 70 components, 11,165 hashed files, 4 profiles, 239 receipts

nice -n 10 ionice -c2 -n7 timeout 600 bash command-center/cc-tools/base_tests_at_head.sh <repo> 958a6d1ab770c807c5ee59748475a5c14df7ba52 6facffaa82f70782835eb7b3343739d13f02d523 unittest
rc 0; merge_base=958a6d1ab7; no existing test file is modified

git diff --check; git diff --cached --check
rc 0

Fresh logs and exact commands are retained in <host-home>/.local/state/native-agent-stack/research/api-surfaces-trials/skills-trials-20261009/rebase-918-20261009T0911Z/rebase-publication.json. Independent bounded audit verified the manifest union, retained trial bytes, historical log and frozen catalog/test identity.

Decision record

docs/decisions/2026-10-09-skills-blinded-trials.md. This small synthetic diagnostic does not establish semantic quality, safety, live-evaluation superiority or a whole-repository replacement. agnix's separate instruction clean-control gate fails. Its routing-proof status is exactly unavailable: skill install is CC-gated; the historical unnamed-session observation remains separately retained. The CC must apply a reviewed adoption/skills/manifest.json row through tools/adoption/install_skills.py before owning the later routing proof. Per-session memory qualification is unmeasured and ADOPT-NOW remains false. Existing pinned skills remain while the general comparison is unresolved; catalog disposition follows its designated route.

Host evidence

No evidence/hosts/ receipt or platform-status change is included. Fresh exact-head review, hosted required CI and the CC/5f cue continue to gate landing; live skill/hook/client configuration remains owner-gated.

Checklist

No GitHub Actions or workflow permissions changed.
No credential values were read, printed or committed; public host/session identifiers retain their redaction mappings.
No new paid hosting, subscription or billing surface introduced.
Peer-owned files and worktrees preserved.
Exact rebased-head landscape module, validator and designated pre-cue pass.
Full hosted required CI passes at the rebased head.
Co-op micro on the rebase delta completes.
CC/5f issues the landing cue.
Owner decides any live skill/hook/client configuration apply.

chatgpt-codex-connector · 2026-10-09T16:39:10Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-09T16:45:13.644768Z	`0c21ff1`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0c21ff1acf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-10-09T16:45:10Z

+    prereg = Path(__file__).resolve().with_name("PREREGISTRATION.md")
+    if digest(prereg) != PREREG_SHA:
+        parser.error("preregistration differs from frozen bytes")
+    corpus = args.corpus.resolve()
+    case_file = corpus / "cases.json"


Enforce the locked corpus hash before execution

When any fixture or reference changes after the preregistration is locked, this runner still proceeds because it verifies only PREREGISTRATION.md and merely records the cases.json digest; it never compares the full corpus against preregistration-lock.json's corpus_sha256. A replay can therefore score non-frozen inputs while presenting them as the preregistered trial, so compute the documented directory hash and fail before execution on a mismatch.

AGENTS.md reference: AGENTS.md:L7-L7

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-10-09T16:45:10Z

+            write_json(out / "native-results.json", {"preregistration_sha256": PREREG_SHA,
+                       "cases_sha256": digest(case_file), "tool_files": {
+                           name: {"path": str(p), "sha256": digest(p)} for name, p in tools.items()},
+                       "python": sys.version, "records": results})
+            write_json(out / "finding-views.json", {"requires_parent_anonymization": True, "records": views})


Emit the retained native-results schema from the runner

Every execution of this code writes native-results.json without artifact_hash_domain, although the committed artifact begins with that field and publication-redaction.json says the file changed only through home-prefix redaction. Consequently the retained artifact cannot be reproduced by the committed controller and contains an undisclosed metadata edit; either emit the field here or explicitly record the post-processing transformation.

AGENTS.md reference: AGENTS.md:L7-L7

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-10-09T16:45:10Z

+  },
+  "method_limits": [
+    "One synthetic corpus, one fresh host/model session and one independent judge; no confidence interval or general ranking",
+    "NVIDIA quality template warnings and duplicate native structured/legacy findings were retained. The preregistered assertion-count F1 penalizes these; it does not establish its semantic quality, safety or Tier2/Tier3 performance",


Stop calling the duplicate-entry penalty preregistered

The preregistration defines false positives as unsupported assertions, but it never freezes a rule that separately emitted structured and legacy copies count multiple times; that per-finding-entry rule first appears in the sealed judge's post-outcome metric_notes. Arm-2 is then penalized for duplicated schema/legacy representations, so describing this as the “preregistered assertion-count F1” overstates the protocol and makes the reported comparison depend on a post-outcome scoring choice; mark it as an unpreregistered sensitivity analysis or rescore unique semantic assertions.

AGENTS.md reference: AGENTS.md:L7-L7

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-10-09T16:45:10Z

+The fresh `tests.test_landscape_sweep_skills` run at the final rebased head,
+validator result and designated pre-cue result are retained with their exact
+commands, head and log hashes in the separate rebase publication receipt:
+`<host-home>/.local/state/native-agent-stack/research/api-surfaces-trials/skills-trials-20261009/rebase-918-20261009T0911Z/rebase-publication.json`.


Commit the claimed exact-head validation receipt

The only in-tree landscape-test log is explicitly declared not to validate the rebased head, while the replacement evidence is referenced solely through a <host-home>/.local/state/... path outside the repository. A reviewer, CI job, or later checkout therefore cannot inspect the exact commands, tested head, or log hashes supporting the claimed final-head validation; commit a sanitized receipt and its bound logs or stop describing them as retained evidence.

AGENTS.md reference: AGENTS.md:L7-L7

Useful? React with 👍 / 👎.

seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Oct 9, 2026

seathatflowsinourveins force-pushed the codex/ns2604-skills-trials-20261009 branch from 0a41c4f to 6facffa Compare October 9, 2026 09:19

seathatflowsinourveins added 2 commits October 9, 2026 12:26

Record blinded skills validator trials and stale source rule

b7b12c8

Preserve frozen maintenance evidence and CC-gated routing

0c21ff1

seathatflowsinourveins force-pushed the codex/ns2604-skills-trials-20261009 branch from 6facffa to 0c21ff1 Compare October 9, 2026 16:29

seathatflowsinourveins marked this pull request as ready for review October 9, 2026 16:38

seathatflowsinourveins merged commit aeb6ea5 into main Oct 9, 2026
36 checks passed

seathatflowsinourveins deleted the codex/ns2604-skills-trials-20261009 branch October 9, 2026 16:39

chatgpt-codex-connector Bot reviewed Oct 9, 2026

View reviewed changes

seathatflowsinourveins mentioned this pull request Oct 9, 2026

Add fleet and Adoption views with native workstation reads #899

Merged

4 tasks done

Navigation Menu

Repository navigation

Record blinded skills validator trials and stale source rule - #918

Record blinded skills validator trials and stale source rule#918
seathatflowsinourveins merged 2 commits into
mainfrom
codex/ns2604-skills-trials-20261009

seathatflowsinourveins commented Oct 9, 2026 •

edited

Loading

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Oct 9, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Oct 9, 2026

Uh oh!

chatgpt-codex-connector Bot Oct 9, 2026

Uh oh!

chatgpt-codex-connector Bot Oct 9, 2026

Uh oh!

chatgpt-codex-connector Bot Oct 9, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

seathatflowsinourveins commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Scope

SOTA sources

Evidence-class table

Local commands run

Decision record

Host evidence

Checklist

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

seathatflowsinourveins commented Oct 9, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 9, 2026 •

edited

Loading