Repository navigation
Gate A U9: independent frozen-check grader for the #381 token E2E (oracles, blind cross-family judges, controls) - #521
Merged
seathatflowsinourveins merged 64 commits intoSep 29, 2026
Conversation
Stage 1a of the U9 frozen-check grader (#381 token E2E): the tests come first, so each guard and rule has a recorded failing value before tools/token-e2e/frozen_checks.py and grade.py exist (1b). Sources: the U9 build contract, sections a3, a4, b2 (R3-R5, R10, R12, R19, R20) and b5 (the stage-1 list P0, F1-F7, F9, F10, F18, F20, F21, F25a, F26, F32, F33); the binding corrections (item 2: the T0 re-run is the arm's own command in a temporary copy of the tree, because `python3 -I -S` cannot import the tests package; item 3: the T14 query time comes from the first tool_use naming the whole word agentsview, because the parser's closed program set returns null for it); TOON spec v4.1.1 section 2 (equality); docs/acceptance-evidence-policy.md "Discriminating controls". Each negative asserts its exact refusal code or rule id, never an exit status alone: the missing script also exits 2, so the expected E_* line is what fails. Also here, ahead of the b1 table, because 1b needs them: the stage-1 disarmed-guard mutants (the F19 subset M1, M2, M3, M6, M10, M18, M19, M21, M22, M23, M24, each asserting the exact set of tests it flips), the `keys` and `capture` command tests, and the schema and readings table tests. Fixtures are generated at test time outside every work tree; no UUID-shaped value or host path is committed. Run at this parent (only U8 has merged; no grader module exists): TMPDIR=<scratch> PYTHONDONTWRITEBYTECODE=1 python3 -B -m unittest -v tests.test_token_e2e_grader Python 3.13.15, toon 4.1.1 Ran 152 tests: FAILED (failures=13, errors=139) Two tests pass at the parent by design: P0 test_preflight (a precondition; it records that loadShellParser, shellParserStatus, commandInvocations, callLedger, m14State, validateIdentityTable and skill_usage.inside_git_work_tree are absent, so fixtures of the stated shape stand in) and P0 test_isolated_mode_cannot_import_the_tests_package (the control that motivates correction 2). Every other test fails, with these values (one line per test; "ModuleNotFoundError frozen_checks" is: ModuleNotFoundError: No module named 'frozen_checks'): F10_Binding.test_a_path_alone_is_insufficient ERROR ModuleNotFoundError frozen_checks F10_Binding.test_inventory_and_line_count ERROR ModuleNotFoundError frozen_checks F10_Binding.test_no_sentinel_record ERROR ModuleNotFoundError frozen_checks F10_Binding.test_own_sentinel_passes ERROR ModuleNotFoundError frozen_checks F10_Binding.test_sibling_value_in_the_child_message ERROR ModuleNotFoundError frozen_checks F10_Binding.test_sibling_value_in_the_parent_message ERROR ModuleNotFoundError frozen_checks F18_LiteralAnswers.test_file_urls_and_templates_are_literals ERROR ModuleNotFoundError frozen_checks F19_Mutants.test_M10_spec_regeneration_off ERROR ModuleNotFoundError grade F19_Mutants.test_M18_create_only_off ERROR ModuleNotFoundError frozen_checks F19_Mutants.test_M19_work_tree_refusal_off ERROR ModuleNotFoundError frozen_checks F19_Mutants.test_M1_order_insensitive_headings ERROR ModuleNotFoundError frozen_checks F19_Mutants.test_M21_file_url_expansion ERROR ModuleNotFoundError frozen_checks F19_Mutants.test_M22_clause_check_off ERROR ModuleNotFoundError frozen_checks F19_Mutants.test_M23_key_drift_on_bytes ERROR ModuleNotFoundError frozen_checks F19_Mutants.test_M24_environment_dependent_mismatch_as_fail ERROR ModuleNotFoundError frozen_checks F19_Mutants.test_M2_one_equals_true ERROR ModuleNotFoundError frozen_checks F19_Mutants.test_M3_exclusions_off ERROR ModuleNotFoundError frozen_checks F19_Mutants.test_M6_no_strict ERROR ModuleNotFoundError frozen_checks F1_Inventory.test_amendment_one_fixture_counts ERROR ModuleNotFoundError frozen_checks F1_Inventory.test_sealed_preregistration_counts ERROR ModuleNotFoundError frozen_checks F20_GradingBlock.test_a_bogus_reading_is_refused_by_field ERROR ModuleNotFoundError frozen_checks F20_GradingBlock.test_a_changed_check_text_is_refused_through_the_cli ERROR ModuleNotFoundError frozen_checks F20_GradingBlock.test_a_missing_grading_block FAIL refusal line: observed grade.py "can't open file" (exit 2), expected 'E_GRADING_BLOCK field=missing' F20_GradingBlock.test_a_preregistration_that_differs_from_the_newest_seal ERROR ModuleNotFoundError frozen_checks F20_GradingBlock.test_a_stale_registry_hash_is_refused ERROR ModuleNotFoundError frozen_checks F20_GradingBlock.test_an_unknown_block_key_is_refused_by_field ERROR ModuleNotFoundError frozen_checks F20_GradingBlock.test_missing_amendment_four_heading ERROR ModuleNotFoundError frozen_checks F20_GradingBlock.test_no_expected_hash_override_flag FAIL spec --help exit 2, expected 0 F20_GradingBlock.test_the_chosen_reading_is_used_and_the_alternatives_swap ERROR ModuleNotFoundError frozen_checks F20_GradingBlock.test_the_override_flag_is_rejected_with_a_refusal_code ERROR ModuleNotFoundError frozen_checks F20_GradingBlock.test_the_sealed_amendment_three_commit_has_no_block FAIL refusal line: observed grade.py "can't open file" (exit 2), expected 'E_GRADING_BLOCK field=missing' F20_GradingBlock.test_the_spec_writes_canonical_private_bytes ERROR ModuleNotFoundError frozen_checks F20b_SchemaAndReadings.test_a_fixture_block_validates_and_a_stale_field_names_its_path ERROR ModuleNotFoundError frozen_checks F20b_SchemaAndReadings.test_decided_first_then_the_ordered_alternatives ERROR ModuleNotFoundError frozen_checks F20b_SchemaAndReadings.test_the_committed_schema_equals_the_derived_one ERROR ModuleNotFoundError frozen_checks F20b_SchemaAndReadings.test_the_validator_refuses_a_keyword_it_cannot_check ERROR ModuleNotFoundError frozen_checks F21_SpecRegeneration.test_a_hand_edited_spec_is_refused ERROR ModuleNotFoundError frozen_checks F21_SpecRegeneration.test_the_unedited_spec_passes_the_regeneration_check ERROR ModuleNotFoundError frozen_checks F25a_MemoryKeys.test_an_empty_topic_blocks_launch ERROR ModuleNotFoundError frozen_checks F25a_MemoryKeys.test_only_records_before_since_with_the_anchors_are_frozen ERROR ModuleNotFoundError frozen_checks F25a_MemoryKeys.test_since_is_the_earliest_window_start ERROR ModuleNotFoundError frozen_checks F25a_MemoryKeys.test_the_keys_command_exits_2_with_the_task_and_writes_nothing ERROR ModuleNotFoundError frozen_checks F25a_MemoryKeys.test_topics_are_checked_in_sorted_order ERROR ModuleNotFoundError frozen_checks F26_Bind.test_a_missing_base_is_refused FAIL bind fixture: grade.py "can't open file" (exit 2), no bindings written F26_Bind.test_a_missing_root_value_is_refused FAIL bind fixture: grade.py "can't open file" (exit 2), no bindings written F26_Bind.test_a_sentinel_byte_change_is_refused FAIL bind fixture: grade.py "can't open file" (exit 2), no bindings written F26_Bind.test_a_symlink_output_counts_as_existing FAIL bind fixture: grade.py "can't open file" (exit 2), no bindings written F26_Bind.test_an_existing_output_is_never_overwritten FAIL bind fixture: grade.py "can't open file" (exit 2), no bindings written F26_Bind.test_an_input_hash_mismatch_is_refused FAIL bind fixture: grade.py "can't open file" (exit 2), no bindings written F26_Bind.test_an_output_inside_a_work_tree_is_refused_before_anything_is_created FAIL bind fixture: grade.py "can't open file" (exit 2), no bindings written F26_Bind.test_an_unknown_key_is_refused FAIL bind fixture: grade.py "can't open file" (exit 2), no bindings written F26_Bind.test_codex_bindings_are_merged_and_checked FAIL bind fixture: grade.py "can't open file" (exit 2), no bindings written F26_Bind.test_run_record_args_that_differ_from_the_bind_mark_the_task ERROR ModuleNotFoundError frozen_checks F26_Bind.test_the_record_is_private_and_holds_the_token_only_as_a_hash FAIL bind fixture: grade.py "can't open file" (exit 2), no bindings written F27a_KeysCommand.test_an_existing_output_is_refused ERROR ModuleNotFoundError frozen_checks F27a_KeysCommand.test_keys_are_private_deterministic_and_record_missing_inputs ERROR ModuleNotFoundError frozen_checks F27b_CaptureCommand.test_a_failed_download_is_recorded_not_hidden ERROR ModuleNotFoundError frozen_checks F27b_CaptureCommand.test_a_window_capture_needs_a_family ERROR ModuleNotFoundError frozen_checks F27b_CaptureCommand.test_post_arm_discards_a_changed_tree_and_lists_new_processes ERROR ModuleNotFoundError frozen_checks F27b_CaptureCommand.test_post_arm_needs_the_pre_arm_capture ERROR ModuleNotFoundError frozen_checks F27b_CaptureCommand.test_post_window_runs_the_acceptance_commands_in_an_exported_tree ERROR ModuleNotFoundError frozen_checks F27b_CaptureCommand.test_pre_arm_captures_six_identities_twice_and_inventories_the_exec_checkout ERROR ModuleNotFoundError frozen_checks F27b_CaptureCommand.test_the_run_token_must_match_the_bind ERROR ModuleNotFoundError frozen_checks F27b_CaptureCommand.test_window_pages_are_captured_with_curl_fail_silent_show_error_location ERROR ModuleNotFoundError frozen_checks F2_ExactTextRegistry.test_digit_change_is_unmapped ERROR ModuleNotFoundError frozen_checks F2_ExactTextRegistry.test_dropped_task_must_be_listed ERROR ModuleNotFoundError frozen_checks F2_ExactTextRegistry.test_registry_has_52_exact_keys_for_40_templates ERROR ModuleNotFoundError frozen_checks F2_ExactTextRegistry.test_task_declared_by_no_template_is_refused ERROR ModuleNotFoundError frozen_checks F31a_T14QueryTime.test_a_grep_for_the_word_is_not_a_query ERROR ModuleNotFoundError frozen_checks F31a_T14QueryTime.test_heredoc_query ERROR ModuleNotFoundError frozen_checks F31a_T14QueryTime.test_lookalike_words_do_not_count ERROR ModuleNotFoundError frozen_checks F31a_T14QueryTime.test_no_query ERROR ModuleNotFoundError frozen_checks F31a_T14QueryTime.test_query_inside_ctx_execute_code ERROR ModuleNotFoundError frozen_checks F32_TokenLines.test_the_decided_reading_and_its_alternative ERROR ModuleNotFoundError frozen_checks F32_TokenLines.test_the_real_handbook_counts ERROR ModuleNotFoundError frozen_checks F32_TokenLines.test_three_counts ERROR ModuleNotFoundError frozen_checks F33_PageFacts.test_a_changed_retention_fact_is_drift ERROR ModuleNotFoundError frozen_checks F33_PageFacts.test_a_missing_header_token_is_key_missing ERROR ModuleNotFoundError frozen_checks F33_PageFacts.test_capture_records_status_bytes_and_hash_from_curl ERROR ModuleNotFoundError frozen_checks F33_PageFacts.test_different_bytes_with_the_same_facts_do_not_drift ERROR ModuleNotFoundError frozen_checks F33_PageFacts.test_extractors_find_the_frozen_facts ERROR ModuleNotFoundError frozen_checks F3_Clauses.test_every_clause_and_literal_is_verbatim ERROR ModuleNotFoundError frozen_checks F3_Clauses.test_mutated_clause_is_refused ERROR ModuleNotFoundError frozen_checks F3_Clauses.test_mutated_recipe_literal_is_refused ERROR ModuleNotFoundError frozen_checks F3_Clauses.test_web_table_first_clauses_are_class_d ERROR ModuleNotFoundError frozen_checks F4_Keys.test_e1_qmd_wrong_document_fails_t4 ERROR ModuleNotFoundError frozen_checks F4_Keys.test_e1_repomix_answer_fails_t8_at_the_e1_keys ERROR ModuleNotFoundError frozen_checks F4_Keys.test_fixture_numbers ERROR ModuleNotFoundError frozen_checks F4_Keys.test_source_keys_at_the_pinned_revision ERROR ModuleNotFoundError frozen_checks F4_Keys.test_t1_heading_key_and_order_check ERROR ModuleNotFoundError frozen_checks F4_Keys.test_t3_definition_readings ERROR ModuleNotFoundError frozen_checks F4_Keys.test_t6_key_is_the_trust_segment ERROR ModuleNotFoundError frozen_checks F5_SymbolSites.test_attribute_call_site_is_excluded ERROR ModuleNotFoundError frozen_checks F5_SymbolSites.test_duplicate_site ERROR ModuleNotFoundError frozen_checks F5_SymbolSites.test_exact_sites_pass ERROR ModuleNotFoundError frozen_checks F5_SymbolSites.test_import_comment_and_string_sites_are_excluded ERROR subtest: ModuleNotFoundError frozen_checks F5_SymbolSites.test_import_comment_and_string_sites_are_excluded ERROR subtest: ModuleNotFoundError frozen_checks F5_SymbolSites.test_import_comment_and_string_sites_are_excluded ERROR subtest: ModuleNotFoundError frozen_checks F5_SymbolSites.test_missing_site ERROR ModuleNotFoundError frozen_checks F5_SymbolSites.test_synthetic_key ERROR ModuleNotFoundError frozen_checks F5_SymbolSites.test_t11_enclosing_function_readings ERROR ModuleNotFoundError frozen_checks F5_SymbolSites.test_t5_labelled_imports_and_mentions_are_not_invocations ERROR ModuleNotFoundError frozen_checks F5_SymbolSites.test_t5_precision_only_alternative ERROR ModuleNotFoundError frozen_checks F5_SymbolSites.test_unrecognised_sum_is_unparsed_not_failed ERROR ModuleNotFoundError frozen_checks F5_SymbolSites.test_wrong_events ERROR ModuleNotFoundError frozen_checks F5_SymbolSites.test_wrong_value_sum ERROR ModuleNotFoundError frozen_checks F6_StrictDecode.test_a_found_but_undecodable_candidate_is_a_failure ERROR ModuleNotFoundError frozen_checks F6_StrictDecode.test_an_evidence_string_is_a_candidate ERROR ModuleNotFoundError frozen_checks F6_StrictDecode.test_decode_argv_never_carries_no_strict ERROR ModuleNotFoundError frozen_checks F6_StrictDecode.test_every_candidate_must_equal_the_key ERROR ModuleNotFoundError frozen_checks F6_StrictDecode.test_json_root_array_passes ERROR ModuleNotFoundError frozen_checks F6_StrictDecode.test_missing_record ERROR ModuleNotFoundError frozen_checks F6_StrictDecode.test_no_candidate_is_unknown_unparsed ERROR ModuleNotFoundError frozen_checks F6_StrictDecode.test_seventeen_point_zero_equals_seventeen ERROR ModuleNotFoundError frozen_checks F6_StrictDecode.test_single_key_wrapper_is_a_shape_failure_under_the_decided_reading ERROR ModuleNotFoundError frozen_checks F6_StrictDecode.test_string_for_a_number_is_a_type_failure ERROR ModuleNotFoundError frozen_checks F6_StrictDecode.test_toon_root_array_passes ERROR ModuleNotFoundError frozen_checks F6_StrictDecode.test_true_for_one_is_a_type_failure ERROR ModuleNotFoundError frozen_checks F6_StrictDecode.test_unfenced_payload_is_a_candidate ERROR ModuleNotFoundError frozen_checks F6_StrictDecode.test_unordered_keys ERROR ModuleNotFoundError frozen_checks F6_StrictDecode.test_wrapper_with_the_sum_needs_the_extras_reading ERROR ModuleNotFoundError frozen_checks F6_StrictDecode.test_wrong_count_header_is_a_toon_strict_decode_failure ERROR ModuleNotFoundError frozen_checks F6_StrictDecode.test_wrong_latency_sum ERROR ModuleNotFoundError frozen_checks F7_T0.test_a_missing_capture_fails ERROR ModuleNotFoundError frozen_checks F7_T0.test_a_tree_changed_between_captures_discards_the_post_capture_and_never_reruns ERROR ModuleNotFoundError frozen_checks F7_T0.test_all_six_captures_and_the_answer_agree ERROR ModuleNotFoundError frozen_checks F7_T0.test_an_older_subject_first ERROR ModuleNotFoundError frozen_checks F7_T0.test_an_unverified_sandbox_makes_the_unittest_fields_environment_dependent ERROR ModuleNotFoundError frozen_checks F7_T0.test_environment_dependent_skip_count_is_unknown ERROR ModuleNotFoundError frozen_checks F7_T0.test_pre_and_post_disagreement_is_a_conflict ERROR ModuleNotFoundError frozen_checks F7_T0.test_test_count_mismatch_when_not_environment_dependent ERROR ModuleNotFoundError frozen_checks F7_T0.test_words_for_numbers_are_unparsed_until_d_extract_quotes_them ERROR ModuleNotFoundError frozen_checks F7b_T0Rerun.test_the_command_imports_the_tests_package_in_the_copy ERROR ModuleNotFoundError frozen_checks F9_Builder.test_bytes_differ_and_nothing_executes ERROR ModuleNotFoundError frozen_checks F9_Builder.test_empty_diff_at_the_prepared_path ERROR ModuleNotFoundError frozen_checks F9_Builder.test_extra_ignored_file ERROR ModuleNotFoundError frozen_checks F9_Builder.test_extra_untracked_file ERROR ModuleNotFoundError frozen_checks F9_Builder.test_mutated_fixture_hash_makes_the_key_unknown ERROR ModuleNotFoundError frozen_checks F9_Builder.test_observed_tree_other_than_the_prepared_path ERROR ModuleNotFoundError frozen_checks F9_Builder.test_only_before_py_changed_and_equal_to_after_passes ERROR ModuleNotFoundError frozen_checks F9_Builder.test_starting_revision_other_than_the_frozen_execution_revision ERROR ModuleNotFoundError frozen_checks G_TemplateSweep.test_t16_payload_and_citation ERROR ModuleNotFoundError frozen_checks G_TemplateSweep.test_t27_acceptance_summary ERROR ModuleNotFoundError frozen_checks G_TemplateSweep.test_t28_review_diff_counts ERROR ModuleNotFoundError frozen_checks G_TemplateSweep.test_t29_scout_inventory ERROR ModuleNotFoundError frozen_checks G_TemplateSweep.test_t2_recovery_digest_and_bytes ERROR ModuleNotFoundError frozen_checks G_TemplateSweep.test_t30_compile_exit ERROR ModuleNotFoundError frozen_checks G_TemplateSweep.test_t32_blind_verdict_id_and_latency ERROR ModuleNotFoundError frozen_checks G_TemplateSweep.test_t33_positive_control_events ERROR ModuleNotFoundError frozen_checks G_TemplateSweep.test_t34_main_output_count ERROR ModuleNotFoundError frozen_checks G_TemplateSweep.test_t35_agent_path_first_and_last ERROR ModuleNotFoundError frozen_checks G_TemplateSweep.test_t38_all_ten_structure_checks_are_read_from_the_answer ERROR ModuleNotFoundError frozen_checks P0_SiblingPreflight.test_isolated_mode_cannot_import_the_tests_package ok (precondition) P0_SiblingPreflight.test_preflight ok (precondition) Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
First implementation step of stage 1b, in tools/token-e2e/frozen_checks.py and grading-block.schema.json. What: the 52-key exact-text registry mapping to 40 templates (RV-23), each template's class set and its verbatim D clauses and recipe literals (E_CLAUSE), the a3 inventory computed from the preregistration bytes and stamped with their sha256 (E_CHECK_UNMAPPED, E_TEMPLATE_TASKS), the one readings table that feeds both the schema's enums and the spec's alternatives (R2-01 gains wrapper_with_extras for the recheck's R4 finding), a stdlib validator for the schema keyword subset that refuses any other keyword, the pinned sources (git cat-file batch reads, tracked-file directory reads), the linear text scanners of grammar g1, and create-only mode 0600 private writes refused inside a work tree. Sources: repair-u9.design.md a3, a4, b2 R3, R5, R20, R22; JSON Schema 2020-12 validation vocabulary (the keyword subset); the registry constants were generated from the sealed bytes at c0966da (d41152f4), not typed. Tests now passing (dev runner over the 857dc393 test file): F1_Inventory (2), F2_ExactTextRegistry (4), F3_Clauses (4), F20b_SchemaAndReadings (4). Not yet built: keys, oracles, captures, grade.py. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Both are defects in test data, not in an assertion's intent. 1. F5_SymbolSites built its synthetic repository once per subtest and per test in the same directory; the second `git commit` found nothing to commit and the test failed with "git commit failed". The key is now built once per test instance. 2. The synthetic enclosing function was named `caller`, which is also the word in the prose answer "Direct caller: ...", so the sites-only answer named the function by accident and the decided reading could not fail it. The function is now `use_registry`, a name no sentence uses. Verified: F5_SymbolSites 12 tests OK with the implementation of the next commit. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Second implementation step of stage 1b, in tools/token-e2e/frozen_checks.py. What: a key recipe for every graded template, computed from pinned Git content and recorded as unknown with a reason when a source is absent (never guessed); the T1 heading and three-way token-line counts, the T3 definition segment, T5/T11/T26 site analysis (Name calls, attribute calls, imports, comment and string mentions, enclosing functions), the T6 trust() segment, the T8 name sets with their distractors, the T9 frozen input, the T28 patch counts and the memory and catalog keys. Then strict payload decoding per TOON spec v4.1.1 section 2 (CLI 4.1.1 `--decode`, never `--no-strict`; JSON-type identity so a boolean never equals a number; ordered keys unless the reading says otherwise) with the correction-6 candidates (fences, contiguous line blocks that start like a TOON header or a JSON array or object, evidence strings, the whole answer; a prose `[1]: url` is not a payload), and the class A oracles for the templates that need only an answer and a key. Sources: repair-u9.design.md a4, R3, R4, R5, R13; TOON spec v4.1.1 SPEC.md section 2; corrections items 5 and 6. Scanners are linear character scanners, not regular expressions, so an answer cannot stall the grader; an answer is never dereferenced. Tests now passing: F4_Keys (7), F5_SymbolSites (12), F6_StrictDecode (17), F32_TokenLines (3), G_TemplateSweep (11), F18_LiteralAnswers (1): 51 tests, OK, in 22.9 s. Not yet built: T0 and the captures, page facts, memory keys, the T14 query time, the builder and binding checks, grade.py (spec, bind, keys, capture). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…emory Third implementation step of stage 1b, in tools/token-e2e/frozen_checks.py. What: the T0 oracle over six captures per arm, taken before and after the arm, under the arm's conditions and plain (R10, U9-D11): a wrong or older subject fails, a missing capture fails, a pre/post disagreement is unknown(capture_conflict), a field that differs between the two captures (or every field when the Codex sandbox could not be verified, recheck RV-06) is environment-dependent and an answer that disagrees is unknown(env_mismatch), and an unparsed count is unknown until a D-extract quote is verbatim in the answer. Correction 2: the unittest re-run is the arm's own command, `python3 -m unittest tests.test_host_requests`, with PYTHONDONTWRITEBYTECODE=1, in a temporary copy of the tree (isolate the tree, not the interpreter): `python3 -I -S` cannot import the tests package (P0's control shows it), and a post capture is discarded, never re-run, when the tree changed. Git calls set GIT_OPTIONAL_LOCKS=0 so observing a tree never writes its index. Also: the T31 builder check (identity, base, porcelain with ignored files, bytes before any execution, then greeting() in an isolated interpreter), the T36 binding check, correction 3 (the T14 query time is the first tool_use naming the whole word agentsview, in a command, in ctx_execute code or as the MCP server), page facts from html.parser text with key drift compared on facts, curl page captures, the R9 memory keys (records created before SINCE holding the anchors; E_MEMORY_KEY per topic in sorted order), qmd coverage, the Linux process listing and the exported-tree post-window runs (git archive, no write to the exec checkout). Sources: repair-u9.design.md R9, R10, R12, R13, a4; corrections 2 and 3; git-status(1) porcelain v1; ai-memory 2.4.1 and curl help read on this host. Tests now passing: F7_T0 (9), F7b_T0Rerun (1), F9_Builder (8), F10_Binding (6), F31a_T14QueryTime (5), F33_PageFacts (5), F25a_MemoryKeys function tests (3): 37 tests, OK. Not yet built: grade.py (spec, bind, keys, capture, grade's regeneration check). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Independent review of 857dc393 and 9496089a (a second loop of this agent, read from
committed bytes) found gaps; each is verified here by a test that fails first, or by a
disarmed-guard mutant where the guard already exists. Sources: that review (bytecode
caches, the copy, the plain value, R4 duplicate keys, the real `keys` path, the judge
effort rule relayed by the coordinator: the GPT-6 judge runs at max, never on codex_lane's
default `high`).
Failing at the current code (observed, host paths removed):
- F7_T0.test_bytecode_caches_are_not_a_tree_change: an ignored scripts/__pycache__ changes
tree_state (status_sha256 193149b9... instead of the empty-status e3b0c442...).
- F7_T0.test_environment_dependent_skip_count_is_unknown (changed): the plain capture's
value "skipped=0" gives ('pass', set()); expected ('unknown', {'env_mismatch'}). The
arm's own capture is the key; the plain one only marks which fields are
environment-dependent.
- F9_Builder.test_bytecode_caches_are_not_extra_changes: ('fail', {'extra_changes'}) for a
builder tree whose only extra is fixtures/__pycache__/; expected ('pass', set()).
- F7b_T0Rerun.test_a_planted_bytecode_cache_is_never_executed: an unchecked-hash .pyc
planted in tests/__pycache__ ran; facts {'ran': None, 'status': None, ...} instead of
{'ran': 2, 'status': 'OK', ...}.
- F2_ExactTextRegistry.test_a_dropped_task_that_is_still_in_the_preregistration_is_refused:
"expected refusal 'E_TEMPLATE_TASKS reason=dropped_present task=reuse-296-15', nothing
was refused".
- F20_GradingBlock.test_a_judge_that_is_not_pinned_to_max_effort_is_refused and
test_malformed_hex_fields_are_refused: no refusal line yet ('' != 'E_GRADING_BLOCK
field=judges.claude_answers.effort' and '... field=tool.revision'); grade.py is a
marker stub at this commit.
- F27c_KeysOnRealContent.test_keys_at_the_pinned_revision (new: `keys` over this
repository's content at the pinned revision, a local clone plus one fixture commit) and
F27b_CaptureCommand.test_post_arm_discards_a_changed_tree_and_lists_new_processes (now
starts a uniquely named process, so other sleepers on the host cannot flake it): both
end in FileNotFoundError for the missing grade.py outputs.
Passing already, and why it is honest to say so:
- F6_StrictDecode.test_duplicate_keys_and_non_finite_constants_are_refused and
F19_Mutants.test_M25_duplicate_key_refusal_off: the guard exists (16ba45d5); M25 (the
duplicate-key hook replaced by dict(pairs)) flips exactly that test.
- F6_StrictDecode.test_prose_that_looks_like_a_footnote_is_not_a_payload: the guard was
written before this test; it covers `[1]: url` and `[2] a note` next to a real payload.
- F7b_T0Rerun.test_the_run_happens_in_a_copy_not_in_the_tree and
F19_Mutants.test_M26_in_place_run: the copy exists (3c095eb5); M26 (the copy replaced by
a link back to the tree) flips exactly the marker test, while the other F7b test stays
green, which is why the old test alone did not prove the copy.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…view fixes) Fixes for the loop-2 review, each driven by the failing tests of 709f17e4. - Bytecode caches (review F-1 and section 6): a `__pycache__` component or a .pyc/.pyo file is neither part of the tree state nor an extra change, so a child that ran the unittest no longer makes every post capture `tree_changed` and a builder that imported before.py is no longer a false FAIL; every other ignored file still counts. The copy never holds bytecode caches, the re-run sets a fresh empty PYTHONPYCACHEPREFIX (a planted unchecked-hash .pyc runs even under `python3 -I -S`, and -I ignores the environment), and the T31 greeting is compiled from the verified retained bytes in an isolated `-I -S -B` interpreter instead of importing a file. This narrows the design's `--ignored=matching` and R13's `-I -S` on purpose; recorded as a deviation with the review's evidence. - T0 key (F-3): the arm's own capture is the key; the plain capture only says which fields are environment-dependent, so the plain value alone is unknown(env_mismatch). - Judge route (G-1, the coordinator's rule): the grading-block schema pins effort to max for both judges and freezes model, role and agent type as design f2 lists them; a block naming `high` is refused with its dotted field. - Dropped task still present (G-2): E_TEMPLATE_TASKS reason=dropped_present task=<id>. - Hex fields (G-5): tool.revision 40 hex, tool.sha256.* and registry_sha256 64 hex, refused by dotted field after the schema pass. - Git wrappers (G-4): timeouts, a return-code check and an E_GIT refusal for a malformed cat-file stream. normalize() documents that both sides of a comparison must pass through it (G-3). Tests now passing: F7_T0 (11), F7b_T0Rerun (3), F9_Builder (9), F2 (5), F20b (4) and F19 M26: 32 tests, OK. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Both are defects in test data, not in an assertion's intent. 1. F25a test_since_is_the_earliest_window_start built its fixture repository without recipes/host-request-lane.md, so the T7 key was unknown(key_missing) and its memory_records could not be read from the key. The fixture now carries the recipe. 2. F27b test_post_window_runs_the_acceptance_commands_in_an_exported_tree called make_bindings a second time into the same output path, which the create-only rule correctly refused (E_PATH reason=exists). make_bindings takes an out_name. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The command line of the grader, in tools/token-e2e/grade.py, on the kernel of frozen_checks.py. - spec: reads the preregistration blob at a 40-hex commit, verifies its sha256 against the newest "Amendment N seal" table of the README at that commit (no override flag: argparse refuses one as E_ARGS field=usage), requires `## Amendment 4` and the `grading` block, validates it (schema, hex fields, registry hash, pinned judge route), builds the inventory and the graded task array and writes canonical JSON create-only, mode 0600. - grade: regenerates the spec from its own commit and refuses a spec whose bytes differ (E_SPEC_MISMATCH). The grading itself is the next stage, so a matching spec ends in E_STAGE reason=grade_not_built (design deviation: `grade` takes --repo, which the b3 table omits and R20's regeneration needs). - bind: the private launch inputs (per-arm worktree paths and bases, retained inputs with their sha256, sentinels with their hashes, both windows, roots, anchors), refusing unknown keys, missing values and hash mismatches with E_BIND field=<f>; the run token is kept only as a sha256; U10's codex-bindings.json merges by task (a fixture-shaped consumer until U10 merges); roots gain exec_checkout (R13 inventories it; the design's bind contents omit it). - keys: every freeze key from pinned Git content; --memory freezes the historical records before the EARLIEST SINCE of either window, so a page written between the windows is not history for a Codex task (U9-D13); E_MEMORY_KEY refuses before anything is written. - capture: w-open/w-close curl the four frozen pages and record facts; pre-arm/post-arm take the six T0 identities twice per tree (the arm's conditions and plain), inventory the exec checkout with the porcelain options, discard a changed tree's post capture, list processes started since the pre-arm capture and key T11 clones; post-w runs the acceptance commands and py_compile in a git-archive export. RUN_TOKEN must match the bind's hash. Sources: repair-u9.design.md b3, R10, R12, R13, R19, R20, R22 and the corrections; ai-memory, qmd, codex and curl native help read on this host. Tests now passing: the complete tests.test_token_e2e_grader suite, 164 tests, OK, 0 skipped, in 82 s (Python 3.13.15, toon 4.1.1, git 2.43). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
… tools Two changes that keep the suite cheap and portable, found while checking how CI runs it (`python3 -m unittest` over everything, on Linux and macOS runners). - Speed: `keys` for all 75 tasks took about 20 s because every symbol re-walked all 240 scripts and tests. A file that never spells a symbol has no site for it, so only files that do are parsed, once per source object, with the token scan and the enclosing-name map built on demand. All 75 keys now take 1.9 s with identical values (the same four Name-call sites, 16 mentions and 20 subprocess.run calls); the whole suite went from 82 s to 24 s. - Portability: the design says CI may skip the TOON and private tests while the workstation acceptance run shows 0 skips. The tests now skip, not fail, when the TOON CLI or node is absent or Linux /proc is missing (the process listing), and P0 is split into python-interfaces, node-exports and toon-version tests so one missing tool no longer hides the others. Verified on a PATH without toon and node: 166 tests OK (skipped=11); on this workstation: 166 tests OK, 0 skipped. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
… page URLs
Two small hardenings found on a final read of the command line.
- bind: a launch-args file whose own `arm` differs from the ARM= label it is bound to is
refused (E_BIND field=arm); failing first: "'' != 'E_BIND field=arm'".
- capture_page: a URL that is not http or https is never handed to curl (it could parse as
an option); the record is {status: None, error: bad_url}. The guard was written before
its test, so the test passes on first run; it asserts curl was not run at all.
Tests: F26_Bind (11) and F27b_CaptureCommand (9) OK.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Second independent review round (loop 2, from committed bytes of 16ba45d5, 3c095eb5 and
7defe604): every finding below was reproduced by running the committed code, and each
is covered by a test that fails first. Sources: that review (REVIEW-1b notes H-1..H-5,
J-3..J-6, K-1, K-2) and the design's R3 ("FAIL means demonstrably wrong; an absent key
fact is unparsed") and F15/R21 (a deterministic FAIL sends no packet to the semantic judge).
The rule the tests pin (H-1): a single-valued fact passes only when every recognised value
equals the key; when the values disagree and one equals the key (a hedge such as "9 or 10",
"96 (or 66)", "yes or no", "exit code 1 or exit code 0") it is unknown(unparsed), so the
D-extract fallback may decide and the lower bound never rises; when none equals the key it
fails. An absent key fact (a paraphrase) is unknown(unparsed), not a FAIL, so the judge
still sees it; an absent required citation or literal still fails.
Failing at the current code (observed, host paths removed):
F20_GradingBlock.test_a_memory_query_that_could_parse_as_an_option_is_refused: '' != 'E_GRADING_BLOCK field=memory.reuse-296-07.query'
F20_GradingBlock.test_a_page_url_that_could_read_a_local_file_is_refused: '' != 'E_GRADING_BLOCK field=pages.json'
F26_Bind.test_codex_bindings_are_merged_and_checked: KeyError: 'arm'
F26_Bind.test_codex_trees_are_matched_by_arm: '/one' != '/two'
F26_Bind.test_run_record_args_with_another_token_or_arm_conflict_for_every_task: Lists differ: [] != ['reuse-296-00', 'reuse-296-02']
F27b_CaptureCommand.test_a_structurally_broken_spec_is_an_internal_refusal_not_a_traceback: 'Traceback (most recent call last):' != 'E_INTERNAL stage=capture'
F27b_CaptureCommand.test_curl_is_restricted_to_http_protocols: ValueError: '--proto' is not in list
F27b_CaptureCommand.test_post_arm_retains_the_builder_bytes_and_entries: KeyError: 'builders'
F27b_CaptureCommand.test_the_t0_commands_never_see_the_operator_environment: 'FAILED' != 'OK'
F6_StrictDecode.test_a_toon_cli_that_is_not_4_1_1_is_refused: 'toon_strict_decode' != 'E_TOOL tool=toon reason=version'
F7_T0.test_a_hedged_skip_count_is_unparsed: ('pass', set()) != ('unknown', {'unparsed'})
F7_T0.test_the_first_subject_mentioned_must_be_the_newest: ('pass', set()) != ('fail', {'subject_not_newest'})
H_OracleReviewFindings.test_a_catalog_answer_with_the_citation_but_paraphrased_facts_is_unparsed: ('fail', {'fact_missing', 'anti_pattern_missing'}) != ('unknown', {'unparsed'})
H_OracleReviewFindings.test_a_hedged_payload_sum: ('pass', set()) != ('unknown', {'unparsed'})
H_OracleReviewFindings.test_a_paraphrase_of_the_release_pin_reaches_the_judge_instead_of_failing: ('fail', {'fact_missing'}) != ('unknown', {'unparsed'})
H_OracleReviewFindings.test_an_absent_scope_name_is_unparsed_because_the_judge_settles_wording: ('fail', {'scopes_missing'}) != ('unknown', {'unparsed'})
H_OracleReviewFindings.test_t1_heading_count_is_not_satisfied_by_numbered_heading_lines: ('pass', set()) != ('fail', {'heading_count'})
H_OracleReviewFindings.test_t1_token_lines: ('pass', set()) != ('unknown', {'unparsed'})
H_OracleReviewFindings.test_t27_exit_code: ('pass', set()) != ('unknown', {'unparsed'})
H_OracleReviewFindings.test_t28_numbers: ('pass', set()) != ('unknown', {'unparsed'})
H_OracleReviewFindings.test_t32_verdict_and_latency: ('pass', set()) != ('unknown', {'unparsed'})
H_OracleReviewFindings.test_t34_count: ('pass', set()) != ('unknown', {'unparsed'})
H_OracleReviewFindings.test_t35_first_and_last: ('pass', set()) != ('unknown', {'unparsed'})
H_OracleReviewFindings.test_t8_a_stated_count_must_equal_the_number_of_names: ('pass', set()) != ('fail', {'count'})
Ran 191 tests: failures=21, errors=3 (all of the 24 above). Environment: Python 3.13.15,
toon 4.1.1.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Implements the fixes for the loop-2 findings that 73ed708e covers with failing tests
(H-1..H-5, J-3..J-6, K-1, K-2), in tools/token-e2e/frozen_checks.py and grade.py.
Correctness (R3): a single-valued fact now passes only when every recognised value equals
the key. A hedge that contains the key ("9 or 10", "96 (or 66)", "First 64 or 65",
"Verdict: yes or no", "exit code 1 or exit code 0", "Latency sum: 124 or 999") is
unknown(unparsed), so the D-extract fallback may decide and the lower bound never rises;
no equal value fails. Alternatives joined by "or", "/" or "vs" are recognised in either
direction inside one clause. T34 reads a bare integer, else integers beside a count word,
else every integer; T1's heading count reads only the plural label, so "Heading 10:" lines
no longer satisfy it; T8 checks a stated function count; T0 requires that the first subject
the answer names is the newest. An absent key fact (a paraphrase of the release pin, the
catalog facts, a scope name) is unknown(unparsed), not a FAIL, because a deterministic FAIL
sends no packet to the semantic judge; an absent required citation still fails.
Safety: the TOON CLI on PATH must be 4.1.1 before any decode (E_TOOL tool=toon
reason=version) and argv keeps the bare name; curl is restricted to http and https and a
page URL or memory query that could parse as a file scheme or an option is refused at spec
time; the six T0 identities, the unittest that runs the tree's code, the greeting run and
the post-window runs get an allowlisted environment (PATH, LANG, TMPDIR and a HOME that is
not the operator's), so no operator secret nor RUN_TOKEN reaches them; an unexpected
exception is one `E_INTERNAL stage=<command>` line with exit 2 instead of a traceback with a
host path and exit 1 (GRADE_DEBUG=1 adds the traceback after it); binding_conflicts
also compares the run token and the arm; the post-arm capture keeps each builder tree's
porcelain entries and before.py; Codex trees are matched by arm and an ambiguous task is
refused (E_CAPTURE field=codex_tree).
Sources: the loop-2 review notes; repair-u9.design.md R3, R19, R22, F15/R21; TOON spec
v4.1.1; the real ai-memory CLI accepts `--` before a positional query (checked).
Tests: the complete tests.test_token_e2e_grader suite, 194 tests, OK, 0 skipped; three new
disarmed-guard mutants (M27 a hedge that contains the key passes, M28 the version pin off,
M29 the operator environment reaches the commands) each flip exactly their tests.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Coverage the stage lacked, found while preparing the handoff. - F25c_QmdCoverage: `keys --qmd` was implemented but untested. A fake qmd 2.8.3 prints listing lines that end in qmd://<collection>/<file>; the coverage of adoption/update.md is read from them, an unreadable index is recorded as not queried (never guessed), and the keys command puts the record into the T4 key. The tests passed on first run because the code existed; M30 (coverage never found) flips exactly the two that assert a hit. - F27c: the spec over this repository's real sealed preregistration bytes (with a fixture Amendment 4 grading block, since the sealed bytes have none) is written twice and compared byte for byte. The pipeline's acceptance item 3 asks for `grade.py spec` twice with the sealed hash "as the expected value"; the design forbids any expected-hash flag (R20, F20) and the sealed bytes carry no grading block, so `spec` at c0966da refuses with E_GRADING_BLOCK field=missing (design acceptance 4); this test is what can be checked. Tests: 198 tests in the complete tests.test_token_e2e_grader suite, OK, 0 skipped. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Closes the last stage-1 gap found while mapping every review and recheck finding to a
disposition (recheck new defect 13: a tracked file that an earlier arm's child modified or
deleted in the shared exec checkout changes the same keys as a new file does).
The pre-arm exec-checkout inventory already lists untracked and modified entries. The test
now also deletes a tracked file and expects ("D", path) beside ("??", path) and ("M", path).
porcelain_entries runs `git status --porcelain=v1 -z --untracked-files=all --ignored=matching`
(design R13), so the code needed no change and the test passed on its first run: it is a
coverage addition, not a failing-first guard.
Stage 1 of the U9 frozen-check grader is complete at this commit (registry, spec, bind, keys,
capture and the pure oracles; corrections 2 and 3 applied): tests.test_token_e2e_grader ran
198 tests, OK, 0 skipped, at the parent. Stages 2 and 3 (evidence and grading; judges,
controls, export and docs) build on it.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Found in the final read of the capture conditions. The T0 identities (six commands, and the
unittest that runs the tree's tests in a copy) get an allowlisted environment with a
throwaway HOME (review J-5, K-3). That HOME also hides the operator's per-user Python
packages. tests.test_host_requests skips one test unless PyYAML is importable, so on a host
where PyYAML sits in the user site the child (real HOME) reports skipped=0 while both grader
captures (arm conditions and plain, same allowlist) report skipped=1. The skip count is then
not marked environment-dependent, and a correct answer fails T0. Correction 2 says the arm's
own environment; the allowlist must keep the user base visible and nothing else.
On this workstation PyYAML is not installed anywhere, so the two runs already agree here
(49 tests, OK, skipped=1 in both, checked in place versus the allowlisted capture); the
defect needs a host with a user-site package, which the tests build: a fake per-user base
holding one module (u9userpkg), a tree test that skips unless it imports.
Failing at the current code (observed):
F7b_T0Rerun.test_a_user_site_package_stays_visible_under_the_allowlisted_environment:
{'ran': 1, 'status': 'OK', 'skipped': 1, ...} != {'ran': 1, 'status': 'OK', 'skipped': 0, ...}
F27b_CaptureCommand.test_both_t0_conditions_keep_the_operators_user_site_packages:
the same facts through `capture --phase pre-arm`, arm and plain conditions.
Passing at this parent by design: F19_Mutants.test_M31_user_base_dropped_from_the_allowlist
(the mutant is the current allowlist, so it flips exactly the two tests above and leaves the
operator-environment control green).
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Fixes the defect the tests of 9e2453a5 fail on: the allowlisted environment of every grader
subprocess (minimal_env) replaced HOME with a throwaway directory, which hid the operator's
per-user Python packages from the T0 runs. tests.test_host_requests skips one test unless
PyYAML imports, so on a host with PyYAML in the user site the child's skip count differed from
both grader captures, and a correct answer failed T0 without being marked environment-dependent.
minimal_env now also sets PYTHONUSERBASE to the base the operator's own interpreter uses (the
variable when set, else site.getuserbase()). It is a package directory, not a secret; every
other operator variable, RUN_TOKEN included, stays out (F27b's operator-environment test and
mutant M29 still hold). Sources: correction 2 ("the arm's environment"), review J-5 and K-3
(the allowlist), Python site documentation for PYTHONUSERBASE and getuserbase().
Measured before the change on this workstation, where PyYAML is not installed anywhere:
tests.test_host_requests in place with the operator shell gives {ran 49, OK, skipped 1} and the
allowlisted capture gives the same facts, so the recorded T0 keys of this host do not change.
Tests: F7b_T0Rerun, F27b_CaptureCommand and the M29 and M31 mutants, 20 tests, OK; the two
tests of 9e2453a5 now pass and M31 (the old allowlist) flips exactly them.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Stage 2a, group 1. Design repair-u9 R1 (attempt classes), R2 (answer carriers), R6 (task, blind and control rules), R15 (M8), R16 (G-Q), R17 (G-C denominators), tests F11-F14 with the D1 matrix (26 cases, mutants MUT-A..E) and the binding corrections 1 (Agent-tool frame and trailer, observed on 33 of 33 foreground results of this host's client: a frame line, every body line indented by two spaces, an agentId line, a usage block) and 4 (a superseded run is classified by its own transcript's final row; a run that ended with an answer or a task-caused failure is a graded attempt). Cross-check outcomes are kept apart (advisor note on the runner log): the logged response equal, differing (carrier_mismatch) or unavailable (absent or unparseable, never a mismatch). Observed at the parent: 52 tests, 118 errors counting subtests, none passing. Each fails with AttributeError: module 'evidence' has no attribute '<name>' (workflow_attempts 26, decide_task 57, codex_exec_attempt 11, agent_child_attempt 7, follow_persisted 4, attempt_effect 2, blind_effect 2, g_c_denominators 2, strip_agent_result 1; g_q and m8_lane fail the same way). There is no tools/token-e2e/evidence.py at the parent, so the name resolves to the repository's evidence/ namespace directory and the error is an AttributeError instead of a ModuleNotFoundError. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Stage 2a, group 2. Design repair-u9 R7 (M12: five tests per blind attempt,
join through the U4 ledger row and the U2 run-mode child, strict transcripts
through the kernel bridge, the U4 m12_inputs cross-check, status), R8
(retrieval of each frozen URL by the child itself, in the family window) and
R9 (a historical hit counts only when it resolves to a record frozen before
SINCE); tests F22, F24, F25b. Each contamination case names its decided and
alternative reading (R2-11..R2-14, R2-18).
Sibling shapes are fixtures of the stated ones: U2 callLedger records
({session_id, owner, tool_use_id, tool, server, state, cause, native_status,
background}) and run-mode children (agent_id, label, lanes.measurement.
hook_context.inserted and by_hook), U4 join-ledger rows. measureTranscript is
the one kernel function that exists at this base, so the strict-transcript
test runs the real one through the bridge.
Observed at the parent: 39 tests, 39 errors, none passing, each
AttributeError: module 'evidence' has no attribute '<name>' (m12_attempt 15,
claude_calls 18, codex_calls 2, kernel_hook_context 1, m12_hook_source 1,
m12_cross_check 1, m12_status 1; retrieval_check and memory_check fail the
same way). The name resolves to the repository's evidence/ namespace
directory because tools/token-e2e/evidence.py does not exist yet.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Stage 2a, group 3. Design repair-u9 R14 (M7: seeded encodes, strict round trips, ineligible encodes, thresholds), R11 (T14 reported sessions, query time, owned survivors), R13 (tree drift), R2-17 (T2 recovery), T31 graded from the post-arm capture, and the binding corrections 5 (toon --stats prints the document on stdout and two status lines on stderr, both merged by Bash, ctx and Codex results; the lines are dropped by their documented shape, read from the installed CLI 4.1.1 source) and 6 (D-extract values join the payload candidates); tests F28, F31b, F34, F9b, F4b. Four defects of the stage-1 build are found here and each has its test: - T27 with a recorded acceptance run raises NameError (`summary` is not a name in oracle_T27; it must read key["summary"]). - capture files are named arm-<ARM>-<phase>, so the Claude and Codex arm B collide in one directory; the tests name them arm-<family>-<ARM>-<phase>. The seven existing F27b tests that read the old names are updated. - list_processes records no pid, so a survivor's descendants cannot be found. - the spec carries no thresholds; R14 reads /thresholds/M7/criteria and M8 needs the 0.8 rate and the minimum of five attempts, all from the sealed bytes. Observed at the parent: 63 tests, 1 failure and 62 errors, none passing. - 50 tests: AttributeError: module 'evidence' has no attribute '<name>' (apply_tree_drift 10, claude_calls 9, t14_check 9, isolate_toon_document 7, m7 7, toon_facts 4, grade_builder_capture 4, builder_observed_tree, command_programs, owned_survivors, recovery_check). - NameError: name 'summary' is not defined (oracle_T27). - KeyError: 'thresholds' (spec) and KeyError: 'pid' (list_processes). - The six renamed F27b tests: FileNotFoundError for arm-claude-B-*.json. - test_arm_captures_are_named_by_family...: AssertionError: 2 != 0 : E_PATH reason=exists (the second family's capture is refused). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Stage 2a, group 4. Design repair-u9 R19 (identity: rows from started labels, launch records and U10's Codex rows, guesswork refused by kind, arm-T rows accepted as reported-only, the validator's code surfaced as E_IDENTITY_INVALID), R22 (private side: create-only, work-tree refusal, the canary over keys, shapes and gathered values), F16 (grade twice then regrade give identical bytes and call no model), F17 (privacy), correction 9 (M11 for Codex role children: role_child_state with the four outputs of U13's reference specification, arm counts) and R2 codex_subagent (child rollout copy, the seed-binding-5 boundary), plus the stage-2 mutants of F19 (M4, M5, M14-M17, M18b, M19b, M20, M32 superseded-as-inadmissible, M33 trailer not removed, M34 whole merged result decoded, M36 drift ignored), each asserting the exact set of tests it flips. `grade` needs its inputs after the spec check, so the stage-1 expectation of E_STAGE reason=grade_not_built is replaced by E_ARGS field=bindings (test_the_grade_needs_its_inputs_after_the_spec_check); the two F21 tests keep passing because the regeneration check runs first. Observed at the parent: 56 tests, 39 failures and 26 errors (subtests counted), the two F21 tests passing. - CLI tests: AssertionError: 'E_ARGS field=usage' != 'E_IDENTITY_SOURCE kind=<main|agent|team|workflow|run_token|codex_rows>' and != 'E_IDENTITY_INVALID code=E_ROW'; AssertionError: 2 != 0 (the grade and regrade commands do not exist); 'E_STAGE reason=grade_not_built' != 'E_ARGS field=bindings'. - AttributeError: module 'evidence' has no attribute 'role_child_state' (8), 'codex_subagent_attempt' (3), 'assert_no_private', 'm11_summary', 'parse_role_toml'; the mutant tests fail on the missing attribute they patch. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
evidence.py part 1 and 2 and node_bridge.mjs. Design repair-u9 R1 (attempt classes from launch-level signals), R2 (workflow_child, codex_exec, agent_child and transcript carriers; the Agent tool_result frame and trailer removed structurally, correction 1; persisted output followed only under the declared roots), R6 (task, blind and control outcomes through attempt_effect and blind_effect), R15-R17 (M8, G-Q, G-C), R7 (M12: five tests, the join through the U4 ledger row and the U2 run-mode child, the m12_inputs cross-check), R8 (retrieval by the child itself, in the window), R9 (a hit resolves to a frozen record) and R2-17 (T2 recovery evidence). Superseded runs are classified by their own transcript's final row (correction 4): a rate limit or a login failure before any model turn is inadmissible, an answer or a task-caused failure is a completed attempt, a run that only stopped is an interrupted driver. The runner's logged response is a three-outcome cross-check (equal, mismatch, unavailable): an unparseable or missing log is never a mismatch. node_bridge.mjs imports the kernel as a namespace and reports a name that a sibling has not merged (U1 loadShellParser, shellParserStatus, commandInvocations; U2 callLedger, m14State; U4 validateIdentityTable) as unavailable; at this base only measureTranscript is real, and F22 runs the real one on a strict transcript. Also fixes one fixture defect of the group-1 tests found here: the wait-notice subtest built two worlds in one directory. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
evidence.py parts 3-6, grade.py identity, grade and regrade, and the stage-1 gaps found while building stage 2. Design repair-u9 R14 (M7: the seeded CLI encode, strict round trips and ineligible encodes; toon --stats merges the document on stdout with the two status lines on stderr, which are dropped by their documented shape before the strict decode, correction 5, and a D-extract payload joins the candidates only with verbatim quotes, correction 6), R11 (T14 reported sessions against the table rows that started before the first archive query, and an owned-survivor filter: started inside the child's lifetime and named like a program it ran, or a child of such a process; other in-lifetime processes are counted apart and never fail the child), R13 (tree drift: modified, deleted, untracked and ignored entries in a key scope turn a FAIL into unknown; T31 graded from the post-arm capture, running the key's own after.py bytes), R19 (identity: rows from journals, launch records and U10's Codex rows; a local twin of U4's R5 validation that accepts arm-T rows as reported-only; guesswork refused by kind), R22 (canary over keys, shapes and gathered values; create-only private writes), R16-R18 and correction 9 (role_child_state). grade collects one private record per run and evaluates them as a pure function; regrade reads only the private directory, so the aggregate and grades.jsonl are byte-identical and no model or launcher is ever called. Fixes to the stage-1 build, each with a failing-first test in the group-3 and group-4 commits: - oracle_T27 read an undefined `summary` when an acceptance run was recorded. - capture files are named arm-<family>-<ARM>-<phase>: the Claude and Codex arm B collided in one directory. - the spec carries the frozen thresholds; list_processes records the pid; Refusal takes its code positionally so a field may be named `code`. - keys.json is no longer key-sorted: canonical form lost the field order of the table records that R2-02 compares (found by the F16 end-to-end run). The positive control is measured only for its Read rows, never as blind evidence (its own hook row is what qualifies it). Two fixture defects of the earlier test commits are fixed here (subTest ids in failing_with, two rollout copies in one directory). Full module: 396 tests OK, 0 skipped, on the workstation. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Observed on this host's real Codex rollouts (120 rollouts, counts only): MCP calls are recorded as event_msg item_completed items of type McpToolCall with a server field, and never as function calls named mcp__<server>__<tool>; the turn_context payload carries model, effort, sandbox_policy and cwd as the M11 grader reads them. role_child_state read only the exec-events spelling (mcp_tool_call), so a child that called a server outside the parent's set passed the bindings check. Also adds two P0 checks: the bridge reports which kernel names exist at this base, and the real summarizeRun output has the run-mode child shape the grader consumes (agent_id, label, lanes.measurement.hook_context.inserted and by_hook). Observed at the parent: test_an_mcp_item_in_the_rollout_shape_is_read_too fails with (True, []) != (False, ['bindings']). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Design c1: the private row records descriptive fields beside the grade: the
payload bytes (T9, T16-T20), the main-transcript bytes (T34), the conversion
state and bytes with the scope terms it lost (T13) and the T14 reported and
expected counts (RV-30, ledger item). They decide nothing.
Observed at the parent: KeyError: 'recorded' (two tests: the mini-run rows have
no recorded object), and ('pass', {}) != ('pass', {'reported': 1, 'expected': 2})
for the T14 counts.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Makes the two failing-first commits before it pass. role_child_state reads an MCP call as an item_completed McpToolCall item (the spelling of this client's rollouts, observed on real ones) as well as the exec-events spelling. The private rows gain the descriptive recorded object of design c1: payload bytes (T9, T16-T20), main-transcript bytes (T34), the conversion call's tool, state and bytes with the scope terms its output lost (T13), and the T14 reported and expected counts, which t14_check now returns as the detail of its result. Real-data checks (aggregate scans on this host, counts only, not committed): - agent_child_attempt over 350 real Agent-tool calls: 36 foreground results all pass with the inline carrier (the frame and trailer removed structurally), the 274 asynchronous launches fail not_inline, 20 children were not found, 19 ended inadmissible (11 usage_limit, 8 interrupted_driver), 1 api_error. - workflow_attempts over 934 labels of 120 real Workflow runs: no exception (879 carry another workflow's response schema, hence schema_invalid). - m12_attempt over the 17 real blind transcripts: the attachment kinds before the first answer are exactly the frozen allowlist, the 15 Agent-tool ones fail the first-prompt and marker tests (the routing marker), the 2 Workflow ones pass. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Correction 9 (M11 for Codex role children) is wired end to end: the role file
at exec_rev, the parent's effective server set from the bindings and the
parent's rollout copy feed role_child_state, and the aggregate publishes the
per-arm counts with their unknowns. `bind` must keep the parent server set of
U10's bindings file (an assumed field: U10's codex-bindings.json is not merged).
Also covers the judgments path that stage 3 will feed: a judgment that holds
passes the D component, one that does not fails it, an unavailable one leaves it
unknown with its reason, and `regrade --judgments` equals a fresh grade with the
same judgments. Two more mutants: M37 (judgments ignored) and M38 (the role
child check never runs).
Observed at the parent: KeyError: 'parent_servers' (bind drops the field), and
the role-child counts come out {A: unknown 1, B: unknown 1} instead of {A: fail 1,
B: pass 1}. The judgments tests already pass (their guards exist); M37 and M38
give the condition-absent evidence once the wiring is in.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Makes the role-child wiring tests of the previous commit pass: `bind` keeps the parent_servers list of U10's bindings file (refusing a non-list) so role_child_state can compare a role child's MCP servers with the parent's effective set. The field name is an assumption until U10's file merges. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
scripts/validate.py scans tracked files for private-content shapes and failed on two literals of the F17 canary test: a UUID-shaped value and a Windows user path. The test still needs both shapes to prove the shape scans, so it now builds them from parts, and no committed line has either. validate.py exits 0 again (status passed, 8083 hashed files). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Source: the advisor's second review of stage 2, plus two checks of my own
against U10 (its design section 3.1 and codex_launch.copy_rollouts keep the
rollout copies in attempts/<identity>/rollouts/, not beside them) and the
retained exec streams (command_execution items are observed there, 4 of 4
completed, fields aggregated_output, command, exit_code, id, status, type).
The test fixtures move to the U10 layout; MiniRun.codex_exec now also writes the
thread id of U10's row, command items and an optional rollout copy.
Failing at 28d4604f, first differing value:
- role children: m11_summary KeyError 'not_applicable'; the rollouts under
rollouts/ are not found, so m11_roles is {} where the child count is expected
- M12: status keyed on the correctness of the answers gives ('fail',
'blind_task_without_clean_verdict') for five clean wrong verdicts; the sums
count rows 2 and hook_rows 1 for a superseded run beside its final run
- a call with no ledger record graded as a failure: memory ('fail',
no_historical_hit) not ('unknown', call_state_unknown); the same for recovery,
a toon encode ('not_encoded'), a child decode ([]) and a small encode (0, 0)
- M7: a Codex seeded attempt gives (0, 0, 0) not (1, 1, 0); a T9 answer payload
gives (0, 0, 0) not (0, 1, 1); the wrapper encode is ineligible under both
R2-01 readings; six natural payloads counted where five carry a payload
- T14: a count with no parsed identity fails count_mismatch not unknown; a Codex
attempt fails no_archive_query where its rollout copy times the query;
t14_check has no q_status or survival_observed, t14_facts and
codex_archive_query_time do not exist
- an exec mcp_tool_call with status completed and a non-null error: 'succeeded'
- identity: a dedicated main session that called the Agent tool once is refused
(E_IDENTITY_SOURCE kind=main)
- M8: (O, R) = (5, 0) where the join ledger gives (6, 2); regrade loses the ledger
(FileNotFoundError on join-claude.jsonl)
- bridge validate_identity: (True, False) not (True, None) for an exception the
validator did not code; validator_state does not exist
- a dotted mcporter call (ai-memory.memory_query) is not read as an ai-memory call
Passing at HEAD as controls: the retained exec item spellings, a Codex command
item encoding like a Claude call, a seeded attempt encoding only the seeded array.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Stage 3a of the U9 frozen-check grader (design repair-u9.design.md R21 and R22; tests F15, F19 stage-3 mutants, F23, F29, F30, F35; binding corrections 7 and 10). 81 tests are appended to tests/test_token_e2e_grader.py: F30 judge builders (strict-mode schema fixed point, Workflow syntax check, the blind-lane-reviewer role bound on every agent call, claude-args and collect), F23 web-table clauses and calibration files, F35 packets, scrubbing, routes, isolation, timing and privacy (with the exact argv element model_reasoning_effort=max of correction 10), F15 the judge contract (judge_quote, judge_leak, judge_calibration, judge_refuted, retry cap, usage-limit pause and resume, extraction), F29 export and check-html, the `controls` and `differential` commands, and 14 disarmed-guard mutants. The judge routes run against a scripted fake `codex` written by the test (the precedent is tests/test_codex_lane.py); no model is ever called. Observed at the parent 8766a39a (git stash of nothing; base suite 468 tests OK): 80 of the 81 fail, 1 passes by design. - 54 tests: ModuleNotFoundError: No module named 'judge'. - 5 tests: FileNotFoundError for tools/token-e2e/frozen-check-judge.js, judge-template.md and calibration/oracle-controls.json (the syntax check reports ENOENT from node); 1 test: 0 calibration files, expected >= 22. - 9 tests: 2 != 0 or 2 != 1 with E_ARGS field=usage from the missing subcommands export, check-html, controls, differential. - 13 refusal checks: 'E_ARGS field=usage' != 'E_EXPORT_DIR reason=exists', 'E_EXPORT_DIR reason=canary', 'E_PRIVACY' (10 including seven subtests) and 'E_DIFFERENTIAL field=inputs'. - grade context has no `command`: KeyError: 'command'. - Mutants: AttributeError for grade.export_dir_issue and grade.html_values; M17c and M51 fail on the expected-set comparison because their base test fails at the parent with or without the mutant. - Passes at the parent by design: F23 test_the_first_clause_of_every_web_table _check_is_class_d (the five clauses were registered as class D in stage 1). Helper changes: make_bindings gains `windows=`, MiniRun gains `windows=` and `files=` (defaults unchanged, the 468 stage-1 and stage-2 tests still pass). One lapse: three small expectation fixes made before this commit used a Bash Python replace instead of Edit; the content equals an Edit. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Stage 3a, second commit (design e8 and correction 7, repair-u9.design.md R21 and b5 F30, F35). Five tests for `grade.py judge rehearse`: two planted controls (one known pass, one known fail) through a route before Amendment 4 and before any window, with no bindings and no packets index. The codex route prints a receipt of counts, versions and booleans and exits 1 when the judge holds the known fail; a missing native credential is an isolation refusal before any call; an existing rehearsal directory is refused; the claude route prepares two Workflow items and later collects a clean run. The real routes need Codex capacity and a Workflow run, so acceptance e8 stays open: these tests run against the scripted fake `codex` only. Observed at the parent 8766a39a plus the previous test commit: all 5 fail, 'E_ARGS field=usage' for the missing subcommand (2 != 0, 2 != 1, and 'E_ARGS field=usage' != 'E_JUDGE_ISOLATION check=codex_home' or 'E_PATH reason=exists'). Expectation corrections found while building stage 3b, none of them loosening a guard (each was checked against the implementation before this commit; the tests that changed passed for a wrong reason or asserted the wrong function): - The fake codex ignored the quota probe only when `app-server` came first in argv; scripts/codex_quota.py runs `codex -c ... app-server`, so any position is now treated as the probe. - test_collect_refuses_a_result_that_consumed_another_prompt_or_packet_hash built two worlds in one test (the second commit of the fixture repo fails); it is now two tests. - M7 (quote verification off) no longer expects the two extraction-quote tests to flip: an extraction quote has a second guard, its mapping back to the raw answer text, so it stays refused with the first disarmed. - M40 (calibration off) no longer expects the extraction-control test; the extraction calibration has its own function, and M40b disarms it. - Judge test fixtures: JudgeWorld gains `embedded=` for a payload that no deterministic candidate detects. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The static files of the class D judges (design repair-u9.design.md R21 and
b1) and the frozen calibration data written before Amendment 4.
- judgment.schema.json: the judgment ({clauses[{id, holds, answer_quote,
source_quote}], extractions[{component, values, answer_quotes}], leak,
leak_text}), every field typed and every object closed with required equal
to its properties, so codex_lane.strict_output_schema leaves it unchanged
(checked: a fixed point) and Codex strict structured output accepts it.
- judge-template.md: the judge section and, after `<!-- refuter -->`, the
refuter section, each with one {PACKET_BLOCK}; the packet is inline for the
Codex route and a `Packet file:` line for the Claude route.
- frozen-check-judge.js: a saved Workflow in the pattern of
tools/sota-convergence/adjudication-lane.js. One blind-lane-reviewer judge
per packet at model opus and effort max, then one refuter per passing
judgment (passes_only); no evidence-reviewer (its carrier block would void
the judgment, correction 7). check-syntax.mjs prints SYNTAX_OK; the script
holds exactly two agent calls, each naming the role, model and effort.
- calibration/T*.json: 21 files (the 19 class D templates, and T0 and T9 for
extraction only) with clauses equal to the registry's, one reference-correct
control built from a source excerpt, one paraphrased-correct control and two
planted wrong controls per clause template, one that breaks each clause of a
two-clause template; the five web-table files hold the frozen contradictions
("ensure_ascii defaults to False", "allow_nan=false emits NaN", "sort_keys
sorts arrays", "raises TypeError", "loads accepts only str"); T0, T9 and the
web templates carry two extraction controls each. Excerpts marked verbatim
in the authoring are the pinned exec revision's own lines (trust(), the
opening hunk, before.py and after.py, the release-pin lines, the catalog
lines and anti-pattern cells).
- calibration/oracle-controls.json: 116 planted controls for acceptance e5
(79 class A across 30 templates, 20 class B for T31, T14, T0 and T27, 17 class
C for the answer carriers), each authored with the design's intent and
checked against the real oracle before it was frozen.
Fixtures carry placeholder ids only (tok7fixture, fx9); no UUID, home path or
tool-call id shape.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
judge.py (design R21; corrections 7 and 10; sources tools/sota-convergence codex_lane.py build_command, ISOLATION_ARGS, isolated_codex_home, child_env, blind_path_issue, blind_child_argv, blind_audit and transcript_audit.py audit, all reused unchanged, and adjudication-lane.js and adjudicate.py as the pattern for a saved blind Workflow with a collect step): - Packets: one per graded record whose class D clause is pending (not one that fails deterministically), plus unparsed payload and unittest facts when d_extract is on. A packet holds the sealed task text, the verbatim clause texts, extraction requests, deterministic source excerpts and the scrubbed answer; under 50,000 bytes; ids follow the content hash, so controls are mixed blind. A memory task carries only the frozen record digest and the anti-pattern row. - Scrubber: one linear left-to-right pass replacing exact run values, UUID and tool-call id shapes, absolute paths and the 81 denylist, model, family and arm names, keeping spans so an extraction quote maps back to the raw answer; the canary (values and id shapes) runs at packet time and again at dispatch. - Checks: every quote must be verbatim in the packet (unknown judge_quote), a leak is unknown judge_leak, a passing judgment gets one same-family refuter (a stood refutation is unknown judge_refuted), a command, MCP, web or file item is unknown judge_audit, at most one retry and only for infrastructure failure, a usage limit pauses the run (exit 75, resumable, the failed attempt retained). Calibration per template and route: any misjudged or unjudged control makes that template's judged clauses unknown judge_calibration; extraction controls gate the extractions only. - Codex route: gpt-6-astra at the literal max effort (JUDGE_EFFORT; the lane default is never referenced), ISOLATION_ARGS, a run-scoped home with an empty HOME, BLIND_CHILD_PATH, the strict schema, the packet inline and an empty per-item -C directory; E_JUDGE_ISOLATION for a home, PATH or launcher problem; the live quota is recorded, never a gate. - Claude route: claude-args (scriptPath, Workflow args, snapshot, the coordinator request that reads one small export file and runs one Grep first), E_JUDGE_ROLE unless the user-scope blind-lane-reviewer copy is byte-identical (checked: both copies exist and match), collect (snapshot, prompt, repo and item binding, transcript_audit, zero hook rows). - Windows: every dispatch refuses while either run window is open (E_WINDOW_OPEN window=W_C or W_X); rehearse runs before any window. - rehearse: two planted controls (one known pass, one known fail) through a route; a receipt of counts and booleans, the raw judgments and versions private. The real routes are not run here (no network): acceptance e8 open. evidence.py: id_shape (UUID, tool_use or call id with eight or more characters, home slug); judges.routes in the aggregate (calls, judgments, refuted, leaks, calibration_failed_templates, unavailable, usage_recorded, the last recorded outside G-C); the optional controls block; run_controls, the planted-control runner for acceptance e5 (class A by oracle, class B for T31, T14, T0 and T27, class C for the carrier functions). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The remaining grade.py subcommands (design b3, R21 and R22).
- judge {packets, codex, claude-args, collect, rehearse}: thin dispatch to
judge.py with grade.py's shared refusals (private outputs, spec
regeneration). `grade --judgments` and `regrade --judgments` take a file or
a judge directory holding judgments-*.jsonl; route summaries travel in the
judgments and publish under judges.routes.
- controls: planted class A, B and C controls from calibration/oracle-controls
.json, the two committed E1 answers graded at their pinned keys (the repomix
47-name list fails T8, the qmd wrong-document answer fails T4) and the
calibration files checked against the registry and the grading block's
minimums; counts only, exit 0 when every class landed where it was planted,
1 otherwise. `--out` keeps the counts and `grade --controls` publishes
them under `controls`; regrade repeats them from the private copy.
- differential: this grader's T0 oracle against the retained check.py of
rtk-296 and rtk-343 on the original baseline and three mutations (subject,
test count, result), each retained check run from a temporary copy with its
hard-coded BASE path pointed at the copy; counts only. Run here on the real
retained inputs: `rtk-296 agree 4/4; rtk-343 agree 4/4`.
- export: one returned_results record (schema_version 1 with the top-level
captured_at and scope the token-report importer requires) whose attachment
is the ID-free aggregate, in a fresh neutral directory; the canary runs over
the aggregate and the whole manifest, and the directory is refused
(E_EXPORT_DIR) inside a work tree, when it exists, or when its own path names
a run value, the home directory or the user name. The recorded command is
the grade or regrade command with a placeholder for every private path
(context.json now keeps it).
- check-html: the value canary (with --from a grade run's private directory,
the run's own values; always the home directory and user name), UUIDs,
tool_use and call ids and the home-slug shape. The '/' shape is not applied:
URLs and tags carry it. Without --from no run value is known (recheck low
finding on the canary source).
The fake codex in the tests answers `--version`.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The blind calibration controls must show the judge the same source sections its real packets carry, so a control that passes or fails says something about the real route. The controls of T5, T12, T13, T14, T16-T25 and T28 are regenerated through judge.source_sections (the section labels now read "... lines naming the facts"), and the T13 packet gains the recorded conversion facts (call state, converted bytes, scope terms missing from the conversion output) without the tool name. F29b gains a guard that the real report templates of tools/token-report (token_manifest.html.in, token_manifest.full.html.in) and returned_results.js pass the check-html canary with neutral values. It passes on first run by design: it pins a property of upstream files, not new code. Sources: tools/token-e2e/judge.py source_sections, tools/token-report templates at this base, the U9 design R7 (calibration) and c2. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Two design items stage 2 left unbuilt, each with its tests first.
F41 (R2-06): the decided reading grades the answer as Markdown (the ten
structure checks); the published alternative grades the content of the same
ten elements wherever they sit, read from the fixture at the exec revision.
F42 (M9): the optional block reports, per seeded optional task, the arm-B
attempts and whether each used the task's lane tool (repomix, markitdown).
U4's join lanes are the eight required tool lanes, so the lane is read from
the attempt's own calls and the call ledger's state.
Failing values at the parent (8 tests, 4 failures, 4 errors):
- F41 key test: KeyError: 'contents'
- F41 plain report: ('fail', {'elements_missing'}) != ('pass', set())
- F41 left-out element: missing lists all ten kinds, expected ['image']
- F41 whole word: missing lists all ten kinds, expected ['table']
- F41 key without contents: ('fail', {'elements_missing'}) != ('unknown', {'key_missing'})
- F41 published alternative: KeyError: 'T38' (ORACLE_READINGS has no entry)
- F42 both tests: KeyError: 'use'
Sources: preregistration M9 (report use and correctness, never gating),
scripts/native_token_ci.py markdown_elements (unchanged upstream), and
fixtures/markitdown-multi-element.html.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
R2-06: key_T38 now carries the content of the fixture's ten elements (html.parser over the exec-revision fixture, document order, script, style and comment text excluded), and oracle_T38 grades the published alternative element_contents by reading each content string as a whole word in the normalized answer. The decided reading (the ten markdown_elements checks) is unchanged. ORACLE_READINGS gains T38, so the alternative is evaluated and published, not a copy of the decided result. A fixture without one of the ten elements leaves the key unknown instead of making its check vacuous. M9: the optional block of the aggregate gains `use` per seeded optional task (lane, attempts, adopted, not_adopted, unknown), counted per arm-B identity from the lane tool the attempt's own calls used (repomix or markitdown as a whole command word, or an MCP server of that name) with the call ledger's state deciding success. U4's join lanes are the eight required tool lanes, so they carry no state for these two lanes. Reported beside correctness, never gating (sealed M9). The eight F41 and F42 tests pass (5 s). Sources: preregistration M9 and R2-06 of the U9 design (R5 register), scripts/native_token_ci.py markdown_elements (unchanged upstream). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The design's traceability table (c4) promises aggregate fields that no earlier test named: blind.verdict_parity, team_T.unmapped_teammates, attempts.superseded_runs, representation_sizes.payload_bytes, m13_organic and the optional block's use shape. F43 asserts each of them on a synthetic run: a workflow and strict verdict that agree, disagree and are absent; a teammate whose transcript is gone at grading time (tasks 1, unknown 1, unmapped 1, outside the G-Q population); a superseded usage-limit run beside its final attempt; the payload size of the web answer; the empty organic binding block; and one binding child without a sentinel record (unknown 1). These describe behaviour that was already built, so the six tests are guards and pass on their first run (7 s). The own and sibling counts of m13_organic are not driven end to end: the shared bindings fixture carries no sentinel record for seed-binding-1, and the T36 oracle itself is covered by F10_Binding. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
tools/token-e2e/README.md gains one section for grade.py (design f6): the command flow, the four grading classes and the attempt classes, the two blind judge routes and the reasons a judgment becomes unknown, the planted controls and the T0 differential, export and check-html, the privacy rules and exit codes, the limits (real judge routes not run, Amendment 4 pending, recorded residual on quoted hit text) and the obligation on the report producer: neutral origin and path values in the data handed to render_reports, then check-html --from on the result (recheck RV-14). The section holds no table row that starts with a backticked id, so U8's README catalogue test is unaffected. docs/decisions/2026-09-29-frozen-check-grader.md records the choice of standard-library Python with unittest over promptfoo 0.123.1 (llm-rubric, g-eval) and Inspect (model_graded_fact, model_graded_qa), names the rejected scorers with their evidence, the overturn conditions, the evidence classes and the calibration alignment (review RV-28). Sources read on 2026-09-29: npm view promptfoo (0.123.1, modified 2026-09-18), pip index inspect-ai (0.3.272), the promptfoo llm-rubric and g-eval documents, the Inspect scorers document, and the U9 design a5 and f6. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The README limits, the decision record and a test comment said the unit had no network. That was wrong: the unit made read-only fetches (npm view, pip index, the promptfoo and Inspect documents). The real reason is scope: its brief allowed read-only document fetches only, so no judge model call was made. Acceptance e8 (grade.py judge rehearse on both routes) needs real judge calls and stays open. Wording only; no behaviour changes. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
scripts/validate.py refuses a tracked file that contains a personal home path (the pattern exempts /home/example). The scrubber test used a synthetic path under another user name to check that home paths become <path> and map back to the raw text; it now uses /home/example/work/file.py. The scrubber treats every absolute path alike, so the expectations are unchanged and the two scrubber tests pass. Found by running validate.py for the manifest registration, the first time the U9 files were in scope of that check. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Last commit of the branch, per the hot-file protocol of docs/lanes.md: the
entries come from host_receipts.register_file, so files[] stays sorted and
only the new entries and the README's hash and size change (33 entries added,
tools/token-e2e/README.md updated because U8 registered it; +167 -2 lines).
Registered: the ten tools/token-e2e sources, schemas, template and README,
the 22 calibration files, tests/test_token_e2e_grader.py and, taking design
e7 ("the new files") at face value, the decision record.
python3 scripts/validate.py passes ({"status": "passed"}) and git diff
--check is clean. component_matrix.py --check and new_host_grand_list.py
--check also pass, so neither generated report needed a rewrite.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Two high and six medium findings of the independent review of head 705df138.
Each test below fails at that head for the stated reason; guard tests that
pass at the parent are marked (guard). Command: python3 -B -m unittest -v
tests.test_token_e2e_grader.<class>; observed values, host paths removed.
M7 answer payloads (high 1) and toon commands (medium 5), class F28_M7:
- unfenced TOON answer plus a "latency sum" line: answer items
['equal', 'unequal'] != ['equal']; a wrong or broken-header unfenced payload:
['unequal', 'unequal'] != ['unequal']; m7 round trip
(checked, equal, rate_lower, rate_upper) = (2, 1, 0.5, 0.5) != (1, 1, 1.0, 1.0).
- `toon --stats seed.json && toon -d out.toon` and the tee chain:
(cli_encodes, encode, ineligible) = (0, 'not_encoded', 0) != (1, 'encoded', 0).
A pipeline ending in `toon -d`: (0, 'not_encoded', None) != (1, 'unknown',
'unparsed'). `curl -d ...; toon --stats`: (0, 'not_encoded', None); `sort -o
...; toon --stats`: (1, 'unknown', 'output_file') != (1, 'encoded', None).
Fields named by their field name (high 2), classes H3_FieldNamedFacts and
H3_PositiveControlOracle (medium 3):
- T32 "id 1, latency_ms 17", "record_id: 1, latency_ms: 17", "id=1 latency_ms=17",
JSON keys: ('unknown', {'unparsed'}) != ('pass', set()); latency_ms 18:
unknown != fail(latency). Sums "latency_sum: 124", "Sum of latency_ms: 124",
"total_latency_ms = 124", "The total latency is 124 ms", "Sum of latency: 124":
('unknown', {'unparsed'}) != ('pass', set()); "latency_sum: 125" is unknown,
not fail(latency_sum). spaced_identifiers does not exist.
- T33 "all INFO (no ERROR among them)" and five more negations:
('fail', {'levels'}) != ('pass', set()); quoted raw rows in four shapes:
('fail', {'events'}) != ('pass', set()); raw rows with one ERROR level:
{'events', 'levels'} != {'levels'}; levels list before events list: unknown.
D-extract contract (medium 4), F7_T0 and F28_M7:
- number words through the production check_judgment:
('unknown', {'unparsed'}) != ('pass', set()); a stub whose values are not
inside its quote: ('pass', set()) != ('unknown', {'judge_quote'}), for the
unittest and the payload component; spelled_numbers_to_digits is missing
(9 subtests error). The stub of the old F7 test and the header-only quote of
the old payload test are replaced by extractions the production contract
accepts (guard).
Historical memory (medium 4), F25b_MemoryHits: a bare slug, a later page that
names recipes/<slug>.md, and three longer paths that contain the frozen path:
('pass', set()) != ('fail', {'no_historical_hit'}). The old test that expected
a bare slug to resolve is inverted.
Canary scope (medium 8), F17b_CanaryScope, F29b_CheckHtml, F35_JudgeRoutes:
'workflow_child' and the anchor line are in the canary list;
ev.IDENTIFIER_FIELDS is missing; check-html and `judge packets` exit
2 != 0 with E_PRIVACY for an actor name and for a quoted instruction line.
Argv sanitizer (medium 6), F29c_ArgvSanitizer: `--flag=value` is recorded
verbatim for all 17 path flags ('--spec=<path>'), a full grade run writes the
path into context.json, an abbreviated `--out-priv` is accepted ('' !=
'E_ARGS field=usage', exit 1) and export copies a recorded '--spec=/x' into
the manifest ('' != 'E_EXPORT_INPUT reason=argv', exit 0).
Guards that pass at the parent: an unfenced payload alone is one equal trip,
a fenced broken header is one unequal trip, the toon command's own -o stays
unknown, a heredoc decode after `cut -d`, the underscore stays a word
character, a count field is not a sum, hedged and wrong values still fail.
Sources: the review and recheck findings of the U9 stage-3 head; TOON CLI
4.1.1 strict decode (content after a root array is refused); check_judgment in
judge.py (the production contract of an extraction).
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
High finding: _answer_toon_items counted the failed strict decode of the whole-answer candidate as an unequal round trip. A correct unfenced TOON answer with the requested latency sum after it (TOON CLI 4.1.1 refuses content after a root array in strict mode) therefore had one equal and one unequal item, and M7 failed on both bounds. Now only a structural candidate (a fence or a line block) that starts like a TOON header and does not decode is a failed trip; a bare candidate (the whole answer, an evidence string) that does not decode adds no item, as frozen_checks.grade_payload already ignores it. Without a frozen original the same split applies (one unknown per payload, not one per candidate). Medium finding: toon_facts read `-d`, `--decode`, `-o` and `--output` from every word of a call, so `toon --stats seed.json && toon -d out.toon` counted as a decode only (no encode, no round trip) and `curl -d` or `sort -o` beside an encode changed its meaning. toon_invocations now reads each simple command of the call separately (the existing _shell_commands and _program_of scanners): a call may hold an encode and a decode; an option belongs to the command it follows; a heredoc feeds a round-trip item only when the decode command itself carries it. A script inside `bash -c '...'` still falls back to the words of the whole text. Sources: R14 of the U9 design (a seeded encode is a succeeded toon call whose result holds a document that strictly decodes to the seeded array; an unresolvable result is unknown), correction 5 and 6 of the build corrections, toon CLI 4.1.1 (strict decode, assertFullyConsumed). Tests: F28_M7 (9 new cases and the whole class: 40 tests, all pass except the payload-extraction case that belongs to the D-extract commit). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
High finding: word_positions keeps '_' a word character (T1's whole-word
reading, grep -w semantics, depends on it), so a label never matched inside a
field name. A correct blind answer "Verdict: yes. id 1, latency_ms 17" gave
latencies [] and unknown(unparsed), and a correct web-table answer
"latency_sum: 124" or "Sum of latency_ms: 124" left the sum unparsed after the
payload had decoded. The D-extract fallback cannot decide either case (it
covers T0 and the payload only), so an arm-B unknown failed G-Q clause 1 on the
lower bound.
The word rule is unchanged. The label sets now name the facts the way the
frozen key does:
- T32 reads `id` and `record_id`, and `latency`, `ms` and `latency_ms`; a label
that is a field name (it holds an underscore) takes its value after it only,
so `id=1 latency_ms=17` never reads the 1 as a latency.
- ints_by_label skips the closing quote of a quoted field name, so a JSON
answer `{"id": 1, "latency_ms": 17}` is read.
- The stated sum reads its line through spaced_identifiers (an underscore
between two letters reads as a space: latency_sum, total_latency_ms) and lets
the sum or total label be followed by the latency nouns and the linking words
("Sum of latency_ms: 124", "The total latency is 124 ms"). A count field
(total_records: 8) is still no sum; a hedge ("124 or 125") stays unparsed and
a wrong value still fails.
Sources: design R3 (recognised forms; an unparsed key fact is unknown, a
contradicting value is a failure), R4 (the requested latency sum), the frozen
key fields record_id and latency_ms of fixtures/table.json. Tests: the new
H3_FieldNamedFacts (7 cases) and the oracle classes G_TemplateSweep,
H_OracleReviewFindings, F6_StrictDecode, F32_TokenLines, F5, F9, F10 and F18
(81 tests, all pass).
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Medium finding: oracle_T33 failed correct answers to seed-blind-positive, which
G-Q clause 1 counts, and a FAIL counts on both bounds (R3: FAIL means
demonstrably wrong). "Events 1, 2, 3, 4, 5, all INFO (no ERROR among them)" gave
fail(levels) because any whole-word ERROR failed, and an answer that quoted the
raw rows `[{"event":1,"level":"INFO","value":1,...}, ...]` gave fail(events)
because every integer of the first bracket span was taken and the fixture rows
carry value == event.
- level_is_negated: a level word is the object of a negation when walking back
from it over separators (spaces, commas, slashes), filler words and other
level words reaches a negator (no, not, none, nothing, without, zero, never,
neither, nor, 0, n't). Any other word or punctuation stops the walk, so
"the third is not INFO, it is ERROR" and "none of them is WARN, but the fourth
is ERROR" are still levels that are there.
- The events are read from quoted raw rows (`"event": N`, either quote), else
from the first bracketed list that holds only integers, else from the lines
that name events. A levels list before the events list no longer hides the
events; a wrong event or level in raw rows still fails.
Both scanners are linear character scanners.
Sources: design a4 row T33 ("Events 1-5, all INFO"), R3; the sealed check text
of seed-blind-positive and the fixtures/events.jsonl row shape at c0966da.
Tests: H3_PositiveControlOracle (5 cases) and the oracle classes; F36_Controls
(planted controls) still lands every control at its expected status.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Medium finding: the F7 D-extract pass rested on a stub judgment that the production contract rejects. Its values ['49 tests', 'OK', 'skipped=1'] are not substrings of its quote 'forty-nine tests, OK, 1 skipped', which judge.check_judgment answers with judge_quote. With verbatim values the grader read the same words with the same grammar, and count_before_noun leaves hyphenated number words unparsed, so in production the T0 fallback could never change an outcome. Neither _apply_extraction nor _extracted_payloads checked that a value lies inside a quote, although check_judgment's docstring says the oracle verifies them again; a `grade --judgments` or `regrade --judgments` file could therefore inject a fact the answer never states. - verified_extraction: a D-extract item is usable only when every quote is verbatim in the raw answer and every value is a substring of one of its quotes; otherwise the component is unknown(judge_quote). Used by the unittest and the payload fallback. - spelled_numbers_to_digits: the extraction-only reader of number words up to 999 (forty-nine, twenty one, one hundred and five). Grammar g1 is unchanged, so an answer stated in words is still unknown(unparsed) until a judge quotes it verbatim, which is the design's F7 case; the quote is then read with this reader. The old F7 stub and the header-only quote of the payload test are replaced by extractions the production contract accepts (F7 now builds its extraction with judge.check_judgment); the injected variants of both are new tests. Sources: design R3 and R21 (an unparsed key fact goes to D-extract; the oracle verifies quotes again), F7 of the design, judge.check_judgment. Tests: F7_T0, F7b_T0Rerun, F28_M7, F15_JudgeContract and F23_WebTableJudges (81 tests, all pass). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Medium finding: resolves_to_record also matched a frozen record by its bare path stem, so a later page that only mentions the same slug counted as a historical hit. The retained ai-memory output shape uses title-slug paths (`decisions/<slug>.md`, title `<slug>`); with a frozen `decisions/host-request-lane.md`, a later session page whose snippet reads `... recipes/host-request-lane.md ...` matched on `/host-request-lane.` and memory_check returned ok. R9 and U9-D13 count a hit only when the result names a frozen record, precisely so that a later arm's page never counts (arms B, A and A0 share one store). The F25b fixture used the id-like stem `mem-fx-1`, which hid it. resolves_to_record now accepts the record's content digest or its whole path. The path is a unit: nothing that belongs to a name (a word character, '-', '.' or '/') may sit before it or after it, so `archive/<path>`, `old-<path>` and `<path>-old` are other pages. The bare stem no longer resolves. The freeze records a path and a digest per record because `ai-memory search --json` rows carry no id (retained receipt: rows hold path, title and snippet; memory_query hits hold id and path), so no id field is added here. The F25b test that expected a bare slug to resolve is inverted; a later page naming another file with the same base name, and three longer paths that contain the frozen path, fail with no_historical_hit, and the whole path and the digest still pass. Sources: design R9 and U9-D13; the retained receipt evidence/artifacts/gap-wave2-20260923/recovery-portability (ai-memory search and memory_query output shapes). Tests: F25b_MemoryHits, F25a_MemoryKeys, F4b_RecoveryEvidence (23 tests, all pass). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Medium finding: sanitize_argv replaced a path only when the token was exactly a PATH_FLAGS key followed by a separate value. argparse also accepts `--spec=/tmp/run/spec.json` and any unambiguous prefix of a long option (`--out-priv /p`), so a `grade --spec=/some/private/spec.json ...` run stored the path in context.json and `export` copied it into the argv of returned-results.json. The export canary missed it: has_private_shape flags only a string that starts with '/' or contains ' /', and a private path outside the home directory is not a gathered value. R22 and U9-D17 require the exported argv to hold only placeholders. - sanitize_argv reads both spellings (`--flag value` and `--flag=value`, with the `FAMILY=FILE` value keeping its family: `--join-ledger=claude=<join-ledger>`). - Parser sets allow_abbrev off for the parser and every sub-parser (they inherit the class), so an abbreviated flag is E_ARGS field=usage before anything is written and the recorded flag table is complete by construction. - export refuses (E_EXPORT_INPUT reason=argv, nothing created) a recorded command that is not a list of strings or whose elements after the script path hold a '/', a backslash, a home shorthand or a drive prefix: the fail-closed backstop for a context.json written by any other means. evidence.has_drive_prefix is the former private `_has_drive_prefix`, now shared. Sources: design R22 and U9-D17 (export argv holds only placeholders), argparse (allow_abbrev: "allows long options to be abbreviated if the abbreviation is unambiguous"). Tests: F29c_ArgvSanitizer (5 cases: both spellings for all 17 path flags, a full grade and export with `=` spellings, an abbreviation, the backstop); with F29_Export and F29b_CheckHtml 18 tests run and 17 pass, the remaining one being the canary-scope case of the next commit. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Medium finding: canary_values gathered strings that are not identifiers, so legitimate outputs were refused with E_PRIVACY. It added every string field of every identity row, which includes the actor vocabulary of 8 or more characters (workflow_child, strict_process, agent_child, codex_exec, codex_subagent, team_teammate), and the instruction-file anchor lines. R22's canary covers only the run token, identities, labels, ids, events file names, input paths and roots, the home directory and the user name. So `check-html --from <private dir>` refused a report that printed an actor name, and in `judge packets` one answer that quoted a line of AGENTS.md or CLAUDE.md refused the whole batch (no packets written, every D clause pending). - evidence.IDENTIFIER_FIELDS is the one closed list of identifier fields of an identity row (identity, the locators, ids and the label); canary_values reads only those (a path-like locator also by its last name, now including parent_events_file), plus the run token, the records' identity and agent id, the roots, the memory roots, the exec checkout, the arm and Codex paths, the sentinel values, the home directory and the user name. - judge.ID_KEYS is that same tuple, so a value the packet scrubber removes is a value the canary checks; the canary stays the second guard behind the scrubber. - The instruction anchors stay in the bindings for M12 test 3 and are no longer gathered. Sources: design R22 (aggregate canary values). Tests: F17b_CanaryScope (3 cases), F29b_CheckHtml (actor vocabulary passes, a planted run token still refuses), F35_JudgeRoutes (a quoted instruction line still yields packets), F17, F29 and F35 (37 tests, all pass; the canary-off mutants M17, M17c, M48 and M50 are untouched). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
U9-D19: every guard gets a condition-absent run. Each mutant patches one repair guard back to the behaviour the review found and asserts the exact set of failing tests; the guard tests beside it must stay green, so a mutant is minimal (11 mutants, all pass on their own expectation): - M52 a bare candidate that does not decode counts as an unequal trip (the stage-3 head's _answer_toon_items): 3 tests flip. - M53 toon options read from every word of the call (toon_invocations off): 3 tests flip; the toon command's own -o and a heredoc decode stay green. - M54 an extraction value outside its quote is accepted (quotes-only verified_extraction): the unittest and the payload injection tests flip. - M55 a frozen record matched by its bare stem (the head's resolves_to_record): the bare slug, the later page and the three longer paths flip. - M56 `--flag=value` recorded verbatim (the head's sanitize_argv): 3 flip; M56b abbreviations accepted (allow_abbrev on): 1 flips; M56c export backstop off (argv_issue): 1 flips. - M57 the canary gathers every row string and the anchors: the vocabulary test, check-html and `judge packets` flip. - M58 field names not read through their words (spaced_identifiers off), M59 a negated level word is a level, M60 number words left as words. Sources: design b5 F19 (one disarmed-guard mutant per guard) and U9-D19; the stage-1 to stage-3 mutant classes F19, F19b and F19c (same failing_with harness). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The README section of tools/token-e2e/README.md (the grader's own section only) now states what the repair round changed for a reader of the tool: - how a fact is read (labels, field names, a quoted JSON key, negated level words, quoted raw rows), and exactly where the D-extract fallback applies: the T0 test-run words (read with an extraction-only number-word reader) and the payload of T9 and the web-table tasks. The web-table sum and the blind tasks' facts have no fallback and stay unknown(unparsed); a quote is used only when it is verbatim in the answer and every value lies inside a quote, whichever path the judgment came from. - the canary scope (identifiers and paths, not actor vocabulary or instruction lines), the two flag spellings, refused abbreviations and the export backstop. - two limits: a historical memory hit is a whole path or a digest (real result shapes for the frozen records are unmeasured), and a payload extraction may be requested for an answer whose payload decoded and only whose sum is unparsed, which the grader ignores. The decision record gets a "Repair round" entry: the eight findings, the choices, and the withdrawal of the earlier claim that the D-extract fallback may decide an unparsed sum or blind-task fact (it covers neither). Extending it to them needs new extraction controls and is left to the Amendment 4 review. Sources: the review of head 705df138 and the design's R3 and R21. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Found by the independent check of the repair round before the manifest commit. R2-16 regression of 6f280d97 (F28_M7): with no frozen original, an undecodable structural block now added an unknown round-trip item whatever R2-16 says. The stage-3 head added such an item only under `unknown_in_denominator`, so the published "excluded" alternative counted payloads it must exclude. The existing R2-16 test used a document that decodes, which hid it. New test, one payload with no original in three shapes (unfenced broken block plus a sum line, the same block valid, a fenced broken block): decided reading ['unknown'], excluded reading []. Observed at this head: ['unknown'] != [] for the unfenced and the fenced broken block (the valid block passes). At 705df138 the unfenced case gave two unknown items for one payload. Scanner cost of the T33 repair commit (H3_PositiveControlOracle), whose message calls both scanners linear: - integer_lists searched again for `]` from every `[`: 6001 str.find calls on `[` * 3000 + "7, 8]" (bound 50). - level_is_negated walked back over every earlier level word (quadratic on `no ERROR ERROR ...`): with 9 words the walk still reaches the negator, observed True != False (a walk over more than eight words is no longer one clause). Sources: design R14 and R2-16 of the U9 register; the repair-round check. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
R2-16 regression of 6f280d97: _answer_toon_items decoded every candidate first, so with no frozen original an undecodable structural block added an unknown item even under the "excluded" reading. The records-None case is decided before any decode again, by the reading alone: unknown_in_denominator adds one unknown item per payload, anything else adds none. A bare candidate (the whole answer, an evidence string) counts there only when it decodes, so an unfenced payload followed by a sum line is one payload, not two (at 705df138 it gave two unknown items). A payload with an original is unchanged: a structural block that does not decode is one unequal trip, a bare one is no trip. Scanner cost of the T33 repair commit, whose message called both scanners linear: - integer_lists is one pass now: each `]` closes the innermost `[` still open, so a run of open brackets no longer searches again from each one (6001 find calls on 3000 open brackets before; the pairing is the same). - level_is_negated stops after eight words (_NEGATION_WALK): a negator further back is no longer in the same clause, and a walk no longer costs time in every earlier level word (quadratic on `no ERROR ERROR ...`). Eight words reach every negation of the tests, including `nothing at WARN or ERROR level`. The remaining quadratic scanners named in the review, code_spans and _or_links on very long lines, are unchanged and stay listed as open. Sources: design R14 and the R2-16 row of the U9 register. Tests: F28_M7, H3_PositiveControlOracle, H3_FieldNamedFacts and the mutant classes F19b and F19d (81 tests, all pass). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
U9-D19 for the guards of the previous commit, in F19d_RepairRoundMutants: - M52b decodes before asking R2-16 (the regression itself): the no-original round-trip test flips and the two unfenced-payload tests stay green. - M61 the quadratic integer_lists of the T33 repair commit: the one-pass test flips; M61b an unbounded negation walk (_NEGATION_WALK patched to 10**9): the clause-bound test flips. The held test in each is the list-reading test. 13 mutants in the class, all pass on their own expectation. Sources: design b5 F19 and U9-D19. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Last commit of the repair round, per the hot-file protocol of docs/lanes.md: the
seven registered files that changed since 705df138 get their new hash and size
through host_receipts.register_file (files[] stays sorted; 14 lines change, no
entry is added or removed): frozen_checks.py, evidence.py, grade.py, judge.py,
tools/token-e2e/README.md, tests/test_token_e2e_grader.py and the decision
record. The calibration files, both schemas, the judge template, the Workflow and
node_bridge.mjs are unchanged, so their entries stay.
python3 scripts/validate.py passes ({"status": "passed"}), git diff --check is
clean, and component_matrix.py --check and new_host_grand_list.py --check pass.
Integration: this branch now holds two manifest commits (705df138 and this one).
At rebase take main's manifests/evidence.json and re-register the U9 files again
(scripts/host_receipts.register_file), as the protocol says; never hand-merge it.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
seathatflowsinourveins
deleted the
claude/pra-u9-frozen-checks-2d-20260929
branch
September 29, 2026 23:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
grade.pywith spec, bind, keys, capture, identity, grade, regrade, judge, controls, differential, export and check-html, overfrozen_checks.py,evidence.pyandjudge.py, a Node bridge to the kernel, two schemas, a blind-judge template and Workflow, calibration controls and a unittest suite). Deterministic oracles grade what a file, a Git content or a transcript can decide; a blind judge from the other model family settles the semantic clauses. It changes no preregistration byte, no agent definition and no setting.3c9438d6(main after U8, U1, U13 and U1d)lane:foundation(Gate A unit U9; owner native-agent-stack-2d)tools/token-e2e/{frozen_checks,evidence,judge,grade}.py,tools/token-e2e/node_bridge.mjs,tools/token-e2e/frozen-check-judge.js,tools/token-e2e/{judge-template.md,judgment.schema.json,grading-block.schema.json},tools/token-e2e/calibration/(new),tests/test_token_e2e_grader.py(new),tests/fixtures/token_e2e_grader/(new),tools/token-e2e/README.md(one grader section),docs/decisions/2026-09-29-frozen-check-grader.md(new),manifests/evidence.json(hot file, last commit)SOTA sources
npm view promptfoo, 2026-09-29; llm-rubric, g-eval) and inspect-ai 0.3.272 (PyPI, scorers): compared as the grading framework and rejected, with the deciding criteria (byte-identical results, keys surviving in results, a blind cross-family judge with isolation, quote checks and audit) in the decision record's table.ast,tokenize,json,hashlib,unicodedata;git;scripts/native_token_ci.pymarkdown_elements; the packaged Codex lane (tools/sota-convergence/codex_lane.py,transcript_audit.py) and the saved-Workflow pattern ofadjudication-lane.js. No dependency is added.evidence/artifacts/token-adoption-e2e-20260926(Amendment 3 seald41152f4…; the R5 register, M9, the gates), read-only here. Claude Code 2.1.285 (read 2026-09-29).docs/decisions/2026-09-29-frozen-check-grader.md.Evidence-class table
local_integration(synthetic hosts, a scriptedcodex, recorded Workflow results, planted controls)python3 -B -m unittest tests.test_token_e2e_grader→Ran 624 tests in 278 s,OK(0 skipped)local_integrationpython3 -B -m unittest tests.test_token_e2e_preregistration tests.test_token_e2e_receipt_checks tests.test_skill_usage tests.test_token_measurement tests.test_freeze_snapshot→Ran 394 tests in 256 s,OKlocal_integration(private host inputs reduced to counts)grade.py controls --spec <fixture spec> --repo <checkout>;grade.py differential --inputs <retained private inputs>(builder handoff, at the pre-rebase head)specrefuses a preregistration without a grading block, as designed; with the sealed bytes (d41152f4…) plus a grading block it exits 0 twice withcmpequallocal_integrationgrade.py specatc0966da2→E_GRADING_BLOCK field=missing, exit 2 (expected before Amendment 4); the two-runcmpon a scratch repo (independent re-verify)source_reviewgpt-6-astrathrough the Codex lane, Opus throughfrozen-check-judge.js) return strict-schema judgments and pass their calibration controlsLocal commands run
Builder record before the rebase (pre-rebase head
fb2fa16c):Ran 624 tests in 342 s / OK, 0 skipped; the baseline of the reviewed head (705df138) ran 571 tests with 1 failure and 2 errors, alltoon --encode/gittimeouts at a host load average of 148 to 180 (environment, not code), and passed at lower load.Residuals (bounded review loop: one review, one repair round; each is in the README limits or the decision record):
specandkeysrun waits for Amendment 4.levels) on both bounds of G-Q clause 1. A follow-up unit (U9b) replaces the fixed filler walk by a clause-scoped negation rule with failing-first tests, and must land before Amendment 4 is sealed.sum(latency_ms) = 124) grades unknown and costs the G-Q lower bound. U9b either adds the extraction controls per template or Amendment 4 records it as unknown by design. The builder's claim that D-extract may decide these was withdrawn.NODE_FIXTURE_AT_BASEand theinside_git_work_treeallowance are removed in a consistency pass. U11's HTML is not built, socheck-htmlon it is unobserved.code_spansand_or_linkscan go quadratic on very long lines;toon_decodecatches onlyValueError; a nestedbash -cwrapper can hide an encode from the M7 call classifier andcat <<EOF | toon -dis not counted as a child-side round trip; a web-table answer whose payload decoded and whose sum is unparsed can request a dead extraction packet (one wasted judge call).grade.pyparser setsallow_abbrev=False, so a flag must be written in full (an abbreviation isE_ARGS).Decision record
docs/decisions/2026-09-29-frozen-check-grader.md(framework comparison with the rejected alternatives promptfoollm-rubric/g-evaland Inspectmodel_graded_*, the choices inside the grader and the overturn conditions).Host evidence
Not applicable (no files under
evidence/hosts/).Checklist
permissions: contents: read(no workflow changes)./home/examplepaths and no UUID-shaped values, private outputs are create-only mode 0600 outside every work tree, and the export canary refuses host paths.🤖 Generated with Claude Code