Skip to content

kernel residual check: scan section payloads and symbols, never executable bytes - #572

Closed
luisleo526 wants to merge 41 commits into
mainfrom
scripts/kernel-residual-section-scan
Closed

luisleo526 wants to merge 41 commits into
mainfrom
scripts/kernel-residual-section-scan

Conversation

@luisleo526

Copy link
Copy Markdown
Collaborator

Kernel residual check: exclude executable bytes and retain unknown data

The checker fed the entire debug-stripped kernel archive to strings -a,
allowing instruction bytes to look like forbidden names. It now scans section
payloads and retains the independent nm -C symbol surface.

Duplicate archive member names raise InfrastructureError naming the member,
before stripping and before extracting the stripped members. ar p <name>
cannot uniquely select a later member with the same name.

The section policy scans by default: skip executable flags, sections with no
file bytes, and explicit debug/linker metadata deny-lists; scan every other
section, including unfamiliar names. Text-like names without executable flags
are scanned. ELF exclusions use metadata types and named DWARF/debug sections;
Mach-O exclusions use instruction/debug attributes and the __DWARF and
__LINKEDIT metadata segments. No object/member exception is added.

Hand-built byte fixtures are authored for duplicate members, unknown data
that must be flagged, metadata that must be ignored, and data/symbol retention
paired with code exclusion. Additional rows exercise 32-bit and 64-bit
ELF/Mach-O in little- and big-endian layouts, fat/fat64 records in both byte
orders with data in the second architecture, and ELF extended counts/name
indexes separately and together. These are synthetic fixtures, not compiler
output or a claim of verified platform compatibility.

The malformed-member row requests named infrastructure errors for unsupported
magic/ELF class/byte order, truncated headers and tables, invalid Mach-O load
commands/segment widths, bad ELF names, and out-of-bounds payloads/fat slices.
It does not claim exhaustive parser or tool-error branch coverage.

Evidence (Linux)

  • Red/green on this exact tree, in a Linux container (Python 3.12, system ar/strings/nm):
    all ten ObjectSectionTests rows FAIL against the previous check_kernel_residuals.py
    (113 failing sub-cases, caused by leaked code bytes or by the missing duplicate / unknown-section
    / malformed-member handling) and all ten PASS against this change.
  • The whole test_check_kernel_residuals.py, ci_preflight, and the kernel profile with the real
    checker on the real Linux kernel archive run in CI on this head.

Not covered

No macOS compiler output is used: the Mach-O fixtures are hand-built byte layouts. 1.8.0 is built
and tested for Linux only, so no CI job scans a real macOS archive with this change.

🤖 Generated with Claude Code

luisleo526 and others added 30 commits October 28, 2026 00:50
…ith a doc-lint rule that holds it

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
… drop the stale 180 s wording

test_selected_window_ownership_manifest_controls_shard_1of4 took 74 s on an idle
8-vCPU box while shards 2-4 took 41-50 s (the modulo split over sorted case ids
put the heaviest controls together). The four rows carry no explicit TIMEOUT
any more (#504 removed it) but the tree still described a 180 s budget and the
CMake guard still exempted the exact old statement.

- Shards are now assigned largest-first to the lightest shard, with a weight per
  case read from its AST (checker subprocess launches, through self helpers and
  literal loops). No measured seconds are stored. The partition stays exact,
  order independent and deterministic; --shard I/N validation and the
  "ran N tests as selected" fail-closed check are unchanged.
- The two 60 s subprocess deadlines inside the controls are removed.
- scripts/check_cr150_cmake_d.py no longer exempts the old six-row
  "TIMEOUT 180" statement, and its test now requires the refusal.
- docs/ci.md, tests/ownership/README.md and the CMake comment describe the
  balancing and no deadline.
- New rows: balance bound (max/min <= 5/4, violated by the old split) and an
  estimator row; the partition row now checks input-order independence.

Documentation sentences removed from docs/ci.md, all introduced by 375795a (the
change that split the ownership manifest controls into four CTest shards): the
sentence saying that change splits the controls into four CTest shards
(`test_selected_window_ownership_manifest_controls_shard_1of4` through the
fourth), the sentence saying it adds three public registered/run rows to every
engine profile in both fixture worlds and only raises the counts above the
pinned minimum floors, and the sentence saying the private-fixture and receipt
row populations stay the same. The replacement paragraph states the balanced
four-shard split and the unchanged row populations.

Measured on a throwaway 8-vCPU spot VM: shards 51/57/53/55 s after, 74/42/41/50 s
before.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…er-platform stream memory bound

The shared descriptor guard compares every status and descriptor flag at each
boundary and masks only Darwin's kernel-maintained "descriptor was written"
bit (FWASWRITTEN) under __APPLE__. A new test plants a real F_SETFL, F_SETFD
and close after a pipe write and requires the guard to reject each; two more
ctest rows run those planted modes through a small CMake driver that accepts
only exit status 1 together with the guard's own message.

The 64 MiB stream memory test keeps its 1 MiB bound everywhere except Darwin,
where it allows 2 MiB for the 16 KiB page size.

CHANGELOG notes the macOS report-sink limit is no longer listed (re-examined,
not re-measured). 1.8.0 is Linux only, so no CI job exercises the Darwin mask
or the Darwin bound.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Attribute exclusive opens and fork-descendant scratch files to their creating
process, keep unrelated co-tenants out of the witness, and report the offending
path with its PID or an explicit unattributable reason. Add three public Linux
CTest cases for exclusive open, inherited evidence, and diagnostics.

Carry the three rows through verifier floors and mock accounting. Preserve
documentation line counts and refresh the floor-table citations.

Documentation replacements from 3ea5a0d name the old wording:
"Their default runs now enter strict exclusion accounting"
"Verifier 8 + preflight 8 + command-after 8 + report 4 add 28 public rows; plain Debug-type runs enforce the increased registration floors."
"Release 938 with the overlay or 631 without it"
"The floor itself remains fixed at 631/334 absent or 938/365 present"
"The native PR registration floor is 1000"
"Both numbers are constants of the profile"
"The full-run release and kernel floors are the table's own numbers above"
"EXCLUDED_REGISTERED_MIN holds 904 for debug, 895 for sanitizers"
"The run floors in the table above"
The release and kernel table rows from 3ea5a0d use the new floors.

The population wording and table header from b97c409 are replaced:
"The registered column is why the absent floor is not the absent registration, and it is to be re-measured after the registration fix lands."
"registered (present / absent)"

The four citation edits retain the passages named by:
"The floors' numbers live in one place"
"Each profile counts the rows that"
"the minimum number of CTest rows"
"The profile carries a row floor"

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Corrects four comments that no longer match the code they describe, and one
stale line number in a script comment:

- scripts/ci_verify.py: the selected-sink memory test is registered at its
  current line of tests/CMakeLists.txt.
- tests/test_scratch_create_pinned_text.cpp: the header names the files that
  produce the pinned create diagnostics instead of a release.
- src/source/pine_adapter.cpp: the placement table scans the retained
  placements and consults the historical minima for erased rows; it has no
  per-id index.
- include/pineforge/native_host.hpp: the live-order read comment no longer
  claims the reads allocate nothing, since a spilled row is read into a buffer.

No code changes.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
… path it pins

The header named two producing files; the test also pins the id-pair scratch
file and the successor-roots create. It now lists all four scratch paths by
what they are, and drops a stale note about an unobserved prediction. Comment
text only.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…labels

Reword published documentation that carried internal task and card labels so each
passage says what the thing is. Lines that only changed a label keep their meaning.

Sentences replaced or dropped, named by the commit that introduced each:
- 91353de: "Read-only census for CR-150 L22u / A6 and A5-WIDER." and
  "All eight vectors use the three-term verdict;" (arena-bound pages), and the two
  passages of cr150-contracts.md that cite the external design lane's work files
  and its revision 6 amendment.
- d093560: the paragraph of native-refactor-progress.md that quotes the supervisor
  cap and the decision that stopped the R5 performance wave.
- a77fb73: "R4-D L1 advanced BacktestEngine, NativeStrategyHost, and the private
  consumer to engine_script_run_v17;" (abi-stability.md).
- 44bb687: "R5 L4 adds NativeRunSpec::margin, folded only when it is set;" and
  "R5 L5 adds calculation, max_recalculations_per_point and open_ba..." (abi-stability.md).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…ds instead of internal tracking labels

Replace internal task, card and work-lane labels in source comments, docstrings,
workflow comments and human-readable progress lines of test programs with plain
descriptions. About 620 text lines in 285 files; no identifier, file name, CTest
selector, CMake fence, assertion or data file is renamed or edited.

Files whose exact bytes are pinned or whose text is data stay unchanged: the
frozen run_json fixture (its digest is asserted), fixture sources hashed by the
heap census, the ci/ gate data, the broker-state waiver list, the ownership probe
notes, the heap census report heading and the run-failure label passed to
note_run_failure. Printed lines that changed are listed in the pull request; each
old text was searched across the engine and the other repositories and nothing
reads them.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…table bytes

check_kernel_residuals.py fed the whole debug-stripped kernel archive to
`strings -a`, so instruction bytes in a text section could spell a forbidden
name by chance (four code bytes were reported on a macOS Release object). The
`strings` surface is now built per archive member from section payloads:
executable sections, sections without file bytes and an explicit list of
debug/linker metadata sections are skipped, every other section (including
unfamiliar names) is scanned. `nm -C` still reads the whole archive.

Duplicate member names in the archive are an InfrastructureError, because
`ar p <name>` cannot select a later member with the same name. A truncated or
unrecognised object structure is an InfrastructureError as well; there is no
fallback to scanning the whole object and no object or offset exception.

Tests: hand-built ELF and Mach-O byte fixtures (thin, fat and fat64, 32 and
64 bit, both byte orders, ELF extended section counts) show that planted code
bytes are ignored while the same word in a data section, an unknown section or
a symbol is still flagged. The macOS fixtures are synthetic, not compiler
output; 1.8.0 is Linux only.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…ors them

The CI workflows carry a verbatim authorization job and comments that scripts/ci_preflight.py
and its test expect byte for byte (the contract check "must start with the verbatim canon
authz job" failed on the reworded comments). Restore .github/, scripts/ci_preflight.py and
scripts/test_ci_preflight.py to their previous text.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
… capital export edit

The export added to the report writer moved the argument parser and the
selected-window helpers of docker/run_json.py by 16 lines (17 for the parser's
flags), so eleven citations in the public contract page pointed at neighbouring
lines. They cite the same functions and flags again.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…e fence text

Four CTest rows pin these bytes or fence markers: the oracle SHA-256 pin hashes tests/oracle,
the cr150 CMake-directory check and the private-fixture reads check read the fence text and the
per-lane CMake fragments, and the source gate and its fixtures mirror them. Restore tests/oracle,
tests/cr150.d, tests/CMakeLists.txt, scripts/check_cr150_cmake_d.py,
scripts/check_private_fixture_reads.py, scripts/cr150_source_gate.py and scripts/fixtures to their
previous text.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Two overlapping passages are resolved to the preceding members' text, which this
branch only re-worded: the manifest control shard paragraph of docs/ci.md and the
cleanup inspection line of tests/default_scratch_tmpfs_fixture.hpp.
…uard

CHANGELOG.md: both branches add a bullet at the same place of the compatibility
section; keep both, the earlier members' bullet first.
luisleo526 and others added 11 commits October 28, 2026 08:19
…main

The native order ABI check compares this file with the frozen base commit and
fails when a single byte differs. The wording pass changed one comment line
in it; restore main's bytes so the file stays pinned. The comment is cleaned
together with the pin in a later joint change.
Main now carries the squashed first five members of the previous stack.
The only conflict was in CHANGELOG.md: keep main's lines and this branch's
own macOS report-sink note, which sits between the walk-forward entry and
the supported-platforms entry. The merged CHANGELOG.md equals this branch's
file; nothing else needed a hand edit.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Plain merge of the member's branch onto the stack so far. No conflict, no
hand edit.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
luisleo526 added a commit that referenced this pull request Oct 28, 2026
…port-sink vectors, release environments and parallel preflight (#609)

This lands a stack of ten engine pull requests as one change, verified and measured once on the combined tree:

- #567 tests: descriptor report-sink guard rows, Darwin write bookkeeping, per-platform stream memory bound
- #572 kernel residual check: scan section payloads and symbols, never executable bytes
- #511 ci: bind App-token minting jobs to release environments
- #568 engine: Pine strings and maps stored on the paged object layer, flat in the number of bars
- #574 metrics: carry the monthly return key as 64 bits, so extreme timestamps no longer overflow int
- #575 abi: native_order_v8 and PF_ABI_VERSION 5, so a stale native module or a strategy library built for ABI 4 is refused
- #558 docker: add the pf-report-sink/v1 golden vectors with a generator, a native capture tool and a drift test
- #595 entrypoint: forward PINEFORGE_RUN_ID to run_json.py --run-id
- #604 ci: route same-repository PR proof through member-only admission
- #609 ci: parallelize independent preflight stages and balance unittest shards

The member pull requests are closed as landed here.
@luisleo526

Copy link
Copy Markdown
Collaborator Author

Landed in #609 (squash f85ce18 on main), which carries this change with one verify and one parity run on the combined tree.

@luisleo526 luisleo526 closed this Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant