Skip to content

Commit fc7aad6

Browse files
luisleo526claude
andcommitted
Gate: kernel residuals read the kernel sources' string literals, so a label GCC splits stays live (INT16b)
PR #278 failed on every Linux job that judges the kernel archive (kernel-only's kernel-residuals stage; the test_kernel_residuals row of build ubuntu-24.04 Release and of native-live): check_kernel_residuals: stale '__kernel_liquidation__': ruled in ADR-0001 but no longer in the archive or the installed headers; drop its row Mechanism, reproduced on spark with x86_64-linux-gnu-g++-13 (13.3.0-6ubuntu2~ 24.04.1, the runner's exact compiler) and the kernel profile's own configure argv -- 5105 nm -C lines, 20886 strings -a lines, 173 archive hits, the same finding. src/native_execution_consumer.cpp declares `constexpr char kNativeLiquidationLabel[] = "__kernel_liquidation__"` and uses it only as the default branch of a std::string ternary in NativeExecutionConsumer::kernel_submit_liquidation. Its address is never taken, so GCC emits no array and builds the 22-byte string inline: movq $0x16,-0x240(%rbp) ; length 22, then _M_create movdqa .LC168(%rip),%xmm0 ; "__kernel_liquida" (.rodata.cst16) movabs $0x5f5f6e6f69746164,%rcx ; "dation__" movups %xmm0,(%rax) mov %rcx,0xe(%rax) ; overlapping 8-byte store at +14 The first 16 bytes sit in .rodata.cst16 packed beside other vector constants (strings prints "native-consumer/native-driver/v5native-calendar/ __kernel_liquidaMargin liquidati"), the tail is an instruction immediate, and no contiguous label exists in the object. kNativeRiskLabel survives because it is passed as `const char*` to kernel_submit_liquidation, which forces the array into .rodata. clang (macOS: 146 hits) and GCC on aarch64 (spark: 159 hits) keep the liquidation array too. Across the three archives the ruled names present differ by exactly that one label (126 distinct names on x86-64, 127 on the other two); the raw hit counts differ only by duplicate lines (GCC's .cold clones, types repeated in demangled signatures), and every hit is ruled on all three. The other 193 identifier rulings and the three vocabulary texts are live everywhere; the eight remaining ADR phrases match no vocabulary pattern and are never judged stale. Fix, in the gate: a third surface, the string literals of the translation units the archive was built from -- CMakeLists.txt's set(PINEFORGE_KERNEL_SOURCES ...), checked to be exactly the archive's `ar t` members, plus the src/ headers they include -- comments stripped and include paths skipped (split_code_and_literals, the lexer the header surface uses). It is judged in both directions like the other two: a ruling stays live while any surface carries it, and a vocabulary literal a kernel TU compiles needs a row even where the compiler split it. Why not the code: forcing the array into the object (taking its address, external linkage) would change the kernel's hottest TU to satisfy one scanner on one compiler, and the next literal GCC splits would fail the same way; the literal is real either way, so the gate reads it where it is written. Not weaker: a row whose text left every surface is still stale on every platform, a comment or an include path never makes one live, a source-layer TU's literal is never read (the list is the archive's), and the source surface adds the forward check (8 hits, all ruled). Self-tests (scripts/test_check_kernel_residuals.py, KernelSourceSurfaceTests, 7 cases): PR #278's strings line alone reads stale; the TU's literal makes it live; a ruling no surface carries is still stale; a comment or an include path is not a literal; an unruled kernel-source literal fails; the source set is the archive's own list (a missing or extra member fails closed); the real kernel sources carry no unruled literal. Fail-before against HEAD's checker: 6 of 7 error (no source surface), the reproduction case passes on both; a mutant that drops the source hits fails 3. Suite: Ran 39 tests OK on macOS, spark aarch64 and spark x86-64 (each with its --archive). macOS: ... 146 archive hits (143 identifiers, 3 texts), 269 installed-header hits and 8 kernel-source hits ... 0 findings ... OK x86-64: ... 173 archive hits (170 identifiers, 3 texts), 269 installed-header hits and 8 kernel-source hits ... 0 findings ... OK aarch64: ... 159 archive hits (156 identifiers, 3 texts), 269 installed-header hits and 8 kernel-source hits ... 0 findings ... OK ADR-0001's residual section and contributing-llm.md name the third surface. No src/ or include/ change. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent ca0aac5 commit fc7aad6

4 files changed

Lines changed: 275 additions & 19 deletions

File tree

‎docs/adr/0001-kernel-adapter-boundary.md‎

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -365,8 +365,15 @@ underscore is not a second name) and `strings -a` over a copy whose debug
365365
information has been stripped, so a `-g` build's DWARF names are not mistaken
366366
for residue (gap lane P2b) — and, since R5 lane F6, every header the kernel
367367
profile installs (the CMake install rule's own exclusions), comments stripped.
368-
It matches the residual vocabulary, whole identifiers only, against the **first
369-
column** of the tables in this section. The vocabulary is fixed in the script
368+
Since lane INT16b it also reads the string literals of the translation units the
369+
archive was built from (`PINEFORGE_KERNEL_SOURCES` in `CMakeLists.txt`, which
370+
must be exactly the archive's members) and of the `src/` headers they include:
371+
whether a literal's bytes stay contiguous is the compiler's choice, and GCC 13 on
372+
x86-64 builds the kernel's liquidation ticket from a 16-byte vector constant and
373+
an overlapping 8-byte immediate, so that one archive's `strings -a` carries no
374+
`__kernel_liquidation__` although its code writes it (clang and aarch64 GCC keep
375+
the array). It matches the residual vocabulary, whole identifiers only, against
376+
the **first column** of the tables in this section. The vocabulary is fixed in the script
370377
(`IDENTIFIER_PATTERNS`, `PHRASE_PATTERNS`): an identifier containing `pine` (not
371378
`pineforge`), `tradingview`, `barmerge`, `coof`, `pooc`, `market_admission`,
372379
`calc_on_order_fills`, `process_orders_on_close` or a `tv` segment, and since
@@ -381,9 +388,9 @@ underscore spelling of a Pine member whose namespace word is generic
381388
is a Pine member and not a C/C++ file suffix (`ta.ema` is a call, `ta.hpp` is
382389
this project's header, which a sanitizer build writes into rodata as a real
383390
literal). Every match must be listed exactly — an identifier by its name, a text
384-
by a phrase it contains — and every listed match must still be in the archive
385-
or the installed headers, so a lane that adds a name adds its row and a lane
386-
that removes one removes its row. A name that is not in these tables fails the
391+
by a phrase it contains — and every listed match must still be in the archive,
392+
the installed headers or a kernel source literal, so a lane that adds a name
393+
adds its row and a lane that removes one removes its row. A name that is not in these tables fails the
387394
`kernel` profile. The audit's original probe is a subset of that vocabulary and
388395
still reads exactly the seventeen `pine_*` names:
389396

‎docs/pages/contributing-llm.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -52,8 +52,9 @@ finding to report, not a step to take.
5252
`src/compat/pine/` or in codegen. *Ruled:* ADR 0001, "Boundary rules",
5353
rule 1. *Enforced:* `scripts/check_kernel_residuals.py` runs `strings` and
5454
`nm` over the built `libpineforge_kernel.a`, reads the headers the kernel
55-
profile installs, and fails on a TradingView-shaped name with no row in
56-
ADR 0001's residual table.
55+
profile installs and the string literals of the kernel's own translation
56+
units, and fails on a TradingView-shaped name with no row in ADR 0001's
57+
residual table.
5758

5859
2. **A kernel capability is opt-in.** A new `NativeRunSpec` field, a new
5960
request kind, or a new virtual with an empty default — never a changed

‎scripts/check_kernel_residuals.py‎

Lines changed: 151 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,27 @@
5757
stripped: code identifiers and string literals are what a consumer
5858
compiles against, a comment is not.
5959
60+
R5 INT16b adds a third surface, because whether a string literal's bytes stay
61+
contiguous in the archive is the compiler's decision, not the code's. GCC 13
62+
on x86-64 (the hosted ubuntu-24.04 runner) builds the std::string the kernel
63+
books its own liquidation under from a 16-byte vector constant and an
64+
overlapping 8-byte immediate -- `movdqa .LC168(%rip),%xmm0` ("__kernel_liquida",
65+
packed into .rodata.cst16 beside other constants) and `movabs
66+
$0x5f5f6e6f69746164,%rcx` ("dation__"), in
67+
NativeExecutionConsumer::kernel_submit_liquidation -- because the constexpr
68+
array's address is never taken; clang (macOS) and GCC on aarch64 keep the
69+
array. `strings -a` finds no `__kernel_liquidation__` in that one archive, so a
70+
gate that read only the archive called the ruling stale on Linux x86-64 and
71+
live everywhere else. So the gate also reads
72+
73+
* the KERNEL SOURCES -- the string literals of the translation units the
74+
archive was built from (CMakeLists.txt's `set(PINEFORGE_KERNEL_SOURCES
75+
...)`, which must be exactly the archive's members) and of the src/
76+
headers they include, comments stripped and include paths skipped. They
77+
are judged in both directions, like the other two surfaces: a ruling
78+
stays live while any surface carries it, and a vocabulary literal a
79+
kernel TU compiles must have a row even where the compiler split it.
80+
6081
Exit 0 when every match is ruled and every ruling is live, 1 on a finding,
6182
2 when the archive, the tools (including the strip tool: the gate fails CLOSED
6283
rather than read debug information), the install rule or the ADR section
@@ -82,6 +103,11 @@
82103
HEADER_SUFFIXES = (".h", ".hpp")
83104
DEFAULT_ARCHIVE = ROOT / "build-ci-kernel" / "lib" / "libpineforge_kernel.a"
84105
SECTION_HEADING = "## Residual TradingView-named surface in the kernel-only archive"
106+
# The kernel's translation units as CMakeLists.txt lists them -- the one list
107+
# both archives compile -- and the private headers they reach: a quoted
108+
# include resolved beside the including file or under src/.
109+
KERNEL_SOURCES = re.compile(r"set\(\s*PINEFORGE_KERNEL_SOURCES\s+([^)]*)\)")
110+
INCLUDE_DIRECTIVE = re.compile(r"^\s*#\s*include\b")
85111

86112
# Pine v6 member names, for the underscore spellings of calls whose namespace
87113
# word the kernel shares (see IDENTIFIER_PATTERNS). A closed vocabulary, like
@@ -336,22 +362,28 @@ def symbol_name(token: str, source: str) -> str:
336362

337363

338364
def evaluate(strings_lines: list[str], nm_lines: list[str], ruled: Ruled,
339-
header_lines: tuple[list[str], list[str], list[str], list[str]] | None = None
365+
header_lines: tuple[list[str], list[str], list[str], list[str]] | None = None,
366+
source_literals: tuple[list[str], list[str]] | None = None
340367
) -> tuple[list[Finding], dict[str, int]]:
341368
"""Judge the archive's two surfaces and, when given, the installed headers'
342369
(code lines and their locations, literal texts and theirs; see
343-
read_headers)."""
370+
read_headers) and the kernel sources' string literals (texts and their
371+
locations; see read_source_literals)."""
344372
hits = scan(strings_lines, "strings") + scan(nm_lines, "nm")
345373
header_hits: list[Hit] = []
346374
if header_lines is not None:
347375
code, code_where, literals, literal_where = header_lines
348376
header_hits = (scan(code, "header", texts=False, locations=code_where)
349377
+ scan(literals, "header-literal", locations=literal_where))
378+
source_hits: list[Hit] = []
379+
if source_literals is not None:
380+
texts, where = source_literals
381+
source_hits = scan(texts, "source-literal", locations=where)
350382
findings: list[Finding] = []
351383
present: set[str] = set()
352384
covered: set[str] = set()
353385
reported: set[tuple[str, str]] = set()
354-
for hit in hits + header_hits:
386+
for hit in hits + header_hits + source_hits:
355387
if hit.kind == "identifier":
356388
name = symbol_name(hit.token, hit.source)
357389
ruled_as = next((candidate for candidate in (hit.token, name)
@@ -383,20 +415,22 @@ def matched(identifier: str) -> bool:
383415
if matched(identifier):
384416
findings.append(Finding(
385417
"stale", identifier,
386-
"ruled in ADR-0001 but no longer in the archive or the installed headers; "
387-
"drop its row"))
418+
"ruled in ADR-0001 but no longer in the archive, the installed headers or "
419+
"a kernel source literal; drop its row"))
388420
for phrase in sorted(ruled.phrases):
389421
if phrase in covered:
390422
continue
391423
if any(pattern.search(phrase) for _, pattern in PHRASE_PATTERNS):
392424
findings.append(Finding(
393425
"stale", phrase,
394-
"ruled in ADR-0001 but no archive or header text contains it; drop its row"))
426+
"ruled in ADR-0001 but no archive, header or kernel source text contains it; "
427+
"drop its row"))
395428
summary = {
396429
"hits": len(hits),
397430
"identifierHits": sum(1 for hit in hits if hit.kind == "identifier"),
398431
"phraseHits": sum(1 for hit in hits if hit.kind == "phrase"),
399432
"headerHits": len(header_hits),
433+
"sourceHits": len(source_hits),
400434
"ruledIdentifiersPresent": len(present),
401435
"ruledPhrasesCovered": len(covered),
402436
"ruledIdentifiers": len(ruled.identifiers),
@@ -552,6 +586,100 @@ def read_headers(include_root: Path, headers: list[Path]
552586
return code, code_where, literals, literal_where
553587

554588

589+
def kernel_translation_units(cmake_text: str) -> list[str]:
590+
"""The kernel TUs, repository-relative, as `set(PINEFORGE_KERNEL_SOURCES ...)`
591+
lists them in CMakeLists.txt -- the one list both archives compile."""
592+
calls = KERNEL_SOURCES.findall(cmake_text)
593+
if len(calls) != 1:
594+
raise InfrastructureError("CMakeLists.txt: expected exactly one "
595+
"set(PINEFORGE_KERNEL_SOURCES ...), found " + str(len(calls)))
596+
units = calls[0].split()
597+
if not units:
598+
raise InfrastructureError("CMakeLists.txt: PINEFORGE_KERNEL_SOURCES lists no "
599+
"translation unit")
600+
return units
601+
602+
603+
def archive_members(archive: Path) -> list[str]:
604+
"""The object members `ar t` lists; an archive's symbol index is not one."""
605+
return [line.strip() for line in run_tool(["ar", "t", str(archive)]).splitlines()
606+
if line.strip().endswith(".o")]
607+
608+
609+
def kernel_source_files(root: Path, cmake_text: str, members: list[str]) -> list[Path]:
610+
"""The translation units `members` were compiled from, then the src/ headers
611+
they include.
612+
613+
The archive must be exactly CMakeLists.txt's kernel list -- CMake names
614+
each member `<file>.o` -- so the literals read are the ones this archive
615+
compiled, never a source-layer TU's (src/compat/pine/ has a
616+
reservation_expansion.cpp too). A header outside src/ is either an
617+
installed one, already its own surface, or a system header."""
618+
units = kernel_translation_units(cmake_text)
619+
expected = [Path(unit).name + ".o" for unit in units]
620+
if len(set(expected)) != len(expected):
621+
raise InfrastructureError("PINEFORGE_KERNEL_SOURCES lists two translation units "
622+
"of one file name")
623+
missing = sorted(set(expected) - set(members))
624+
extra = sorted(set(members) - set(expected))
625+
if missing or extra:
626+
raise InfrastructureError(
627+
"the archive's members are not CMakeLists.txt's PINEFORGE_KERNEL_SOURCES ("
628+
+ "; ".join(part for part in ("missing " + " ".join(missing) if missing else "",
629+
"extra " + " ".join(extra) if extra else "")
630+
if part) + ")")
631+
source_root = (root / "src").resolve()
632+
files: list[Path] = []
633+
seen: set[Path] = set()
634+
pending = [root / unit for unit in units]
635+
while pending:
636+
path = pending.pop(0)
637+
resolved = path.resolve()
638+
if resolved in seen:
639+
continue
640+
if not path.is_file():
641+
raise InfrastructureError("kernel source is missing: " + str(path))
642+
seen.add(resolved)
643+
files.append(path)
644+
text = path.read_text(encoding="utf-8")
645+
lines = text.split("\n")
646+
for number, literal in split_code_and_literals(text)[1]:
647+
if not INCLUDE_DIRECTIVE.match(lines[number - 1]):
648+
continue
649+
for base in (path.parent, root / "src"):
650+
candidate = (base / literal).resolve()
651+
if candidate.is_file():
652+
if candidate.is_relative_to(source_root) and candidate not in seen:
653+
pending.append(candidate)
654+
break
655+
return files
656+
657+
658+
def repository_path(path: Path, root: Path) -> str:
659+
"""`path` as a location under `root`, or as given when it lies elsewhere."""
660+
try:
661+
return path.resolve().relative_to(root.resolve()).as_posix()
662+
except ValueError:
663+
return str(path)
664+
665+
666+
def read_source_literals(root: Path, files: list[Path]) -> tuple[list[str], list[str]]:
667+
"""The string literals of `files` with their `file:line` locations: comments
668+
stripped (split_code_and_literals), an include directive's path skipped."""
669+
literals: list[str] = []
670+
where: list[str] = []
671+
for path in files:
672+
text = path.read_text(encoding="utf-8")
673+
lines = text.split("\n")
674+
relative = repository_path(path, root)
675+
for number, literal in split_code_and_literals(text)[1]:
676+
if INCLUDE_DIRECTIVE.match(lines[number - 1]):
677+
continue
678+
literals.append(literal)
679+
where.append(f"{relative}:{number}")
680+
return literals, where
681+
682+
555683
def run_tool(argv: list[str]) -> str:
556684
if shutil.which(argv[0]) is None:
557685
raise InfrastructureError(argv[0] + " is not on PATH")
@@ -606,22 +734,30 @@ def read_archive(archive: Path, workdir: Path) -> tuple[list[str], list[str], st
606734

607735

608736
def check(archive: Path, adr: Path = ADR, *, evidence_dir: Path | None = None,
609-
include_root: Path = INCLUDE_ROOT) -> int:
737+
include_root: Path = INCLUDE_ROOT, source_root: Path = ROOT) -> int:
610738
ruled = ruled_entries(adr.read_text(encoding="utf-8"))
611-
headers = kernel_profile_headers(include_root)
739+
cmake_text = (source_root / "CMakeLists.txt").read_text(encoding="utf-8")
740+
headers = kernel_profile_headers(include_root, cmake_text)
612741
header_lines = read_headers(include_root, headers)
613742
with tempfile.TemporaryDirectory(prefix="pineforge-kernel-residuals-") as workdir:
614743
strings_lines, nm_lines, tool = read_archive(archive, Path(workdir))
615-
findings, summary = evaluate(strings_lines, nm_lines, ruled, header_lines=header_lines)
744+
sources = kernel_source_files(source_root, cmake_text, archive_members(archive))
745+
source_literals = read_source_literals(source_root, sources)
746+
units = len(kernel_translation_units(cmake_text))
747+
findings, summary = evaluate(strings_lines, nm_lines, ruled, header_lines=header_lines,
748+
source_literals=source_literals)
616749
if evidence_dir is not None:
617750
evidence_dir.mkdir(parents=True, exist_ok=True)
618751
(evidence_dir / "strings.txt").write_text("\n".join(strings_lines) + "\n")
619752
(evidence_dir / "nm.txt").write_text("\n".join(nm_lines) + "\n")
620753
(evidence_dir / "headers.txt").write_text(
621754
"\n".join(header.relative_to(include_root).as_posix() for header in headers) + "\n")
755+
(evidence_dir / "sources.txt").write_text(
756+
"\n".join(repository_path(path, source_root) for path in sources) + "\n")
622757
(evidence_dir / "summary.json").write_text(json.dumps({
623758
"archive": str(archive), "adr": str(adr), "stripTool": tool,
624759
"includeRoot": str(include_root), "headers": len(headers),
760+
"sources": len(sources), "sourceLiterals": len(source_literals[0]),
625761
"summary": summary,
626762
"findings": [finding.__dict__ for finding in findings]}, indent=2) + "\n")
627763
for finding in findings:
@@ -630,10 +766,13 @@ def check(archive: Path, adr: Path = ADR, *, evidence_dir: Path | None = None,
630766
print(f"kernel residuals: {archive.name}: linkable surface = "
631767
f"{len(nm_lines)} nm -C lines and {len(strings_lines)} strings -a lines "
632768
f"of a {tool} debug-stripped copy; installed surface = {len(headers)} "
633-
f"kernel-profile headers under {include_root}")
769+
f"kernel-profile headers under {include_root}; source surface = "
770+
f"{len(source_literals[0])} string literals of the archive's {units} "
771+
f"translation units and the {len(sources) - units} src/ headers they include")
634772
print(f"kernel residuals: {archive.name}: {summary['hits']} archive hits "
635-
f"({summary['identifierHits']} identifiers, {summary['phraseHits']} texts) and "
636-
f"{summary['headerHits']} installed-header hits "
773+
f"({summary['identifierHits']} identifiers, {summary['phraseHits']} texts), "
774+
f"{summary['headerHits']} installed-header hits and "
775+
f"{summary['sourceHits']} kernel-source hits "
637776
f"against {summary['ruledIdentifiers']} ruled identifiers and "
638777
f"{summary['ruledPhrases']} ruled texts in {adr.name}: "
639778
f"{summary['findings']} findings ... {verdict}")

0 commit comments

Comments
 (0)