Skip to content

Commit 2cd229f

Browse files
authored
report sink: read trades and equity in one raw sequential pass each (#529)
## What this changes The run wrapper writes its report from a verified sink file. After the strict verification pass (header, every record, trailer, counts, digest, run id and receipt, unchanged and still required), the trade records and the equity records were each read back through the strict reader a second time. Both are now read in one fresh raw sequential stream, sharing one scan: - one unbuffered read of the sink in 64 KiB windows, no seek and no line reader; - the SHA-256 of every byte before the trailer is computed from the same chunks that are cut into lines and decoded, so what is decoded is exactly what is hashed; - trade lines are decoded with `RecordParser.parse`; equity lines with `RecordParser.parse_verified`, which decodes after the strict pass has verified the sink: same framing refusals (line feed, 1 MiB cap, blank line, byte-order mark), one `json.loads` with the shared token hooks, and the shared field coercion (exact integer time in the 53-bit range, optional finite double whose text equals `repr`, `-0.0` kept). It does not repeat the strict parser's byte-by-byte lexical walk; - other record bodies are skipped without being walked or retained, including lines whose tag is written with a JSON escape; - needle search keeps a 5-byte carry across window edges (the longest needle is `\n["T",`, 6 bytes); one pending line is held at a time, capped at 1 MiB including the LF; the scan loop is one function, `_tagged_records(tag)`, for both tags; - at exhaustion: framing, trailer line feed and end of file are checked first, then the digest against the verified sink's digest, then the record count against the verified count. An inode/size mismatch at open is refused early. Read and open errors are typed refusals, never reported as a file change. Every other `records(tag)`, and `records()` with no tag, still uses the strict reader. ## Behaviour - Report output is byte-identical (frozen-bytes test on a real-sink corpus subset with equity arrays). - Memory stays flat in bars and trades: one 64 KiB window, a 5-byte carry, one pending line. - A changed byte after verification is refused with `the report sink changed after verification: digest differs` for both trades and equity. For equity this replaces the strict parser's text; a tag mismatch in the verified decoder says `record tag is 'T', expected 'E'`. - No speed figure is claimed here: only a four-build measurement on a quiet box can support one. ## Tests New `docker/run_json_trade_scan_test.py` and `docker/run_json_equity_scan_test.py` (real sinks, with the independent strict reader, `RecordParser.parse` and the frozen writer as oracles): window-edge sweeps for literal, escaped and trailer needles at every offset around the 64 KiB boundary, carry mutants, decoder equality with the strict records (types, float bits, serialized bytes, every accepted `repr` spelling and its refused alternates), typed hostile-byte refusals, one raw stream per scan, same-chunk hashing with injected read flips, identity/digest/count ordering, early close, I/O errors, publication cleanup. Existing rows in `tests/test_run_json_stream_fastpath.py`, `docker/run_json_bounded_transport_test.py` and `tests/test_run_strategy_sink.py` that pinned the old strict passes follow the new accounting (one strict open for verification and raw opens for the trade and equity scans); no bound was loosened and no row removed. The new pytest rows have no `fast` label (PR Release runs `ctest -L '^fast$'` without pytest). Files: `docker/pf_sink.py`, `docker/run_json.py`, `docker/run_json_trade_scan_test.py` (new), `docker/run_json_equity_scan_test.py` (new), `docker/run_json_bounded_transport_test.py`, `tests/test_run_json_stream_fastpath.py`, `tests/test_run_strategy_sink.py`, `tests/cr150.d/p2cx-rjf1.cmake` (new), `tests/cr150.d/p2cx-rjf2.cmake` (new). ## Evidence (test runs on the project's cloud runner) - All eight verify steps (preflight, release, debug, sanitizers, kernel, docs, parity, subset) pass on this exact tree with main merged in. - The new equity rows against the base without the equity change: 118 of 249 fail (the rows were written before the implementation); the new stream-fastpath rows against the trade-pass change alone: 30 of 280 fail, exactly the five tests that pin the old accounting. - Window-edge mutants: a carry of 2 bytes fails 7 trade-scan rows (3 boundary rows, 3 trade-edge rows, the escaped edge at pad 65534); the clean run passes all 159. - Frozen-bytes test on a real-sink corpus subset with equity arrays: passes. - Whole-population parity sweep of this exact tree (clean engine and codegen main, 134 of 134 cases, 7989 graded probes): the gate passes with no improvement and no regression: coverage lost 0, band 1 +0/-0, band 2 +0/-0, hard regressions 0, leavers 0. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
1 parent 6c95d82 commit 2cd229f

9 files changed

Lines changed: 2337 additions & 64 deletions

‎docker/pf_sink.py‎

Lines changed: 67 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -23,13 +23,14 @@
2323
import sys
2424
from typing import Any, BinaryIO, NamedTuple
2525

26-
__all__ = ["I53_MAX", "MAX_LINE_BYTES", "Record", "SinkFormatError", "SinkReader"]
26+
__all__ = ["I53_MAX", "MAX_LINE_BYTES", "Record", "RecordParser", "SinkFormatError", "SinkReader"]
2727

2828
I53_MAX = (1 << 53) - 1
2929
MAX_LINE_BYTES = 1024 * 1024
3030
_ORDER = "ETOVKXH"
3131
_HEX64 = re.compile(r"[0-9a-f]{16}\Z")
3232
_SHA256 = re.compile(r"[0-9a-f]{64}\Z")
33+
_UNPARSED = object()
3334

3435

3536
class SinkFormatError(ValueError):
@@ -232,6 +233,66 @@ def _field(value: Any, wire_type: str) -> Any:
232233
raise SinkFormatError("unknown wire type")
233234

234235

236+
class RecordParser:
237+
"""Parse record lines against one header, without receipt or file state.
238+
239+
``parsed`` lets SinkReader forward the value it already decoded while
240+
checking for a trailer, preserving one JSON parse per line.
241+
"""
242+
243+
def __init__(self, header: dict[str, Any]):
244+
self._descriptors = _header(header)
245+
self._equity_descriptor = self._descriptors["E"]
246+
247+
def parse(self, line: bytes, *, parsed: Any = _UNPARSED) -> Record:
248+
value = _parse(line) if parsed is _UNPARSED else parsed
249+
if type(value) is not list or not value or type(value[0]) is not str:
250+
raise SinkFormatError("record must be a nonempty array with a string tag")
251+
tag = value[0]
252+
if tag not in self._descriptors:
253+
raise SinkFormatError("record tag is absent from the header")
254+
descriptor = self._descriptors[tag]
255+
if len(value) != len(descriptor.fields) + 1:
256+
raise SinkFormatError("record field count differs from its descriptor")
257+
values = tuple(_field(item, kind) for item, kind in zip(value[1:], descriptor.types))
258+
if tag == "T" and values[1] not in ("long", "short"):
259+
raise SinkFormatError("trade side must be long or short")
260+
return Record(tag, descriptor.fields, values)
261+
262+
def parse_verified(self, line: bytes) -> Record:
263+
"""Decode E after strict sink verification, without the lexical walk.
264+
265+
The caller must prove the scanned bytes still match that verified
266+
sink. Keep framing and field checks, including the shared token hooks.
267+
"""
268+
if not line.endswith(b"\n"):
269+
raise SinkFormatError("line does not end with LF")
270+
if len(line) > MAX_LINE_BYTES:
271+
raise SinkFormatError("line exceeds 1 MiB including LF")
272+
body = line[:-1]
273+
if not body:
274+
raise SinkFormatError("blank line")
275+
if body.startswith(b"\xef\xbb\xbf"):
276+
raise SinkFormatError("UTF-8 BOM is forbidden")
277+
try:
278+
value = json.loads(body.decode("utf-8"), object_pairs_hook=_pairs,
279+
parse_int=_int_token, parse_float=_float_token,
280+
parse_constant=_constant)
281+
except SinkFormatError:
282+
raise
283+
except (UnicodeError, ValueError, RecursionError) as error:
284+
raise SinkFormatError(f"invalid UTF-8 or JSON: {error}") from error
285+
if type(value) is not list or not value or type(value[0]) is not str:
286+
raise SinkFormatError("record must be a nonempty array with a string tag")
287+
if value[0] != "E":
288+
raise SinkFormatError(f"record tag is {value[0]!r}, expected 'E'")
289+
descriptor = self._equity_descriptor
290+
if len(value) != len(descriptor.fields) + 1:
291+
raise SinkFormatError("record field count differs from its descriptor")
292+
values = tuple(_field(item, kind) for item, kind in zip(value[1:], descriptor.types))
293+
return Record("E", descriptor.fields, values)
294+
295+
235296
class SinkReader:
236297
"""One-pass, bounded-memory iterator. Exhaustion verifies the entire sink."""
237298

@@ -249,8 +310,8 @@ def __init__(self, path: str | Path | None = None, *, stream: BinaryIO | None =
249310
try:
250311
self._file = stream if stream is not None else open(path, "rb")
251312
value, line = self._readline()
252-
self._descriptors = _header(value)
253-
self._counts = {tag: 0 for tag in self._descriptors}
313+
self._parser = RecordParser(value)
314+
self._counts = {descriptor["type"]: 0 for descriptor in value["records"]}
254315
self.header = value
255316
self._hash.update(line)
256317
except (OSError, SinkFormatError) as error:
@@ -309,22 +370,13 @@ def __next__(self) -> Record:
309370
if type(value) is dict:
310371
self._finish(value)
311372
raise StopIteration
312-
if type(value) is not list or not value or type(value[0]) is not str:
313-
raise SinkFormatError("record must be a nonempty array with a string tag")
314-
tag = value[0]
315-
if tag not in self._descriptors:
316-
raise SinkFormatError("record tag is absent from the header")
317-
descriptor = self._descriptors[tag]
318-
if len(value) != len(descriptor.fields) + 1:
319-
raise SinkFormatError("record field count differs from its descriptor")
320-
values = tuple(_field(item, kind) for item, kind in zip(value[1:], descriptor.types))
321-
if tag == "T" and values[1] not in ("long", "short"):
322-
raise SinkFormatError("trade side must be long or short")
373+
record = self._parser.parse(line, parsed=value)
374+
tag = record.tag
323375
if self._counts[tag] == I53_MAX:
324376
raise SinkFormatError("record count exceeds the exact JSON integer bound")
325377
self._counts[tag] += 1
326378
self._hash.update(line)
327-
return Record(tag, descriptor.fields, values)
379+
return record
328380
except (OSError, SinkFormatError) as error:
329381
self._refuse(error)
330382

‎docker/run_json.py‎

Lines changed: 175 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6018,6 +6018,7 @@ class _ReportSink:
60186018
def __init__(self, lib, state, *, directory=None, run_id="", path=None):
60196019
self.directory = _disk_report_directory(directory)
60206020
self.header, self.counts, self.sha256 = {}, {}, ""
6021+
self._identity = None
60216022
self.summary = self.summary_failure = self.diagnostics = None
60226023
self.named, self.run_id = path is not None, run_id
60236024
self.keep, self.closed = False, False
@@ -6075,9 +6076,22 @@ def verify(self, lib, state, report):
60756076
or bytes(receipt.sha256_hex) != self.sha256.encode("ascii")
60766077
or int(report.total_trades) != self.counts["T"]):
60776078
raise _engine_invariant("the report sink trailer differs from its engine receipt")
6079+
try:
6080+
status = os.stat(self.path)
6081+
except OSError as error:
6082+
raise _engine_invariant(f"the engine's report sink is invalid: {error}") from error
6083+
# Identity is an early check alongside the verified receipt; the
6084+
# subsequent scan still proves content against A's digest.
6085+
self._identity = status.st_ino, status.st_size
60786086
self.summary, self.summary_failure, self.diagnostics = summary, summary_failure, diagnostics
60796087

60806088
def records(self, tag=None):
6089+
if tag == "T":
6090+
yield from self._trade_records()
6091+
return
6092+
if tag == "E":
6093+
yield from self._tagged_records(tag)
6094+
return
60816095
from pf_sink import SinkFormatError
60826096
try:
60836097
with _open_uncached_sink(self.path) as reader:
@@ -6089,6 +6103,157 @@ def records(self, tag=None):
60896103
except SinkFormatError as error:
60906104
raise _engine_invariant(f"the engine's report sink is invalid: {error}") from error
60916105

6106+
def _trade_records(self):
6107+
yield from self._tagged_records("T")
6108+
6109+
def _tagged_records(self, tag):
6110+
"""Hash and extract one tag from the same fresh sequential raw reads."""
6111+
import pf_sink
6112+
try:
6113+
file = _open_uncached_sink(self.path, raw=True)
6114+
try:
6115+
status = os.fstat(file.fileno())
6116+
if (status.st_ino, status.st_size) != self._identity:
6117+
raise _engine_invariant(
6118+
"the report sink changed after verification: file identity differs")
6119+
parser = pf_sink.RecordParser(self.header)
6120+
decode = parser.parse_verified if tag == "E" else parser.parse
6121+
needle = f'\n["{tag}",'.encode("ascii")
6122+
digest = hashlib.sha256()
6123+
count = offset = processed = 0
6124+
carry = b""
6125+
pending = bytearray()
6126+
kind = "header"
6127+
tag_escaped = False
6128+
last_byte = trailer_start = None
6129+
done = False
6130+
while not done:
6131+
chunk = file.read(file.window)
6132+
if not chunk:
6133+
if pending or (last_byte is not None and last_byte != 0x0A):
6134+
raise pf_sink.SinkFormatError("line does not end with LF")
6135+
raise pf_sink.SinkFormatError("missing header or trailer")
6136+
data = carry + chunk
6137+
base, end = offset - len(carry), len(data)
6138+
last_byte = chunk[-1]
6139+
if trailer_start is None:
6140+
marker = data.find(b"\n{")
6141+
if marker >= 0:
6142+
trailer_start = base + marker + 1
6143+
# Only new bytes enter the hash. Remember the absolute cut
6144+
# even while a trailer continues across later windows.
6145+
hash_end = (len(chunk) if trailer_start is None
6146+
else max(0, trailer_start - offset))
6147+
if hash_end:
6148+
digest.update(chunk[:hash_end])
6149+
cursor = len(carry) if kind is not None else max(0, processed - base)
6150+
record_at = escaped_at = None
6151+
while cursor < end:
6152+
if kind == "tag":
6153+
# The fallback decodes only the first string token;
6154+
# skipped bodies are never walked or retained.
6155+
while cursor < end:
6156+
byte = data[cursor]
6157+
pending.append(byte)
6158+
cursor += 1
6159+
if len(pending) > pf_sink.MAX_LINE_BYTES:
6160+
raise pf_sink.SinkFormatError("line exceeds 1 MiB including LF")
6161+
if byte == 0x0A:
6162+
raise pf_sink.SinkFormatError(
6163+
"invalid UTF-8 or JSON: unterminated record tag")
6164+
if tag_escaped:
6165+
tag_escaped = False
6166+
elif byte == 0x5C:
6167+
tag_escaped = True
6168+
elif byte == 0x22:
6169+
try:
6170+
line_tag = json.loads(bytes(pending[1:]).decode("utf-8"))
6171+
except (UnicodeError, ValueError, RecursionError) as error:
6172+
raise pf_sink.SinkFormatError(
6173+
f"invalid UTF-8 or JSON: {error}") from error
6174+
if line_tag == tag:
6175+
kind = "record"
6176+
else:
6177+
kind = None
6178+
pending.clear()
6179+
processed = base + cursor
6180+
del line_tag
6181+
break
6182+
if kind == "tag":
6183+
break
6184+
continue
6185+
if kind is not None:
6186+
limit = min(end, cursor + pf_sink.MAX_LINE_BYTES + 1 - len(pending))
6187+
newline = data.find(b"\n", cursor, limit)
6188+
stop = newline + 1 if newline >= 0 else limit
6189+
pending.extend(data[cursor:stop])
6190+
cursor = stop
6191+
if len(pending) > pf_sink.MAX_LINE_BYTES:
6192+
raise pf_sink.SinkFormatError("line exceeds 1 MiB including LF")
6193+
if newline < 0:
6194+
break
6195+
if kind == "trailer":
6196+
# A verified its receipt; B/C need only framing
6197+
# and EOF, without parsing or hashing the trailer.
6198+
if cursor != end or file.read(file.window):
6199+
raise pf_sink.SinkFormatError("bytes after the trailer")
6200+
done = True
6201+
break
6202+
if kind == "header":
6203+
# Parse the raw header exactly once: malformed
6204+
# JSON must be a format refusal before digest.
6205+
pf_sink._parse(bytes(pending))
6206+
record = None
6207+
else:
6208+
record = decode(bytes(pending))
6209+
count += 1
6210+
pending.clear()
6211+
kind = None
6212+
processed = base + cursor
6213+
if record is not None:
6214+
yield record
6215+
del record
6216+
continue
6217+
# Cache each next needle so a skipped escaped tag does
6218+
# not rescan the window for a distant matching record.
6219+
# Literal skipped lines need no per-line Python work.
6220+
if record_at is None or record_at < cursor:
6221+
marker = data.find(needle, max(0, cursor - 1))
6222+
record_at = marker + 1 if marker >= 0 else end
6223+
if escaped_at is None or escaped_at < cursor:
6224+
marker = data.find(b'\n["\\', max(0, cursor - 1))
6225+
escaped_at = marker + 1 if marker >= 0 else end
6226+
trailer_at = (trailer_start - base
6227+
if trailer_start is not None
6228+
and trailer_start >= base + cursor else end)
6229+
cursor = min(record_at, escaped_at, trailer_at)
6230+
if cursor == end:
6231+
break
6232+
if cursor == trailer_at:
6233+
kind = "trailer"
6234+
elif cursor == record_at:
6235+
kind = "record"
6236+
else:
6237+
kind = "tag"
6238+
pending.extend(data[cursor:cursor + 2])
6239+
cursor += 2
6240+
tag_escaped = False
6241+
offset += len(chunk)
6242+
carry = data[-5:]
6243+
# Streamed records remain provisional until exhaustion.
6244+
if digest.hexdigest() != self.sha256:
6245+
raise _engine_invariant(
6246+
"the report sink changed after verification: digest differs")
6247+
if count != self.counts[tag]:
6248+
raise _engine_invariant(
6249+
f"the report sink {tag} scan disagrees with its verified count")
6250+
finally:
6251+
file.close()
6252+
except (OSError, pf_sink.SinkFormatError) as error:
6253+
# Includes raw read failures (EIO or a mid-scan short read), not
6254+
# just framing/parser failures; never relabel these as changes.
6255+
raise _engine_invariant(f"the engine's report sink is invalid: {error}") from error
6256+
60926257
def close(self):
60936258
if not self.closed:
60946259
if not self.named and not self.keep:
@@ -6303,7 +6468,7 @@ def close(self):
63036468

63046469

63056470
class _UncachedSinkFile(_BoundedJSONFile):
6306-
"""Binary readback after the fixed sink reader has validated its header."""
6471+
"""Uncached binary readback with an independent bounded window."""
63076472
def __init__(self, file):
63086473
position = file.tell()
63096474
super().__init__(file) # path sinks are unjailed only
@@ -6333,12 +6498,20 @@ def readline(self, limit):
63336498
return b"".join(pieces)
63346499

63356500

6336-
def _open_uncached_sink(path):
6501+
def _open_uncached_sink(path, *, raw=False):
63376502
"""Reuse the strict reader; only its bounded header uses buffered I/O.
63386503
63396504
The header's existing 1 MiB line bound also bounds that initial cache
63406505
footprint. All subsequent reads bypass cache through a fixed 64 KiB window.
63416506
"""
6507+
if raw:
6508+
# B owns a new FileIO at byte zero, including the original header.
6509+
file = open(path, "rb", buffering=0)
6510+
try:
6511+
return _UncachedSinkFile(file)
6512+
except BaseException:
6513+
file.close()
6514+
raise
63426515
from pf_sink import SinkReader
63436516
reader = SinkReader(path)
63446517
try:

0 commit comments

Comments
 (0)