Skip to content

Commit 6b33c47

Browse files
luisleo526ci-cxclaudeci-cxci-cx
authored
run_json: verify and copy path-sink equity records in bulk (#553)
* run_json: verify and copy the equity records of a path report sink in bulk, with exact fallback to the strict reader The default report reads the engine's sink twice in Python, decoding about two JSON documents per equity point (214k json.loads for a 105k-bar run) and validating every field per record; that is most of the 5.4 s harness time. Runs of canonical equity lines inside a bounded window are now accepted in bulk: one regular expression per window matches the exact line grammar (ASCII, no whitespace, integer within +-(2^53-1) and not -0, two finite float tokens whose repr round-trips, checked with map(float) and map(repr)), the SHA-256 is updated per window, and the equity array is produced by one byte substitution into the compact point objects. The first line or window that does not match, a null, an overflow of the E count or any other tag sends the rest of the file through the existing strict per-line code, so every refusal keeps its text, code and priority; the strict reader, the trade path, the fd route and the jailed route are unchanged, and pf_sink.py, pineforge_sink_finalize.py and pineforge_sink_to_json.py are byte-identical. The bulk path is selected only by a concrete path sink report and can be switched off by _BULK_E_ENABLED. New tests (docker/run_json_bulk_equity_test.py, 161 rows): byte parity with the bulk path off, including a point cut at every offset of a small window; the same exception type, text and typed code on 35 corrupted sinks; completed-prefix and write-failure behaviour; a count of json.loads and parse_verified calls (O(1) per run instead of one per record); bounded buffers and hash updates. The existing strict-parse row is split into [strict] and [bulk]. Not run on the author's machine. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * tests: run the two sink pass-count tests with the bulk equity path off and on test_selected_validation_reads_only_the_first_equity_record counted the sink.records("E") passes and expected the anchor check plus the writer's full pass; test_strict_reader_and_trade_pass_share_one_parser_each expected every record, equity included, to go through RecordParser.parse. Both describe the strict reader. With the bulk equity path the writer copies the equity array from the raw sink bytes and canonical equity lines are verified in bulk, so neither call happens for them. Both tests now run twice. The strict row keeps the original expectations unchanged. The bulk row expects only the anchor pass of records("E"), and in the parser test every non-equity record parsed once, in file order, through the one shared parser. Not run on the author's machine. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * tests: differential fuzz of the bulk equity path of run_json against the strict reader The bulk equity path accepts only canonical lines and replays everything else through the strict reader. This file attacks that claim with generated inputs: 40 deterministic seeds (fixed integer seeds, no clock, no urandom) of valid sinks with interleaved trade, event, output and constant rows, and 39 mutation kinds applied to the sink bytes (single kinds and random chains of one to three), each with the digest and receipt recomputed ("fresh") or left stale. For every case the report entry runs once with the bulk path off and once on, on the same file and window, and the two must end alike: the same bytes and write-call boundaries, or the same exception class, text, typed code and arguments. The strict reader must also fire the check the mutation is named for, so the rows cannot pass by corrupting nothing, and a count of json.loads and per-point decodes proves the bulk path really engages on the valid sinks. Three controls show the comparison can fail: a copy of the bulk acceptance without the repr-identity proof, a grammar that also accepts an integer -0, and a bulk path that never engages must each be flagged. Not run on the author's machine. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * tests: bulk equity fuzz covers the direct-I/O page boundary, the time-range proof and the reader bookkeeping The differential fuzz of the bulk equity path gains the cases the review of the first fuzz commit left open: - faults placed around the 64 KiB page boundary of the sink file (on the boundary line, before it, after it, and at rotating positions), each at the production window and at a 97-byte window, so a line carried across the page edge is compared with the strict reader; - control D: a bulk proof without the +-(2**53-1) time range test is detected; - control E: a bulk reader that skips its own record count or line number after an accepted batch is detected. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * run_json: bound the lines per regex call of the bulk equity path The bulk equity path matched a whole 64 KiB window of canonical E lines with one regular-expression call. The engine keeps the backtracking state of a repeated group on a heap stack until the call returns, about 2 KiB per line here (every alternation and bounded repeat of a line nests a frame). A full window of 32-byte lines is 2,048 lines, so that stack alone reached about 4.4 MB, while a 1,000-point sink, one run of 1,000 lines, reached about 2.2 MB. The traced Python peak therefore followed the window and the file size, and the selected sink memory row saw a fixed 2.3 MB step between its 1,000-point run and the larger ones. The run pattern is now (?:LINE){1,128}: a window is consumed in batches of at most 128 lines, so the stack and every list built from a batch stay a few hundred KB whatever the window or the file. The proof, the accept-only rule, the exact fallback to the strict reader, the 64 KiB geometry and the write boundaries are unchanged; a failed proof now replays at most one batch instead of a window. Tests: a tracemalloc test over 1,000 and 20,000 point sinks through the verify and copy passes (each peak below 1 MiB, the two within 256 KiB, every line accepted in bulk), a mutation control that restores a whole-window batch and must fail both bounds, and a test of the batch contract. The fuzz control for the integer -0 grammar follows the new run pattern. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * tests: bulk equity fuzz places faults at the batch edges and asserts the replayed prefix The bulk equity path verifies canonical E lines in batches of at most 128 lines and replays a batch whose proof fails through the strict reader. The fuzz now puts the two proof faults, a negative-zero time and a space at run-start +127, +128, +129 and +256 (positions read off the sink bytes, not hard-coded), at both windows, and a premise test fails loudly if the fixture's runs stop fitting those labels. A fifth label (+256 in the 280-line run) puts the +256 offset on a real batch edge, because the longest run is cut by the 64 KiB page. The decoded-prefix assertion is no longer skipped for proof faults at the production window: a rejected batch is at most one batch long, so the bulk side must still have skipped the decode of every earlier batch (saved >= (accepted - batch) // 2, with the batch size read from run_json). Co-Authored-By: Claude Sonnet 5.5 <[email protected]> --------- Co-authored-by: ci-cx <[email protected]> Co-authored-by: Claude Sonnet 5.5 <[email protected]> Co-authored-by: ci-cx <[email protected]> Co-authored-by: ci-cx <ci-cx@invalid>
1 parent 4319d52 commit 6b33c47

6 files changed

Lines changed: 2547 additions & 31 deletions

‎docker/run_json.py‎

Lines changed: 270 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -5962,6 +5962,172 @@ def _disk_report_directory(directory=None):
59625962
+ "; set TMPDIR or --spill-dir to a disk-backed directory", "arguments")
59635963

59645964

5965+
_BULK_E_ENABLED = True
5966+
_BULK_E_WINDOW = 64 * 1024
5967+
# A finite double's repr is at most 24 ASCII bytes. The lookahead bounds
5968+
# rejected tokens too, avoiding unbounded regex work on a malformed line.
5969+
_BULK_E_FLOAT = (
5970+
rb"(?=[-+0-9.e]{1,24}[,\]])-?(?:0|[1-9][0-9]*)"
5971+
rb"(?:\.[0-9]+(?:e[+-]?[0-9]+)?|e[+-]?[0-9]+)"
5972+
)
5973+
_BULK_E_LINE = re.compile(
5974+
rb'\["E",(0|-?[1-9][0-9]{0,15}),(' + _BULK_E_FLOAT
5975+
+ rb"),(" + _BULK_E_FLOAT + rb")\]\n")
5976+
# Lines per regex call. The engine matches a repeated group with its whole
5977+
# backtracking state on a heap stack that is freed only when the call returns:
5978+
# every alternation and bounded repeat of a line nests a frame, about 2 KiB per
5979+
# canonical line. A call over a full window of 33-byte lines (about 2,000)
5980+
# would therefore hold several MiB, and a call over a shorter file less, so the
5981+
# transient memory would follow the window and the file instead of staying flat.
5982+
# The bound keeps that stack, and every list built from one batch, the same for
5983+
# any window and any file length; a window is consumed in successive batches.
5984+
_BULK_E_BATCH = 128
5985+
_BULK_E_RUN = re.compile(rb"(?:" + _BULK_E_LINE.pattern + (rb"){1,%d}" % _BULK_E_BATCH))
5986+
_BULK_E_LINE_BYTES = 80
5987+
5988+
5989+
class _BulkEquity:
5990+
"""One accepted batch of lines, with only its two decoded endpoints retained."""
5991+
def __init__(self, raw, count, first, last):
5992+
self.raw, self.count, self.first, self.last = raw, count, first, last
5993+
5994+
5995+
def _bulk_equity(raw):
5996+
"""Accept a strict subset; a failed proof never supplies an error.
5997+
5998+
``raw`` is one run of at most _BULK_E_BATCH lines; a longer one is no match.
5999+
"""
6000+
if not raw or _BULK_E_RUN.fullmatch(raw) is None:
6001+
return None
6002+
times, equities, profits = zip(*_BULK_E_LINE.findall(raw))
6003+
timestamps = list(map(int, times))
6004+
if min(timestamps) < -((1 << 53) - 1) or max(timestamps) > (1 << 53) - 1:
6005+
return None
6006+
texts = list(map(bytes.decode, equities + profits))
6007+
numbers = list(map(float, texts))
6008+
if not all(map(math.isfinite, numbers)) or list(map(repr, numbers)) != texts:
6009+
return None
6010+
count = len(timestamps)
6011+
first = dict(time_ms=timestamps[0], equity=numbers[0], open_profit=numbers[count])
6012+
last = dict(time_ms=timestamps[-1], equity=numbers[count - 1], open_profit=numbers[-1])
6013+
return _BulkEquity(raw, count, first, last)
6014+
6015+
6016+
class _BulkEquityFile:
6017+
"""Look ahead within a loaded direct page; replay all other bytes strictly."""
6018+
def __init__(self, file):
6019+
self.file, self.data, self.cursor, self.eof = file, b"", 0, False
6020+
6021+
def _fill(self):
6022+
self.data, self.cursor = self.data[self.cursor:], 0
6023+
room = _BULK_E_WINDOW - len(self.data)
6024+
if not room or self.eof:
6025+
return
6026+
first = self.file.read(1) # populate exactly the page readline would use
6027+
if not first:
6028+
self.eof = True
6029+
return
6030+
self.data += first
6031+
available = self.file.read_base + self.file.read_size - self.file.position
6032+
take = min(room - 1, available)
6033+
if take:
6034+
self.data += self.file.read(take)
6035+
6036+
def window(self):
6037+
if self.cursor == len(self.data):
6038+
self._fill()
6039+
# A canonical line can cross a page. Carry only its bounded tail;
6040+
# longer or malformed lines go back to the strict readline limit.
6041+
while (not self.eof and len(self.data) - self.cursor < _BULK_E_LINE_BYTES
6042+
and len(self.data) - self.cursor < _BULK_E_WINDOW
6043+
and self.data.find(b"\n", self.cursor) < 0):
6044+
self._fill()
6045+
return self.data, self.cursor
6046+
6047+
def readline(self, limit):
6048+
stop = min(len(self.data), self.cursor + limit)
6049+
newline = self.data.find(b"\n", self.cursor, stop)
6050+
if newline >= 0:
6051+
stop = newline + 1
6052+
head = self.data[self.cursor:stop]
6053+
self.cursor = stop
6054+
if newline >= 0 or len(head) == limit:
6055+
return head
6056+
return head + self.file.readline(limit - len(head))
6057+
6058+
def read(self, count):
6059+
stop = min(len(self.data), self.cursor + count)
6060+
head = self.data[self.cursor:stop]
6061+
self.cursor = stop
6062+
return head if len(head) == count else head + self.file.read(count - len(head))
6063+
6064+
def close(self):
6065+
self.data = b""
6066+
self.file.close()
6067+
6068+
6069+
class _BulkEquityHash:
6070+
"""Keep the strict reader's byte order, updating SHA only by window."""
6071+
def __init__(self, digest):
6072+
self.digest, self.pending = digest, bytearray()
6073+
6074+
def update(self, data):
6075+
view, offset = memoryview(data), 0
6076+
while offset < len(view):
6077+
take = min(_BULK_E_WINDOW - len(self.pending), len(view) - offset)
6078+
self.pending.extend(view[offset:offset + take])
6079+
offset += take
6080+
if len(self.pending) == _BULK_E_WINDOW:
6081+
self.digest.update(self.pending)
6082+
self.pending.clear()
6083+
6084+
def hexdigest(self):
6085+
if self.pending:
6086+
self.digest.update(self.pending)
6087+
self.pending.clear()
6088+
return self.digest.hexdigest()
6089+
6090+
6091+
def _bulk_equity_records(reader):
6092+
"""Advance the real strict reader only after a complete E-batch proof."""
6093+
import pf_sink
6094+
# Alternate readers retain their own iteration and refusal semantics.
6095+
if (not _BULK_E_ENABLED or not isinstance(reader, pf_sink.SinkReader)
6096+
or not isinstance(reader._file, _UncachedSinkFile)):
6097+
yield from reader
6098+
return
6099+
file = reader._file = _BulkEquityFile(reader._file)
6100+
reader._hash = _BulkEquityHash(reader._hash)
6101+
while not reader.complete:
6102+
try:
6103+
data, cursor = file.window()
6104+
except OSError as error:
6105+
# _readline increments only after a successful read.
6106+
reader._refuse(error)
6107+
match = _BULK_E_RUN.match(data, cursor)
6108+
if match is not None:
6109+
batch = _bulk_equity(data[cursor:match.end()])
6110+
if batch is None or reader._counts["E"] + batch.count > pf_sink.I53_MAX:
6111+
yield from reader
6112+
return
6113+
reader._counts["E"] += batch.count
6114+
reader._line_number += batch.count
6115+
reader._hash.update(batch.raw)
6116+
file.cursor = match.end()
6117+
yield batch
6118+
continue
6119+
try:
6120+
record = next(reader)
6121+
except StopIteration:
6122+
return
6123+
yield record
6124+
if record.tag == "E":
6125+
# Includes nulls, escaped tags and accepted noncanonical integers.
6126+
# No second failed proof is attempted later in this file.
6127+
yield from reader
6128+
return
6129+
6130+
59656131
class _SinkReportDiagnostics:
59666132
"""Only the two E endpoints, their count and the first result refusal."""
59676133

@@ -5984,12 +6150,17 @@ def __getitem__(self, index):
59846150

59856151
def trades(self, reader):
59866152
# The existing strict reader exhausts its trailer in this same pass;
5987-
# its uncached backing reads fixed 64 KiB windows, never all rows.
6153+
# canonical E windows need only their count and decoded endpoints.
59886154
# O/V/K/X/H are deliberately dropped from this fold: the old trade
59896155
# fold read only records("T"). The reader still hashes every record
59906156
# as it iterates, supplying verify()'s digest for the entire sink.
5991-
for record in reader:
5992-
if record.tag == "E":
6157+
for record in _bulk_equity_records(reader):
6158+
if isinstance(record, _BulkEquity):
6159+
if not self.count_e:
6160+
self.first = record.first
6161+
self.last = record.last
6162+
self.count_e += record.count
6163+
elif record.tag == "E":
59936164
point = record.as_dict()
59946165
if not self.count_e:
59956166
self.first = point
@@ -6119,9 +6290,10 @@ def records(self, tag=None):
61196290
def _trade_records(self):
61206291
yield from self._tagged_records("T")
61216292

6122-
def _tagged_records(self, tag):
6293+
def _tagged_records(self, tag, *, bulk=False):
61236294
"""Hash and extract one tag from the same fresh sequential raw reads."""
61246295
import pf_sink
6296+
bulk = bulk and _BULK_E_ENABLED and tag == "E"
61256297
try:
61266298
file = _open_uncached_sink(self.path, raw=True)
61276299
try:
@@ -6141,7 +6313,7 @@ def _tagged_records(self, tag):
61416313
last_byte = trailer_start = None
61426314
done = False
61436315
while not done:
6144-
chunk = file.read(file.window)
6316+
chunk = file.read(min(file.window, _BULK_E_WINDOW) if bulk else file.window)
61456317
if not chunk:
61466318
if pending or (last_byte is not None and last_byte != 0x0A):
61476319
raise pf_sink.SinkFormatError("line does not end with LF")
@@ -6162,6 +6334,19 @@ def _tagged_records(self, tag):
61626334
cursor = len(carry) if kind is not None else max(0, processed - base)
61636335
record_at = escaped_at = None
61646336
while cursor < end:
6337+
if bulk and kind == "record" and not pending:
6338+
match = _BULK_E_RUN.match(data, cursor)
6339+
if match is not None:
6340+
batch = _bulk_equity(data[cursor:match.end()])
6341+
if batch is not None:
6342+
cursor = match.end()
6343+
processed = base + cursor
6344+
kind = None
6345+
count += batch.count
6346+
yield batch
6347+
del batch
6348+
continue
6349+
bulk = False
61656350
if kind == "tag":
61666351
# The fallback decodes only the first string token;
61676352
# skipped bodies are never walked or retained.
@@ -6218,8 +6403,13 @@ def _tagged_records(self, tag):
62186403
pf_sink._parse(bytes(pending))
62196404
record = None
62206405
else:
6221-
record = decode(bytes(pending))
6222-
count += 1
6406+
record = _bulk_equity(bytes(pending)) if bulk else None
6407+
if record is None:
6408+
bulk = False
6409+
record = decode(bytes(pending))
6410+
count += 1
6411+
else:
6412+
count += record.count
62236413
pending.clear()
62246414
kind = None
62256415
processed = base + cursor
@@ -6323,6 +6513,13 @@ def __getitem__(self, index):
63236513
return self.item(index)
63246514

63256515

6516+
class _JSONEquityArray(_JSONRecordArray):
6517+
"""The ordinary record interface plus an opt-in encoded path-sink scan."""
6518+
def __init__(self, sink):
6519+
super().__init__(lambda: (row.as_dict() for row in sink.records("E")), sink.counts["E"])
6520+
self.sink = sink
6521+
6522+
63266523
class _JSONUTF8:
63276524
"""A borrowed event message, decoded with the existing replacement rule."""
63286525
_strlen = ctypes.CDLL(None).strlen
@@ -6421,18 +6618,24 @@ def write(self, text):
64216618
if self.reading:
64226619
raise ValueError("a prepared report artifact is read-only")
64236620
for offset in range(0, len(text), self.window):
6424-
data = text[offset:offset + self.window].encode("ascii")
6425-
position = 0
6426-
while position < len(data):
6427-
count = min(self.window - self.used, len(data) - position)
6428-
self.buffer[self.used:self.used + count] = data[position:position + count]
6429-
self.used += count
6430-
self.size += count
6431-
position += count
6432-
if self.used == self.window:
6433-
self._sync_window()
6621+
self.write_ascii(text[offset:offset + self.window].encode("ascii"))
64346622
return len(text)
64356623

6624+
def write_ascii(self, data):
6625+
"""Append already encoded ASCII through the same aligned I/O window."""
6626+
if self.reading:
6627+
raise ValueError("a prepared report artifact is read-only")
6628+
position = 0
6629+
while position < len(data):
6630+
count = min(self.window - self.used, len(data) - position)
6631+
self.buffer[self.used:self.used + count] = data[position:position + count]
6632+
self.used += count
6633+
self.size += count
6634+
position += count
6635+
if self.used == self.window:
6636+
self._sync_window()
6637+
return len(data)
6638+
64366639
def flush(self):
64376640
self.file.flush() # retain checked flush/close failure behavior
64386641
if not self.reading:
@@ -6636,13 +6839,60 @@ def _json_fast_value(value):
66366839
return sum(len(item) for item in value.values() if type(item) is str) <= _STREAM_FAST_STR_LIMIT
66376840

66386841

6842+
def _write_json_ascii(destination, data):
6843+
if isinstance(destination, _BoundedJSONFile):
6844+
destination.write_ascii(data)
6845+
else:
6846+
destination.write(data.decode("ascii"))
6847+
6848+
6849+
def _stream_equity_array(value, destination):
6850+
destination.write("[")
6851+
pending, count, first = [], 0, True
6852+
# Fixed E fields, a <=17-byte integer and two <=24-byte reprs make an
6853+
# encoded row plus comma <=103 bytes. The ordinary 512-row boundary
6854+
# therefore precedes its 256 KiB byte boundary, including strict fallback.
6855+
try:
6856+
for record in value.sink._tagged_records("E", bulk=True):
6857+
if isinstance(record, _BulkEquity):
6858+
# sub and splitlines walk the entire batch in C. No Python
6859+
# loop, decoder or serializer visits its individual points.
6860+
rows = _BULK_E_LINE.sub(
6861+
rb'{"time_ms":\1,"equity":\2,"open_profit":\3},' + b"\n",
6862+
record.raw).splitlines()
6863+
else:
6864+
# The existing verified decoder owns all noncanonical cases.
6865+
rows = [json.dumps(record.as_dict(), separators=(",", ":"),
6866+
allow_nan=False).encode("ascii") + b","]
6867+
offset = 0
6868+
while offset < len(rows):
6869+
take = min(_STREAM_BATCH_RECORDS - count, len(rows) - offset)
6870+
chunk = b"".join(rows[offset:offset + take])[:-1]
6871+
pending.append(chunk if first else b"," + chunk)
6872+
first = False
6873+
count += take
6874+
offset += take
6875+
if count == _STREAM_BATCH_RECORDS:
6876+
chunk = b"".join(pending)
6877+
pending.clear() # never retry a failed write
6878+
count = 0
6879+
_write_json_ascii(destination, chunk)
6880+
finally:
6881+
# Match the ordinary array writer's completed prefix on producer error.
6882+
if pending:
6883+
_write_json_ascii(destination, b"".join(pending))
6884+
destination.write("]")
6885+
6886+
66396887
def _stream_json_value(value, destination):
66406888
if isinstance(value, _JSONColumn):
66416889
destination.write("[")
66426890
value.file.seek(0)
66436891
while chunk := value.file.read(64 * 1024):
66446892
destination.write(chunk)
66456893
destination.write("]")
6894+
elif isinstance(value, _JSONEquityArray) and _BULK_E_ENABLED:
6895+
_stream_equity_array(value, destination)
66466896
elif isinstance(value, _JSONRecordArray):
66476897
destination.write("[")
66486898
first = True
@@ -6892,6 +7142,9 @@ def _build_sink_report(report, ohlcv_path, n_bars, first_ts, last_ts, elapsed,
68927142
lambda: (_sink_trade(row) for row in sink.records("T")), sink.counts["T"])
68937143
equity_curve = _JSONRecordArray(
68947144
lambda: (row.as_dict() for row in sink.records("E")), sink.counts["E"])
7145+
if _BULK_E_ENABLED and type(sink) is _ReportSink:
7146+
# Preserve subclass-provided readers and the non-path report routes.
7147+
equity_curve = _JSONEquityArray(sink)
68957148
out["equity_curve"] = equity_curve
68967149
first_bar_trade = ({"entry_bar_index": 0},) if diagnostics.trade_at_start else ()
68977150
out["diagnostics"].update(

0 commit comments

Comments
 (0)