Repository navigation
Commit 6b33c47
run_json: verify and copy path-sink equity records in bulk (#553)
* run_json: verify and copy the equity records of a path report sink in bulk, with exact fallback to the strict reader
The default report reads the engine's sink twice in Python, decoding about
two JSON documents per equity point (214k json.loads for a 105k-bar run) and
validating every field per record; that is most of the 5.4 s harness time.
Runs of canonical equity lines inside a bounded window are now accepted in
bulk: one regular expression per window matches the exact line grammar
(ASCII, no whitespace, integer within +-(2^53-1) and not -0, two finite
float tokens whose repr round-trips, checked with map(float) and map(repr)),
the SHA-256 is updated per window, and the equity array is produced by one
byte substitution into the compact point objects. The first line or window
that does not match, a null, an overflow of the E count or any other tag
sends the rest of the file through the existing strict per-line code, so
every refusal keeps its text, code and priority; the strict reader, the
trade path, the fd route and the jailed route are unchanged, and
pf_sink.py, pineforge_sink_finalize.py and pineforge_sink_to_json.py are
byte-identical. The bulk path is selected only by a concrete path sink
report and can be switched off by _BULK_E_ENABLED.
New tests (docker/run_json_bulk_equity_test.py, 161 rows): byte parity with
the bulk path off, including a point cut at every offset of a small window;
the same exception type, text and typed code on 35 corrupted sinks;
completed-prefix and write-failure behaviour; a count of json.loads and
parse_verified calls (O(1) per run instead of one per record); bounded
buffers and hash updates. The existing strict-parse row is split into
[strict] and [bulk].
Not run on the author's machine.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
* tests: run the two sink pass-count tests with the bulk equity path off and on
test_selected_validation_reads_only_the_first_equity_record counted the
sink.records("E") passes and expected the anchor check plus the writer's full
pass; test_strict_reader_and_trade_pass_share_one_parser_each expected every
record, equity included, to go through RecordParser.parse. Both describe the
strict reader. With the bulk equity path the writer copies the equity array
from the raw sink bytes and canonical equity lines are verified in bulk, so
neither call happens for them.
Both tests now run twice. The strict row keeps the original expectations
unchanged. The bulk row expects only the anchor pass of records("E"), and in
the parser test every non-equity record parsed once, in file order, through
the one shared parser.
Not run on the author's machine.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
* tests: differential fuzz of the bulk equity path of run_json against the strict reader
The bulk equity path accepts only canonical lines and replays everything else
through the strict reader. This file attacks that claim with generated inputs:
40 deterministic seeds (fixed integer seeds, no clock, no urandom) of valid
sinks with interleaved trade, event, output and constant rows, and 39
mutation kinds applied to the sink bytes (single kinds and random chains of
one to three), each with the digest and receipt recomputed ("fresh") or left
stale. For every case the report entry runs once with the bulk path off and
once on, on the same file and window, and the two must end alike: the same
bytes and write-call boundaries, or the same exception class, text, typed
code and arguments. The strict reader must also fire the check the mutation
is named for, so the rows cannot pass by corrupting nothing, and a count of
json.loads and per-point decodes proves the bulk path really engages on the
valid sinks.
Three controls show the comparison can fail: a copy of the bulk acceptance
without the repr-identity proof, a grammar that also accepts an integer -0,
and a bulk path that never engages must each be flagged.
Not run on the author's machine.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
* tests: bulk equity fuzz covers the direct-I/O page boundary, the time-range proof and the reader bookkeeping
The differential fuzz of the bulk equity path gains the cases the review of the first fuzz commit left open:
- faults placed around the 64 KiB page boundary of the sink file (on the boundary line, before it, after it, and at rotating positions), each at the production window and at a 97-byte window, so a line carried across the page edge is compared with the strict reader;
- control D: a bulk proof without the +-(2**53-1) time range test is detected;
- control E: a bulk reader that skips its own record count or line number after an accepted batch is detected.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
* run_json: bound the lines per regex call of the bulk equity path
The bulk equity path matched a whole 64 KiB window of canonical E lines with one
regular-expression call. The engine keeps the backtracking state of a repeated
group on a heap stack until the call returns, about 2 KiB per line here (every
alternation and bounded repeat of a line nests a frame). A full window of
32-byte lines is 2,048 lines, so that stack alone reached about 4.4 MB, while a
1,000-point sink, one run of 1,000 lines, reached about 2.2 MB. The traced
Python peak therefore followed the window and the file size, and the selected
sink memory row saw a fixed 2.3 MB step between its 1,000-point run and the
larger ones.
The run pattern is now (?:LINE){1,128}: a window is consumed in batches of at
most 128 lines, so the stack and every list built from a batch stay a few
hundred KB whatever the window or the file. The proof, the accept-only rule,
the exact fallback to the strict reader, the 64 KiB geometry and the write
boundaries are unchanged; a failed proof now replays at most one batch instead
of a window.
Tests: a tracemalloc test over 1,000 and 20,000 point sinks through the verify
and copy passes (each peak below 1 MiB, the two within 256 KiB, every line
accepted in bulk), a mutation control that restores a whole-window batch and
must fail both bounds, and a test of the batch contract. The fuzz control for
the integer -0 grammar follows the new run pattern.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
* tests: bulk equity fuzz places faults at the batch edges and asserts the replayed prefix
The bulk equity path verifies canonical E lines in batches of at most 128 lines and replays a batch whose proof fails through the strict reader. The fuzz now puts the two proof faults, a negative-zero time and a space at run-start +127, +128, +129 and +256 (positions read off the sink bytes, not hard-coded), at both windows, and a premise test fails loudly if the fixture's runs stop fitting those labels. A fifth label (+256 in the 280-line run) puts the +256 offset on a real batch edge, because the longest run is cut by the 64 KiB page.
The decoded-prefix assertion is no longer skipped for proof faults at the production window: a rejected batch is at most one batch long, so the bulk side must still have skipped the decode of every earlier batch (saved >= (accepted - batch) // 2, with the batch size read from run_json).
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
---------
Co-authored-by: ci-cx <[email protected]>
Co-authored-by: Claude Sonnet 5.5 <[email protected]>
Co-authored-by: ci-cx <[email protected]>
Co-authored-by: ci-cx <ci-cx@invalid>1 parent 4319d52 commit 6b33c47
6 files changed
Lines changed: 2547 additions & 31 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5962 | 5962 | | |
5963 | 5963 | | |
5964 | 5964 | | |
| 5965 | + | |
| 5966 | + | |
| 5967 | + | |
| 5968 | + | |
| 5969 | + | |
| 5970 | + | |
| 5971 | + | |
| 5972 | + | |
| 5973 | + | |
| 5974 | + | |
| 5975 | + | |
| 5976 | + | |
| 5977 | + | |
| 5978 | + | |
| 5979 | + | |
| 5980 | + | |
| 5981 | + | |
| 5982 | + | |
| 5983 | + | |
| 5984 | + | |
| 5985 | + | |
| 5986 | + | |
| 5987 | + | |
| 5988 | + | |
| 5989 | + | |
| 5990 | + | |
| 5991 | + | |
| 5992 | + | |
| 5993 | + | |
| 5994 | + | |
| 5995 | + | |
| 5996 | + | |
| 5997 | + | |
| 5998 | + | |
| 5999 | + | |
| 6000 | + | |
| 6001 | + | |
| 6002 | + | |
| 6003 | + | |
| 6004 | + | |
| 6005 | + | |
| 6006 | + | |
| 6007 | + | |
| 6008 | + | |
| 6009 | + | |
| 6010 | + | |
| 6011 | + | |
| 6012 | + | |
| 6013 | + | |
| 6014 | + | |
| 6015 | + | |
| 6016 | + | |
| 6017 | + | |
| 6018 | + | |
| 6019 | + | |
| 6020 | + | |
| 6021 | + | |
| 6022 | + | |
| 6023 | + | |
| 6024 | + | |
| 6025 | + | |
| 6026 | + | |
| 6027 | + | |
| 6028 | + | |
| 6029 | + | |
| 6030 | + | |
| 6031 | + | |
| 6032 | + | |
| 6033 | + | |
| 6034 | + | |
| 6035 | + | |
| 6036 | + | |
| 6037 | + | |
| 6038 | + | |
| 6039 | + | |
| 6040 | + | |
| 6041 | + | |
| 6042 | + | |
| 6043 | + | |
| 6044 | + | |
| 6045 | + | |
| 6046 | + | |
| 6047 | + | |
| 6048 | + | |
| 6049 | + | |
| 6050 | + | |
| 6051 | + | |
| 6052 | + | |
| 6053 | + | |
| 6054 | + | |
| 6055 | + | |
| 6056 | + | |
| 6057 | + | |
| 6058 | + | |
| 6059 | + | |
| 6060 | + | |
| 6061 | + | |
| 6062 | + | |
| 6063 | + | |
| 6064 | + | |
| 6065 | + | |
| 6066 | + | |
| 6067 | + | |
| 6068 | + | |
| 6069 | + | |
| 6070 | + | |
| 6071 | + | |
| 6072 | + | |
| 6073 | + | |
| 6074 | + | |
| 6075 | + | |
| 6076 | + | |
| 6077 | + | |
| 6078 | + | |
| 6079 | + | |
| 6080 | + | |
| 6081 | + | |
| 6082 | + | |
| 6083 | + | |
| 6084 | + | |
| 6085 | + | |
| 6086 | + | |
| 6087 | + | |
| 6088 | + | |
| 6089 | + | |
| 6090 | + | |
| 6091 | + | |
| 6092 | + | |
| 6093 | + | |
| 6094 | + | |
| 6095 | + | |
| 6096 | + | |
| 6097 | + | |
| 6098 | + | |
| 6099 | + | |
| 6100 | + | |
| 6101 | + | |
| 6102 | + | |
| 6103 | + | |
| 6104 | + | |
| 6105 | + | |
| 6106 | + | |
| 6107 | + | |
| 6108 | + | |
| 6109 | + | |
| 6110 | + | |
| 6111 | + | |
| 6112 | + | |
| 6113 | + | |
| 6114 | + | |
| 6115 | + | |
| 6116 | + | |
| 6117 | + | |
| 6118 | + | |
| 6119 | + | |
| 6120 | + | |
| 6121 | + | |
| 6122 | + | |
| 6123 | + | |
| 6124 | + | |
| 6125 | + | |
| 6126 | + | |
| 6127 | + | |
| 6128 | + | |
| 6129 | + | |
| 6130 | + | |
5965 | 6131 | | |
5966 | 6132 | | |
5967 | 6133 | | |
| |||
5984 | 6150 | | |
5985 | 6151 | | |
5986 | 6152 | | |
5987 | | - | |
| 6153 | + | |
5988 | 6154 | | |
5989 | 6155 | | |
5990 | 6156 | | |
5991 | | - | |
5992 | | - | |
| 6157 | + | |
| 6158 | + | |
| 6159 | + | |
| 6160 | + | |
| 6161 | + | |
| 6162 | + | |
| 6163 | + | |
5993 | 6164 | | |
5994 | 6165 | | |
5995 | 6166 | | |
| |||
6119 | 6290 | | |
6120 | 6291 | | |
6121 | 6292 | | |
6122 | | - | |
| 6293 | + | |
6123 | 6294 | | |
6124 | 6295 | | |
| 6296 | + | |
6125 | 6297 | | |
6126 | 6298 | | |
6127 | 6299 | | |
| |||
6141 | 6313 | | |
6142 | 6314 | | |
6143 | 6315 | | |
6144 | | - | |
| 6316 | + | |
6145 | 6317 | | |
6146 | 6318 | | |
6147 | 6319 | | |
| |||
6162 | 6334 | | |
6163 | 6335 | | |
6164 | 6336 | | |
| 6337 | + | |
| 6338 | + | |
| 6339 | + | |
| 6340 | + | |
| 6341 | + | |
| 6342 | + | |
| 6343 | + | |
| 6344 | + | |
| 6345 | + | |
| 6346 | + | |
| 6347 | + | |
| 6348 | + | |
| 6349 | + | |
6165 | 6350 | | |
6166 | 6351 | | |
6167 | 6352 | | |
| |||
6218 | 6403 | | |
6219 | 6404 | | |
6220 | 6405 | | |
6221 | | - | |
6222 | | - | |
| 6406 | + | |
| 6407 | + | |
| 6408 | + | |
| 6409 | + | |
| 6410 | + | |
| 6411 | + | |
| 6412 | + | |
6223 | 6413 | | |
6224 | 6414 | | |
6225 | 6415 | | |
| |||
6323 | 6513 | | |
6324 | 6514 | | |
6325 | 6515 | | |
| 6516 | + | |
| 6517 | + | |
| 6518 | + | |
| 6519 | + | |
| 6520 | + | |
| 6521 | + | |
| 6522 | + | |
6326 | 6523 | | |
6327 | 6524 | | |
6328 | 6525 | | |
| |||
6421 | 6618 | | |
6422 | 6619 | | |
6423 | 6620 | | |
6424 | | - | |
6425 | | - | |
6426 | | - | |
6427 | | - | |
6428 | | - | |
6429 | | - | |
6430 | | - | |
6431 | | - | |
6432 | | - | |
6433 | | - | |
| 6621 | + | |
6434 | 6622 | | |
6435 | 6623 | | |
| 6624 | + | |
| 6625 | + | |
| 6626 | + | |
| 6627 | + | |
| 6628 | + | |
| 6629 | + | |
| 6630 | + | |
| 6631 | + | |
| 6632 | + | |
| 6633 | + | |
| 6634 | + | |
| 6635 | + | |
| 6636 | + | |
| 6637 | + | |
| 6638 | + | |
6436 | 6639 | | |
6437 | 6640 | | |
6438 | 6641 | | |
| |||
6636 | 6839 | | |
6637 | 6840 | | |
6638 | 6841 | | |
| 6842 | + | |
| 6843 | + | |
| 6844 | + | |
| 6845 | + | |
| 6846 | + | |
| 6847 | + | |
| 6848 | + | |
| 6849 | + | |
| 6850 | + | |
| 6851 | + | |
| 6852 | + | |
| 6853 | + | |
| 6854 | + | |
| 6855 | + | |
| 6856 | + | |
| 6857 | + | |
| 6858 | + | |
| 6859 | + | |
| 6860 | + | |
| 6861 | + | |
| 6862 | + | |
| 6863 | + | |
| 6864 | + | |
| 6865 | + | |
| 6866 | + | |
| 6867 | + | |
| 6868 | + | |
| 6869 | + | |
| 6870 | + | |
| 6871 | + | |
| 6872 | + | |
| 6873 | + | |
| 6874 | + | |
| 6875 | + | |
| 6876 | + | |
| 6877 | + | |
| 6878 | + | |
| 6879 | + | |
| 6880 | + | |
| 6881 | + | |
| 6882 | + | |
| 6883 | + | |
| 6884 | + | |
| 6885 | + | |
| 6886 | + | |
6639 | 6887 | | |
6640 | 6888 | | |
6641 | 6889 | | |
6642 | 6890 | | |
6643 | 6891 | | |
6644 | 6892 | | |
6645 | 6893 | | |
| 6894 | + | |
| 6895 | + | |
6646 | 6896 | | |
6647 | 6897 | | |
6648 | 6898 | | |
| |||
6892 | 7142 | | |
6893 | 7143 | | |
6894 | 7144 | | |
| 7145 | + | |
| 7146 | + | |
| 7147 | + | |
6895 | 7148 | | |
6896 | 7149 | | |
6897 | 7150 | | |
| |||
0 commit comments