Repository navigation
Commit 2cd229f
authored
report sink: read trades and equity in one raw sequential pass each (#529)
## What this changes
The run wrapper writes its report from a verified sink file. After the strict verification pass (header, every record, trailer, counts, digest, run id and receipt, unchanged and still required), the trade records and the equity records were each read back through the strict reader a second time. Both are now read in one fresh raw sequential stream, sharing one scan:
- one unbuffered read of the sink in 64 KiB windows, no seek and no line reader;
- the SHA-256 of every byte before the trailer is computed from the same chunks that are cut into lines and decoded, so what is decoded is exactly what is hashed;
- trade lines are decoded with `RecordParser.parse`; equity lines with `RecordParser.parse_verified`, which decodes after the strict pass has verified the sink: same framing refusals (line feed, 1 MiB cap, blank line, byte-order mark), one `json.loads` with the shared token hooks, and the shared field coercion (exact integer time in the 53-bit range, optional finite double whose text equals `repr`, `-0.0` kept). It does not repeat the strict parser's byte-by-byte lexical walk;
- other record bodies are skipped without being walked or retained, including lines whose tag is written with a JSON escape;
- needle search keeps a 5-byte carry across window edges (the longest needle is `\n["T",`, 6 bytes); one pending line is held at a time, capped at 1 MiB including the LF; the scan loop is one function, `_tagged_records(tag)`, for both tags;
- at exhaustion: framing, trailer line feed and end of file are checked first, then the digest against the verified sink's digest, then the record count against the verified count. An inode/size mismatch at open is refused early. Read and open errors are typed refusals, never reported as a file change.
Every other `records(tag)`, and `records()` with no tag, still uses the strict reader.
## Behaviour
- Report output is byte-identical (frozen-bytes test on a real-sink corpus subset with equity arrays).
- Memory stays flat in bars and trades: one 64 KiB window, a 5-byte carry, one pending line.
- A changed byte after verification is refused with `the report sink changed after verification: digest differs` for both trades and equity. For equity this replaces the strict parser's text; a tag mismatch in the verified decoder says `record tag is 'T', expected 'E'`.
- No speed figure is claimed here: only a four-build measurement on a quiet box can support one.
## Tests
New `docker/run_json_trade_scan_test.py` and `docker/run_json_equity_scan_test.py` (real sinks, with the independent strict reader, `RecordParser.parse` and the frozen writer as oracles): window-edge sweeps for literal, escaped and trailer needles at every offset around the 64 KiB boundary, carry mutants, decoder equality with the strict records (types, float bits, serialized bytes, every accepted `repr` spelling and its refused alternates), typed hostile-byte refusals, one raw stream per scan, same-chunk hashing with injected read flips, identity/digest/count ordering, early close, I/O errors, publication cleanup. Existing rows in `tests/test_run_json_stream_fastpath.py`, `docker/run_json_bounded_transport_test.py` and `tests/test_run_strategy_sink.py` that pinned the old strict passes follow the new accounting (one strict open for verification and raw opens for the trade and equity scans); no bound was loosened and no row removed. The new pytest rows have no `fast` label (PR Release runs `ctest -L '^fast$'` without pytest).
Files: `docker/pf_sink.py`, `docker/run_json.py`, `docker/run_json_trade_scan_test.py` (new), `docker/run_json_equity_scan_test.py` (new), `docker/run_json_bounded_transport_test.py`, `tests/test_run_json_stream_fastpath.py`, `tests/test_run_strategy_sink.py`, `tests/cr150.d/p2cx-rjf1.cmake` (new), `tests/cr150.d/p2cx-rjf2.cmake` (new).
## Evidence (test runs on the project's cloud runner)
- All eight verify steps (preflight, release, debug, sanitizers, kernel, docs, parity, subset) pass on this exact tree with main merged in.
- The new equity rows against the base without the equity change: 118 of 249 fail (the rows were written before the implementation); the new stream-fastpath rows against the trade-pass change alone: 30 of 280 fail, exactly the five tests that pin the old accounting.
- Window-edge mutants: a carry of 2 bytes fails 7 trade-scan rows (3 boundary rows, 3 trade-edge rows, the escaped edge at pad 65534); the clean run passes all 159.
- Frozen-bytes test on a real-sink corpus subset with equity arrays: passes.
- Whole-population parity sweep of this exact tree (clean engine and codegen main, 134 of 134 cases, 7989 graded probes): the gate passes with no improvement and no regression: coverage lost 0, band 1 +0/-0, band 2 +0/-0, hard regressions 0, leavers 0.
🤖 Generated with [Claude Code](https://claude.com/claude-code)1 parent 6c95d82 commit 2cd229f
9 files changed
Lines changed: 2337 additions & 64 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
| 26 | + | |
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
| 33 | + | |
33 | 34 | | |
34 | 35 | | |
35 | 36 | | |
| |||
232 | 233 | | |
233 | 234 | | |
234 | 235 | | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
235 | 296 | | |
236 | 297 | | |
237 | 298 | | |
| |||
249 | 310 | | |
250 | 311 | | |
251 | 312 | | |
252 | | - | |
253 | | - | |
| 313 | + | |
| 314 | + | |
254 | 315 | | |
255 | 316 | | |
256 | 317 | | |
| |||
309 | 370 | | |
310 | 371 | | |
311 | 372 | | |
312 | | - | |
313 | | - | |
314 | | - | |
315 | | - | |
316 | | - | |
317 | | - | |
318 | | - | |
319 | | - | |
320 | | - | |
321 | | - | |
322 | | - | |
| 373 | + | |
| 374 | + | |
323 | 375 | | |
324 | 376 | | |
325 | 377 | | |
326 | 378 | | |
327 | | - | |
| 379 | + | |
328 | 380 | | |
329 | 381 | | |
330 | 382 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6018 | 6018 | | |
6019 | 6019 | | |
6020 | 6020 | | |
| 6021 | + | |
6021 | 6022 | | |
6022 | 6023 | | |
6023 | 6024 | | |
| |||
6075 | 6076 | | |
6076 | 6077 | | |
6077 | 6078 | | |
| 6079 | + | |
| 6080 | + | |
| 6081 | + | |
| 6082 | + | |
| 6083 | + | |
| 6084 | + | |
| 6085 | + | |
6078 | 6086 | | |
6079 | 6087 | | |
6080 | 6088 | | |
| 6089 | + | |
| 6090 | + | |
| 6091 | + | |
| 6092 | + | |
| 6093 | + | |
| 6094 | + | |
6081 | 6095 | | |
6082 | 6096 | | |
6083 | 6097 | | |
| |||
6089 | 6103 | | |
6090 | 6104 | | |
6091 | 6105 | | |
| 6106 | + | |
| 6107 | + | |
| 6108 | + | |
| 6109 | + | |
| 6110 | + | |
| 6111 | + | |
| 6112 | + | |
| 6113 | + | |
| 6114 | + | |
| 6115 | + | |
| 6116 | + | |
| 6117 | + | |
| 6118 | + | |
| 6119 | + | |
| 6120 | + | |
| 6121 | + | |
| 6122 | + | |
| 6123 | + | |
| 6124 | + | |
| 6125 | + | |
| 6126 | + | |
| 6127 | + | |
| 6128 | + | |
| 6129 | + | |
| 6130 | + | |
| 6131 | + | |
| 6132 | + | |
| 6133 | + | |
| 6134 | + | |
| 6135 | + | |
| 6136 | + | |
| 6137 | + | |
| 6138 | + | |
| 6139 | + | |
| 6140 | + | |
| 6141 | + | |
| 6142 | + | |
| 6143 | + | |
| 6144 | + | |
| 6145 | + | |
| 6146 | + | |
| 6147 | + | |
| 6148 | + | |
| 6149 | + | |
| 6150 | + | |
| 6151 | + | |
| 6152 | + | |
| 6153 | + | |
| 6154 | + | |
| 6155 | + | |
| 6156 | + | |
| 6157 | + | |
| 6158 | + | |
| 6159 | + | |
| 6160 | + | |
| 6161 | + | |
| 6162 | + | |
| 6163 | + | |
| 6164 | + | |
| 6165 | + | |
| 6166 | + | |
| 6167 | + | |
| 6168 | + | |
| 6169 | + | |
| 6170 | + | |
| 6171 | + | |
| 6172 | + | |
| 6173 | + | |
| 6174 | + | |
| 6175 | + | |
| 6176 | + | |
| 6177 | + | |
| 6178 | + | |
| 6179 | + | |
| 6180 | + | |
| 6181 | + | |
| 6182 | + | |
| 6183 | + | |
| 6184 | + | |
| 6185 | + | |
| 6186 | + | |
| 6187 | + | |
| 6188 | + | |
| 6189 | + | |
| 6190 | + | |
| 6191 | + | |
| 6192 | + | |
| 6193 | + | |
| 6194 | + | |
| 6195 | + | |
| 6196 | + | |
| 6197 | + | |
| 6198 | + | |
| 6199 | + | |
| 6200 | + | |
| 6201 | + | |
| 6202 | + | |
| 6203 | + | |
| 6204 | + | |
| 6205 | + | |
| 6206 | + | |
| 6207 | + | |
| 6208 | + | |
| 6209 | + | |
| 6210 | + | |
| 6211 | + | |
| 6212 | + | |
| 6213 | + | |
| 6214 | + | |
| 6215 | + | |
| 6216 | + | |
| 6217 | + | |
| 6218 | + | |
| 6219 | + | |
| 6220 | + | |
| 6221 | + | |
| 6222 | + | |
| 6223 | + | |
| 6224 | + | |
| 6225 | + | |
| 6226 | + | |
| 6227 | + | |
| 6228 | + | |
| 6229 | + | |
| 6230 | + | |
| 6231 | + | |
| 6232 | + | |
| 6233 | + | |
| 6234 | + | |
| 6235 | + | |
| 6236 | + | |
| 6237 | + | |
| 6238 | + | |
| 6239 | + | |
| 6240 | + | |
| 6241 | + | |
| 6242 | + | |
| 6243 | + | |
| 6244 | + | |
| 6245 | + | |
| 6246 | + | |
| 6247 | + | |
| 6248 | + | |
| 6249 | + | |
| 6250 | + | |
| 6251 | + | |
| 6252 | + | |
| 6253 | + | |
| 6254 | + | |
| 6255 | + | |
| 6256 | + | |
6092 | 6257 | | |
6093 | 6258 | | |
6094 | 6259 | | |
| |||
6303 | 6468 | | |
6304 | 6469 | | |
6305 | 6470 | | |
6306 | | - | |
| 6471 | + | |
6307 | 6472 | | |
6308 | 6473 | | |
6309 | 6474 | | |
| |||
6333 | 6498 | | |
6334 | 6499 | | |
6335 | 6500 | | |
6336 | | - | |
| 6501 | + | |
6337 | 6502 | | |
6338 | 6503 | | |
6339 | 6504 | | |
6340 | 6505 | | |
6341 | 6506 | | |
| 6507 | + | |
| 6508 | + | |
| 6509 | + | |
| 6510 | + | |
| 6511 | + | |
| 6512 | + | |
| 6513 | + | |
| 6514 | + | |
6342 | 6515 | | |
6343 | 6516 | | |
6344 | 6517 | | |
| |||
0 commit comments