Repository navigation
run_json: cut the wrapper's report-sink passes from ~11 to 3 (part of S-272) - #517
Merged
Merged
Conversation
… in bounded batches (S-272 part 1) The 1.7.0 report writer took about 11 full strict passes over the report sink and wrote every record with dozens of tiny Python calls, which cost about 0.22 ms per bar of user CPU (100,000 bars: 22 s against 1.36 s for 1.6.3, the engine's own part being 1.57 s of it). - The verify pass, which already reads the whole sink for its digest, now also folds the trade summary, the first and last equity points and the equity finiteness check. The annualization block gets a two-point object instead of five full re-reads (point_at). Failures found there are kept as the exception object (traceback dropped) and raised at the old points, so which error surfaces first is unchanged. - _stream_json_value writes flat records of exact scalar types with one json.dumps and joins them in batches fixed in both record count (512) and bytes (256 KiB); a string field longer than 4096 characters keeps the windowed path; non-finite values replay the old piecewise path so the partial bytes and the exception are identical. Output bytes are unchanged (the tests keep the old code path verbatim as the oracle). Memory stays flat: every batch is a fixed size. pf_sink.py, the strict reader, is untouched. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…ests that pinned the old pass pattern
- _validate_report_result is back at its original line count (docstring and one tuple reflowed, no
behaviour change), so every docker/run_json.py line the design docs cite by number (BARS-BIN-INPUT.md)
stays valid: the doc-anchors stage had flagged 37 anchors shifted by 3 lines.
- tests/test_run_json_selected_sink.py: records("E") now sees only the one-point anchor check and the
writer's complete pass ([1, count]); the finite-result scan and both annualization endpoint reads are
folded into the verify pass, which reads through the strict reader and not records().
- tests/test_run_json_stream_fastpath.py: a record whose strings exceed the fast-path limit is written in
windows, so the write bound is derived from the window, and a 4,000 x 1,000-character row keeps the
byte-bounded batch tested.
- Comments: the fold drops O/V/K/X/H records on purpose; the equity count is still checked against the
receipt by the strict reader's trailer comparison and verify().
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Brings in the A5-STORE change (#515) and the docs/CHANGELOG commits that landed after b0f4341. First parent is 0638354 (the wrapper head). The wrapper change touches docker/run_json.py and its tests only; none of main's files overlap. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Brings in #516 (test_check_doc_reverts: do not inherit the caller's PINEFORGE_DOC_REVERT_BASE): one file, scripts/test_check_doc_reverts.py. First parent is b913faa (the PR head). No file of the run_json change is touched. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| return False | ||
|
|
||
|
|
||
| class _StubSink(run_json._ReportSink): |
| monkeypatch, _build_sink_report, _OldSink, rows, writer=_stream_json) | ||
| actual, _, actual_bytes, failure = _exercise_sink( | ||
| monkeypatch, run_json._build_sink_report, _StubSink, rows) | ||
| assert failure == old_failure is None |
| sink, actual, actual_bytes, failure = _exercise_sink( | ||
| monkeypatch, run_json._build_sink_report, _StubSink, rows, named=named, | ||
| window_anchor=window_anchor) | ||
| assert failure == old_failure is None |
| writer=_stream_json) | ||
| _, actual, actual_bytes, failure = _exercise_sink( | ||
| monkeypatch, run_json._build_sink_report, _StubSink, rows, window_anchor=window_anchor) | ||
| assert failure == old_failure is None |
luisleo526
changed the title
luisleo526
added a commit
that referenced
this pull request
Oct 26, 2026
…nch was made) Merge commit, no rebase: first parent is the branch head, second parent is main. The page docs/pages/run-failure-codes.md is unchanged on main since the branch base, so the merge result is main's tree plus this branch's one-page insertion; no conflict. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
luisleo526
added a commit
that referenced
this pull request
Oct 26, 2026
…_fastpath (#522) #517 labelled this pytest-backed CTest row "fast", so the PR Release job's `ctest -L '^fast$'` (which installs no pytest, by the ci_preflight rule at scripts/ci_preflight.py:2014-2016) runs it and fails on any runner whose tool-cache Python lacks pytest (pf-gcp-pr-3/-4). Like every other pytest row (tests/CMakeLists.txt:2263-2276), it now carries no label and runs in the full profiles, which install pytest. Tests only; no engine behaviour change. Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
luisleo526
added a commit
that referenced
this pull request
Oct 26, 2026
## What Adds the `## 1.8.0 — 2026-10-27` section (with migration notes) to `CHANGELOG.md`, above `## 1.7.0`. `CHANGELOG.md` is the only file changed (+131/-0). Documentation only: no `src/`, `include/`, `docker/`, `scripts/`, `tests/` or workflow file is touched. ## Why The engine release body is generated from git-log subjects only and carries no migration notes; the release notes users read are this file (and the GitHub release body edited from it after publish). The notes must be on main before the release dispatch. ## Content - Compatibility and migration: exact pairing with codegen 1.8.0, regenerate C++ and relink every strategy library, no public header change, the new codegen `color` input type, the changed harness file `docker/run_json.py`. - Faster `run_json` report handling (#517) with the measured table against 1.6.3 and 1.7.0 and the statement that the gap to 1.6.3 is not closed. - Scratch placement restated (#518); the unused storage building block (#515; its allocation-gate sites are unreviewed, no bound claimed); a known limit (memory with never-filled pending orders, "a fix is planned for a later release"). - CI and release process: every merged change since 1.7.0 not listed above (#505-#510, #513, #514, #516, #519, #520, #522, #469, #481, #504). Every line is taken from the merged commit body; none claims a behaviour change for a tests-only or CI-only change. Not in this commit (not on main at base `641349a`): the release gate, tmpwatch and the TvJO fix. Each gets one line from its merged PR body as a follow-up commit on this branch before this PR merges; this PR is meant to merge after those three. ## Checks - Route: docs-only, `lab verify --via ci-only` (no sweep). - Read as a committed diff by an independent reviewer: GO. All 18 engine commits `v1.7.0..641349a` have a bullet, none extra. - Numbers: the #517 timings and RSS are the measurements in the #517 body; the ratios in the text were recomputed (1.77/0.36 = 4.9, 8.26/1.35 = 6.1, 8.94/5.39 = 1.66; 4.61/0.36 = 12.8, 22.31/1.35 = 16.5, 22.99/5.39 = 4.27). - The heading date `2026-10-27` is the planned tag date and is corrected before the tag if the date moves. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
docker/run_json.py: cut the wrapper's repeated passes over the report sink from about eleven to three per run: the record stream is read in fixed-size chunks and the derived sections (including the annualization diagnostics that calledpoint_atfive times) reuse those passes. Output bytes are unchanged.Part of the 1.7.0 slowdown report (S-272, part 1: the wrapper). The engine's own per-record sink parse and the dense-script close path are separate fixes.
Measured on one spot box (4 reps each, min-max within 2 percent, same tag build and
.so, onlydocker/run_json.pyreplaced; trades identical in every cell):At 100k fixed ids: 8.3 s wall, of which about 7.2 s CPU, vs 22.3 s unpatched. This removes the wrapper's multi-pass cost, NOT the remaining gap to 1.6.3 (still about 6x on fixed ids). Table measured on the v1.7.0 tag engine with a byte-identical wrapper file.
Peak RSS unchanged (72.9 vs 72.7 MB at 100k). Report bytes identical to unpatched 1.7.0 under the accepted normalization (timing fields replaced by a token) in all six cells; a control of two unpatched runs hashes equal.
Tests:
tests/test_run_json_stream_fastpath.py(new; 10 mutations of the fast path each turn a row red),tests/test_run_json_selected_sink.py, wired viatests/cr150.d/p2cx-rjfast.cmake.🤖 Generated with Claude Code