Repository navigation
Commit 4dc08f4
authored
run_json: buffer the jailed and fd report text as UTF-8 bytes (#559)
## What this changes
`docker/run_json.py` built the report text of the jailed sink-off route and of the fd route in an `io.StringIO` and then read it back for publication. CPython's `StringIO` keeps the written text (one byte per ASCII character) and realizes a four-byte-per-character buffer at its first read, so the resident text was about five times the stdout bytes.
The text is now built in `io.TextIOWrapper(io.BytesIO(), encoding="utf-8", newline="\n")`: the same characters and the same bytes on stdout, one byte of resident text per output byte for the ASCII `json.dumps` output, and non-ASCII text still encodes as UTF-8. Serialization still completes before the buffer is returned, so a serialization error publishes no report prefix; `_write_report` is unchanged. The remaining `io.StringIO` in `run_json.py` serves the capped failure diagnostics and is not touched.
The change in the code is four lines in `_prepare_report_json`. The rest of the PR is the rows that prove it and the documentation.
## Measured effect
Kernel `ru_maxrss`, 8,000,000 versus 2,000,000 bars (mean of two slopes), stdout byte-equal in every pair, on build hosts:
| module | stdout per bar | slope before | slope after |
| --- | --- | --- | --- |
| zero-trade codegen module | 63 B | 363.0 B/bar | 191.0 B/bar |
| hand-written trading module | 143.88 B | 767.4 B/bar | 289.7 B/bar |
The previous slope is exactly the chart (48 B) plus five times the stdout bytes per bar. What remains after this change is not decomposed here and no claim is made that it is explained.
## Tests
`docker/run_json_stringio_test.py` (17 tests, collected by the release harness test stage):
- allocation rows trace the preparer plus the real `_write_report` read-back on two documents (one 8 MiB string, one header with 200,000 small equity records), in both profiles, with a ceiling of 2.5 times the encoded document derived from the two mechanisms (the previous code is at least 5 times by construction, measured 5.000; the new code about 1.5 times);
- a slope row compares an 8 MiB and a 16 MiB document and requires at most 2.5 per added byte (previous code 5.0, new 1.05);
- a diagnostic row prints the peak of one 1 MiB read-back chunk;
- stdout bytes equal to the previous code for zero-trade, trading and non-ASCII input; a mid-document serialization error publishes nothing; a mid-write `OSError` and a short write keep the accepted prefix with a stderr-only reason and status 1; the fd route keeps its bounded summary.
The rows were red on the previous preparer and green on this one, on Python 3.12.3 and inside the release image (Python 3.11.2, image pinned by digest).
## Documentation
- `docs/pages/memory-management.md` states that the sink-off jailed route holds the report text at about one byte per output byte plus bounded encoding and read-back buffers (`O(output)`), that `--report-sink-fd` avoids the full report text buffer, and adds the caller contract for the report-sink descriptor (what the engine does with the fd, which reasons are typed, what a closed reader looks like).
- `docs/design/run-json-report-text-rows.md` records the commands, the measured numbers and the history of the allocation rows.
## Verification
Preflight, release, docs, debug, sanitizers and kernel legs and the corpus parity and subset legs run on the merge head; a full parity sweep on the same tree is measured against the current codegen main.1 parent ff05c78 commit 4dc08f4
4 files changed
Lines changed: 814 additions & 8 deletions
File tree
- docker
- docs
- design
- pages
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6545 | 6545 | | |
6546 | 6546 | | |
6547 | 6547 | | |
6548 | | - | |
| 6548 | + | |
6549 | 6549 | | |
6550 | 6550 | | |
6551 | 6551 | | |
| |||
7008 | 7008 | | |
7009 | 7009 | | |
7010 | 7010 | | |
7011 | | - | |
| 7011 | + | |
7012 | 7012 | | |
7013 | 7013 | | |
7014 | 7014 | | |
| |||
0 commit comments