Repository navigation
docker: faster report wrapper (per-header sink validation, bulk equity encoding, packed bar tapes) - #649
Conversation
The strict sink reader dispatched every field of every trade line through a generic per-field function and walked each string character by character in Python. Build one validation plan per verified header instead: an ordinary trade line now costs three parser frames rather than about fifty, with the same accepted set, the same left-to-right refusal order and the same error text. The surrogate-range test of the string fields becomes one compiled character-class search with the same exact-str and range semantics. The lexical guard that refuses whitespace and control bytes outside JSON strings runs its Python byte walk only when a compiled search finds a byte at or below 0x20 in the line; compact lines (the producer's output) skip it. The refusal itself, its text and its order against the UTF-8 and JSON errors are unchanged. The generic field function stays for verified equity decoding and for malformed numeric tokens. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…olume bar tapes Equity: an accepted bulk batch was walked by four regular-expression passes (run match, fullmatch, findall, sub) and the sub expanded a replacement template once per row. Keep the run match and one findall, prove coverage by arithmetic over the disjoint line matches (the accepted set equals what fullmatch of the run pattern accepted, including for a replaced run pattern), and join the captured number text with fixed keys. The saving applies to both the fused spool route and the streamed route; write boundaries and bytes are unchanged. Directly constructed batches keep their original substitution encoding. Bars: the packed loader declined every tape whose volume column holds integers (the grammar required a decimal point or exponent in all five numeric columns) and, for fixed-decimal prices such as 100.00, rejected the slab on a repr round trip that the DictReader path never required. Both paths convert each cell with the same float() and pack the same record, so admit integer, fixed-decimal and exponent spellings, keep the finite check, the negative-zero guard and a 24-byte cell envelope (so a cell over the csv field limit still reaches the reader that refuses it), and decline a tape without a final newline before scanning its body. The array bytes, the count and the feed hash equal the DictReader result for every admitted spelling. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Pin the wrapper's per-row Python work with counters taken from the interpreter (sys.settrace and sys.setprofile on the real frames, and wrapped pattern objects), each with a positive control on a verbatim copy of the previous code: lexical walk events per compact line, regular-expression passes and template expansions per equity batch on both encoding routes, Python calls and line events per trade, and DictReader conversions and pre-decline scan work for bar tapes. Pin output identity against the same verbatim copies for complete reports (stdout bytes, sink bytes, sink digest, receipt), refusals (text, args, order), surrogate boundaries, replaced run patterns, and every bar-cell spelling and envelope case. Mutation rows put each removed piece of work and each dropped validation back and require a named failure. The shared references live in a support module that the harness glob does not collect. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Only the file:line pins into docker/run_json.py and docker/pf_sink.py change, following the lines the previous two commits moved; no sentence is edited. A few pins that already pointed at a neighbouring line (a help string, a docstring, a handle close) now point at the statement their sentence names. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| _REFERENCE_E_RUN = re.compile(rb"(?:" + _REFERENCE_E_LINE.pattern + rb"){1,128}") | ||
|
|
||
|
|
||
| class _ReferenceEquity(run_json._BulkEquity): |
| _REFERENCE_MAX_LINE_BYTES = 1024 * 1024 | ||
| _REFERENCE_HEX64 = re.compile(r"[0-9a-f]{16}\Z") | ||
| _REFERENCE_UNPARSED = object() | ||
| _UNICODE_BOUNDARIES = ( |
| f"and constructors; budget is 31 + 4 per trade ({count} trades)") | ||
|
|
||
|
|
||
| def _record_result(parse, parser, line): |
| _record_pair(parser, line) | ||
|
|
||
|
|
||
| def _string_result(validate, value): |
| return _REFERENCE_BAR_HEADER + b"".join(rows) | ||
|
|
||
|
|
||
| def _bars_result(load, path): |
|
|
||
| import pf_sink | ||
| import run_json | ||
| import run_json_wrapper_test_support as original |
… source The bar spelling admission helper chose its expectation by reading the source of the packed slab function, so on the previous grammar it accepted the old result and the rows could not tell the two loaders apart. Assert the new admission only: spellings the packed path now takes (integers, fixed decimals, trailing dot, underflow to positive zero) must use it, and the verbatim reference loader keeps its own expectation as a separate control. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 1 advisory finding(s) below merit a look before merge.
Not checked: tests were not run; formal verification proved 0 of 9 changed function(s) (3 sampled, not proven, 6 not verified).
Formal verification. PR-changed functions: 3/9 verified (0 proven, 3 may-equivalent, 0 distinguished) · 6 not verified (6 vacuous).
Not verified on this run: \_\_init\_\_ (vacuous: never exercised), \_parse (vacuous: never exercised), parse (vacuous: never exercised), \_bulk\_equity\_ascii (vacuous: never exercised), \_pack\_bar\_bulk\_slab (vacuous: never exercised), \_stream\_equity\_array (vacuous: never exercised).
Graphify review — findings
Speeds up pf_sink record parsing by skipping the lexical string-walk when a line has no byte at or below 0x20, and by precomputing per-tag column plans so _record_values dispatches on predecoded kinds and only calls _field to produce refusals. That fast path now owns the f64 and integer column handling, and the plan table is built from a fixed set of wire types in RecordParser.__init__. Widens the bulk bar loader in run_json.py to accept integer and exponent spellings, replacing the repr round-trip check with a 24-byte token cap and declining the bulk path when the final line lacks an LF, so CSV fallback still handles those inputs. Replaces the fullmatch proof in _bulk_equity with a findall coverage check that sums capture lengths plus nine literal bytes per line, reads the batch limit from the _BULK_E_RUN pattern, and keeps the captured columns on _BulkEquity for later encoding, with an import-time assert tying the line bounds to the grammar.
Worth a look
- f64 fallback discards _field's result and then dereferences .value, which can raise AttributeError —
docker/pf_sink.py:255· Escalate · medium- NOT verified (no proof, no reproducing execution) — needs human review
Review partial — this diff was larger than one review pass covers, so later files were not reviewed; some findings may be missing.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 896 functions depend on the 848 functions this change touches.
Health — this change adds coupling hotspots:
- new:
test_fused_report_has_bounded_python_lexical_work()— 0 callers, 6 callees
Verification — 896 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 863 function(s) in the blast radius were not formally verified this run
Docs that may be stale (advisory)
AGENTS.md§ REQUIRED before claiming any change is done (lines 5-45): references changed symbolsreplaceCHANGELOG.md§ 1.6.0 — 2026-10-17 (lines 1507-1672): references changed symbolswriteCHANGELOG.md§ 1.5.0 — 2026-10-12 (lines 1790-1865): references changed symbolsinstallCHANGELOG.md§ 1.0.0 — 2026-09-30 (lines 3171-3198): references changed symbolsinstallCONTRIBUTING.md§ Development setup (lines 72-102): references changed symbolsinstallLEGAL.md§ Optional benchmark harness (benchmarks/) (lines 31-45): references changed symbolsinstallLEGAL.md§ TradingView trade-list exports (tv_trades.csv) (lines 46-54): references changed symbolspublishREADME.md§ From source (lines 117-131): references changed symbolsinstallREADME.md§ Prebuilt library (lines 132-135): references changed symbolsinstallREADME.md§ 1. PineScript, through codegen (lines 170-189): references changed symbolsinstall
…and 10 more.
Formal verification
Could not verify: Could not verify \_\_init\_\_.
The verifier did not have enough to check \_\_init\_\_, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 22 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly __init__ raised on the synthesized receiver args — names the real obstacle, not a sampling gap)
Could not verify: Could not verify \_parse.
The verifier did not have enough to check \_parse, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 15 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly SinkFormatError — names the real obstacle, not a sampling gap)
No difference found (not proven): No behavior difference found in \_string (not a proof).
The verifier ran both versions of \_string on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify parse.
The verifier did not have enough to check parse, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 37 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly __init__ raised on the synthesized receiver args — names the real obstacle, not a sampling gap)
No difference found (not proven): No behavior difference found in \_bulk\_equity (not a proof).
The verifier ran both versions of \_bulk\_equity on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify \_bulk\_equity\_ascii.
The verifier did not have enough to check \_bulk\_equity\_ascii, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 73 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly AttributeError — names the real obstacle, not a sampling gap)
No difference found (not proven): No behavior difference found in \_load\_bars\_bulk (not a proof).
The verifier ran both versions of \_load\_bars\_bulk on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify \_pack\_bar\_bulk\_slab.
The verifier did not have enough to check \_pack\_bar\_bulk\_slab, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 73 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly TypeError — names the real obstacle, not a sampling gap)
Could not verify: Could not verify \_stream\_equity\_array.
The verifier did not have enough to check \_stream\_equity\_array, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 108 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly AttributeError — names the real obstacle, not a sampling gap)
· 1 grounded finding(s) anchored inline below.
|
|
||
|
|
||
| @pytest.mark.parametrize("count", (1, 257)) | ||
| def test_fused_report_has_bounded_python_lexical_work(tmp_path, monkeypatch, count): |
There was a problem hiding this comment.
test_fused_report_has_bounded_python_lexical_work()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Faster report wrapper (docker/run_json.py, docker/pf_sink.py): four commits.
docker: validate sink records with a per-header plan
The strict sink reader dispatched every field of every trade line through a
generic per-field function and walked each string character by character in
Python. Build one validation plan per verified header instead: an ordinary
trade line now costs three parser frames rather than about fifty, with the
same accepted set, the same left-to-right refusal order and the same error
text. The surrogate-range test of the string fields becomes one compiled
character-class search with the same exact-str and range semantics.
The lexical guard that refuses whitespace and control bytes outside JSON
strings runs its Python byte walk only when a compiled search finds a byte
at or below 0x20 in the line; compact lines (the producer's output) skip it.
The refusal itself, its text and its order against the UTF-8 and JSON errors
are unchanged. The generic field function stays for verified equity decoding
and for malformed numeric tokens.
docker: encode bulk equity rows without a template and pack integer-volume bar tapes
Equity: an accepted bulk batch was walked by four regular-expression passes
(run match, fullmatch, findall, sub) and the sub expanded a replacement
template once per row. Keep the run match and one findall, prove coverage by
arithmetic over the disjoint line matches (the accepted set equals what
fullmatch of the run pattern accepted, including for a replaced run pattern),
and join the captured number text with fixed keys. The saving applies to both
the fused spool route and the streamed route; write boundaries and bytes are
unchanged. Directly constructed batches keep their original substitution
encoding.
Bars: the packed loader declined every tape whose volume column holds
integers (the grammar required a decimal point or exponent in all five
numeric columns) and, for fixed-decimal prices such as 100.00, rejected the
slab on a repr round trip that the DictReader path never required. Both
paths convert each cell with the same float() and pack the same record, so
admit integer, fixed-decimal and exponent spellings, keep the finite check,
the negative-zero guard and a 24-byte cell envelope (so a cell over the csv
field limit still reaches the reader that refuses it), and decline a tape
without a final newline before scanning its body. The array bytes, the count
and the feed hash equal the DictReader result for every admitted spelling.
docker: add work-counter and identity tests for the report wrapper
Pin the wrapper's per-row Python work with counters taken from the
interpreter (sys.settrace and sys.setprofile on the real frames, and wrapped
pattern objects), each with a positive control on a verbatim copy of the
previous code: lexical walk events per compact line, regular-expression
passes and template expansions per equity batch on both encoding routes,
Python calls and line events per trade, and DictReader conversions and
pre-decline scan work for bar tapes. Pin output identity against the same
verbatim copies for complete reports (stdout bytes, sink bytes, sink digest,
receipt), refusals (text, args, order), surrogate boundaries, replaced run
patterns, and every bar-cell spelling and envelope case. Mutation rows put
each removed piece of work and each dropped validation back and require a
named failure. The shared references live in a support module that the
harness glob does not collect.
docs: repin the line anchors that the wrapper change moved
Only the file:line pins into docker/run_json.py and docker/pf_sink.py change,
following the lines the previous two commits moved; no sentence is edited.
A few pins that already pointed at a neighbouring line (a help string, a
docstring, a handle close) now point at the statement their sentence names.
Reports stay byte-identical; the change is outside the engine library.
🤖 Generated with Claude Code