Repository navigation
metrics: stream the bar Sharpe and Sortino through an ordered, packed return replay - #596
luisleo526 wants to merge 5 commits into
Conversation
… return replay A run that needs only the bar Sharpe and Sortino no longer has to keep the per-bar equity curve for them. When the metrics request asks for those two ratios alone (no report sink, no observer, no curve, returns, monthly, summary or trade statistics need), the engine appends each return (`equity_i / equity_(i-1) - 1`, one binary64, 8 bytes, no header) to a packed stream: one 64 KiB resident batch (8,192 doubles) flushed in full batches through the run's scratch store (the attached arena in a jailed run, an anonymous unlinked file otherwise), and at finish runs the existing legacy two-pass fold over that stream. The ratios are the legacy values bit for bit at any length: no tolerance, no certificate, no approximation. Sortino keeps its legacy ordered additions with the risk-free rate taken from the planned extent; the actual extent must match the plan, otherwise the request keeps today's exact curve route. New API (all additive, no existing symbol or struct changes size): - BacktestEngine::set_metrics_streaming_v1 and the C export strategy_set_metrics_streaming_v1: the streaming switch (default on) and the mask of metrics the caller needs; - BacktestEngine::metrics_extent_receipt_v1 and strategy_metrics_extent_receipt_v1: a versioned receipt naming the route (Streaming, LegacyActualCurve, LegacyRerun), the route reason (including the new OrderedReplay), the plan and actual extents and the stop boundary of a run that ended early; - BacktestEngine::prepare_metrics_streaming_v1 and admit_metrics_plain_v1: the planned extent and the plain-run shape predicate (one point per input, an identity presenter on the final amended curve; a module without the declaration keeps the legacy route). The last equity point is held until the range-end amendment that may replace its equity has been consumed and only then released to the fold, so the fold sees the final amended curve the presenter would show. The ordered stream is the only per-bar state: 8 bytes per return, about 8 MB at one million returns, in the arena in a jailed run (typed `report_sink_failed`, reason `arena_exhausted`, store `curve`, when the arena is exhausted; no new cap) or on disk otherwise. A fault while writing or reading the stream is a typed failure and never a ratio from a partial stream. If the scratch file cannot be created at all for an unjailed run, the request keeps the resident curve route, so a run that succeeds today still succeeds. Requests that need monthly ratios, returns, the curve, the equity summary or the trade statistics keep the established curve route. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…he comparison tool Black-box rows for the streaming Sharpe and Sortino, each with an independent exact reference that does not call the production fold: - tests/test_metrics_streaming_ratios.cpp, tests/tool_metrics_streaming_compare.cpp, tests/metrics_streaming_stop_fixture.hpp, tests/metrics_streaming_exact_reference.py (a rational reference): the ratios are bit-identical to the legacy two-pass values on generated series (n = 0, 1, 2, 3, filtered returns, flat, alternating, long-tail, huge-mean, kappa sweeps, overflow order, full int64 timestamp span, noisy and trend series at 1e4, 1e5 and 1e6 points), the stop rows (completed mark, intra-bar, margin callback, cooperative cancel) take the legacy route with the stopped prefix, the held-point release rows (amended commission, a point amended twice, a point held twice before the next mark, a provisional non-finite point cleared) match the final amended curve, the switch, need-mask and determinism rows (repeat, two threads, intervening input) pass, and the loaded-library rows cover a module without the plain declaration and one that overrides the presenter; - tests/test_metrics_ordered_replay.cpp: the replay itself: the curve provider is never read, a scratch write or read fault is typed and never produces a ratio, a write fault while finalizing aborts it, repeated replays are deterministic, the resident state is one batch plus a fixed allowance, and three mutants (skip every 1024th return, reorder two returns, a compensated sum instead of the legacy fold) turn named rows red; - tests/test_metrics_fold.cpp and tests/test_metrics_overflow.cpp: the held-release and overflow-order rows; - tests/equity_stats_reference.hpp: the reference month key computes in 64 bits where it falls back to gmtime_r, because the full int64 timestamp span overflowed the int product (undefined behaviour reported by the sanitizer build); reference helper only. Registration through two lane files (tests/cr150.d/streaming-ratios.cmake and ordered-replay.cmake) and one fenced block in tests/CMakeLists.txt for the Threads link of the determinism row. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…ed replay added The streaming ratio change inserts lines into include/pineforge/engine.hpp, src/engine_report.cpp, src/engine_metrics.cpp and src/native_execution_consumer.cpp, so documentation citations into those files moved and the source allocation-bound gate indexes new sites. - Documentation: the citations in nine pages follow the code they name (part of them mechanically by the anchor checker's own re-anchoring, the rest by hand where the symbol occurs on several lines or the old lines are now comment text). Only citation paths and numbers change (plus one pair of backticks around an owner name that the checker would otherwise take for the cited symbol); no sentence, heading or page line count changes. The content-hash pin on the calibration comment cites its new range. - ci/cr150-source-bounds.yaml: four new unreviewed entries (the fold's append, the held point's two assignments, the finish output, the return stream's append); no bound is claimed for them. The reviewed identity inventory in ci/cr150-source-sites.json follows the code: three stale identities of the held point are replaced by the five current ones, the finish output of the plain-run admission gains its assignment, and the four new entries carry their identities (3,534 records become 3,541). - scripts/test_cr150_source_gate.py: the real-tree test pins the open count of the manifest (1465 becomes 1469; the unreviewed ceiling is 1281). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…s descriptor demand The ordered return replay streams through the run's scratch store: the arena's inherited descriptor in a jailed run, one anonymous unlinked file otherwise. New row `ordered-replay-descriptors` (Linux, counted from /proc/self/fd; an unreadable directory fails the row): the process's open descriptor count after a streaming run equals the count before it, on success and after the typed scratch write failure. The memory page states the demand in one sentence inside its existing paragraph. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The ratio rows compared the Sharpe of the Streaming route with a 1e-12 relative tolerance, a leftover from an approximate design. The Streaming route is the ordered replay, the legacy fold over the same returns in the same order, so every route is now compared bit for bit (the tolerance branch is gone from the shared comparison, and the determinism rows compare the two runs and the legacy values by bits). Two comments that still described the abandoned tolerance design now state why the replay is required. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Graphify reviewed this change.
No issues found by static checks — no coupling regressions or blocking issues in the code graph. That is not the same as safe to merge: see what was not checked.
Not checked: tests were not run; formal verification proved 0 of 13 changed function(s) (13 not verified).
Formal verification. PR-changed functions: 0/13 verified (0 proven, 0 may-equivalent, 0 distinguished) · 13 not verified (13 unsupported).
Not verified on this run: begin\_report\_sink (unsupported), compute\_report\_sink\_metrics (unsupported), fill\_metrics\_section (unsupported), fill\_report\_envelope\_v1 (unsupported), begin\_ready (unsupported), capture\_presentation (unsupported), invoke\_applied\_callback (unsupported), pump\_batch (unsupported), refold\_equity\_tail (unsupported), run\_rich (unsupported), and 3 more.
Graphify review — findings
Adds a streaming v1 path for equity-ratio metrics: BacktestEngine can now fold each equity point into a streaming accumulator as it's recorded (via set_metrics_streaming_v1, admit_metrics_plain_v1 and an extent plan/receipt), instead of computing the ratios from a retained curve afterwards. When no_curve is set, the report sink skips keeping the full equity curve, as SUMMARY_ONLY mode already does, and holds only the in-flight point, which the fold can amend and sync_metrics_streaming_held_v1 copies into the report. With streaming off, the existing curve-based path is unchanged. The CI source-bounds manifest takes four new unreviewed allocation/growth sites (ceiling 1277 → 1281) for the new fold code, and the ADR 0001 line references are updated to match the shifted source.
Review partial — this diff was larger than one review pass covers, so later files were not reviewed; some findings may be missing.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 3537 functions depend on the 1940 functions this change touches.
Health — this change adds coupling hotspots:
- new:
candidate()— 10 callers, 4 callees - new:
run_suite()— 2 callers, 17 callees - new:
run_probe()— 5 callers, 6 callees - new:
reference_source()— 2 callers, 7 callees - new:
scratch_write_fault_finalize_row()— 1 callers, 8 callees - new:
scratch_write_fault_row()— 1 callers, 8 callees - new:
held_synthetic_rows()— 1 callers, 7 callees - new:
corpus()— 1 callers, 7 callees - …and 6 more — each is listed as a finding
Verification — 3537 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 3356 function(s) in the blast radius were not formally verified this run
Docs that may be stale (advisory)
CHANGELOG.md§ CI and release process (lines 150-211): references changed symbolscheckCHANGELOG.md§ Report sink and scratch spill (lines 270-481): references changed symbolsbinary64CHANGELOG.md§ 1.1.0 — 2026-10-04 (lines 2530-2744): references changed symbolscheckCHANGELOG.md§ What a native host must handle (lines 2978-3066): references changed symbolsbinary64CHANGELOG.md§ Behaviour that moves results against 0.13.1 (lines 3092-3416): references changed symbolsbinary64CONTRIBUTING.md§ The two layers, and the one rule (lines 26-59): references changed symbolsGeneratedStrategyCONTRIBUTING.md§ The gates (lines 123-158): references changed symbolscheckREADME.md§ Validation scoreboard (lines 280-294): references changed symbolscheckREADME.md§ Architecture: kernel vs. parity (lines 436-463): references changed symbolsGeneratedStrategyREADME.md§ Building, testing and the gates (lines 464-510): references changed symbolscheck
…and 10 more.
Formal verification
Could not verify: Could not verify begin\_report\_sink.
The verifier did not have enough to check begin\_report\_sink, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver
Could not verify: Could not verify compute\_report\_sink\_metrics.
The verifier did not have enough to check compute\_report\_sink\_metrics, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver
Could not verify: Could not verify fill\_metrics\_section.
The verifier did not have enough to check fill\_metrics\_section, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver
Could not verify: Could not verify fill\_report\_envelope\_v1.
The verifier did not have enough to check fill\_report\_envelope\_v1, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver
Could not verify: Could not verify begin\_ready.
The verifier did not have enough to check begin\_ready, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver
Could not verify: Could not verify capture\_presentation.
The verifier did not have enough to check capture\_presentation, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver
Could not verify: Could not verify invoke\_applied\_callback.
The verifier did not have enough to check invoke\_applied\_callback, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver
Could not verify: Could not verify pump\_batch.
The verifier did not have enough to check pump\_batch, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver
Could not verify: Could not verify refold\_equity\_tail.
The verifier did not have enough to check refold\_equity\_tail, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver
Could not verify: Could not verify run\_rich.
The verifier did not have enough to check run\_rich, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver
Could not verify: Could not verify run\_simple.
The verifier did not have enough to check run\_simple, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver
Could not verify: Could not verify run\_tf.
The verifier did not have enough to check run\_tf, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver
Could not verify: Could not verify stream\_begin.
The verifier did not have enough to check stream\_begin, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver
· 14 grounded finding(s) anchored inline below.
| return result | ||
|
|
||
|
|
||
| def reference_source(count, first, last, generate_points): |
There was a problem hiding this comment.
reference_source()
fans out to 7 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| // An actual deterministic host adapter, not a precomputed-prefix replay. | ||
| // Its curve run repeats the original injection (including real liquidation), | ||
| // verifies the complete partial result, then publishes that result once. | ||
| inline ReplayRun run_replay_adapter(StopSpec stop, const MetricsExtentPlanV1& planned, |
There was a problem hiding this comment.
run_replay_adapter()
fans out to 8 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| } | ||
| }; | ||
|
|
||
| std::size_t run_probe(const std::string& row_name, std::int64_t count, |
There was a problem hiding this comment.
run_probe()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| } | ||
| }; | ||
|
|
||
| void scratch_write_fault_row() { |
There was a problem hiding this comment.
scratch_write_fault_row()
fans out to 8 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| }); | ||
| } | ||
|
|
||
| void scratch_write_fault_finalize_row() { |
There was a problem hiding this comment.
scratch_write_fault_finalize_row()
fans out to 8 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| if (candidate(points, mt::plan(points), cfg, MetricsRatioRouteV1::Streaming, &script)) | ||
| STREAM_CHECK(mt::same(active->candidate->downside_sum, want.down)); | ||
| } | ||
| void held_synthetic_rows() { |
There was a problem hiding this comment.
held_synthetic_rows()
fans out to 7 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| if (pine_bar_index() == 1) throw std::runtime_error("streaming-ratio mid-recalculation error"); | ||
| } | ||
| }; | ||
| void failure_and_monthly_rows() { |
There was a problem hiding this comment.
failure_and_monthly_rows()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| } | ||
| } | ||
|
|
||
| void run_suite() { |
There was a problem hiding this comment.
run_suite()
fans out to 17 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| throw std::runtime_error("incomplete/invalid sink: " + path.string()); | ||
| return points; | ||
| } | ||
| void corpus(const fs::path& directory, std::size_t expected_count) { |
There was a problem hiding this comment.
corpus()
fans out to 7 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| } | ||
| } // namespace streaming_compare | ||
|
|
||
| int main(int argc, char** argv) { |
There was a problem hiding this comment.
main()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
…aling, UDT graphs, metrics replay and the CI admission fix (#616) This lands twelve engine pull requests as one change, verified and measured once on the combined tree: - #593 scratch: a fixed write-through read cache for the adapter's id tables - #594 live runner: a delta report stream by default, and report-path work that does not rescan closed trades - #597 live runner: typed pre-growth stop for the ledger budget and flat pending counts - #608 run_json: reserve the report channel, so strategy output on fd 1 cannot corrupt a report - #586 Pine UDT object graphs on the paged object Store - #578 native order: dormant working book, flat per-point cost for never-filled orders - #579 adapter: skip roster passes and empty-family scans that cannot change anything - #599 adapter: read projected pending orders through one cached duplicate index - #563 adapter: gate the reversal exit cancel on candidate owners instead of every historical id - #591 report sink: retire sealed closed-trade rows on sink routes when the module opts in - #596 metrics: stream the bar Sharpe and Sortino through an ordered, packed return replay - #615 ci: accept release.yml's matrix runner under its fetched path in PR admission Fixes found while testing the combined tree are in the final commits: the retiring ledger's cache sizing, a residency witness that bounds sink-run heap by the declared cache cap and checks zero growth at 100k, 200k and 1M trades, a test fix for a descriptor reuse race, and a standing-orders timing grid whose 20,000-order point is tracked in #621. The member pull requests are closed as landed here.
What this changes
A run that needs only the bar Sharpe and Sortino no longer keeps the per-bar equity curve for them.
When the metrics request asks for those two ratios alone (no report sink, no observer, no curve, returns, monthly, summary or trade-statistics need), the engine appends each return (
equity_i / equity_(i-1) - 1, one binary64, 8 bytes, no header) to a packed stream: one 64 KiB resident batch of 8,192 doubles, flushed in full batches through the run's scratch store (the attached arena in a jailed run, an anonymous unlinked file otherwise). At finish it runs the existing legacy two-pass fold over that stream. The ratios are the legacy values bit for bit, at any length: there is no tolerance, no rounding certificate and no approximation. The risk-free rate for Sortino comes from the planned extent; if the actual extent differs from the plan, or the request needs anything else, the run keeps today's exact curve route.New, additive API:
BacktestEngine::set_metrics_streaming_v1/strategy_set_metrics_streaming_v1(the streaming switch, default on, and the mask of metrics the caller needs),metrics_extent_receipt_v1/strategy_metrics_extent_receipt_v1(a versioned receipt naming the route and the reason, including the newOrderedReplay),prepare_metrics_streaming_v1andadmit_metrics_plain_v1. No existing symbol or struct changes size.The last equity point is held until the range-end amendment that may replace its equity has been consumed, and only then released to the fold, so the fold sees the final amended curve the presenter would show.
Memory
The ordered stream is the only per-bar state: 8 bytes per return, about 8 MB at one million returns, plus one 64 KiB batch. In a jailed run it lives in the attached arena (the typed
report_sink_failederror with reasonarena_exhaustedand storecurveis the only refusal; there is no new cap); otherwise it is an unlinked file on disk. A fault while writing or reading the stream is a typed failure and never produces a ratio from a partial stream. If the scratch file cannot be created at all for an unjailed run, the request keeps the resident curve route, so a run that succeeds today still succeeds.Tests
Black-box rows with independent exact references that do not call the production fold: bit equality with the legacy ratios on generated series (empty, one to three returns, filtered returns, flat, alternating, heavy-tailed, huge-mean, kappa sweeps, overflow order, the full int64 timestamp span, noisy and trend series at 1e4, 1e5 and 1e6 points), stop rows (completed mark, intra-bar, margin callback, cooperative cancel), held-point release rows, switch, need-mask and determinism rows, loaded-library rows, fault rows for the scratch stream, and mutation rows that turn named rows red against product-level mutants.
A descriptor row (Linux) counts the process's open descriptors before and after a streaming run, on success and after the typed scratch write failure, and requires them equal; the memory page states the replay's descriptor demand (none of its own in a jailed run, one transient anonymous file otherwise). Every ratio row compares the Sharpe and Sortino bit for bit on every route, with no tolerance.
tests/equity_stats_reference.hppcomputes the reference month key in 64 bits where it falls back togmtime_r; the full int64 timestamp span overflowed anintproduct there (undefined behaviour reported by the sanitizer build). The engine's own month key has the same expression and is not changed here.🤖 Generated with Claude Code