live runner: a carried report accumulator, so each input updates only the appended rows - #617

…date touches only the appended rows

After the closed-trade scan was fixed, the runner still rebuilt the engine's
whole trade list and re-folded the equity curve and metrics for every input
(fill_report), so total work over N inputs stayed O(N*T).

- A runner-owned accumulator carries the closed-trade folds, the equity high-water,
  the monthly walk state and the logical array lengths, updates them from the rows
  appended since the last input through a new engine-internal entry point, and
  produces the delta from patches with explicit offsets. The batch fill_report and
  every exported C symbol are untouched.
- The four whole-series ratio fields (sharpe_monthly, sharpe_bar, sortino_bar,
  sortino_monthly) depend on the whole-series mean, so they are computed by the
  batch routine: every input by default (--ratio-interval 1, results identical to
  before), or only every K-th input, at the final input and at export with
  --ratio-interval K, which adds a ratios_as_of_input key to the delta document.
  The documentation states the per-input bound of each part.
- Tests compare the accumulator's cumulative report with the batch report at every
  input, check the K > 1 boundaries, and count the rows touched per input over N
  and 10N inputs.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…code apart, index its sites

The first version of the accumulator removed an overload that the generated strategy
libraries resolve from libpineforge.a (pineforge::metrics::compute_equity_stats for
pf_equity_point_s*), so the corpus strategies failed to load. The overload is restored as
main has it and the accumulator code lives in its own source file with an internal header,
so no declaration, template or instantiation of an installed header changes.

The ten growable-container sites of the new source file have rows in the source allocation
manifest (state unreviewed, each with the bound: the rows appended since the last input) and
records in the site inventory; the manifest's unreviewed ceiling rises by exactly ten, which
goes to the reviewers. The documentation citations that the earlier edits moved are re-pointed.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…live-report entries

The accumulator's ten reviewed manifest rows add ten OPEN entries to the real
tree (the unreviewed ceiling went up by the same ten), so the real-tree case
that pins the OPEN count moves from 1465 to 1475. The case still checks that
PASS(bounded) plus OPEN equals the matched entries and that --strict fails on
any OPEN entry.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…ts into the report accumulator branch

The accumulator keeps feeding the existing delta fold, so there is one
production way to build the cumulative document: the accumulator update, the
delta applier and the retained document. Plain report output stays the
cumulative document and the delta stream stays opt-in. Recovery recomputes the
same document for the first cursor, every replayed input and the checkpoint
boundary, so a checkpoint is written and verified against the text a live run
prints. For a ratio interval above one, the freshness marker and the civil month
clock travel with the document and its checkpoint, and an export refreshes the
ratios only as a final read-only step.

New restart rows run the default stream with ratio interval one and with
interval three: the run is killed after a few committed inputs, restarted, and
every later report must equal the independently built document and the
uninterrupted run byte for byte. Two moved citations in the native feature
parity page are re-pointed. Every changelog entry of both sides is kept.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…, keep a refused input out of it, read the replay marker without a parse

The runner keeps the cumulative document, the accumulator curve and the engine
arrays resident in every report mode, so memory grows with the number of equity
points and trades. The README, the changelog and the memory page now say so, and
a new measurement row runs the real runner on the default stream at growing
input counts, prints the runner's own peak resident size for each, and fails on
a sanity ceiling and on a per-input growth ceiling; it is skipped on non-Linux
systems. The measured figures are added to the docs from the lab run.

A run that ends by a ledger storage stop used to skip the end-of-run ratio
refresh, and a refused input had already advanced the retained document. The
patch update now commits through a callback before it is applied, so the
retained document stays at the durable cursor, and the terminal refresh runs on
that committed document after a storage stop without touching the ledger. A new
row proves the three ledger files are byte-identical across that refresh and the
printed document equals the independent batch prefix.

Replay reads the ratio freshness marker of a stored delta without parsing the
whole payload when the envelope has its canonical shape, and falls back to the
previous parse for every other shape, so malformed or missing markers fail as
before; a row compares the two readers over a table of shapes and runs real
streams with a forced final boundary. Another row checks that a report at a
non-multiple input differs from the committed row only in the four ratios and
their marker.

Replaced documentation sentences: "Delta serialization keeps scalar state and
array counts; the report accumulator carries the equity series for ratios, and
full mode also retains the cumulative document." in the changelog, and in the
runner README "All stream modes retain one cumulative document by applying the
same suffix patches used in the ledger.", "Cumulative stdout, legacy report
verification and checkpoint equality serialize that fold; they never rebuild a
separate native report." and "All modes retain the cumulative JSON arrays and
the accumulator's equity history; this is O(history) memory, not a flat-memory
claim." The native report declaration cited by the selected window wire design
moved two lines.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…ranch

The checkpoint branch now says why an optional checkpoint was not written
(budget, build or io, with a count per reason beside the unchanged total), pins
the default-ledger exact limit with a row, and states two cost choices in the
README and the budget guide. The accumulator keeps its own retention, storage
stop and replay changes. The one conflicting citation in the native feature
parity page keeps the line where the native example batch test is declared in
the merged runner build file.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…t rows, publish the measured memory

The storage-stop row used a round budget that the run never reached. The limit is
now derived from the contract arithmetic: the input-only reservation plus half of
the checkpoint margin on top of the file sizes after input seven, checked on an
unbounded control, so input eight alone fits and input eight with its scheduled
checkpoint is refused; the refused input is one ordinary bar. The historical
export row compared two documents whose ratio words were all null, because the
native market example never records equity points. Both rows now run a small
native test host that records the marked equity on every bar, and the feed has
volatile closes, so the as-of and the exact ratios differ for a stated reason.
Every earlier assertion stays; expected documents come from a separate batch
strategy instance.

The runner README, the memory page and the changelog now publish the lab
measurement of the default stream: peak resident size 22,472 KiB at 1,000 inputs
and 29,828 KiB at 10,000 inputs, about 0.82 KiB per additional input, on a
fixture that does not record equity points, with no flat memory claim.

Replaced documentation sentences: "No retained-document RSS figure is published
here before the lab runs that row." in the memory page, and in the changelog
"Delta serialization keeps scalar state and array counts; the report accumulator
carries the equity series for ratios, while every mode keeps the cumulative JSON
document, accumulator curve and ratio workspace, and engine arrays resident with
O(report) growth in equity points and trades as the cost of one
cumulative-document builder, measured by `test_native_live_report_memory`."

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…s equity, publish both measurements

The memory row ran on the native market example, which records no equity points,
so its figures left out the per-bar equity series. It now runs on the native
test host that records one equity point per bar. Lab measurement of the default
stream, the runner's own peak resident size: 22,880 KiB at 1,000 inputs and
40,024 KiB at 10,000 inputs, about 1.9 KiB per additional input, below the 4 KiB
per input ceiling of the row; the earlier no-equity measurement, 22,472 KiB and
29,828 KiB, is kept in the docs for comparison. No flat memory claim.

Replaced documentation sentences, in the runner README, the memory page and the
changelog: "peak RSS 22,472 KiB at 1,000 inputs and 29,828 KiB at 10,000 inputs,
about 0.82 KiB per additional input. These are the runner's own `VmHWM` readings
for one fixture, including the accumulator curve, retained cumulative document
and engine arrays. The measured native-market fixture records no equity points,
so these figures do not measure growth of the accumulator curve." and in the
changelog "the runner's own `VmHWM` peak RSS was 22,472 KiB at 1,000 inputs and
29,828 KiB at 10,000 inputs, about 0.82 KiB per additional input for one
native-market fixture with no recorded equity points".

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…ckpoint, publish nothing before the commit, keep the storage-stop exit

Replay may read the ratio marker of a stored delta without a full parse only for
rows whose report equality is compared (at or after the verified checkpoint);
for every row below it the old full parse stays, so a stored delta with a
damaged body is still refused at startup. A new row corrupts the body of the
stored delta below, at and after the boundary and requires the old diagnostic
and no stdout document.

The patch update now publishes the array sizes, the closed trade high-water mark
and the visit count only after the commit callback returned, so a thrown commit
leaves every member at the last durable cursor; a row throws from the callback
before every successful update and retries the same patch. The final ratio
refresh returns before reading the input cursor when there is no freshness
marker, and an exception of the refresh inside the storage-stop handler is
caught and reported on one stderr line so the exit status stays 3; a row covers
the empty retained document and a refusal of the first input. The soft stop at
the ledger size check cannot be reached with admission before COMMIT (the lines
are named in the report), so it has no row. The two-host memory comparison now
says the per-bar equity series is consistent with being the difference.

Replaced documentation and comment sentences: "so the per-bar equity series
accounts for the difference" in the README, the memory page and the changelog;
in the changelog "example that records none); the process measurement includes
the accumulator" and "immutable. Exact ratio recomputation remains O(series), at
every input for"; the comment "The memory row keeps its original module"; and
the comment "A refused input may already have advanced the engine and suffix
bookkeeping. Keep the retained document at the last durable cursor." Three store
citations that were already stale are re-pointed.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…ator branch

The checkpoint branch now tolerates a failing diagnostic lock without failing the
run, says that the diagnostic reason names the latest failure while the counts
are cumulative, and states what the default-ledger exact-limit row proves. The
accumulator keeps its replay, publication and storage-stop changes. The four
conflicting citations of the ledger commit and the recovered checkpoint report in
the runner README and the budget guide point at the lines where those functions
are declared in the merged runner sources.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…rt accumulator branch

The checkpoints branch now sits on main (which carries the report path and the ledger
budget). This merge brings the accumulator onto it. The accumulator's ten source
container records take the manifest ids U1580 to U1589 because main already uses
U1524 to U1579; the unreviewed ceiling equals the merged entry count (1309), the
pending flat-memory ceiling is 184, and the real-tree test of the source gate pins
1493 open entries. The two moved line citations in the feature-parity page follow
main's lines; no documentation sentence is replaced.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…nto the live runner report accumulator branch

The checkpoints branch merged main (the pre-integrity ledger open fix, #631) and derives the ledger layouts it accepts, with report_checkpoints as the last stage, from one list: 3, 6, 7, 8, 9, 11, 12 and 13 tables are opened, every other count is refused. This merge brings that rule and its store rows, the changelog bullet and the main workflow and preflight changes into the accumulator branch; the merge needed no manual resolution. No documentation sentence was replaced or deleted.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

… a direct engine handle

The runner cast every loaded strategy's opaque state to a BacktestEngine to feed the report accumulator. A module whose handle is not an engine of this layout (the startup stub and legacy modules without execution capabilities, or any wrapped handle) was read through the wrong type: undefined behaviour that crashed under the sanitizers and with a plain SIGSEGV. The engine runtime now exports an optional strategy_stream_engine_layout_version() that returns PF_STREAM_ENGINE_LAYOUT_VERSION (1); the runner uses the accumulator only when the module exports it with that value, and otherwise takes the report through the module's own strategy_stream_fill_report and the report-based delta update, as before the accumulator existed. That update now takes the same optional commit callback and publishes its counters and document only after the commit succeeds. Report bytes for engine-backed modules are unchanged; modules that omit the export or declare another layout keep full snapshots. The export inventory counts (115 declared, 95 implemented) and the README symbol table follow. No documentation sentence was replaced or deleted.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…to the accumulator and opaque modules to the report path

Two end-to-end rows for the layout stamp. native_live_engine_layout_e2e loads the normally linked native example, checks that it exports strategy_stream_engine_layout_version with value 1, runs the live runner over a fixed input stream with an interval of two and asserts the marker only the accumulator branch writes (ratios_as_of_input = cursor / 2 * 2 in every delta and every stored snapshot, one extra end-of-stream refresh) and that stdout mirrors the committed delta bytes. native_live_opaque_report_e2e loads the integer-handle stub module (no engine, no stamp), checks that it exports none, and asserts that the runner takes the module's own report: no ratio marker even at an interval of two, the stub's closed-form values in every report, and ledger and stream bytes equal to a golden captured from main's runner; the golden is produced by tests/generate_native_live_opaque_report_golden.py and the row fails until it is committed. The stub now returns real, count-dependent report values and frees them. The export-count pins read 115 and 95, the stale counts in the documentation follow, and the new export sits after every cited line so no citation moves. No documentation sentence was replaced or deleted.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…main's runner

The opaque-report row compares the runner's ledger and stream bytes for the integer-handle stub module with a golden captured from the runner of main at a34a0d0, so the report path taken for modules without an engine handle is proven to produce the bytes the runner produced before the accumulator existed. The golden was generated on a cloud box with the generator's own checks on: source revision a34a0d0, the stub built from the source in this tree, a Release runner. Its closed-form content agrees with the stub's formulas (net profit 1.25 n (n+1) / 2 after n inputs, state hash n + 1, no ratio marker). No documentation sentence was replaced or deleted.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…nt when a scenario ends with one

When a scenario finishes with a delivered event the runner never acknowledged, the failure said only that pending was 1. The harness now prints, for every unacknowledged event and from the scenario's own ledger opened read-only, its ordinal, event id, input position, attempts and last error, every delivery-log row of that event, what the receiver recorded for it (received, how many times, the status it chose, whether the effect was recorded) and the last forty lines of the runner's stderr, which is now kept in its own file next to the combined log. Nothing is printed when no event is pending, and no assertion, retry, wait or acknowledgement changed: the scenario passes or fails exactly as before. No documentation sentence was replaced or deleted.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…per input (red row)

The accumulator re-supplies the whole equity curve to the exact-ratio routine on every input, so the points it stages grow with the history: n inputs stage about n times the warmup length plus n (n + 1) / 2 points, although only one point is appended per input. The new row drives the real accumulator over a generated daily tape at two populations (64 and 640 inputs), counts the staged points through the accumulator's own work counter rather than time, and requires at most three logical point stores per input (the previous held point, the appended point and the presented tail). It also keeps every output contract: the carried document equals a batch report's document byte for byte at every input, the state hash is unchanged by the counted calls, and no fallback report read happens. The row fails on the current tree because the staging is the quadratic baseline, and passes once only the new suffix is staged. No documentation sentence was replaced or deleted.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…stent workspace instead of re-staging it on every input

At every ratio boundary the accumulator resized a scratch vector and copied the whole raw equity curve into it before calling the exact-ratio routine, so n inputs copied about n times the warmup length plus n (n + 1) / 2 points although each input adds one point. The workspace now carries the raw prefix between inputs: each read restores the previous held point to its raw value, appends only the new raw points, and overlays the presented tail on the last point after presentation, so an input stages at most three points (restored held point, appended point, presented tail) and only the vector's amortised growth is linear. The workspace is kept in step even when the ratio computation is deferred, so a later computation sees the same raw prefix and the same presented tail as the removed full copy. The exact-ratio call receives the same points, count, capital, timezone, prices, exposure and net as before; no metric, fold order, summation order, ratio schedule or output byte changes. The staging counter now counts the points actually stored into the workspace.

The existing replacement-tail case also runs with deferred ratios (interval 7, New York calendar): it alternates held-point replacements with appended points, requires the carried document to equal the separate reference engine's full batch document at every input, and checks a forced refresh at a skipped boundary and its repeat. No documentation sentence was replaced or deleted.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…ised build without sanitizers

The row's deadline distinguished only the thread and address sanitizer builds from everything else, so a plain Debug build kept the release 160 s deadline and 180 s ctest timeout; measured on a spot box with three rows running in parallel it took 152.7 s, 7 s under the deadline. CMake now defines a macro for the Debug build type when neither sanitizer is on and sets the ctest timeout to 460 s, and the row's work deadline for that build is 400 s (2.6 times the measurement, the same factor as the sanitizer budgets, with the figure and its source in the comments). Release keeps 160 s and 180 s, the sanitizer builds are unchanged, and the growth and size assertions are untouched. No documentation sentence was replaced or deleted.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…ess while its ledger actor is blocked (red row)

The delivery worker is one thread that both talks to the ledger and drives curl. While it waits for the ledger mutex, which the input thread holds through a checkpoint, no curl handle is polled, so a request the receiver already accepted can run into curl's total timeout and be recorded as a failed delivery. The new row drives the real worker over a fake transport and a real ledger and requires, by thread identity and operation counts rather than elapsed time, that exactly one transport thread, distinct from the ledger-facing worker thread, constructs, adds to, polls and destroys the transport. With the worker parked at its existing callback, scripted polls must still return a request's result and observe another poll with the second request still active; before the worker is released the ledger must show started rows only, and after it drains one started and one completed row and one acknowledgement per event. Further subcases cover a backlog of 256 completed results, cancellation with a queued result (nothing is recorded as finished and a fresh worker replays the same bytes as attempt 2), an exception in the transport thread (rethrown by finish, nothing recorded), and a retry whose second attempt starts under a new key with the same event and payload. Barriers order the steps; the ctest timeout is only a deadlock watchdog. On the current tree the owner-count check fails first, the fixture opens every gate and the row exits non-zero without hanging. No documentation sentence was replaced or deleted.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

… delivery progress does not wait for the ledger

The delivery worker constructed the curl multi handle, added requests, polled and persisted results on one thread, so every ledger call, including one that waits while the input thread holds the ledger mutex through a checkpoint, stopped curl from making progress. A response the receiver had already accepted could then be reported as a timeout, recorded as a failed delivery and sent again. A second thread now owns curl: it alone constructs, adds, polls and destroys the transport, exchanging data with the worker through two preallocated single-producer single-consumer rings of 256 entries with release/acquire publication, no lock and no wait on the other side. The worker keeps every ledger operation and the existing lock order: it reserves command space and a result credit before the started-row transaction, publishes the command after the row commits, and releases the credit only after the atomic finish and the retry decision. Active work covers commands not yet added, live handles and results awaiting the ledger, so a drain cannot end before every result is persisted; the finish request is still read before the ledger scans; cancellation and the drain deadline stop the owner from submitting queued sends; an exception in the owner is published once and surfaces through the existing worker-failure path; every exit stops and joins the owner after the ledger scopes have unwound. Acknowledgement still comes only from the atomic finish transaction, crash recovery is unchanged, and no timeout, retry, classification, store or checkpoint code changed. The service test's fake now returns no result until all four keys of its stop case exist, keeping every existing assertion. No documentation sentence was replaced or deleted.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>