live runner: report checkpoints so recovery does not replay the whole history - #598

… reads a bounded tail

Exporting the report of an earlier cursor rebuilt it from the whole stored
history, and the runner had no checkpoint structure.

- The ledger writes an immutable cumulative report checkpoint in the same
  transaction as the input commit at every configured boundary (default 1024
  inputs, --checkpoint-interval K). Report export starts from the nearest
  checkpoint at or below the cursor and applies only the deltas after it, so the
  rows read are at most the interval plus the checkpoint row.
- A checkpoint is verified against the report integrity chain before use; a
  payload or witness failure records a typed diagnostic and falls back to full
  reconstruction, which returns the same bytes. Ledgers written by the previous
  version open unchanged and get their first checkpoint at the next boundary.
- The checkpoint holds report state only: the engine has no state restore entry
  point, so restart recovery still replays committed strategy inputs and
  validates their state hashes and actions. This is stated in the documentation.
- tests/test_native_live_checkpoint.cpp compares exports with an independent
  full reconstruction for every cursor, bounds the rows read for a history ten
  times longer, refuses a corrupted checkpoint and falls back, opens an old
  ledger, and checks commit atomicity.
  tests/test_native_live_checkpoint_measurement.cpp is a diagnostic row that
  prints ledger bytes, WAL bytes, peak RSS, open descriptors and recovery and
  export times for growing input counts; it asserts only the structural bound.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…ry status work independent of history

The durable ledger checked its byte budget after a commit had already grown
the file, defaulted to unlimited, and answered pending/unsent counts and the
status aggregation by visiting the action and attempt rows.

- Every ledger writer now reserves its payload and SQLite page overhead under
  the writer mutex before BEGIN and refuses the whole transaction with a typed
  storage-budget failure carrying used, requested and limit; nothing is written
  on refusal. A budget refusal latches for the instance.
- Service mode requires a positive --max-ledger-bytes or an explicit
  --ledger-unbounded; ordinary runs keep their unlimited default. The rule sits
  behind one named function so the policy is a one-line change.
- Pending and unsent counts, the oldest-pending lookup and the delivery
  metrics come from counters and indexes kept in the same transactions as the
  writes. A ledger written by the previous version is migrated once, the
  summaries are rebuilt from its rows and verified.
- tests/test_native_live_store.cpp checks the reservation against real file
  growth, refusal without growth, the service-mode rule, the counters against a
  brute-force recount over a seeded operation sequence, constant rows read as
  the history grows tenfold, and migration of a ledger written by the previous
  code (tests/native_live_ledger_fixture_writer.cpp produces it).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

test_live_report uses the run-failure machinery but its target did not link
the library that defines it, so the kernel profile stopped at the link step.
It now links the same library as the neighbouring runner tests.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…e base answer of report_json

The corruption cases of the checkpoint test changed rows the verifier did not
read, so a corrupted payload and a broken integrity witness were not refused;
they now target the checkpoint row and the witness it is checked against, and
each must produce the typed diagnostic and a fallback to full reconstruction
that returns the right bytes. The checkpoint path also changed what
LedgerView::report_json answers for cursors without a checkpoint; the base
validation is restored, so the existing store test passes unchanged. Each check
prints its operands on failure.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…nd print the operands

The reservation test compared the requested bytes with the real growth of a
commit under a slack that did not follow the database's page size, and did not
persist the page size the fixtures used. The slack is now a function of the
page size and the pages and frames a commit touches, the fixtures set and read
the same page size, and a failing check prints request, before, after, slack and
page size. The estimator is unchanged: it is an upper bound and the test now
checks that it stays within the derived slack.

The page docs/live-ledger-budget.md stated a fixed slack ceiling in the sentence
"Its independent slack ceiling for the two-event input fixture is ...",
introduced by c9bfb29; that sentence is replaced by the page-size derived
ceiling this commit introduces.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…ke the delta stream opt-in

An external consumer reads the first line of the runner's stdout and refuses
the engine unless it is the cumulative pineforge-native-report/v1 document, so
the default of --report-jsonl cannot change shape in this release.

- Plain --report-jsonl and --report-jsonl=full print the cumulative document per
  committed input, byte for byte as before; --report-jsonl=delta prints the
  pineforge-native-report-delta/v1 document. Any other value is refused with the
  usage error. The documentation states each mode's per-input bound: the delta
  stream is O(new rows), the cumulative stream is O(cumulative document).
- The closed-trade high-water mark stays in every mode, so the runner no longer
  rescans the closed-trade prefix per input; the retained cumulative document
  exists only where the cumulative stream needs it.
- New tests: the delta stream's first line carries an integer input_cursor of at
  least 1 at the same commit point as the cumulative stream's first line; a run
  without --report-jsonl prints exactly one stdout line, the exit summary; a
  revised last closed trade, the closed-trade prefix regression error and the
  option values are covered.

This rewrites passages introduced by 4342247: the sentences that called plain
--report-jsonl the delta stream ("Delta JSONL is the default", "request
--report-jsonl=full to retain that cumulative output") and the heading "Per-input
report-path work is flat in the appended rows" now say the default is cumulative,
delta is opt-in, and that the runner no longer rescans the closed-trade prefix
per input while the delta output is linear in the appended rows.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

The delta stream line carries input_cursor as its first top-level key and the
stored ledger row has no such key; the runner README and the changelog said only
"an additional top-level integer". They now state the key order and that the
stored row differs.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…thing that fits, and keep service mode as it was

The reservation multiplied a worst-case B-tree path cost per mutation before BEGIN,
so one input with one action reserved about 36 MB and a 1 GiB budget refused an
input with about 30 actions at under 5% real usage: a false refusal.

- Admission now happens immediately before COMMIT: a finite-budget connection runs
  with cache_spill off, the page cache is sampled after every trigger, split and
  overflow allocation as an upper bound on all the pages the transaction touches,
  and the commit is admitted or refused before the first WAL write; a refusal
  unwinds through the rollback and leaves the file unchanged. The cost is that the
  dirty pages stay in memory until COMMIT, which the documentation states.
  Unlimited runs keep normal spilling. The reservation function takes an extra
  payload margin and returns the admission callback a writer attaches to its
  transaction, so the checkpoint writer can use it.
- The service-mode budget requirement ships off (today's behaviour) behind its one
  named function; --ledger-unbounded stays implemented; the three unrelated e2e
  edits that added it are reverted. The CHANGELOG says service mode does not
  require a budget in this release.
- The migration reserves what it writes, the offline redeliver path goes through the
  reservation, new rows commit 1, 10 and 100 actions under a small budget, initialise
  a ledger under 64 MB and upgrade a base-written ledger under a finite budget; the
  documentation gives the measured order of the headroom and the one-way migration.

This rewrites passages introduced by c9bfb29 and 792b229: the sentences that
described the reservation before BEGIN and "deliberately exceeds" growth, the
"Its independent slack ceiling for the two-event input fixture is" sentence, and
the statement that service mode requires a budget.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…t commit, with verification stated and reachable

Checkpoints held the whole cumulative report every K inputs (O(N^2/K) storage, written
inside the input transaction), and export and restart verified the chain only from the
nearest checkpoint without saying so.

- The schedule is geometric (K, 2K, 4K, ...), so total checkpoint storage is O(N); the
  option keeps its meaning as the base interval, the exact read bound per export is
  derived and asserted, and a test asserts the stored checkpoint bytes against a model.
- A checkpoint is written in a separate small transaction after the input commit: a
  failed write costs only the optimisation (typed diagnostic, the input stays durable and
  the service keeps running). On upgrade of an old ledger the first checkpoint is taken at
  open, not inside a live commit.
- The README states that export and restart verify the chain from the nearest checkpoint,
  `report --verify-full` runs the full verification, and a CheckpointFailure at startup is
  logged and published in the status file instead of being swallowed. Export uses
  MAX(input_index)+1 instead of COUNT(*); a test covers a checkpoint whose payload and hash
  are consistent with each other but wrong versus the deltas.

This rewrites passages introduced by 1df103f and 4c571b8: "This avoids quadratic report
storage", the interval-based tail bound, and the sentence that export re-verifies the
whole committed report-integrity chain.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…me write reports the same request

The admission sampled sqlite3_memory_used(), a process-wide allocation total
that includes clean cached pages, statements and other connections'
allocations, and turned it into the page bound. That total can change without
any ledger file changing (a rollback, a finalised statement), so a write refused
with request R could be admitted at a limit of used + R - 1 on the next attempt.

The ledger now observes the creation and destruction of its own SQLite page
caches (the original cache methods are kept and called), releases unreferenced
clean pages just before admission, and bounds the dirty set by the resident page
count of its own cache, capped by the logical page count. With cache spilling
off every dirty page stays resident and the pinned transaction header is
counted, so the bound still covers what COMMIT can dirty. If another component
has already initialised SQLite, or the page count is unavailable, the bound
falls back to the logical page count (roughly twice the database size of
headroom); the docs say so. No limit, slack or margin is loosened.

The refusal rows now repeat the same limit of used + R - 1 twice in a row for
every writer, a new row holds a 32 MiB unrelated allocation across the first
refusal at three page sizes and requires the same request and unchanged files on
the retries, the exact limit used + R admits the write, and the reservation
headroom must stay under one historical payload so a fix that charges the whole
history fails.

The rewritten passages of docs/live-ledger-budget.md were introduced by 71d675b
("live ledger: admit a commit by the pages it really dirtied, refuse nothing
that fits, and keep service mode as it was"): the memory_used() sampling
sentences, the allocation-total and external-page-cache sentences, the
conservative-fallback wording, the 512-page and 32-page 2 MiB allocation
bounds, and the refusal-row sentence are replaced on purpose by the
resident-page-count description above; the claims they carried (a refusal one
byte below admission leaves all three files unchanged, a limit just above
admission commits within its reservation) are kept and now stated for the
resident-page bound.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

The report path branch keeps the cumulative report stream as the default and
makes the delta stream opt-in. The checkpoint verification of this branch now
recomputes the cumulative document through the same full-document call the live
run uses, at the first cursor, at every replayed input and at the checkpoint
boundary, so a checkpoint is written and verified against the text a live run
prints. A restart row runs the default stream, kills the runner after seven
committed inputs, and requires the recovered report and every later report to
equal an uninterrupted run byte for byte, with the first line of each process
carrying the v1 schema and an integer input cursor of at least one.

Both changelog entries are kept as they were; the checkpoint payload joins the
ledger byte reservation when the budget branch is merged.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

The ledger budget admits every write before COMMIT by the pages it really
dirties. The checkpoint payload now sits inside that budget: the input that is a
checkpoint boundary is admitted together with the cumulative document its
checkpoint will store, the separate checkpoint transaction is admitted against
the ceiling that input reserved, and an open over a new finite budget fails with
the typed stop before any checkpoint is attempted. A checkpoint that cannot fit
its reservation is rolled back and counted as an optional write that failed; the
run continues and the budget is never exceeded.

New cases: the boundary input is refused at one byte under input plus
checkpoint, twice with identical requests and unchanged files, and admitted at
the exact sum; a checkpoint cannot spend past the first reservation; startup and
recovery checkpoints are admitted or refused like any write; an over-budget open
writes no checkpoint. The budget default stays unlimited and the service-mode
requirement stays off.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…ke rows moved it

The merged runner/CMakeLists.txt gained rows ahead of the native example batch
test, so the citation in the native feature parity design page now points at the
line where the test executable is declared. Only the line number changes.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…he default-ledger exact limit, state two cost choices

An optional checkpoint that is not written now reports a reason, one of budget,
build or io, with a count per reason beside the existing total, in the same
stderr line and status object as before; the existing keys are unchanged. A new
row admits a boundary input and its checkpoint at exactly the reported request on
a ledger with the production default auto-vacuum setting, and requires the
observed request not to exceed the request of the same write on a ledger that
uses the logical page bound. The README and the budget guide now state as
intended behaviour that the checkpoint payload is charged to its boundary input,
so an input that would fit alone can stop earlier, and that the startup and
post-replay checkpoints have no input reservation behind them and can spend most
of the remaining headroom without exceeding the budget.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…h reason is the latest, state what the default-ledger row proves

The recorder that counts a failed optional checkpoint takes a mutex, and a mutex
lock can in principle throw, so it can no longer be marked non-throwing. It stays
a mutex and every call site now goes through a wrapper that drops the diagnostic
if the lock throws, so a skipped optional checkpoint can never fail the run; when
the lock succeeds exactly one reason count and the total change together, and
every snapshot still satisfies budget + build + io == count.

The README and the budget guide say that `reason` names the most recent failure
while the three counts are cumulative since the ledger was opened and `count` is
their sum. The comments and the docs about the production-default exact-limit row
now say it shows the observed request is in the contract's allowed set, is not
larger than the logical-page request and is exact-limit safe, and that it does
not pin the estimator to the resident page count. Four citations that the new
lines moved are re-pointed.

The sentence introduced by 85cd873 ("live runner checkpoints: say why an
optional checkpoint failed, pin the default-ledger exact limit, state two cost
choices") in the budget guide, "`budget_count`, `build_count` and `io_count` sum
to the existing total `count`; the same fields appear in the status file and
stderr diagnostic JSON.", is replaced on purpose by the sentence that says
`reason` names the most recent failure while the three counts are cumulative
since the ledger was opened and `count` is their sum; the claim that the fields
appear in the status file and the stderr diagnostic is kept in the diagnostic
field description.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

…nner report checkpoints branch

The ledger open of main derives the table layouts it accepts from one list of the ledger's
tables; this branch added a thirteenth table, report_checkpoints, and a hand-written list of
counts at the same line. The merge keeps the derived rule only and adds report_checkpoints as the
last stage of the list: accepted counts are 3, 6, 7, 8, 9 (older runners), 11 and 12 (the delivery
summary layouts, with and without report_integrity or report_checkpoints) and 13 (this branch's
layout); every other count, including 10 and 14, is refused. The store rows of both sides are
reconciled to that set: the current layout is 13 tables, a pre-integrity layout with or without
checkpoints resumes with one verified replay, and a written layout plus unrelated tables is
refused. The changelog gets one bullet for the derived rule. No documentation sentence was replaced
or deleted.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>