Skip to content

live runner: typed pre-growth stop for the ledger budget and flat pending counts - #597

Closed
luisleo526 wants to merge 4 commits into
mainfrom
runner/ledger-budget
Closed

luisleo526 wants to merge 4 commits into
mainfrom
runner/ledger-budget

Conversation

@luisleo526

Copy link
Copy Markdown
Collaborator

Draft. Live runner ledger budget: a typed stop before the ledger grows past its budget, a service-mode budget rule, and pending counts kept flat instead of rescanned. Docs: docs/live-ledger-budget.md. All 8 lab legs green; review and the merge of current main come before READY.

🤖 Generated with Claude Code

luisleo526 and others added 4 commits October 28, 2026 07:29
…ry status work independent of history

The durable ledger checked its byte budget after a commit had already grown
the file, defaulted to unlimited, and answered pending/unsent counts and the
status aggregation by visiting the action and attempt rows.

- Every ledger writer now reserves its payload and SQLite page overhead under
  the writer mutex before BEGIN and refuses the whole transaction with a typed
  storage-budget failure carrying used, requested and limit; nothing is written
  on refusal. A budget refusal latches for the instance.
- Service mode requires a positive --max-ledger-bytes or an explicit
  --ledger-unbounded; ordinary runs keep their unlimited default. The rule sits
  behind one named function so the policy is a one-line change.
- Pending and unsent counts, the oldest-pending lookup and the delivery
  metrics come from counters and indexes kept in the same transactions as the
  writes. A ledger written by the previous version is migrated once, the
  summaries are rebuilt from its rows and verified.
- tests/test_native_live_store.cpp checks the reservation against real file
  growth, refusal without growth, the service-mode rule, the counters against a
  brute-force recount over a seeded operation sequence, constant rows read as
  the history grows tenfold, and migration of a ledger written by the previous
  code (tests/native_live_ledger_fixture_writer.cpp produces it).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…nd print the operands

The reservation test compared the requested bytes with the real growth of a
commit under a slack that did not follow the database's page size, and did not
persist the page size the fixtures used. The slack is now a function of the
page size and the pages and frames a commit touches, the fixtures set and read
the same page size, and a failing check prints request, before, after, slack and
page size. The estimator is unchanged: it is an upper bound and the test now
checks that it stays within the derived slack.

The page docs/live-ledger-budget.md stated a fixed slack ceiling in the sentence
"Its independent slack ceiling for the two-event input fixture is ...",
introduced by c9bfb29; that sentence is replaced by the page-size derived
ceiling this commit introduces.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…thing that fits, and keep service mode as it was

The reservation multiplied a worst-case B-tree path cost per mutation before BEGIN,
so one input with one action reserved about 36 MB and a 1 GiB budget refused an
input with about 30 actions at under 5% real usage: a false refusal.

- Admission now happens immediately before COMMIT: a finite-budget connection runs
  with cache_spill off, the page cache is sampled after every trigger, split and
  overflow allocation as an upper bound on all the pages the transaction touches,
  and the commit is admitted or refused before the first WAL write; a refusal
  unwinds through the rollback and leaves the file unchanged. The cost is that the
  dirty pages stay in memory until COMMIT, which the documentation states.
  Unlimited runs keep normal spilling. The reservation function takes an extra
  payload margin and returns the admission callback a writer attaches to its
  transaction, so the checkpoint writer can use it.
- The service-mode budget requirement ships off (today's behaviour) behind its one
  named function; --ledger-unbounded stays implemented; the three unrelated e2e
  edits that added it are reverted. The CHANGELOG says service mode does not
  require a budget in this release.
- The migration reserves what it writes, the offline redeliver path goes through the
  reservation, new rows commit 1, 10 and 100 actions under a small budget, initialise
  a ledger under 64 MB and upgrade a base-written ledger under a finite budget; the
  documentation gives the measured order of the headroom and the one-way migration.

This rewrites passages introduced by c9bfb29 and 792b229: the sentences that
described the reservation before BEGIN and "deliberately exceeds" growth, the
"Its independent slack ceiling for the two-event input fixture is" sentence, and
the statement that service mode requires a budget.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…me write reports the same request

The admission sampled sqlite3_memory_used(), a process-wide allocation total
that includes clean cached pages, statements and other connections'
allocations, and turned it into the page bound. That total can change without
any ledger file changing (a rollback, a finalised statement), so a write refused
with request R could be admitted at a limit of used + R - 1 on the next attempt.

The ledger now observes the creation and destruction of its own SQLite page
caches (the original cache methods are kept and called), releases unreferenced
clean pages just before admission, and bounds the dirty set by the resident page
count of its own cache, capped by the logical page count. With cache spilling
off every dirty page stays resident and the pinned transaction header is
counted, so the bound still covers what COMMIT can dirty. If another component
has already initialised SQLite, or the page count is unavailable, the bound
falls back to the logical page count (roughly twice the database size of
headroom); the docs say so. No limit, slack or margin is loosened.

The refusal rows now repeat the same limit of used + R - 1 twice in a row for
every writer, a new row holds a 32 MiB unrelated allocation across the first
refusal at three page sizes and requires the same request and unchanged files on
the retries, the exact limit used + R admits the write, and the reservation
headroom must stay under one historical payload so a fix that charges the whole
history fails.

The rewritten passages of docs/live-ledger-budget.md were introduced by 71d675b
("live ledger: admit a commit by the pages it really dirtied, refuse nothing
that fits, and keep service mode as it was"): the memory_used() sampling
sentences, the allocation-total and external-page-cache sentences, the
conservative-fallback wording, the 512-page and 32-page 2 MiB allocation
bounds, and the refusal-row sentence are replaced on purpose by the
resident-page-count description above; the claims they carried (a refusal one
byte below admission leaves all three files unchanged, a limit just above
admission commits within its reservation) are kept and now stated for the
resident-page bound.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
luisleo526 added a commit that referenced this pull request Oct 28, 2026
…aling, UDT graphs, metrics replay and the CI admission fix (#616)

This lands twelve engine pull requests as one change, verified and measured once on the combined tree:

- #593 scratch: a fixed write-through read cache for the adapter's id tables
- #594 live runner: a delta report stream by default, and report-path work that does not rescan closed trades
- #597 live runner: typed pre-growth stop for the ledger budget and flat pending counts
- #608 run_json: reserve the report channel, so strategy output on fd 1 cannot corrupt a report
- #586 Pine UDT object graphs on the paged object Store
- #578 native order: dormant working book, flat per-point cost for never-filled orders
- #579 adapter: skip roster passes and empty-family scans that cannot change anything
- #599 adapter: read projected pending orders through one cached duplicate index
- #563 adapter: gate the reversal exit cancel on candidate owners instead of every historical id
- #591 report sink: retire sealed closed-trade rows on sink routes when the module opts in
- #596 metrics: stream the bar Sharpe and Sortino through an ordered, packed return replay
- #615 ci: accept release.yml's matrix runner under its fetched path in PR admission

Fixes found while testing the combined tree are in the final commits: the retiring ledger's cache sizing, a residency witness that bounds sink-run heap by the declared cache cap and checks zero growth at 100k, 200k and 1M trades, a test fix for a descriptor reuse race, and a standing-orders timing grid whose 20,000-order point is tracked in #621. The member pull requests are closed as landed here.
@luisleo526

Copy link
Copy Markdown
Collaborator Author

Landed in #616 (squash a991b4e on main), which carries this change with one verify and one parity run on the combined tree.

@luisleo526 luisleo526 closed this Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant