Repository navigation
live runner: typed pre-growth stop for the ledger budget and flat pending counts - #597
Closed
luisleo526 wants to merge 4 commits into
Closed
luisleo526 wants to merge 4 commits into
luisleo526 wants to merge 4 commits into
Conversation
…ry status work independent of history The durable ledger checked its byte budget after a commit had already grown the file, defaulted to unlimited, and answered pending/unsent counts and the status aggregation by visiting the action and attempt rows. - Every ledger writer now reserves its payload and SQLite page overhead under the writer mutex before BEGIN and refuses the whole transaction with a typed storage-budget failure carrying used, requested and limit; nothing is written on refusal. A budget refusal latches for the instance. - Service mode requires a positive --max-ledger-bytes or an explicit --ledger-unbounded; ordinary runs keep their unlimited default. The rule sits behind one named function so the policy is a one-line change. - Pending and unsent counts, the oldest-pending lookup and the delivery metrics come from counters and indexes kept in the same transactions as the writes. A ledger written by the previous version is migrated once, the summaries are rebuilt from its rows and verified. - tests/test_native_live_store.cpp checks the reservation against real file growth, refusal without growth, the service-mode rule, the counters against a brute-force recount over a seeded operation sequence, constant rows read as the history grows tenfold, and migration of a ledger written by the previous code (tests/native_live_ledger_fixture_writer.cpp produces it). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…nd print the operands The reservation test compared the requested bytes with the real growth of a commit under a slack that did not follow the database's page size, and did not persist the page size the fixtures used. The slack is now a function of the page size and the pages and frames a commit touches, the fixtures set and read the same page size, and a failing check prints request, before, after, slack and page size. The estimator is unchanged: it is an upper bound and the test now checks that it stays within the derived slack. The page docs/live-ledger-budget.md stated a fixed slack ceiling in the sentence "Its independent slack ceiling for the two-event input fixture is ...", introduced by c9bfb29; that sentence is replaced by the page-size derived ceiling this commit introduces. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…thing that fits, and keep service mode as it was The reservation multiplied a worst-case B-tree path cost per mutation before BEGIN, so one input with one action reserved about 36 MB and a 1 GiB budget refused an input with about 30 actions at under 5% real usage: a false refusal. - Admission now happens immediately before COMMIT: a finite-budget connection runs with cache_spill off, the page cache is sampled after every trigger, split and overflow allocation as an upper bound on all the pages the transaction touches, and the commit is admitted or refused before the first WAL write; a refusal unwinds through the rollback and leaves the file unchanged. The cost is that the dirty pages stay in memory until COMMIT, which the documentation states. Unlimited runs keep normal spilling. The reservation function takes an extra payload margin and returns the admission callback a writer attaches to its transaction, so the checkpoint writer can use it. - The service-mode budget requirement ships off (today's behaviour) behind its one named function; --ledger-unbounded stays implemented; the three unrelated e2e edits that added it are reverted. The CHANGELOG says service mode does not require a budget in this release. - The migration reserves what it writes, the offline redeliver path goes through the reservation, new rows commit 1, 10 and 100 actions under a small budget, initialise a ledger under 64 MB and upgrade a base-written ledger under a finite budget; the documentation gives the measured order of the headroom and the one-way migration. This rewrites passages introduced by c9bfb29 and 792b229: the sentences that described the reservation before BEGIN and "deliberately exceeds" growth, the "Its independent slack ceiling for the two-event input fixture is" sentence, and the statement that service mode requires a budget. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…me write reports the same request The admission sampled sqlite3_memory_used(), a process-wide allocation total that includes clean cached pages, statements and other connections' allocations, and turned it into the page bound. That total can change without any ledger file changing (a rollback, a finalised statement), so a write refused with request R could be admitted at a limit of used + R - 1 on the next attempt. The ledger now observes the creation and destruction of its own SQLite page caches (the original cache methods are kept and called), releases unreferenced clean pages just before admission, and bounds the dirty set by the resident page count of its own cache, capped by the logical page count. With cache spilling off every dirty page stays resident and the pinned transaction header is counted, so the bound still covers what COMMIT can dirty. If another component has already initialised SQLite, or the page count is unavailable, the bound falls back to the logical page count (roughly twice the database size of headroom); the docs say so. No limit, slack or margin is loosened. The refusal rows now repeat the same limit of used + R - 1 twice in a row for every writer, a new row holds a 32 MiB unrelated allocation across the first refusal at three page sizes and requires the same request and unchanged files on the retries, the exact limit used + R admits the write, and the reservation headroom must stay under one historical payload so a fix that charges the whole history fails. The rewritten passages of docs/live-ledger-budget.md were introduced by 71d675b ("live ledger: admit a commit by the pages it really dirtied, refuse nothing that fits, and keep service mode as it was"): the memory_used() sampling sentences, the allocation-total and external-page-cache sentences, the conservative-fallback wording, the 512-page and 32-page 2 MiB allocation bounds, and the refusal-row sentence are replaced on purpose by the resident-page-count description above; the claims they carried (a refusal one byte below admission leaves all three files unchanged, a limit just above admission commits within its reservation) are kept and now stated for the resident-page bound. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
luisleo526
added a commit
that referenced
this pull request
Oct 28, 2026
…aling, UDT graphs, metrics replay and the CI admission fix (#616) This lands twelve engine pull requests as one change, verified and measured once on the combined tree: - #593 scratch: a fixed write-through read cache for the adapter's id tables - #594 live runner: a delta report stream by default, and report-path work that does not rescan closed trades - #597 live runner: typed pre-growth stop for the ledger budget and flat pending counts - #608 run_json: reserve the report channel, so strategy output on fd 1 cannot corrupt a report - #586 Pine UDT object graphs on the paged object Store - #578 native order: dormant working book, flat per-point cost for never-filled orders - #579 adapter: skip roster passes and empty-family scans that cannot change anything - #599 adapter: read projected pending orders through one cached duplicate index - #563 adapter: gate the reversal exit cancel on candidate owners instead of every historical id - #591 report sink: retire sealed closed-trade rows on sink routes when the module opts in - #596 metrics: stream the bar Sharpe and Sortino through an ordered, packed return replay - #615 ci: accept release.yml's matrix runner under its fetched path in PR admission Fixes found while testing the combined tree are in the final commits: the retiring ledger's cache sizing, a residency witness that bounds sink-run heap by the declared cache cap and checks zero growth at 100k, 200k and 1M trades, a test fix for a descriptor reuse race, and a standing-orders timing grid whose 20,000-order point is tracked in #621. The member pull requests are closed as landed here.
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. Live runner ledger budget: a typed stop before the ledger grows past its budget, a service-mode budget rule, and pending counts kept flat instead of rescanned. Docs: docs/live-ledger-budget.md. All 8 lab legs green; review and the merge of current main come before READY.
🤖 Generated with Claude Code