Repository navigation
scratch: a fixed write-through read cache for the adapter's id tables - #593
Closed
luisleo526 wants to merge 7 commits into
Closed
luisleo526 wants to merge 7 commits into
luisleo526 wants to merge 7 commits into
Conversation
The adapter's cohort and pair id tables answered every lookup with a real pread, so a run that re-reads the same ids paid one system call per lookup however hot the ids were. The tables now read through a small cache of 4 KiB logical pages that is fixed in size and never grows with bars, ids or trades: 64/64/32/32 KiB for the four tables, 192 KiB per adapter (not per engine). The cache is write-through. It never delays or reorders a write: owners invalidate every overlapping line before the real write, including a write that may fail after a prefix; growth, clear, move and a changed backend invalidate everything. Reads past the logical end go straight to the original reader so short-read, EIO and errno text and the copied prefix are unchanged. Slab and arena backends (including unjailed regular-file arenas and inherited or sealed memfds) bypass the cache without allocating. The default capacity of a table is zero; only the adapter opts its tables in. Tests compare every returned value and the resulting file image against the uncached path, with digests recorded from the unchanged engine, and assert that the cached run makes strictly fewer real read calls and never more bytes. They cover write, invalidation, end-of-file, bypass, the byte budget, the default, append and fold-append, file-size changes, eviction pressure and allocation failure. memory-management.md states the new fixed 192 KiB per adapter. The "Cohort id table" row of docs/pages/memory-management.md replaces two statements introduced by 30e9f9a (docs: memory-management page re-read against main after the 1.8.0 changes): "The ordered index lives on disk and is read through one 64 KiB window per scratch file" and the citation "`grow_index`, `include/pineforge/detail/id_pair_store.hpp:N`)". They are replaced because DiskResidentIdTable::find walks the disk tree on every lookup, even for resident rows, and those reads now go through the id-table page cache described in the new row above it; the row now says so, names exit_rank_keys_ as a separate plain DiskIdTable, and cites the index growth by symbol instead of by a line number that moves. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
# Conflicts: # docs/pages/memory-management.md
…he is invalidated The base-golden comments and failure messages still told a person to fill in the constants; they are recorded from the unchanged engine, so say that. The memory-management row said growth invalidates from the old tail page, which is true for appends only: index growth, clear, moves and backend changes invalidate every line. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The cmake.d checker requires every registered test name to have a source of the same name, so the twelve mode rows registered as separate tests (test_p2cx_sc412_write, ..._budget, ...) were refused. test_p2cx_sc412 now runs every row when it is started without an argument and still accepts a row name as the first argument to rerun one row; the source-level program keeps its own test. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The read cache allocates its page buffer lazily with new (std::nothrow), once per cache; the source gate found no entry for that site. U1524 records it as unreviewed (the fixed size is the author's statement: 4 KiB pages, at most 16 lines per table, and is not proven by the gate). The unreviewed ceiling becomes 1278 and the pinned OPEN count 1466 (1465 entries plus this one). ci/cr150-source-sites.json was re-derived with cr150_source_gate.py --emit-site-inventory; the gate over the result reports entries 1466, matched 1466, findings 0. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…the memory-management sentences it replaces The read cache adds three lines before the cited lines of pine_adapter.hpp and moves lines in id_pair_store.hpp, so 18 file:line citations in five pages are re-pointed at the lines that hold the same symbols (numbers only; no page changes its line count). The "Cohort id table" row of memory-management.md replaces two statements introduced by 30e9f9a (docs: memory-management page re-read against main after the 1.8.0 changes): "The ordered index lives on disk and is read through one 64 KiB window per scratch file" and the citation "`grow_index`, `include/pineforge/detail/id_pair_store.hpp:N`)". They are replaced because DiskResidentIdTable::find walks the disk tree on every lookup, even for resident rows, and those reads now go through the id-table page cache described in the new row above it; the row now says so, names exit_rank_keys_ as a separate plain DiskIdTable, and cites the index growth by symbol instead of by a line number that moves. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
luisleo526
added a commit
that referenced
this pull request
Oct 28, 2026
…aling, UDT graphs, metrics replay and the CI admission fix (#616) This lands twelve engine pull requests as one change, verified and measured once on the combined tree: - #593 scratch: a fixed write-through read cache for the adapter's id tables - #594 live runner: a delta report stream by default, and report-path work that does not rescan closed trades - #597 live runner: typed pre-growth stop for the ledger budget and flat pending counts - #608 run_json: reserve the report channel, so strategy output on fd 1 cannot corrupt a report - #586 Pine UDT object graphs on the paged object Store - #578 native order: dormant working book, flat per-point cost for never-filled orders - #579 adapter: skip roster passes and empty-family scans that cannot change anything - #599 adapter: read projected pending orders through one cached duplicate index - #563 adapter: gate the reversal exit cancel on candidate owners instead of every historical id - #591 report sink: retire sealed closed-trade rows on sink routes when the module opts in - #596 metrics: stream the bar Sharpe and Sortino through an ordered, packed return replay - #615 ci: accept release.yml's matrix runner under its fetched path in PR admission Fixes found while testing the combined tree are in the final commits: the retiring ledger's cache sizing, a residency witness that bounds sink-run heap by the declared cache cap and checks zero growth at 100k, 200k and 1M trades, a test fix for a descriptor reuse race, and a standing-orders timing grid whose 20,000-order point is tracked in #621. The member pull requests are closed as landed here.
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces #577; same tree, the first commit's message now names the two replaced sentences.
What
The adapter's cohort and pair id tables now read through a small, fixed-size, write-through page cache. Before this, every lookup in those tables was a real
pread, so a run that re-reads the same ids paid one system call per lookup however hot the ids were.Bounds
nothrow, and falls back to direct reads if it fails.docs/pages/memory-management.mdgains one row for this cache.Correctness
The cache never delays or reorders a write. Owners invalidate every overlapping line before the real write, including a write that may fail after a prefix has landed. Appends invalidate from the old tail page; index growth, clear, moves and a changed backend invalidate every line. Reads past the logical end go to the original reader, so short-read,
EIOanderrnotext and the copied prefix are unchanged.Tests
ctest -R test_p2cx_sc412(13 targets): write, invalidate, eof, bypass, budget, default, append, fold-append, fsize, thrash, alloc-failure, plus a source-level run through the adapter. Each compares every returned value and the final file image with the uncached path, using digests recorded from the unchanged engine (frozen base6b33c47f), and asserts that the cached run makes strictly fewer real read calls and never reads more bytes. The budget case forces evictions (working set larger than the slots) and also revisits a hot subset, so hits are guaranteed; it fails if the budget is ignored. On the unchanged engine the cache rows fail as intended (cache and direct counts equal, 20,761 preads each).Not covered
No behaviour change is intended: the returned bytes are the same and only the number of reads changes. The paired parity sweep for this head is posted separately.
🤖 Generated with Claude Code