Repository navigation
ledger: cheaper closed-trade history reads and fewer scratch writes per closed trade - #648
Conversation
…er closed trade
A script that scans closedtrades.entry_id(i) over every closed trade each
bar, and a streaming run that publishes and retires rows, paid for four
things in v1.9.0 that v1.6.3 never did:
- two calls into the engine and a library comparison per visit: add
TradeLedger::{entry_id,exit_id,entry_comment,exit_comment}_matches (range
check and compare in one call; the old symbols stay), route the four
BacktestEngine::closed_trade_*_equals wrappers through them, compare texts
of up to eight bytes as two words, keep the entry-id length in the cache
entry's generation word so that compare never reads the record line, and
prefetch the next cache slots of a sequential scan;
- a read cache that doubled with the population and discarded every entry at
each doubling, so a script reading the whole history reloaded every row from
disk after each growth (about 195,000 preads for 35,000 trades): keep the
slot mapping fixed (global modulo cache_rows) and allocate the storage in
256-slot pages behind a page table; the last page is clipped to the cap,
retirement frees pages outside the retained window, and the address space at
a given population is never above the doubling figure (a run caching 64
rows or fewer holds one page, about 62 KB more);
- duplicate descriptor writes: the host close-cause pass wrote every row back
even when the cause did not change, and the stored record was rewritten when
it equalled the new one byte for byte; write back only on a change (a row
below the sealed base still goes through replace for its refusal), skip the
identical descriptor store, and fill the cache slot of an appended or
replaced row from the Trade in memory when a reader already created the
cache;
- per-text payload writes: the texts of one appended row go out as one
contiguous write (same bytes, extents and resident tail), and retirement no
longer zeroes on disk the part of a retired text that the same operation's
head release zeroes or truncates again.
resident_cache_bytes(rows) now includes the page table (ceil(rows / 256)
pointers, 2,048 bytes at the default cap); the actual allocation never
exceeds it. Skipping duplicate writes also lowers the byte-keyed fdatasync
count (the write budget syncs every 64 KiB of successful bytes); durability
per byte is unchanged. Reports are byte-identical.
Source gate: one new named site, the page allocation in
State::ensure_cache_entry (manifest entry U1623, ceilings.unreviewed
1338 -> 1339, the OPEN pin in test_cr150_source_gate 1522 -> 1523); the page
table allocation re-uses the reviewed cache entry with its identity refreshed.
Three doc anchors that cite engine_orders.cpp lines are re-pinned.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The read-cache row of the memory page described the doubling cache. Rewrite it for the paged cache with a fixed slot mapping and the charge that includes the page table. This removes three sentences of the old row on purpose: - "Direct-mapped, allocated at the first read at the smaller of the cap and the next power of two" (introduced by a991b4e): the cache is now allocated in 256-slot pages behind a page table and never discards entries on growth; - "The declared bound is resident_cache_bytes(kSinkCacheRows): 65,536 x 312 = 20,447,232 B" (introduced by 1654497): the charge now adds the page table, ceil(cap / 256) pointers, 2,048 bytes at the default cap; - "Chosen by the engine, not settable from the C API; cache allocation grows lazily up to the declared constant cap" (introduced by 1654497): the first read creates the page table and each first-touched page allocates its slots; the cap is unchanged and still not settable from the C API. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…ive the l17b cache bytes The lane check for tests/cr150.d allows only list(APPEND TEST_SOURCES), pineforge_add_unit_test and set_tests_properties in a lane file, so the four new ledger rows register their tests that way and their syscall-wrapping link options live in lane fences, as the existing resident-command-scratch fence does. test_entry_id_equals_cost drops its sanitizer compile definitions: its in-process ratio is only printed, so there is nothing to skip. test_selected_metrics_l17b encoded the doubling read cache. The charge now includes the page table (ceil(cap / 256) pointers) and a 1,024-row run holds the table plus four whole 256-slot pages: add the table term to both expectations. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 2 advisory finding(s) below merit a look before merge.
Not checked: tests were not run; formal verification proved 0 of 6 changed function(s) (6 not verified).
Formal verification. PR-changed functions: 0/6 verified (0 proven, 0 may-equivalent, 0 distinguished) · 6 not verified (6 unsupported).
Not verified on this run: cache\_resident\_bytes (unsupported), replace (unsupported), resident\_cache\_bytes (unsupported), resize (unsupported), set\_cache\_rows (unsupported), on\_native\_applied (unsupported).
Graphify review — findings
Replaces the TradeLedger read cache's single up-front direct-mapped table with a lazily paged one: the first read builds a table of ceil(cap/256) owning pointers, and ensure_cache_entry allocates a 256-slot page only when a global index first lands in it, from within const readers. Retirement frees pages outside the retained window, and writes fill only a cache that already exists, so allocated slots stay under min(cap, (ceil(retained_rows/256)+2)*256), and the page table is sized from the cap at first allocation. Updates the memory-management charge formula, the shifted engine_orders.cpp line references in the docs, and the CR150 bounds ledger, which gains one reviewed-pending page allocation site (U1623) and raises the unreviewed ceiling to 1339.
Worth a look
- Cache page table is not resized when cache_rows changes after first allocation —
src/engine_orders.cpp:470· Escalate · medium- NOT verified (no proof, no reproducing execution) — needs human review
- Lazy cache page allocation races in const readers —
src/engine_orders.cpp:486· Escalate · medium- NOT verified (no proof, no reproducing execution) — needs human review
Review partial — this diff was larger than one review pass covers, so later files were not reviewed; some findings may be missing.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 3289 functions depend on the 1808 functions this change touches.
Health — this change adds coupling hotspots:
- worse:
bounded()— 8 callers, 4 callees - new:
same_texts()— 3 callers, 10 callees - new:
bounded_and_replace_controls()— 1 callers, 15 callees - new:
memory_fill_controls()— 1 callers, 14 callees - new:
append_population()— 1 callers, 14 callees - new:
failed_append_is_unpublished()— 1 callers, 13 callees - new:
refusal_controls()— 1 callers, 12 callees - new:
equivalence_rows()— 1 callers, 11 callees - …and 11 more — each is listed as a finding
Verification — 3289 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 3095 function(s) in the blast radius were not formally verified this run
Formal verification
Could not verify: Could not verify cache\_resident\_bytes.
The verifier did not have enough to check cache\_resident\_bytes, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver
Could not verify: Could not verify replace.
The verifier did not have enough to check replace, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver
Could not verify: Could not verify resident\_cache\_bytes.
The verifier did not have enough to check resident\_cache\_bytes, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver
Could not verify: Could not verify resize.
The verifier did not have enough to check resize, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver
Could not verify: Could not verify set\_cache\_rows.
The verifier did not have enough to check set\_cache\_rows, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver
Could not verify: Could not verify on\_native\_applied.
The verifier did not have enough to check on\_native\_applied, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver
· 18 grounded finding(s) anchored inline below; 1 more finding(s) on lines outside this diff (see the check run).
| } | ||
| check(refused, "Ledger accepted a sealed edit or retired-history read"); | ||
| } | ||
| void refusal_controls(const Directory& directory) { |
There was a problem hiding this comment.
refusal_controls()
fans out to 12 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| row.pnl = static_cast<double>(n); | ||
| return row; | ||
| } | ||
| Count same_packed_row(const TradeLedger& ledger, std::size_t i, const Trade& expected) { |
There was a problem hiding this comment.
same_packed_row()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| return actual.entry_id.size() + actual.entry_comment.size() | ||
| + actual.exit_comment.size() + actual.exit_id.size(); | ||
| } | ||
| void memory_fill_controls(const Directory& directory) { |
There was a problem hiding this comment.
memory_fill_controls()
fans out to 14 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| "Oversized memory-filled row lost numeric caching or skipped the string fallback"); | ||
| } | ||
|
|
||
| void scenario(const Directory& directory, bool read_history = false) { |
There was a problem hiding this comment.
scenario()
fans out to 9 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| == "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad", | ||
| "independent hash known-answer control failed"); | ||
| } | ||
| void scenario(const Directory& directory) { |
There was a problem hiding this comment.
scenario()
fans out to 7 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| } | ||
| } | ||
|
|
||
| void retiring_rewind_releases_suffix_pages() { |
There was a problem hiding this comment.
retiring_rewind_releases_suffix_pages()
fans out to 11 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| } | ||
| } | ||
|
|
||
| void retirement_keeps_only_the_clipped_last_page() { |
There was a problem hiding this comment.
retirement_keeps_only_the_clipped_last_page()
fans out to 9 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| } | ||
| } | ||
|
|
||
| void sparse_page_touches() { |
There was a problem hiding this comment.
sparse_page_touches()
fans out to 8 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| bounded(ledger); | ||
| } | ||
|
|
||
| void append_preserves_cached_rows() { |
There was a problem hiding this comment.
append_preserves_cached_rows()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| bounded(ledger); | ||
| } | ||
|
|
||
| void entry_id_length_oracle() { |
There was a problem hiding this comment.
entry_id_length_oracle()
fans out to 10 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
A script that scans closedtrades.entry_id(i) over every closed trade each
bar, and a streaming run that publishes and retires rows, paid for four
things in v1.9.0 that v1.6.3 never did:
TradeLedger::{entry_id,exit_id,entry_comment,exit_comment}matches (range
check and compare in one call; the old symbols stay), route the four
BacktestEngine::closed_trade*_equals wrappers through them, compare texts
of up to eight bytes as two words, keep the entry-id length in the cache
entry's generation word so that compare never reads the record line, and
prefetch the next cache slots of a sequential scan;
each doubling, so a script reading the whole history reloaded every row from
disk after each growth (about 195,000 preads for 35,000 trades): keep the
slot mapping fixed (global modulo cache_rows) and allocate the storage in
256-slot pages behind a page table; the last page is clipped to the cap,
retirement frees pages outside the retained window, and the cache holds one page, about 62 KB more, for a non-retiring run
of 64 rows or fewer, and up to three pages, about 240 KB, while a
retiring run's retained window crosses page boundaries (a fixed amount,
independent of the number of trades);
even when the cause did not change, and the stored record was rewritten when
it equalled the new one byte for byte; write back only on a change (a row
below the sealed base still goes through replace for its refusal), skip the
identical descriptor store, and fill the cache slot of an appended or
replaced row from the Trade in memory when a reader already created the
cache;
contiguous write (same bytes, extents and resident tail), and retirement no
longer zeroes on disk the part of a retired text that the same operation's
head release zeroes or truncates again.
resident_cache_bytes(rows) now includes the page table (ceil(rows / 256)
pointers, 2,048 bytes at the default cap); the actual allocation never
exceeds it. Skipping duplicate writes also lowers the byte-keyed fdatasync
count (the write budget syncs every 64 KiB of successful bytes); durability
per byte is unchanged. Reports are byte-identical.
Source gate: one new named site, the page allocation in
State::ensure_cache_entry (manifest entry U1623, ceilings.unreviewed
1338 -> 1339, the OPEN pin in test_cr150_source_gate 1522 -> 1523); the page
table allocation re-uses the reviewed cache entry with its identity refreshed.
Three doc anchors that cite engine_orders.cpp lines are re-pinned.
docs: memory page: the ledger read cache is paged, not doubled
The read-cache row of the memory page described the doubling cache. Rewrite
it for the paged cache with a fixed slot mapping and the charge that includes
the page table. This removes three sentences of the old row on purpose:
the next power of two" (introduced by a991b4e): the cache is now allocated
in 256-slot pages behind a page table and never discards entries on growth;
20,447,232 B" (introduced by 1654497): the charge now adds the page table,
ceil(cap / 256) pointers, 2,048 bytes at the default cap;
lazily up to the declared constant cap" (introduced by 1654497): the first
read creates the page table and each first-touched page allocates its
slots; the cap is unchanged and still not settable from the C API.
tests: the new test rows are registered through the repository's lane files and fences, and the
selected-metrics expectations for the ledger cache bytes are derived from the accounting formula
(the page table term included) instead of the old constant.
🤖 Generated with Claude Code