Skip to content

ledger: cheaper closed-trade history reads and fewer scratch writes per closed trade - #648

Merged
luisleo526 merged 3 commits into
mainfrom
perf/closed-trade-ledger
Oct 30, 2026
Merged

luisleo526 merged 3 commits into
mainfrom
perf/closed-trade-ledger

Conversation

@luisleo526

@luisleo526 luisleo526 commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

A script that scans closedtrades.entry_id(i) over every closed trade each
bar, and a streaming run that publishes and retires rows, paid for four
things in v1.9.0 that v1.6.3 never did:

  • two calls into the engine and a library comparison per visit: add
    TradeLedger::{entry_id,exit_id,entry_comment,exit_comment}matches (range
    check and compare in one call; the old symbols stay), route the four
    BacktestEngine::closed_trade
    *_equals wrappers through them, compare texts
    of up to eight bytes as two words, keep the entry-id length in the cache
    entry's generation word so that compare never reads the record line, and
    prefetch the next cache slots of a sequential scan;
  • a read cache that doubled with the population and discarded every entry at
    each doubling, so a script reading the whole history reloaded every row from
    disk after each growth (about 195,000 preads for 35,000 trades): keep the
    slot mapping fixed (global modulo cache_rows) and allocate the storage in
    256-slot pages behind a page table; the last page is clipped to the cap,
    retirement frees pages outside the retained window, and the cache holds one page, about 62 KB more, for a non-retiring run
    of 64 rows or fewer, and up to three pages, about 240 KB, while a
    retiring run's retained window crosses page boundaries (a fixed amount,
    independent of the number of trades);
  • duplicate descriptor writes: the host close-cause pass wrote every row back
    even when the cause did not change, and the stored record was rewritten when
    it equalled the new one byte for byte; write back only on a change (a row
    below the sealed base still goes through replace for its refusal), skip the
    identical descriptor store, and fill the cache slot of an appended or
    replaced row from the Trade in memory when a reader already created the
    cache;
  • per-text payload writes: the texts of one appended row go out as one
    contiguous write (same bytes, extents and resident tail), and retirement no
    longer zeroes on disk the part of a retired text that the same operation's
    head release zeroes or truncates again.

resident_cache_bytes(rows) now includes the page table (ceil(rows / 256)
pointers, 2,048 bytes at the default cap); the actual allocation never
exceeds it. Skipping duplicate writes also lowers the byte-keyed fdatasync
count (the write budget syncs every 64 KiB of successful bytes); durability
per byte is unchanged. Reports are byte-identical.

Source gate: one new named site, the page allocation in
State::ensure_cache_entry (manifest entry U1623, ceilings.unreviewed
1338 -> 1339, the OPEN pin in test_cr150_source_gate 1522 -> 1523); the page
table allocation re-uses the reviewed cache entry with its identity refreshed.
Three doc anchors that cite engine_orders.cpp lines are re-pinned.

docs: memory page: the ledger read cache is paged, not doubled

The read-cache row of the memory page described the doubling cache. Rewrite
it for the paged cache with a fixed slot mapping and the charge that includes
the page table. This removes three sentences of the old row on purpose:

  • "Direct-mapped, allocated at the first read at the smaller of the cap and
    the next power of two" (introduced by a991b4e): the cache is now allocated
    in 256-slot pages behind a page table and never discards entries on growth;
  • "The declared bound is resident_cache_bytes(kSinkCacheRows): 65,536 x 312 =
    20,447,232 B" (introduced by 1654497): the charge now adds the page table,
    ceil(cap / 256) pointers, 2,048 bytes at the default cap;
  • "Chosen by the engine, not settable from the C API; cache allocation grows
    lazily up to the declared constant cap" (introduced by 1654497): the first
    read creates the page table and each first-touched page allocates its
    slots; the cap is unchanged and still not settable from the C API.

tests: the new test rows are registered through the repository's lane files and fences, and the
selected-metrics expectations for the ledger cache bytes are derived from the accounting formula
(the page table term included) instead of the old constant.

🤖 Generated with Claude Code

luisleo526 and others added 3 commits October 30, 2026 15:02
…er closed trade

A script that scans closedtrades.entry_id(i) over every closed trade each
bar, and a streaming run that publishes and retires rows, paid for four
things in v1.9.0 that v1.6.3 never did:

- two calls into the engine and a library comparison per visit: add
  TradeLedger::{entry_id,exit_id,entry_comment,exit_comment}_matches (range
  check and compare in one call; the old symbols stay), route the four
  BacktestEngine::closed_trade_*_equals wrappers through them, compare texts
  of up to eight bytes as two words, keep the entry-id length in the cache
  entry's generation word so that compare never reads the record line, and
  prefetch the next cache slots of a sequential scan;
- a read cache that doubled with the population and discarded every entry at
  each doubling, so a script reading the whole history reloaded every row from
  disk after each growth (about 195,000 preads for 35,000 trades): keep the
  slot mapping fixed (global modulo cache_rows) and allocate the storage in
  256-slot pages behind a page table; the last page is clipped to the cap,
  retirement frees pages outside the retained window, and the address space at
  a given population is never above the doubling figure (a run caching 64
  rows or fewer holds one page, about 62 KB more);
- duplicate descriptor writes: the host close-cause pass wrote every row back
  even when the cause did not change, and the stored record was rewritten when
  it equalled the new one byte for byte; write back only on a change (a row
  below the sealed base still goes through replace for its refusal), skip the
  identical descriptor store, and fill the cache slot of an appended or
  replaced row from the Trade in memory when a reader already created the
  cache;
- per-text payload writes: the texts of one appended row go out as one
  contiguous write (same bytes, extents and resident tail), and retirement no
  longer zeroes on disk the part of a retired text that the same operation's
  head release zeroes or truncates again.

resident_cache_bytes(rows) now includes the page table (ceil(rows / 256)
pointers, 2,048 bytes at the default cap); the actual allocation never
exceeds it. Skipping duplicate writes also lowers the byte-keyed fdatasync
count (the write budget syncs every 64 KiB of successful bytes); durability
per byte is unchanged. Reports are byte-identical.

Source gate: one new named site, the page allocation in
State::ensure_cache_entry (manifest entry U1623, ceilings.unreviewed
1338 -> 1339, the OPEN pin in test_cr150_source_gate 1522 -> 1523); the page
table allocation re-uses the reviewed cache entry with its identity refreshed.
Three doc anchors that cite engine_orders.cpp lines are re-pinned.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The read-cache row of the memory page described the doubling cache. Rewrite
it for the paged cache with a fixed slot mapping and the charge that includes
the page table. This removes three sentences of the old row on purpose:

- "Direct-mapped, allocated at the first read at the smaller of the cap and
  the next power of two" (introduced by a991b4e): the cache is now allocated
  in 256-slot pages behind a page table and never discards entries on growth;
- "The declared bound is resident_cache_bytes(kSinkCacheRows): 65,536 x 312 =
  20,447,232 B" (introduced by 1654497): the charge now adds the page table,
  ceil(cap / 256) pointers, 2,048 bytes at the default cap;
- "Chosen by the engine, not settable from the C API; cache allocation grows
  lazily up to the declared constant cap" (introduced by 1654497): the first
  read creates the page table and each first-touched page allocates its
  slots; the cap is unchanged and still not settable from the C API.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…ive the l17b cache bytes

The lane check for tests/cr150.d allows only list(APPEND TEST_SOURCES),
pineforge_add_unit_test and set_tests_properties in a lane file, so the four
new ledger rows register their tests that way and their syscall-wrapping link
options live in lane fences, as the existing resident-command-scratch fence
does. test_entry_id_equals_cost drops its sanitizer compile definitions: its
in-process ratio is only printed, so there is nothing to skip.

test_selected_metrics_l17b encoded the doubling read cache. The charge now
includes the page table (ceil(cap / 256) pointers) and a 1,024-row run holds
the table plus four whole 256-slot pages: add the table term to both
expectations.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
@luisleo526
luisleo526 marked this pull request as ready for review October 30, 2026 08:42
@luisleo526
luisleo526 merged commit 772766a into main Oct 30, 2026
18 of 20 checks passed

@graphify-labs graphify-labs Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 2 advisory finding(s) below merit a look before merge.

Not checked: tests were not run; formal verification proved 0 of 6 changed function(s) (6 not verified).

Formal verification. PR-changed functions: 0/6 verified (0 proven, 0 may-equivalent, 0 distinguished) · 6 not verified (6 unsupported).

Not verified on this run: cache\_resident\_bytes (unsupported), replace (unsupported), resident\_cache\_bytes (unsupported), resize (unsupported), set\_cache\_rows (unsupported), on\_native\_applied (unsupported).


Graphify review — findings

Replaces the TradeLedger read cache's single up-front direct-mapped table with a lazily paged one: the first read builds a table of ceil(cap/256) owning pointers, and ensure_cache_entry allocates a 256-slot page only when a global index first lands in it, from within const readers. Retirement frees pages outside the retained window, and writes fill only a cache that already exists, so allocated slots stay under min(cap, (ceil(retained_rows/256)+2)*256), and the page table is sized from the cap at first allocation. Updates the memory-management charge formula, the shifted engine_orders.cpp line references in the docs, and the CR150 bounds ledger, which gains one reviewed-pending page allocation site (U1623) and raises the unreviewed ceiling to 1339.

Worth a look

  • Cache page table is not resized when cache_rows changes after first allocation — src/engine_orders.cpp:470 · Escalate · medium
    • NOT verified (no proof, no reproducing execution) — needs human review
  • Lazy cache page allocation races in const readers — src/engine_orders.cpp:486 · Escalate · medium
    • NOT verified (no proof, no reproducing execution) — needs human review

Review partial — this diff was larger than one review pass covers, so later files were not reviewed; some findings may be missing.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 3289 functions depend on the 1808 functions this change touches.

Health — this change adds coupling hotspots:

  • worse: bounded() — 8 callers, 4 callees
  • new: same_texts() — 3 callers, 10 callees
  • new: bounded_and_replace_controls() — 1 callers, 15 callees
  • new: memory_fill_controls() — 1 callers, 14 callees
  • new: append_population() — 1 callers, 14 callees
  • new: failed_append_is_unpublished() — 1 callers, 13 callees
  • new: refusal_controls() — 1 callers, 12 callees
  • new: equivalence_rows() — 1 callers, 11 callees
  • …and 11 more — each is listed as a finding

Verification — 3289 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 3095 function(s) in the blast radius were not formally verified this run

Formal verification

Could not verify: Could not verify cache\_resident\_bytes.

The verifier did not have enough to check cache\_resident\_bytes, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver

Could not verify: Could not verify replace.

The verifier did not have enough to check replace, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver

Could not verify: Could not verify resident\_cache\_bytes.

The verifier did not have enough to check resident\_cache\_bytes, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver

Could not verify: Could not verify resize.

The verifier did not have enough to check resize, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver

Could not verify: Could not verify set\_cache\_rows.

The verifier did not have enough to check set\_cache\_rows, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver

Could not verify: Could not verify on\_native\_applied.

The verifier did not have enough to check on\_native\_applied, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: unsupported: member function — receiver/self (`this`) not synthesizable from the simple-type literal pools — abstaining rather than guessing a receiver

· 18 grounded finding(s) anchored inline below; 1 more finding(s) on lines outside this diff (see the check run).

}
check(refused, "Ledger accepted a sealed edit or retired-history read");
}
void refusal_controls(const Directory& directory) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — refusal_controls()

fans out to 12 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

row.pnl = static_cast<double>(n);
return row;
}
Count same_packed_row(const TradeLedger& ledger, std::size_t i, const Trade& expected) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — same_packed_row()

fans out to 6 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

return actual.entry_id.size() + actual.entry_comment.size()
+ actual.exit_comment.size() + actual.exit_id.size();
}
void memory_fill_controls(const Directory& directory) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — memory_fill_controls()

fans out to 14 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

"Oversized memory-filled row lost numeric caching or skipped the string fallback");
}

void scenario(const Directory& directory, bool read_history = false) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — scenario()

fans out to 9 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

== "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad",
"independent hash known-answer control failed");
}
void scenario(const Directory& directory) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — scenario()

fans out to 7 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

}
}

void retiring_rewind_releases_suffix_pages() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — retiring_rewind_releases_suffix_pages()

fans out to 11 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

}
}

void retirement_keeps_only_the_clipped_last_page() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — retirement_keeps_only_the_clipped_last_page()

fans out to 9 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

}
}

void sparse_page_touches() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — sparse_page_touches()

fans out to 8 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

bounded(ledger);
}

void append_preserves_cached_rows() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — append_preserves_cached_rows()

fans out to 6 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

bounded(ledger);
}

void entry_id_length_oracle() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — entry_id_length_oracle()

fans out to 10 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant