Skip to content

Commit 2e9c083

Browse files
luisleo526claude
andcommitted
Native journal: restore the one-page event refusal; the memory caches no longer carry the multi-page event
A single serialized journal event larger than one spill page is refused with EFBIG and stays resident again, exactly as on main. This removes the extent writer and reader added with the cache work (ExtFile write_pending and publish_append, the ByteWriter extent constructor and finish, the ByteReader extent constructor, the 64-bit remaining count and its representability checks), the continued-page selection and validation in the journal store, the test-only resident-threshold seam and set_journal_residency_for_test, the test that drove oversized events, and its registration. A separate change implements oversized events on its own. Everything else of the memory caches stays: the tail-window read-through caches and their statistics, the per-kind capacity policy, the successor-root cache, and the ordinary page-index arithmetic hardening. docs/pages/memory-management.md again states the one-page refusal in the places that described continuation chunks. Retire the sentences introduced by 878af53: "Older events read back through four decoded slots using 32,768 B transfer chunks" (kSpillPageBytes), "A larger serialized event continues across as many chunks as it needs, with one directory entry for the full event", "Page size does not limit event size; scratch backing and address-space exhaustion retain their typed failures", and "An event larger than the 32,768 B transfer buffer is stored across continuation chunks and does not fail merely because it leaves residency". They are replaced by the one-page EFBIG refusal of main. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
1 parent 878af53 commit 2e9c083

5 files changed

Lines changed: 54 additions & 674 deletions

File tree

‎docs/pages/memory-management.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -349,7 +349,7 @@ in backing.
349349
| --- | --- | --- | --- |
350350
| Trade ledger (`TradeLedger`), descriptor window | 64 rows (`kResidentRows`, `include/pineforge/trade_ledger.hpp:42`), and a fixed payload tail of the same size | 168 B per row descriptor (`kRecordBytes`), so 10,752 B for the rows and 10,752 B for the tail; the four strings of a row are variable | Retiring sink runs erase published rows and return complete head chunks, keeping at most one head chunk per extent plus the unsealed suffix; each head retains fewer than 4,096 B of the newest retired index and payload bytes per file (about 24 trade descriptors) until the next zero-fill block completes or the chunk holding them is returned; other runs retain 168 B plus text per closed trade |
351351
| Trade ledger, read cache | Default cap of 65,536 occupied entries with or without a report sink (`kSinkCacheRows = kDefaultCacheRows`; the row-setting ceiling `kMaxCacheRows` is 1,048,576). The ledger cap is policy-controlled: frozen host bytes for `PF_CACHE_KIND_LEDGER_CACHE` select the largest row count R with `R*kLedgerEntryBytes + ceil(R/256)*sizeof(void*) <= capacity_bytes`, including the private page directory. The host and standalone byte setters share `TradeLedger::cache_rows_for_capacity` (`include/pineforge/trade_ledger.hpp:61`); run begin applies the host result through `TradeLedger::set_cache_rows` (`src/native_execution_consumer.cpp:3207-3208`). Zero bypasses; nonzero settings below one entry plus one pointer and values converting above the row ceiling are typed setting errors. Storage is private: global index modulo the fixed cap chooses a slot; the first read creates `ceil(cap/256)` owning pointers, and the first touch of page `p` allocates `min(256,cap-256*p)` slots (`TradeLedger::cache_resident_bytes`, `include/pineforge/trade_ledger.hpp:126-132`). Page growth preserves entries through the 64/128/256 boundaries. Writes publish to backing before an optional cache fill. Retirement frees pages outside the retained window, bounding allocated slots by `min(cap,(ceil(retained_rows/256)+2)*256)`. Directory/page allocation failure permanently bypasses further cache allocation for that ledger; reads, appends and replacements continue through backing | 312 B per slot (`kLedgerEntryBytes`: 168 B record, 16 B key and generation, 128 inline string bytes); up to 79,872 B per page, with the last page clipped to the cap. The charge `resident_cache_bytes(cap)` is `sizeof(TradeLedger)+sizeof(State)+cap*312+ceil(cap/256)*sizeof(void*)`: the default optional allowance is 20,449,280 B on LP64, comprising 20,447,232 entry bytes (19.5 MiB) plus 2,048 table bytes. The fixed base is separate. `cache_resident_bytes()` reports the fixed base, allocated directory and allocated pages once in cache resident bytes. The base includes the descriptor window, payload tail and zero-row read workspace; allocator overhead, returned owning Trades and explicit legacy materialization are excluded. Accounted allocation never exceeds the charge. The cap is independent of trade and bar counts | Not applicable (a cache over the retained rows) |
352-
| Command journal | 2,048 events (`JournalStore::kMaxResident`, `src/native_order.cpp:1826`), in 32-event chunks (`kChunk`); spilling stops once the resident count is at most 128 (`kKeepResident`, a stopping threshold rather than a floor) | Not a constant: an event is a variant with variable payload and `sizeof` of a resident slot is not stated in the source and was not compiled. Plus one lazy 163,840 B spill buffer per journal: a 32,768 B write buffer and four 32,768 B reload buffers (`SpillStorage`) | Older events read back through four decoded slots using 32,768 B transfer chunks (`kSpillPageBytes`, `src/native_order.cpp:1846`). A larger serialized event continues across as many chunks as it needs, with one directory entry for the full event. Page size does not limit event size; scratch backing and address-space exhaustion retain their typed failures |
352+
| Command journal | 2,048 events (`JournalStore::kMaxResident`, `src/native_order.cpp:1757`), in 32-event chunks (`kChunk`); spilling stops once the resident count is at most 128 (`kKeepResident`, a stopping threshold rather than a floor) | Not a constant: an event is a variant with variable payload and `sizeof` of a resident slot is not stated in the source and was not compiled. Plus one lazy 163,840 B spill buffer per journal: a 32,768 B write buffer and four 32,768 B reload buffers (`SpillStorage`) | Older events, in pages of 32,768 B (`kSpillPageBytes`, `src/native_order.cpp:1778`) read back through four decoded page slots. An event whose serialized form exceeds one page is refused with `EFBIG` (about 1,360 external openings) and stays resident |
353353
| Order book | 4,096 rows | See the drain plateau below | Older rows |
354354
| Command-store resident prefix / read cache | Up to five 64 KiB payloads, each with at most 128 direct-mapped lines in `ReadThroughCache`. Default K is `5 * (65536 + 128 * (ReadThroughCache::line_bytes(512) - 512) + 2048)` bytes (`CachedScratchBacking::kCommandScratchMemoryBytes`, `src/command_scratch_cache.hpp`): the slab includes all shared tags, pins and replacement metadata; the 2 KiB allowance bounds each decorator's fixed control. For another representable build default C = `kCacheBytes`, the same bound is `K = 5 * (C + floor(C/512) * (ReadThroughCache::line_bytes(512) - 512) + 2048)`. Owners are trade, dependency, exit, lazy dormant book and lazy retained-sequence index. Pages larger than the selected payload (64 KiB by default) or smaller than 512 B bypass it | Independent of extent and bars. For internal payload capacity C and page size P, N = floor(C/P) and optional slab bytes are `C + N * (ReadThroughCache::line_bytes(P) - P)`; control bytes are separate. Capacities are `std::size_t` and page ids are 64-bit; no capacity-sized array enlarges the fixed object when the default is raised. Zero, sub-page or unrepresentable capacity bypasses. The host policy supplies independent inclusive allowances for the supported owners. Optional slab allocation failure selects direct backing I/O. The former 330 KiB K did not include the shared cache's larger metadata and no longer applies | Each disk store holds its prefix in the same slab until its extent exceeds the selected prefix capacity, then spills the prefix once and lends the slab to the shared read cache. Shared tags are constructed only after the entire prefix has landed. Failed spills retain the authoritative prefix for rollback and retry. Direct mode still charges logical extent against the same byte budget. Attached arenas retain direct mapped admission |
355355
| Id-table read caches | Defaults remain 192 KiB of page bytes **per adapter, not per engine** (`IdTableReadCache::kAdapterCacheBytes`, `include/pineforge/detail/id_table_read_cache.hpp`): the cohort AVL and exit-rank AVL use 64 KiB each, and erased minima and consumed cycles use 32 KiB each (data/index share one cache per pair store). Other instances default to zero capacity and allocate nothing. The detail constructors accept `IdTableReadCache::Policy{line_bytes, lines}`; defaults are 4 KiB lines and zero lines. Payload capacity is the product, independent of logical ids and backing extent. Typed `CacheCapacity` constructors and setup setters reserve the shared line metadata within the inclusive byte allowance and forward `Policy{4096, lines}`; zero and sub-line allowances bypass, and line counts are clipped only to representable allocation geometry. The existing byte-count constructors retain their 4 KiB alignment and 64 KiB ceiling. Each adapter uses the shared `ReadThroughCache` with direct mapping and immediate owner writes. Lazy optional-allocation failure takes the exact original backing read; a later logical read can retry allocation. Slab and arena paths bypass allocation and cache lookup, including unjailed regular-file arenas and inherited sealed memfds | The shared allocation is exactly `lines * ReadThroughCache::line_bytes(line_bytes)`, including payload, tags, pins and replacement metadata; the 192 KiB default names payload only. Add `sizeof(IdTableReadCache)` per instance (including the shared cache control), plus a 4 KiB stack transfer workspace per active read. Policies wider than 4 KiB also use one optional line-sized transfer allocation, separately counted; its allocation failure also reads directly. No separate private tags or payload cache remain, and no cache allocation grows with ids. One adapter owns each enabled cache; no sharing across threads. Every write reaches the file immediately; overlapping writes invalidate before I/O, appends invalidate from the old tail page onward, and index growth, clear, moves and backend changes invalidate all lines. Writes spanning more pages than the configured slots conservatively invalidate the following pages of that file too. These are configuration/layout bounds, not measurements | Not applicable (copy-in/copy-out reads over private disk files; formats and logical folds unchanged) |
@@ -594,7 +594,7 @@ or bounds the row.
594594
| `scratch_capacity_exceeded`, argument `resource` | A slab region ran out, more than 4,096 cohorts are live in the bounded profile, or an id is longer than 64 bytes in the bounded profile (refused before any store change). Admit larger maxima or shorter ids before another attempt |
595595
| `report_sink_failed`, reason `arena_exhausted` | The inherited arena filled. A host may re-place the run on a larger arena; one that makes a single attempt returns the failure |
596596
| `scratch_io_failed` | A scratch read, write, create or resize failed, including real syscall `ENOMEM`, or the command-scratch byte cap was reached |
597-
| `report_sink_failed`, reason `scratch_io` | The native command journal's scratch read or write failed, or its stored bytes or page directory are damaged. An event larger than the 32,768 B transfer buffer is stored across continuation chunks and does not fail merely because it leaves residency. Genuine file-offset exhaustion, disk errors and missing continuation bytes remain typed failures |
597+
| `report_sink_failed`, reason `scratch_io` | The native command journal's scratch read or write failed, or its stored bytes or page directory are damaged. An event whose serialized form exceeds the 32,768 B page is refused with `EFBIG` and stays resident. Genuine file-offset exhaustion, disk errors and missing event bytes remain typed failures |
598598
| `scratch_configuration_invalid` | `PINEFORGE_COMMAND_SCRATCH_MAX_BYTES` is not a positive decimal |
599599
| `out_of_memory` | A required C++ allocation failed, including script scratch, arena directories/metadata, store workspace, owning decode, caller-owned output values or error-diagnostic construction. It is not relabelled as I/O or a capacity failure |
600600

‎src/native_execution_consumer.hpp‎

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2718,11 +2718,6 @@ class NativeExecutionConsumer final : public IExecutionConsumer {
27182718
// The transcript witness reads the Limit-class resident watermark only.
27192719
std::uint64_t dormant_watermark_for_test() const noexcept;
27202720

2721-
// Internal fixture seam, frozen before the first event. Production uses
2722-
// the journal's existing defaults; these sizes never enter run state.
2723-
static void set_journal_residency_for_test(native_order::WorkingRequestCore&,
2724-
std::size_t maximum, std::size_t keep);
2725-
27262721
using DormantReach = bool (*)(bool ge, double level, double lo,
27272722
double hi, double delta) noexcept;
27282723
// GE means level >= lo - delta; LE means level <= hi + delta.

0 commit comments

Comments
 (0)