|
| 1 | +# Live ledger admission and delivery status |
| 2 | + |
| 3 | +`pineforge-live run --max-ledger-bytes N` budgets the SQLite database, WAL and |
| 4 | +SHM files together. The default remains 0 (unlimited) for ordinary runs. |
| 5 | +`--status-file` enables service mode, which requires a positive budget or the |
| 6 | +explicit `--ledger-unbounded` opt-out. A positive budget still applies when both |
| 7 | +options are supplied. This deployment policy is controlled by the single |
| 8 | +`service_requires_ledger_budget` constant in `runner/store.hpp`. |
| 9 | + |
| 10 | +Admission runs under the ledger's writer mutex before the transaction begins. |
| 11 | +It reads file sizes and SQLite page metadata, computes a conservative |
| 12 | +reservation, and refuses if `used + requested > limit`. Saturating arithmetic |
| 13 | +prevents overflow from admitting an oversized request. An already committed, |
| 14 | +identical input, routing document, report or redelivery request needs no write |
| 15 | +reservation. Initialization and upgrades reserve space before changing the |
| 16 | +journal mode or schema. Opening also reserves missing WAL/SHM space, using the |
| 17 | +existing WAL length and SQLite's minimum page size to bound index rebuilding |
| 18 | +before any SQL reader can allocate it. |
| 19 | + |
| 20 | +The refusal is `LedgerBudgetExceeded`, a `coded<std::runtime_error>` carrying |
| 21 | +the existing `scratch_capacity_exceeded` code with `{"resource":"layout"}`. |
| 22 | +Its separate integer fields `used`, `requested` and `limit` describe the |
| 23 | +ledger admission; they also appear in the diagnostic JSON. They are not new |
| 24 | +arguments in the engine's unchanged failure-code catalog. Startup policy |
| 25 | +refusals use `run_request_invalid` with `{"option":"arguments"}` and exit 1; |
| 26 | +storage admission refusals exit 3. An admission refusal cancels delivery and |
| 27 | +stops the advanced strategy instance. The first refusal is latched under the |
| 28 | +ledger mutex, so another writer thread cannot add rows while shutdown catches |
| 29 | +up. Restart replays the committed prefix. Finite budgets disable SQLite's |
| 30 | +implicit checkpoint on close; the durable WAL is retained for recovery, and |
| 31 | +checkpoint work during admitted commits is included in their reservations. |
| 32 | +If admission fails during initialization, status publication may not yet have |
| 33 | +started. The existing readiness field `storage_below_budget` becomes false |
| 34 | +when a running service stops for storage, including failed admission. |
| 35 | + |
| 36 | +The reservation deliberately exceeds the expected file growth. SQLite may |
| 37 | +reuse pages or WAL frames, split a B-tree, or checkpoint previous WAL pages into |
| 38 | +the main file. Available disk space can therefore exceed the remaining ledger |
| 39 | +budget when admission stops. Size the budget for this headroom as well as the |
| 40 | +durable history; an estimate is never a promise of successful storage I/O. |
| 41 | + |
| 42 | +The accounting uses these quantities: |
| 43 | + |
| 44 | +| Quantity | Meaning | |
| 45 | +|---|---| |
| 46 | +| `B` | Sum of text byte lengths being inserted or rewritten. Input commits include input JSON, the decimal state hash, event IDs/payloads/routes/delivery IDs, the supplied report delta and its 64-byte digest. Event updates include the existing event ID, payload and error bytes. | |
| 47 | +| `W` | Conservative number of table/index mutations, including summary triggers: 2 per input, 24 per event, 6 per report/digest pair; 4 for a legacy attempt/failure, 24 for acknowledgement, 32 for starting an audited attempt, 64 for completion, 2 for routing, 4 per redelivery request. | |
| 48 | +| `P`, `C` | SQLite page size and logical page count. | |
| 49 | +| `T` | `floor(8*B/(P-64)) + 1` pages for text, copies in indexes, overflow pages and rounding. | |
| 50 | +| `H` | `floor(log2(C + T + 512*W)) + 1`, a binary-tree upper bound on height including space for growth. | |
| 51 | +| `D` | `T + W*8*(H+2)` dirty pages. The per-row overhead covers headers, sibling balancing, parent paths and splits. | |
| 52 | + |
| 53 | +For the physical reservation, count `2*D*(P+24) + 32` bytes for WAL frames |
| 54 | +(24-byte frame headers), the WAL header and possible simultaneous checkpoint |
| 55 | +growth. Add `max(C*P - main_file_bytes, 0)` for checkpointing earlier logical |
| 56 | +pages. Reserve WAL-index regions up to |
| 57 | +`(floor(8*(floor(wal_file_bytes/(P+24)) + D)/32768) + 2)*32768`, |
| 58 | +less the existing SHM size when positive. Page-number and hash slots explain |
| 59 | +the eight bytes per indexed frame; the extra regions cover allocation |
| 60 | +rounding. Schema creation/migration additionally reserves the existing |
| 61 | +logical database bytes and `256 + 32*events + 16*delivery_log_rows` mutations. |
| 62 | +This is conservative per-row overhead, not measured SQLite packing density. |
| 63 | + |
| 64 | +The ledger test exercises 512, 4096 and 65536-byte pages, short text and |
| 65 | +overflow records up to 800000 bytes. It compares reservations against real |
| 66 | +database/WAL/SHM growth, tries limits one byte below and above admission, and |
| 67 | +compares all three files byte for byte after refusals. Its independent slack |
| 68 | +ceiling for the two-event input fixture is |
| 69 | +`32*B + 56*16*66*(P+24) + 3*32768`; it also requires actual growth to fit the |
| 70 | +reservation and the admitted limit. The extra writer cases cover reports, |
| 71 | +routing, legacy delivery, audited attempts/results and redelivery requests. |
| 72 | + |
| 73 | +Delivery status uses transactional `delivery_totals`, `delivery_targets` and |
| 74 | +`delivery_state` tables. Global pending/unsent counts require one singleton |
| 75 | +lookup. A partial index finds the first pending or unsent ordinal, and another |
| 76 | +finds the earliest known creation/attempt time per pending target. Unknown |
| 77 | +legacy times remain null. Service metrics visit only pending targets; |
| 78 | +`status` reads per-target totals and at most one indexed error row per target. |
| 79 | +Work is independent of input, backlog and attempt history, apart from B-tree |
| 80 | +lookups and the number of targets being returned. |
| 81 | + |
| 82 | +Migration rebuilds these structures once from source rows and verifies counts |
| 83 | +before committing. A failed migration rolls back its counters. The read-only |
| 84 | +`status` command retains its old scan fallback for ledgers that have not yet |
| 85 | +been upgraded by `run`. The pre-existing refusal for incompatible chart-feed |
| 86 | +ledgers remains unchanged. Export/recovery report formats and event payloads |
| 87 | +do not change. See the [runner guide](../runner/README.md#service-operation) |
| 88 | +for lifecycle, delivery and shutdown behavior. |
0 commit comments