Skip to content

Commit c9bfb29

Browse files
luisleo526claude
andcommitted
live runner: reserve ledger growth before it happens, and keep delivery status work independent of history
The durable ledger checked its byte budget after a commit had already grown the file, defaulted to unlimited, and answered pending/unsent counts and the status aggregation by visiting the action and attempt rows. - Every ledger writer now reserves its payload and SQLite page overhead under the writer mutex before BEGIN and refuses the whole transaction with a typed storage-budget failure carrying used, requested and limit; nothing is written on refusal. A budget refusal latches for the instance. - Service mode requires a positive --max-ledger-bytes or an explicit --ledger-unbounded; ordinary runs keep their unlimited default. The rule sits behind one named function so the policy is a one-line change. - Pending and unsent counts, the oldest-pending lookup and the delivery metrics come from counters and indexes kept in the same transactions as the writes. A ledger written by the previous version is migrated once, the summaries are rebuilt from its rows and verified. - tests/test_native_live_store.cpp checks the reservation against real file growth, refusal without growth, the service-mode rule, the counters against a brute-force recount over a seeded operation sequence, constant rows read as the history grows tenfold, and migration of a ledger written by the previous code (tests/native_live_ledger_fixture_writer.cpp produces it). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
1 parent 482b000 commit c9bfb29

14 files changed

Lines changed: 1112 additions & 72 deletions

‎CHANGELOG.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,16 @@ diagnostics keys, described below.
3636
- The release harness file `docker/run_json.py` changed (below). The hub image
3737
for this pair carries the 1.8.0 copy.
3838

39+
### Live ledger resource control
40+
41+
- Ledger writes now reserve payload and SQLite overhead before growth and
42+
refuse a whole transaction with a typed storage-budget failure. Service mode
43+
requires a positive `--max-ledger-bytes` or explicit `--ledger-unbounded`;
44+
ordinary runs keep their unlimited default.
45+
- Delivery status uses transactional counters and oldest-pending indexes,
46+
keeping status work independent of action and attempt history. Existing
47+
ledgers acquire verified summaries in an automatic migration.
48+
3949
### Run profile in the report diagnostics
4050

4151
- **Two additive success keys.** Every executed success report of

‎docs/design/native-feature-parity.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,7 @@ Columns: **Feature** (adapter mechanism, cited) · **Native today** (`yes` / `pa
121121
| CT4 | Partial-bar view / open-bar lookahead guard | **no** — the open hook receives the *complete* script bar: `invoke_bar_open_callback(engine, bar, point)` native_execution_consumer.cpp:8403 with `engine.current_bar_ = open_view` native_execution_consumer.cpp:8454 (the complete bar unless `OpenOnly`) **Closed:** `current_partial_bar()` native_host.hpp:1093 is the lookahead-free bar so far, through the last path point consumed (a fill inside a segment reads it at the segment's origin), and `NativeOpenBarView::OpenOnly` native_run_spec.hpp:109 masks `on_native_bar_open`'s bar to `H = L = C = open` as presentation only — the complete bar is restored before the open match (`tests/test_native_calc_timing.cpp`). | K | L5 | F:E3 | (single-source) The adapter needs the full bar, so the guard is opt-in. |
122122
| CT5 | Raw input observation before aggregation | **yes** — `on_native_input` native_host.hpp:895, `NativeInputContext` native_host.hpp:793-798 | K | — | O:F3 | (single-source row) |
123123
| CT6 | Calculate on order fills (scheduler `PineScheduler::recalculate` pine_scheduler_native.cpp:767, `coof_recalculation_due` pine_scheduler_native.cpp:733, the first-open chain `kFirstOpenLoopGuard` pine_scheduler_native.cpp:888; `begin_coof_recalc` pine_adapter.cpp:7114, `flush_coof_tail` pine_adapter.cpp:7536, `end_coof_recalc` pine_adapter.cpp:7171) | **partial** — `on_native_applied` native_host.hpp:958 fires mid-path after each fill native_execution_consumer.cpp:8307, native_execution_consumer.cpp:8307; commands are legal there `commands_allowed` native_execution_consumer.cpp:1966; a request born there is eligible on the unconsumed rest of the bar `born_on_remaining_path` native_execution_consumer.cpp:6857, `remaining_path_delivery` native_order.cpp:4347-4355. Missing: calculation re-entry, a documented chronology, a bar-so-far view **Closed:** `NativeCalculationTrigger::BarCloseAndFills` native_run_spec.hpp:96 drives one `OrderFill` recalculation per applied execution at that event's own cursor, delivered as `on_native_recalculate(..., OrderFill, cause)` and bounded by `max_recalculations_per_point` (`tests/test_native_calc_timing.cpp`). | K+A | L5 | F:E4 O:F5 S:C2 | **Disagree:** F partial / O, S no; F, S v1 / O deferrable. R5-5: v1, sub-bar hook + chronology contract, adapter path unchanged. TV parts → CT11. |
124-
| CT7 | Calculate on every tick | **partial** — `on_native_tick` native_host.hpp:898 fires per accepted realtime print native_execution_consumer.cpp:8542; no per-tick calculation in batch; the guide pinned "close-only" (`docs/pages/native-engine.md`, "Batch OHLCV vs ticks vs quiet") **Closed:** `NativeCalculationTrigger::EveryModeledPoint` native_run_spec.hpp:97 adds one `Tick` recalculation at every modeled point of the delivered path and at every observed print, in batch and in a stream alike (`tests/test_native_calc_timing.cpp`). | K | L5 | F:E5 O:F4 S:C3 | **Disagree:** F yes / O partial / S no. R5-5: v1. Not an adapter feature either: absent from `PineStrategyConfig` pine_adapter.hpp:67-81, the runner rejects it (`calc_on_every_tick` main.cpp:259), the Pine stream route refuses `calc_on_order_fills` pine_strategy_host.cpp:608. |
124+
| CT7 | Calculate on every tick | **partial** — `on_native_tick` native_host.hpp:898 fires per accepted realtime print native_execution_consumer.cpp:8542; no per-tick calculation in batch; the guide pinned "close-only" (`docs/pages/native-engine.md`, "Batch OHLCV vs ticks vs quiet") **Closed:** `NativeCalculationTrigger::EveryModeledPoint` native_run_spec.hpp:97 adds one `Tick` recalculation at every modeled point of the delivered path and at every observed print, in batch and in a stream alike (`tests/test_native_calc_timing.cpp`). | K | L5 | F:E5 O:F4 S:C3 | **Disagree:** F yes / O partial / S no. R5-5: v1. Not an adapter feature either: absent from `PineStrategyConfig` pine_adapter.hpp:67-81, the runner rejects it (`calc_on_every_tick` main.cpp:264), the Pine stream route refuses `calc_on_order_fills` pine_strategy_host.cpp:608. |
125125
| CT8 | Historical intrabar calculation (per lower-TF sub-bar / magnifier sample) | **no** — with an `IntrabarPath` the callback still fires once per script-bar calculation (`invoke_callback` native_execution_consumer.cpp:9308). **Closed:** `EveryModeledPoint` recalculates at each intrabar sample, and `on_native_sub_bar` native_host.hpp:948 fires once after each retained lower-timeframe sub-bar's whole matching path (`tests/test_native_calc_timing.cpp`). | K | L5 | F:E6 O:F8 S:C4 | — |
126126
| CT9 | Intrabar (magnifier) matching path + path-order policy (`volume_weighted_cap` pine_adapter.cpp:3120) | **yes** — `IntrabarPath` native_run_spec.hpp:362-403, delivery `deliver_confirmed_script` native_execution_consumer.cpp:8752-9018; `NativePathOrder` native_run_spec.hpp:335-339 | K+A | — | F:E7 F:G5 O:F7 S:C4 | **Disagree:** S partial (it wants callback cadence → CT8). TV sampler choice is A: TradingView's own intrabars at its table's timeframe, each owned by the chart bar of its last minute, built from the finer feed (`tradingview_magnifier_bars` magnifier_intrabars.hpp:46, R5 lane MAG-INTRABAR). |
127127
| CT10 | Batch → warmup → realtime forward execution | **yes** — `stream_begin` engine.hpp:1998, `stream_push_bar` engine.hpp:2004, `stream_end` engine.hpp:2008, `NativeRunPhase` native_host.hpp:41, C ABI `strategy_stream_begin` c_abi.cpp:781 | K+A | — | F:E8 O:J3 S:C5 | **Disagree:** S partial. R5-1: the lifecycle is native; Pine trade-start suppression and realtime-tail / probe rules (`trading_window_active` pine_strategy_commands.cpp:15-21, `prepare_native_begin` pine_strategy_host.cpp:548-702) are A. S's "neutral tail + trade-window policy" is single-source and unassigned (§4 Q8). |

‎docs/live-ledger-budget.md‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# Live ledger admission and delivery status
2+
3+
`pineforge-live run --max-ledger-bytes N` budgets the SQLite database, WAL and
4+
SHM files together. The default remains 0 (unlimited) for ordinary runs.
5+
`--status-file` enables service mode, which requires a positive budget or the
6+
explicit `--ledger-unbounded` opt-out. A positive budget still applies when both
7+
options are supplied. This deployment policy is controlled by the single
8+
`service_requires_ledger_budget` constant in `runner/store.hpp`.
9+
10+
Admission runs under the ledger's writer mutex before the transaction begins.
11+
It reads file sizes and SQLite page metadata, computes a conservative
12+
reservation, and refuses if `used + requested > limit`. Saturating arithmetic
13+
prevents overflow from admitting an oversized request. An already committed,
14+
identical input, routing document, report or redelivery request needs no write
15+
reservation. Initialization and upgrades reserve space before changing the
16+
journal mode or schema. Opening also reserves missing WAL/SHM space, using the
17+
existing WAL length and SQLite's minimum page size to bound index rebuilding
18+
before any SQL reader can allocate it.
19+
20+
The refusal is `LedgerBudgetExceeded`, a `coded<std::runtime_error>` carrying
21+
the existing `scratch_capacity_exceeded` code with `{"resource":"layout"}`.
22+
Its separate integer fields `used`, `requested` and `limit` describe the
23+
ledger admission; they also appear in the diagnostic JSON. They are not new
24+
arguments in the engine's unchanged failure-code catalog. Startup policy
25+
refusals use `run_request_invalid` with `{"option":"arguments"}` and exit 1;
26+
storage admission refusals exit 3. An admission refusal cancels delivery and
27+
stops the advanced strategy instance. The first refusal is latched under the
28+
ledger mutex, so another writer thread cannot add rows while shutdown catches
29+
up. Restart replays the committed prefix. Finite budgets disable SQLite's
30+
implicit checkpoint on close; the durable WAL is retained for recovery, and
31+
checkpoint work during admitted commits is included in their reservations.
32+
If admission fails during initialization, status publication may not yet have
33+
started. The existing readiness field `storage_below_budget` becomes false
34+
when a running service stops for storage, including failed admission.
35+
36+
The reservation deliberately exceeds the expected file growth. SQLite may
37+
reuse pages or WAL frames, split a B-tree, or checkpoint previous WAL pages into
38+
the main file. Available disk space can therefore exceed the remaining ledger
39+
budget when admission stops. Size the budget for this headroom as well as the
40+
durable history; an estimate is never a promise of successful storage I/O.
41+
42+
The accounting uses these quantities:
43+
44+
| Quantity | Meaning |
45+
|---|---|
46+
| `B` | Sum of text byte lengths being inserted or rewritten. Input commits include input JSON, the decimal state hash, event IDs/payloads/routes/delivery IDs, the supplied report delta and its 64-byte digest. Event updates include the existing event ID, payload and error bytes. |
47+
| `W` | Conservative number of table/index mutations, including summary triggers: 2 per input, 24 per event, 6 per report/digest pair; 4 for a legacy attempt/failure, 24 for acknowledgement, 32 for starting an audited attempt, 64 for completion, 2 for routing, 4 per redelivery request. |
48+
| `P`, `C` | SQLite page size and logical page count. |
49+
| `T` | `floor(8*B/(P-64)) + 1` pages for text, copies in indexes, overflow pages and rounding. |
50+
| `H` | `floor(log2(C + T + 512*W)) + 1`, a binary-tree upper bound on height including space for growth. |
51+
| `D` | `T + W*8*(H+2)` dirty pages. The per-row overhead covers headers, sibling balancing, parent paths and splits. |
52+
53+
For the physical reservation, count `2*D*(P+24) + 32` bytes for WAL frames
54+
(24-byte frame headers), the WAL header and possible simultaneous checkpoint
55+
growth. Add `max(C*P - main_file_bytes, 0)` for checkpointing earlier logical
56+
pages. Reserve WAL-index regions up to
57+
`(floor(8*(floor(wal_file_bytes/(P+24)) + D)/32768) + 2)*32768`,
58+
less the existing SHM size when positive. Page-number and hash slots explain
59+
the eight bytes per indexed frame; the extra regions cover allocation
60+
rounding. Schema creation/migration additionally reserves the existing
61+
logical database bytes and `256 + 32*events + 16*delivery_log_rows` mutations.
62+
This is conservative per-row overhead, not measured SQLite packing density.
63+
64+
The ledger test exercises 512, 4096 and 65536-byte pages, short text and
65+
overflow records up to 800000 bytes. It compares reservations against real
66+
database/WAL/SHM growth, tries limits one byte below and above admission, and
67+
compares all three files byte for byte after refusals. Its independent slack
68+
ceiling for the two-event input fixture is
69+
`32*B + 56*16*66*(P+24) + 3*32768`; it also requires actual growth to fit the
70+
reservation and the admitted limit. The extra writer cases cover reports,
71+
routing, legacy delivery, audited attempts/results and redelivery requests.
72+
73+
Delivery status uses transactional `delivery_totals`, `delivery_targets` and
74+
`delivery_state` tables. Global pending/unsent counts require one singleton
75+
lookup. A partial index finds the first pending or unsent ordinal, and another
76+
finds the earliest known creation/attempt time per pending target. Unknown
77+
legacy times remain null. Service metrics visit only pending targets;
78+
`status` reads per-target totals and at most one indexed error row per target.
79+
Work is independent of input, backlog and attempt history, apart from B-tree
80+
lookups and the number of targets being returned.
81+
82+
Migration rebuilds these structures once from source rows and verifies counts
83+
before committing. A failed migration rolls back its counters. The read-only
84+
`status` command retains its old scan fallback for ledgers that have not yet
85+
been upgraded by `run`. The pre-existing refusal for incompatible chart-feed
86+
ledgers remains unchanged. Export/recovery report formats and event payloads
87+
do not change. See the [runner guide](../runner/README.md#service-operation)
88+
for lifecycle, delivery and shutdown behavior.

‎runner/CMakeLists.txt‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ if(TARGET SQLite3::SQLite3)
2525
else()
2626
set(_pf_sqlite_target SQLite::SQLite3)
2727
endif()
28-
target_link_libraries(pineforge_live_support PUBLIC ${_pf_sqlite_target} CURL::libcurl OpenSSL::Crypto Threads::Threads)
28+
target_link_libraries(pineforge_live_support PUBLIC PineForge::pineforge ${_pf_sqlite_target} CURL::libcurl OpenSSL::Crypto Threads::Threads)
2929
target_compile_options(pineforge_live_support PRIVATE -Wall -Wextra -Wpedantic)
3030
add_library(pineforge_live_startup STATIC native_startup.cpp)
3131
target_compile_features(pineforge_live_startup PUBLIC cxx_std_17)
@@ -173,7 +173,10 @@ if(PINEFORGE_BUILD_TESTS)
173173
target_include_directories(pineforge_live_legacy_test_support PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})
174174
target_compile_definitions(pineforge_live_legacy_test_support PUBLIC PINEFORGE_LIVE_LEGACY_TEST_API)
175175
target_link_libraries(pineforge_live_legacy_test_support PUBLIC
176-
${_pf_sqlite_target} CURL::libcurl OpenSSL::Crypto Threads::Threads)
176+
PineForge::pineforge ${_pf_sqlite_target} CURL::libcurl OpenSSL::Crypto Threads::Threads)
177+
add_executable(native_live_ledger_fixture_writer EXCLUDE_FROM_ALL
178+
${PROJECT_SOURCE_DIR}/tests/native_live_ledger_fixture_writer.cpp)
179+
target_link_libraries(native_live_ledger_fixture_writer PRIVATE pineforge_live_legacy_test_support)
177180
foreach(native_test test_native_live_store test_native_live_websocket test_native_live_routing test_native_live_service)
178181
add_executable(${native_test} ${PROJECT_SOURCE_DIR}/tests/${native_test}.cpp)
179182
if(native_test STREQUAL "test_native_live_store")

‎runner/README.md‎

Lines changed: 21 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -177,13 +177,22 @@ pineforge-live probe --status-file status.json --max-age 3 --ready
177177
(plus, after a fatal error or a storage-budget stop, the delivery drain); there is no claim that arbitrary strategy code is
178178
interruptible. Container liveness probes can detect a stalled control loop.
179179
- `--max-ledger-bytes N` budgets database + WAL + SHM bytes (0 disables it;
180-
default 0; integer 0..INT64_MAX). Crossing the budget removes readiness,
181-
stops intake after the current whole message and exits 3 without deleting
182-
inputs, actions or reports. Recovery still verifies the ledger before
183-
deciding to stop. It is a stop threshold, not a quota: one atomic message
184-
and the bounded delivery drain can exceed it. Reserve free space for both;
185-
an actual I/O failure exits 1. No automatic pruning or report retention
186-
policy is applied.
180+
default 0; integer 0..INT64_MAX). With `--status-file` (service mode), an
181+
omitted or zero budget is refused at startup with `run_request_invalid`
182+
unless `--ledger-unbounded` explicitly opts out. Ordinary runs retain the
183+
unlimited default. A positive budget remains effective even with the opt-out.
184+
Before each write transaction, the runner reserves payload bytes and
185+
conservative SQLite row/index, WAL, checkpoint and SHM overhead. Insufficient
186+
space refuses the whole transaction before writing, carries `used`,
187+
`requested` and `limit` in a typed `scratch_capacity_exceeded` failure, removes
188+
readiness and exits 3. Earlier committed inputs, actions and reports stay
189+
intact. Delivery is cancelled on an admission refusal; unfinished attempts
190+
remain recoverable. The existing post-growth check and
191+
`storage_below_budget` readiness field remain in use. An actual I/O failure
192+
exits 1. Initialization and migrations also need admission space; a refusal
193+
there can precede status publication. No automatic pruning or report
194+
retention policy is applied. See [ledger budget accounting](../docs/live-ledger-budget.md)
195+
for the reservation formula and its conservative headroom.
187196

188197
### Live redelivery control files
189198

@@ -236,6 +245,11 @@ ledger budget. Without `--control-dir`, `redeliver` remains offline, reads only
236245
the selected target's secret and refuses while the runner holds the lock.
237246

238247
The additive ledger tables/columns are migrated while holding its writer lock.
248+
Delivery counts and oldest-pending indexes are rebuilt once from the existing
249+
rows and verified in that migration transaction. Later delivery and status work
250+
uses the summaries and indexes; status visits targets rather than historical
251+
actions or attempts. Counters update in the same transaction as their source
252+
rows, including failed redelivery and legacy acknowledgements.
239253
This is a one-way runner upgrade: back up the ledger before upgrading; earlier
240254
runner versions cannot open the migrated database. Strategy/deployment identity
241255
and historical action payload bytes remain unchanged.

‎runner/delivery.cpp‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -68,8 +68,12 @@ DeliveryWorker::~DeliveryWorker() {
6868
}
6969

7070
void DeliveryWorker::check() const {
71-
if (worker_failed_.load(std::memory_order_acquire))
71+
if (worker_failed_.load(std::memory_order_acquire)) {
72+
try { std::rethrow_exception(error_); }
73+
catch (const LedgerBudgetExceeded&) { throw; }
74+
catch (...) {}
7275
throw std::runtime_error("webhook delivery worker failed; committed actions remain in the ledger");
76+
}
7377
}
7478

7579
void DeliveryWorker::limit_drain() {
@@ -141,15 +145,23 @@ void DeliveryWorker::run() {
141145
retries.clear();
142146
}
143147
if (!deadline && DeliveryClock::now() >= control_due) {
148+
std::exception_ptr budget_failure;
144149
controls.poll(deployment_, [&](const Json& request) {
150+
if (budget_failure) std::rethrow_exception(budget_failure);
145151
const auto target = request.at("target").text();
146152
const auto from = request.at("from").integer<std::uint64_t>();
147153
if (!targets_.count(target) || !from || request.at("failed_only").kind != Json::Kind::Bool)
148154
throw std::invalid_argument("invalid redelivery selection");
149-
return ledger_.request_redelivery(request.at("request_id").text(), target, from,
150-
request.at("failed_only").value == "true");
155+
try {
156+
return ledger_.request_redelivery(request.at("request_id").text(), target, from,
157+
request.at("failed_only").value == "true");
158+
} catch (const LedgerBudgetExceeded&) {
159+
budget_failure = std::current_exception();
160+
throw;
161+
}
151162
});
152163
control_errors_ = controls.errors();
164+
if (budget_failure) std::rethrow_exception(budget_failure);
153165
control_due = DeliveryClock::now() + std::chrono::milliseconds(100);
154166
}
155167
bool exhausted = false;

0 commit comments

Comments
 (0)