Repository navigation
Commit ba14721
Add selected-capacity bounds, bounded drawings and native history storage
Integrated engine branch for the next release. It adds the selected-capacity input driver with three of its five bounds (primary rows, auxiliary rows and script bars), and it carries the work of the earlier draft pull request #734 (the head of its 1.11 engine batch is an ancestor of the integration line that this one commit squashes): native history storage, bounded ledger caching, bounded drawing creation with typed failures, and several runtime-scope and request fixes. It supersedes #734 and #745, which carry this work on older bases; the parts that are deferred are named below.
Strategy modules built against 1.10.x must be rebuilt (the existing engine and codegen pairing rule): the C++ script ABI epoch moves from `engine_script_run_v19` to `engine_script_run_v20`, an explicit exception to the earlier promise of one epoch throughout 1.x. The C API only gains functions, and the caller-allocated C report and trade layouts are unchanged.
## Selected-capacity inputs
A caller can bound the actual CSV inputs of a selected run. The driver (`src/selected_capacity_driver.*`) reads every supplied stream through a private single-pass chart CSV reader (`src/selected_capacity_csv_reader.*`), validates each record, and offers it to the observer of every bound that is present, on a tentative state. The primary sink receives a row only after every reservation for that row has succeeded, so a refused row is never materialized and neither a counter nor the stream state is committed for it. An absent bound imposes nothing; an explicit zero is finite. Source and auxiliary rows never reach the sink.
Each owner reserves through the shared `reserve()` in `src/selected_capacity_core.cpp` and raises a canonical typed failure (class resource, not retryable, since 1.11.0; string arguments `limit`, `used` and `requested`, where `limit` and `used` are the bound and the count before the attempted reservation and `requested` is the count asked for):
| bound | owner | code | English |
| --- | --- | --- | --- |
| primary rows | `src/selected_capacity_primary_rows.cpp` | `selected_primary_rows_limit_exceeded` (85) | Selected primary-row capacity exceeded. |
| auxiliary rows | `src/selected_capacity_auxiliary_rows.cpp` | `selected_auxiliary_rows_limit_exceeded` (83) | Selected auxiliary-row capacity exceeded. |
| script bars | `src/selected_capacity_script_bars.cpp` | `selected_script_bars_limit_exceeded` (86) | Selected script-bar capacity exceeded. |
The script-bar bound counts the script buckets that the supplied primary rows open, by their native script bucket key: pre-roll rows, rows that are trimmed later and an unsealed final bucket all count; session-anchored buckets follow the session open; a bucket that continues from one primary stream into the next counts once, because the previous bucket is kept in the run state.
The codes `selected_source_rows_limit_exceeded` (87) and `selected_bytes_limit_exceeded` (84) are in the catalog and the registry. Their owners, and the production entry that binds all five bounds (`materialize_selected_capacity` and `make_product_observer_table`, declared in `src/selected_capacity_driver.hpp`), are not part of this change. The ownership manifest, the release diff of the failure catalog and the generated tables of the failure-code page are brought up to date with the catalog: they list the five selected-capacity codes, the two runtime history storage codes and the map capacity code.
Tests: `test_selected_capacity_primary_rows`, `test_selected_capacity_auxiliary_rows` and `test_selected_capacity_script_bars`, registered under the `selected_capacity` label. Their expected values come from the tests' own oracles over timestamps written into their CSV text; the script-bar test checks the exact limit, the limit minus one, an absent bound, an explicit zero, the roles that must not charge, the early stop, session anchoring, equal input and script timeframes, a bucket split over two primary streams, strict feeds, the failure class, the arguments, the English text and the committed run state.
## Also in this branch
These changes were made on the batch branch before the selected-capacity work; they are listed here by their commit subjects, in order.
- Map capacity is a parameter, default no limit; RE10070 pine_map_capacity_exceeded at the parameter
- Assemble staged ledger, cache and provider source proposals
- Use technical terminology for ledger cache configuration
- Keep the descriptor layout docstring technical
- Track runtime context storage and refusal argument ownership
- Serialize quota observations and cover runtime context ownership
- Clarify cursor contract and implementation detail assertions
- Add deterministic two-host runtime scope isolation controls
- Clarify cursor oracle provenance and characterization coverage
- Read a deleted drawing as missing, return a typed status for a stale borrow, and add an opt-in bounded slot arena
- Report applied grading policies and missing observations
- Create drawings through a bounded arena: a refused creation returns the na handle and records a typed diagnostic
- Validate request timeframes before registration
- Regenerate the map failure catalog delta
- Keep timezone conversion coherent across sequential engine modules
- Describe the timezone mutex scope across separate engine images
- Preserve scratch-file layout (the cold-book frame of the same commit is not in this change; see below)
- Fix timeframe test registration-state observation
- Code the drawing slot-bound and arena-misuse throws as engine invariants
- Reconcile numeric catalog contracts with the selected release union
- Code the drawing id-exhaustion and declared-count throws
- Extend existing drawing failure vocabularies and messages
- Regenerate the additive drawing failure registry and release diff
- Avoid scanning unchanged chunks during object growth
One part of the map capacity change is deferred to the next release: the 120,000-pair map command line row (`test_cr559_map_capacity_cli`, with its script `tests/cr559_map_capacity_cli.py` and fixture module) is not in this change, because it needs the bounded scratch cache redesign. It runs the command line runner with maps of 120,000 pairs, 50,001 and 50,000 pairs and two typed refusals; in a Debug build on x86-64 its cost grows by about 2.5 for each doubling of the pairs (5,000 pairs 26.0 s, 10,000 pairs 66.1 s, 20,000 pairs 181.6 s), because the bounded cache mode of `include/pineforge/detail/scratch_file.hpp` discards the whole file's page cache after every 64 KiB of clean reads and every 64 KiB of writes and a map insertion touches about 17 pages. The row is not kept under a `slow` label to hide that cost. The other eight map capacity rows (default, resident, limit, rehash, rollback, error context, run and calc on fill) are in this change and pass.
The cold-book frame is deferred to the next release too. The commit "Preserve scratch-file layout and encode complete cold-book archive bytes" also stored every cold order book row behind a 10-byte frame (version, mode, decoded length, then a raw or run-length payload) with its own codec and unit test, and two later commits tested its malformed-row faults against real native reads; none of that is in this change. The row `test_native_dormant_book_cost` compares the real `pread` calls of the last eight ticks of a forced eager run with those of an always-eager reference, within the spread of the reference. With the frame it fails ("persistent eager fallback preads differ from the eager reference") on the commit that adds it and on every later state that was measured; it passes on the four earlier states that were measured and on the branch with the codec replaced by the identity (no frame header), and it still fails with raw frames and with the previous scratch file header. Measured per tick (Debug build, x86-64, the book of 4,608 rows): in both measured states the forced run costs 44,648 reads in every tick of that window while the always-eager reference costs 35,998 in its ordinary ticks, and the row passes only if a burst of about 45,000 extra reads of the reference falls into the window (it does with the codec replaced by the identity, it does not with the frame). The two states differ only in the byte layout of the stored rows, and the bursts fall in different ticks. So the pass without the frame does not show that the two runs cost the same; the difference and the bursts are not explained yet, and the frame stays out until the row passes for a reason. The scratch-file layout change of the same commit is in this change: it keeps the scratch file inside the size that the id-pair store requires.
The calendar-bucket commit of the earlier draft #745, "Handle initial incomplete calendar request buckets", is not in this change either. It kept partial initial daily, weekly and monthly request buckets out of the historical batch state for continuous-session intraday chart input (edits in `src/source/pine_security_eval.cpp` and `src/source/pine_strategy_host.cpp`, and the row `test_security_initial_calendar_bucket`). On the integrated tree two rows fail with it: its own row and `test_adapter_security_route_conditions`, which exists on main and passes there. It is taken again with its own proof in a later release.
## Fixes made while integrating the branch
The branch was built and run as a whole for the first time: every target of the Debug profile, then every non-slow test row, on x86-64 and on aarch64. That run found the following, which is fixed in this change. None of it alters the selected-capacity behavior.
- **Script ABI epoch.** `PF_STREAM_ENGINE_LAYOUT_VERSION` moves from 3 to 4 and this branch changes the layout of the public `CacheDefaults` struct (`ledger_cache_max_bytes` and the cache policy classes), so the epoch moves: the inline namespaces of the engine, the host and the execution consumer are `engine_script_run_v20`, and the host capability macro is `PINEFORGE_HAS_NATIVE_STRATEGY_HOST_V20`. `tests/test_ledger_cache_epoch_layout.cpp` is registered and refuses a changed layout under an unchanged epoch; the state-hash domain tags keep their names. The public contract keeps the earlier promise visible as history and states the exception; the ABI page and the changelog say it too.
- **Live runner.** With the layout stamp at 4, `runner/main.cpp` (`Strategy::load`, described in `runner/README.md`) refuses a module that declares a nonzero engine layout stamp other than the runtime's before it creates a strategy; a module that omits the stamp or declares 0 keeps the public report path. Two rows that exist only in the profiles that build the live runner were brought to that policy: `native_live_engine_layout_e2e` pins stamp 4, and `native_live_old_layout_report_e2e` now asserts the refusal of its stamp-2 control module (exit status 1, the message line `strategy engine layout mismatch: module=2 runtime=4`, no output, no ledger), where it used to compare that module's report with the output of main.
- **Runtime context.** The runtime context service follows the engine's copy rule: a copy keeps its options and its enabled switch and nothing of a run (no contexts, no borrowed batch). A native stream keeps its begin mode for the whole run, so a stream feeds the live runtime history instead of being refused at `stream_begin`. Two fixtures of its tests state what the product does: a canceled quota ticket keeps the tracked endpoint, snapshot and factory JSON alive, so the ticket alone observes the same complete storage as with its client; and the managed race fixture finishes its client after the rejected overflow, because a host that sees a disconnect without a finish records a protocol refusal.
- **Pins and counts.** The C ABI count pins are 124 declarations and 104 implementations, with the documents and unit tests that state them; the ABI layout probe pins the sizes and offsets measured on Linux x86-64; the source-bounds inventory knows the owned deque as a custom container and the owned allocator's record pool, and raises its unreviewed groups from 1,354 on main to 1,403, nothing marked bounded; the failure-code count pin of `tests/test_run_failure_codes.cpp` is 88 (87 codes and the none value), a row that runs only where the private fixtures are checked out; two slow-labelled rows are named in the pinned list of `scripts/ci_verify.py`.
- **Memory-filesystem predicate.** The runtime context and the issued range state use the canonical predicate in `scratch_directory.hpp` instead of spelling the tmpfs and ramfs magic values again.
- **Ledger cache tests.** The expectations of the tests of the former cache (an omitted setting holds 65,536 rows, one slot per ordinal, a rewind frees only the pages outside the retained window, a move starts cold) are reconciled with the cache this branch builds: a provisional 64 MiB byte default that holds 215,070 entries in four-way sets, every page released on retirement, a move that hands over the hot entries. The expected values come from independent oracles in `tests/ledger_default_cache_rows.hpp`, not from the product. The test of the removed history-sized cache (`test_ledger_auto_cache_growth`) is deleted with its pin, and the cache's own bound test, `test_ledger_cache_bound`, which had never been built, is built and registered as one slow row that runs three seeded populations.
- **Owned allocator.** It keeps its allocation records in one vector that is reused after deallocation, where it allocated one record per allocation: the steady state of the RCI, median and mode windows no longer makes temporary allocations.
- **Documentation and rulings.** Citations of source lines are re-pointed; the failure-code page counts the 87 named codes and announces the eight new ones; ADR-0001's residual table rules `pine_drawing_id_exhausted` and the engine-invariant counter text; the failure-code vocabulary guard has a reviewed entry for the cache policy setter's pass-through of the engine's code; the C header describes the ledger cache default as the byte default.
- **Runtime history spill file.** The file that holds the runtime history once it no longer fits in memory is opened as an unnamed file (`O_TMPFILE`). Where the filesystem has no unnamed files (it answers `EOPNOTSUPP`, or `EISDIR` or `EINVAL` on older kernels), the run ended with `runtime history create spool: Operation not supported`. It now uses a named file that is removed as soon as it is open, the way the build without `O_TMPFILE` already did, and every other refusal is still raised. `test_runtime_history_spool_fallback` forces each of the three answers and checks that the history reads back from the file and that nothing is left under a name, and forces four other refusals and checks that each is raised. It is a Debug-build row, because it forces the refusals through a seam that release builds compile out.
- **Two-module timezone row.** `test_timezone_multi_module` runs the Python interpreter under the sanitizer runtime, and with leak detection on, LeakSanitizer reported the interpreter's own exit-time allocations as leaks. The row now runs with leak detection off, as the stop-equivalence row that also hosts Python already does. The instrumented modules are still checked for memory errors, and the C++ rows keep leak detection.
## Documentation
`docs/coverage.md` describes `tz_util::ScopedTimezone` in its module table and in the section "Time / Session / Timezone". Three sentences there are replaced because the guard changed: they said that it is a mutex-guarded `TZ` swap for thread-safe formatting, that it grabs a process-wide mutex and skips the `setenv` / `tzset` pair for a same-zone request, and that this is the only reason `pine_str_format_time` and the session helpers are safe to call from a multi-strategy harness (introduced by aa6ac08 and 6aa1d13). The page now says that the guard holds a per-image mutex for its whole scope, compares the actual `TZ` value, leaves `TZ` in place when the guard ends, and that separately linked engine images do not share the mutex.
`docs/pages/memory-management.md` describes the ledger read cache. Nine sentences there are replaced because the cache changed (introduced by 69cddc9 and 772766a): the engine defaults to 65,536 ledger entries, applied through `TradeLedger::set_cache_rows`; the default cap is 65,536 occupied entries (`kSinkCacheRows = kDefaultCacheRows`), with an allowance of 20,449,280 B on LP64 for 65,536 entries; the host and standalone byte setters share `TradeLedger::cache_rows_for_capacity` and run begin applies the host result through `set_cache_rows`; retirement frees pages outside the retained window, bounding allocated slots by `min(cap,(ceil(retained_rows/256)+2)*256)`; a directory or page allocation failure permanently bypasses further cache allocation for that ledger, while reads, appends and replacements continue through backing. The page now says that an omitted setting keeps the provisional byte default of 67,108,864 B (64 MiB), which holds 215,070 entries, applied through `CacheDefaults::apply_ledger_cache`; that a global ordinal chooses one four-way set; that retirement and a rewind release every optional page; and that an allocation failure bypasses the cache until `clear()` for an explicit row setting and until the retained population has doubled for the byte default. It also states that a copy of a ledger starts with a cold cache and that a move hands the whole cache over, so the target is as hot as the source was (on main a move left the target cold; `test_p2cx_tledger_budget` states the new behavior).
## Verification of the selected-capacity part
On the tree of this change (Ubuntu 24.04, GNU 13.3.0, CMake 3.28.3; x86-64 and aarch64): `ctest -N -L selected_capacity` lists exactly `test_selected_capacity_auxiliary_rows`, `test_selected_capacity_primary_rows` and `test_selected_capacity_script_bars` (main has no test with that label), and all three pass in the Debug profile of `scripts/ci_verify.py`; the kernel archive defines the script-bar owner (`observe_script_bars`) once, and the script-bar test links that definition and carries no stand-in of its own; `scripts/check_run_failure_codes.py --no-history` reports "87 codes, registry in sync, 94 files scanned for uncoded throws, OK". Six deliberate breakages of the script-bar owner (no role or bucket filter, rows counted instead of buckets, no enforcement, the wrong state field, a wrong code, a wrong English text) each make its test fail at the assertion that names the defect, and a test that gets a setup answer from the driver exits with status 2 instead of reporting a bound failure. Those breakages were run when the owner was written; the selected-capacity sources, the three tests, their fixture and their lane files are byte-identical in this change to the files they were run on.
The rest of the suite, the parity runs and the performance and memory results are those of this pull request's checks.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>1 parent 3080b40 commit ba14721
225 files changed
Lines changed: 21708 additions & 2357 deletions
File tree
- ci
- docker
- docs
- adr
- design
- pages
- examples/native
- include/pineforge
- detail
- source
- runner
- scripts
- src
- detail
- source
- tests
- cr150.d
- ownership
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
10 | 19 | | |
11 | 20 | | |
12 | 21 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
131 | 131 | | |
132 | 132 | | |
133 | 133 | | |
| 134 | + | |
134 | 135 | | |
135 | 136 | | |
136 | 137 | | |
137 | 138 | | |
138 | 139 | | |
| 140 | + | |
139 | 141 | | |
140 | 142 | | |
141 | 143 | | |
| |||
163 | 165 | | |
164 | 166 | | |
165 | 167 | | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
166 | 174 | | |
167 | 175 | | |
168 | 176 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
36 | | - | |
| 36 | + | |
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
| |||
129 | 129 | | |
130 | 130 | | |
131 | 131 | | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
132 | 144 | | |
133 | 145 | | |
134 | 146 | | |
| |||
529 | 541 | | |
530 | 542 | | |
531 | 543 | | |
532 | | - | |
533 | | - | |
534 | | - | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
535 | 547 | | |
536 | | - | |
| 548 | + | |
537 | 549 | | |
538 | 550 | | |
539 | 551 | | |
| |||
571 | 583 | | |
572 | 584 | | |
573 | 585 | | |
| 586 | + | |
574 | 587 | | |
575 | 588 | | |
576 | 589 | | |
| |||
708 | 721 | | |
709 | 722 | | |
710 | 723 | | |
711 | | - | |
| 724 | + | |
712 | 725 | | |
713 | 726 | | |
714 | 727 | | |
| |||
0 commit comments