Skip to content

Commit f81e1f6

Browse files
authored
Merge branch 'main' into ci/doc-reverts-locale
2 parents 449c8b6 + 0a8ca1e commit f81e1f6

240 files changed

Lines changed: 22941 additions & 2388 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,15 @@ version number follows semantic versioning over the surfaces the
77

88
## Unreleased
99

10+
- **Strategy modules built against 1.10.x must be rebuilt.** The C++ script
11+
ABI epoch moves from `engine_script_run_v19` to `engine_script_run_v20`
12+
(and `PINEFORGE_HAS_NATIVE_STRATEGY_HOST_V19` to `..._V20`) because
13+
`PF_STREAM_ENGINE_LAYOUT_VERSION` moves from 3 to 4 and the layout of
14+
`CacheDefaults` changed. This is an explicit exception to the earlier
15+
promise of one epoch throughout 1.x, and the existing engine and codegen
16+
pairing rule applies. The C API only gains functions; the caller-allocated
17+
C report and trade layouts are unchanged, and the state-hash domain tags
18+
keep their names.
1019
- `pineforge-live` status files now report `metrics.feed_idle_ms`, the
1120
milliseconds since the last feed arrival, or null before the first; the
1221
consumer owns freshness. HTTP and WebSocket feeds now also default to no
@@ -132,6 +141,21 @@ version number follows semantic versioning over the surfaces the
132141
`copy`, `snapshot` or `restore`) instead of at creation, with the same typed failure
133142
`report_sink_failed` and `reason: arena_exhausted`. Scripts that create maps
134143
they never fill can now run where they used to fail at `map.new`.
144+
- The native runner seeks the next unsent delivery through the existing
145+
partial index, excluding delivered, journal-only and completed-failure
146+
history from the scheduler's cursor walk.
147+
- Native runner status reports database + WAL bytes, measured bytes per
148+
committed input, available disk and a growth-based exhaustion projection.
149+
`--ledger-warn-free-fraction F` (default 0.8) reports an advisory `WARNING`
150+
without changing readiness or exit status. The default allowance stays
151+
unlimited.
152+
- `--report-jsonl=delta-v2` adds versioned change-only report records with
153+
`ratios.as_of_input` on every record, outside the cumulative document, and
154+
a raw `--delta-base` bootstrap. The envelope bounds emitted bytes and rows
155+
serialized per input on the 10,000/100,000-input work populations.
156+
Ratio scheduling is a separate later change; whole-curve passes remain.
157+
Plain/full and v1 delta bytes are preserved. Ledger tables and checkpoint
158+
layout are unchanged.
135159

136160
## 1.9.1 — 2026-10-31
137161

‎CMakeLists.txt‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,11 +131,13 @@ set(PINEFORGE_KERNEL_SOURCES src/array.cpp
131131
src/reservation_expansion.cpp
132132
src/engine_run.cpp
133133
src/engine_security.cpp
134+
src/runtime_context.cpp
134135
src/engine_state_hash.cpp
135136
src/engine_stream.cpp
136137
src/engine_trade_accessors.cpp
137138
src/pending_order_mirror.cpp
138139
src/native_order.cpp
140+
139141
src/native_journal_codec.cpp
140142
src/engine_consumer.cpp
141143
src/market_driver.cpp
@@ -163,6 +165,12 @@ set(PINEFORGE_KERNEL_SOURCES src/array.cpp
163165
src/native_bar_views.cpp
164166
src/selected_report_view.cpp
165167
src/selected_window_plan.cpp
168+
src/selected_capacity_core.cpp
169+
src/selected_capacity_csv_reader.cpp
170+
src/selected_capacity_driver.cpp
171+
src/selected_capacity_primary_rows.cpp
172+
src/selected_capacity_auxiliary_rows.cpp
173+
src/selected_capacity_script_bars.cpp
166174
src/metrics_spill_curve.cpp
167175
src/report_scratch.cpp
168176
src/detail/o1_store.cpp

‎README.md‎

Lines changed: 19 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ The separate PineForge compiler, [`pineforge-codegen`](https://github.com/pinefo
3333
- **Open runtime.** The engine and native live runner are Apache-2.0. The separately distributed [PineForge compiler](https://github.com/pineforge-4pass/pineforge-codegen-oss/blob/main/LICENSE) is source-available under the PineForge Source License 1.2 from its release 1.3.0 (release 1.2.0 keeps the PineForge Source License 1.1, and releases up to 1.1.0 keep the PolyForm Noncommercial terms they shipped with); personal trading is free, and commercial use requires a separate license; the capital in an account a proprietary-trading firm or funded-trader program provides or allocates, including a challenge, evaluation or simulated account, is investment capital. Public reference strategies, benchmarks and validation tooling are available in their respective repositories; the community-script test set is not redistributed.
3434
- **Fast.** In-process, no interpreter: median **36× faster than PyneCore** on the 200 strategies both engines time (100 corpus probes and 100 held-out private strategies), measured at engine `35db01c8` on 2026-09-29 on an AWS c7a.8xlarge with PyneCore timed as a subprocess, its interpreter start-up included (PineForge runs a median 807k bars/s per strategy over its 201 slots with the bar magnifier on; [method](benchmarks/results/speed.md)). Parameter sweeps re-run a loaded `.so` with new inputs — no recompile, no fork.
3535
- **Deterministic to the bit.** Two runs with the same inputs produce identical trade lists.
36-
- **Yours to embed.** 167 `PF_API` declarations across six installed headers — `pineforge.h`'s 111 (91 of the 102 runtime exports and up to 20 per-strategy functions a generated module defines), eleven more runtime declarations in `execution_observer.h`, `selected_window.h` and `report_sink.h` (three, three and five), `native_c_api.h`'s 43 native-host declarations and `selected_window_plan.h`'s two planner declarations — an ABI that is append-only within a major version from 1.0 on. Call it from C, Python, Rust, Go, Node, Julia — or let an AI agent drive it over MCP.
36+
- **Yours to embed.** 169 `PF_API` declarations across six installed headers — `pineforge.h`'s 113 (93 of the 104 runtime exports and up to 20 per-strategy functions a generated module defines), eleven more runtime declarations in `execution_observer.h`, `selected_window.h` and `report_sink.h` (three, three and five), `native_c_api.h`'s 43 native-host declarations and `selected_window_plan.h`'s two planner declarations — an ABI that is append-only within a major version from 1.0 on. Call it from C, Python, Rust, Go, Node, Julia — or let an AI agent drive it over MCP.
3737

3838
Recorded outputs and their additional API declarations described here are
3939
part of **1.5.0**.
@@ -129,6 +129,18 @@ The [shared local/CI verifier](docs/ci.md) includes source guards, tests and ins
129129

130130
Prerequisites: CMake ≥ 3.16 to build (the verifier and `ctest --test-dir` need 3.20), a C++17 compiler whose standard library has floating-point `std::to_chars` (libstdc++ from GCC 11 or later, or libc++ 14 or later; CI builds with GCC on Ubuntu 24.04), Eigen 3.3+ (fetched automatically if absent), Python 3 for the tests (`-DPINEFORGE_BUILD_TESTS=OFF -DPINEFORGE_BUILD_TUTORIAL=OFF` for a library-only build). `cmake --install build --prefix /usr/local` installs `lib/libpineforge.a` and `lib/libpineforge_kernel.a`, `include/pineforge/`, and the `find_package(PineForge)` config.
131131

132+
Script maps have no pair limit by default. Their entry arenas grow in steps of
133+
4,096 slots over the spilling Store, and their bucket tables scale with capacity.
134+
Map sizes and slot links use 64-bit integers.
135+
Use `docker/run_json.py --limit-map-pairs 50000` for TradingView's pair policy;
136+
omitted or zero selects unlimited. C callers install the same creation policy
137+
with `pf_map_pairs_limit_set_checked` before the strategy factory and restore
138+
NULL afterward. C++ callers can override one map with
139+
`PineMap<Key, Value>::new_(maximum)`. A refused distinct insertion raises
140+
`MapCapacityError` with `RE10070`, `maxSize`, `currSize` and `bar_index`, exposed
141+
through the `pine_map_capacity_exceeded` run-failure code. Replacing an existing
142+
key remains valid at the configured maximum.
143+
132144
### Prebuilt library
133145

134146
The [v1.3.0 release](https://github.com/pineforge-4pass/pineforge-engine/releases/tag/v1.3.0) attaches the library prebuilt: `pineforge-v1.3.0-linux-x86_64.tar.gz`, `pineforge-v1.3.0-linux-aarch64.tar.gz` and `pineforge-v1.3.0-macos-universal.tar.gz`, each with a `.sha256`. Each unpacks to one directory holding what the install above writes (the two archives, `include/pineforge/` and the `find_package(PineForge)` config) plus `LICENSE`, `NOTICE` and `VERSION`; point `CMAKE_PREFIX_PATH` at it. The package asks for Eigen 3.3+: install it on Linux, while the macOS tarball carries Eigen 3.4.0 ([install guide](docs/pages/install.md)).
@@ -529,11 +541,11 @@ is in [CHANGELOG.md](CHANGELOG.md).
529541

530542
The counts below describe the current API, including recorded outputs, the report sink and scratch-table selection; recorded outputs are not part of 1.4.0.
531543

532-
A newly generated strategy `.so` exposes 118 compiled-strategy `PF_API` declarations
533-
(102 runtime implementations plus sixteen generated exports) plus 43 native-host
534-
declarations: 161 `PF_API` exports in total; `use_bar_magnifier = true` adds
544+
A newly generated strategy `.so` exposes 120 compiled-strategy `PF_API` declarations
545+
(104 runtime implementations plus sixteen generated exports) plus 43 native-host
546+
declarations: 163 `PF_API` exports in total; `use_bar_magnifier = true` adds
535547
`strategy_declares_bar_magnifier` and a module that records [outputs](docs/outputs.md) its
536-
three outputs exports, bringing the inventories to at most 122 and 165. Older modules lack the six opt-in
548+
three outputs exports, bringing the inventories to at most 124 and 167. Older modules lack the six opt-in
537549
[checked-settings exports](docs/checked-settings.md). Older modules can also lack
538550
the two opt-in [execution-capability exports](docs/strategy-capabilities.md).
539551
In an optimized build `nm -gU` also shows libc++'s
@@ -571,6 +583,7 @@ inventories are pinned by `scripts/check_c_abi_runtime.py`:
571583
| `strategy_execution_contract` / `strategy_configure_native_v1` / `strategy_configure_native_fx_curve_v1` / `_fx_curve_ext_v1` | Query Legacy vs NativeMarketV1; apply the versioned native run specification (an invalid or refused base call returns `-1` and leaves the handle Failed; a cooperative `Aborted` handle may be reused with the same key and a higher run number); stage or clear an immutable native FX curve, `_fx_curve_ext_v1` also writing the typed refusal (`pf_native_fx_curve_error_e`) and the offending point's index |
572584
| `strategy_request_abort` / `strategy_last_run_status` | Cooperative abort of a run in progress; `0`=completed, `1`=aborted |
573585
| `strategy_set_scratch_tables_v1` | Copy disk-scratch or bounded-memory table policy and its eleven admission maxima while idle; NULL restores disk defaults (since 1.7.0) |
586+
| `pf_map_pairs_limit_set_checked` | Set the calling thread's creation-time limit for distinct pairs in each map before the strategy factory; NULL or zero selects unlimited. Restore NULL after the factory; existing hosts retain their captured policy |
574587
| `strategy_set_cache_policy_v1` | Copy optional per-kind resident cache bytes before run admission; omitted kinds use internal defaults, zero bypasses, and idle changes apply to the next run. See [memory controls](docs/pages/memory-management.md#memory_controls) for supported kinds |
575588
| `pf_run_abort_install_sigterm_v1` / `pf_run_abort_requested_v1` / `pf_run_abort_restore_sigterm_v1` | Opt-in child invocation SIGTERM flag; stop after a completed input bar, retaining typed progress and an incomplete sink; install/restore with SIGTERM blocked |
576589
| `strategy_set_realtime_tail` | Live-runtime surface (ABI v4): the array's last bar is a still-forming tail — `barstate.islast=false`, `last_bar_index`/`last_bar_time` frozen at the horizon bar, no range-end row |
@@ -708,7 +721,7 @@ Documentation: [C ABI reference](https://cdocs.pineforge.dev) · [Getting starte
708721

709722
## Releases
710723

711-
- **[v1.7.0](https://github.com/pineforge-4pass/pineforge-engine/releases/tag/v1.7.0)** (2026-10-25) — report-sink transport, scratch-backed history and bounded storage admission, with new C++ views and command-line report routes. `get_trade()` and `closed_trade()` now return `Trade` by value, working-book, journal and cohort accessors return views, and some reads can throw on scratch I/O; closing a trade writes `TradeLedger` scratch even without a report sink, so a run needs a disk-backed `TMPDIR` or an explicit scratch directory; the C++ epoch namespace is still `engine_script_run_v19` although the layout of `BacktestEngine` changed, so rebuild every C++ object that includes an engine header, regenerate strategy C++ with the matching codegen and relink. Pair with [codegen 1.7.0](https://github.com/pineforge-4pass/pineforge-codegen-oss/releases/tag/v1.7.0). The known limits of this release (slower than 1.6.3 on some scripts, the closed-trade cache and its size by entry point) are in [CHANGELOG.md](CHANGELOG.md#170--2026-10-25), section "Costs that remain".
724+
- **[v1.7.0](https://github.com/pineforge-4pass/pineforge-engine/releases/tag/v1.7.0)** (2026-10-25) — report-sink transport, scratch-backed history and bounded storage admission, with new C++ views and command-line report routes. `get_trade()` and `closed_trade()` now return `Trade` by value, working-book, journal and cohort accessors return views, and some reads can throw on scratch I/O; closing a trade writes `TradeLedger` scratch even without a report sink, so a run needs a disk-backed `TMPDIR` or an explicit scratch directory; the C++ epoch namespace was still `engine_script_run_v19` although the layout of `BacktestEngine` changed, so rebuild every C++ object that includes an engine header, regenerate strategy C++ with the matching codegen and relink. Pair with [codegen 1.7.0](https://github.com/pineforge-4pass/pineforge-codegen-oss/releases/tag/v1.7.0). The known limits of this release (slower than 1.6.3 on some scripts, the closed-trade cache and its size by entry point) are in [CHANGELOG.md](CHANGELOG.md#170--2026-10-25), section "Costs that remain". <!-- verified HEAD -->
712725
- **[v1.6.3](https://github.com/pineforge-4pass/pineforge-engine/releases/tag/v1.6.3)** (2026-10-23) — CI, release-gate and test-only changes after 1.6.2: the engine's CI runs on the GCP fleet (`pf-gcp-pr`, `pf-gcp-main`), fork-gated; the PR wall is split from full-event verification (the CI split); a release requires full verify and parity green first (C2); the live test matches delivery receipts by request sequence; a session tape comment no longer names a private repository. Nothing under `src/`, `include/`, `cmake/` or `runner/` changed since v1.6.2, and in `docker/` only the run-failure code catalog diff (`run_failure_codes_diff.json`) did. Pair with codegen 1.6.3. See [CHANGELOG.md](CHANGELOG.md#163--2026-10-23).
713726
- **[v1.6.2](https://github.com/pineforge-4pass/pineforge-engine/releases/tag/v1.6.2)** (2026-10-21) — selected-window contract revision `1.3` again, declared with the completeness proof (`capabilities.selected_window_contract_revisions: ["1.3"]` on a supported runtime and `[]` otherwise; 1.6.1 declared `1.3-parity`); proportional annualisation for runs shorter than a day, never null (short-run annualisation); the ownership probes fixed so proof B reads 118/118 and the ownership manifest-controls test sharded four ways; CI changes: Linux x64 jobs on the pf-ci-spot pool and release jobs on pf-ci-release, `timeout-minutes` removed, the actions of `promote-baseline.yml` pinned by SHA, the PR tier skipping the slow rows (D1). It ships the [v1.6.2-rc.1](https://github.com/pineforge-4pass/pineforge-engine/releases/tag/v1.6.2-rc.1) candidate (2026-10-19). Pair with codegen 1.6.2, regenerate and relink. See [CHANGELOG.md](CHANGELOG.md#162--2026-10-21).
714727
- **[v1.6.1](https://github.com/pineforge-4pass/pineforge-engine/releases/tag/v1.6.1)** (2026-10-18) — selected-window contract revision `1.3-parity` (`capabilities.selected_window_contract_revisions: ["1.3-parity"]` on a supported runtime and `[]` otherwise) rather than `1.3` until a completeness proof exists (EN-115), with the four typed value-ownership codes (`strategy_value_invalid`, `engine_command_value_invalid`, `report_result_invalid` and `report_post_execution_failed`) emitted and value parity with the 1.4.0 reference on the tested cases; per-stage `startedAt` and `seconds` in `preflight-summary.json`; `noindex` on raw Doxygen source listings and their omission from `sitemap.xml`; verifier-tests sharded across parallel `unittest` subprocesses; the lower-feed timing ratio skipped only under AddressSanitizer. The C layouts stayed at ABI 4, `PF_ABI_VERSION` was still 4, the selected-window wire version stayed 1, the phase version 1 and the fingerprint/schema version 2. Pair with codegen 1.6.1, regenerate and relink. See [CHANGELOG.md](CHANGELOG.md#161--2026-10-18). <!-- verified HEAD -->

0 commit comments

Comments
 (0)