Skip to content

Commit 62a0b70

Browse files
authored
Combined 1.8.0 batch 2: reversal fix, report iteration, replay companion, metrics-only mode (#632)
Combined second batch for 1.8.0. Each member was reviewed and proven on its own branch; this branch merges them onto main and is verified as one tree. ## Members (merged in this order) - Keep a reversal's newly credited opening when its entry id had opposite-side history (fixes wrong positions, uncapped pyramiding and missing closed-trade rows after a reversal on a reused entry id; affected releases v1.1.0 through v1.7.0). - Window buffers for technical-analysis series kept on main. - Report channel contract and hosting jail budget contract (docs; #613, #614). - Annualization diagnostic uses the engine's initial capital on every report path. - Opt-in report replay companion (#607). - Metrics-only run mode (#630). - Comment-only rewording of shipped source comments. - Source clones skip the in-repo `.ccache` directory, so a whole-tree copy no longer races a running compile. - In-process iteration of the default-path report columns: `report["trades"]` and `report["equity_curve"]` iterate lazily from the report spool again, report their length and support integer indexing; iteration must happen before the report's resources close. - CHANGELOG entries for the above, and re-pointed source-line citations in two documentation pages (line numbers only). - The report-sink trade-identity test reference is re-recorded: the reversal fix changes the per-bar state hash input (open-position bookkeeping), while no trade or equity row changes; the CHANGELOG says so. ## Not in this batch The live runner's report checkpoints (#598) and report accumulator (#617) move to the next minor release: testing showed a timing case that can leave a delivered order unacknowledged and a crash for modules whose handle is not an engine. Their pull requests stay open as drafts. ## Verification - Native live suite (all native shards, thread and address sanitizer legs) on this tree's code. - Full verify and the corpus parity sweep on this exact head. - Every member was reviewed independently before it was merged here. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
1 parent a34a0d0 commit 62a0b70

135 files changed

Lines changed: 40216 additions & 25413 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,17 @@ ABI number in `strategy_library_incompatible`'s message. The report keys are
2020
unchanged except for two additive diagnostics keys, described below.
2121

2222
- Monthly Sharpe/Sortino month keys for timestamps beyond about 34,800 years from the epoch no longer overflow; in-range results are unchanged.
23+
- The report's annualization diagnostic now uses the engine's initial capital
24+
on every report path. The path-sink report route (named or unnamed)
25+
previously omitted it and could drop the diagnostic when the compounded
26+
return overflowed; metric values are unchanged.
27+
- Strategies that reversed on an entry id the other side had used before kept the wrong position: the reversal's new lot was erased from its entry, so later percentage exits were sized from zero units and closed nothing, the position stayed open, later entries could exceed the pyramiding limit, and the closed-trade rows for those exits were missing. Affected releases: v1.1.0 through v1.7.0. Fixed in 1.8.0. Release 1.0.0 handles percentage exits after a reversal differently for an older, unrelated reason, so it is not a safe version to stay on either. For strategies that reverse on a reused entry id, the per-bar state hash values change (the fix changes position bookkeeping); compare state hashes only between runs of one release, as before.
28+
- In-process callers can again iterate the report's trades and equity curve on
29+
the default path, as well as the outputs rows, and take their length. Rows
30+
decode lazily; trades and equity also support `[i]` (new for sink trades).
31+
The first read freezes the report spool, which then accepts no more rows.
32+
Iterate before the report's resources close; afterwards iteration raises
33+
`ValueError`.
2334

2435
### Compatibility and migration
2536

@@ -133,6 +144,27 @@ unchanged except for two additive diagnostics keys, described below.
133144
per-record sink parse and the dense-script close path are separate work, not in
134145
this release.
135146

147+
### Metrics-only `run_json` request
148+
149+
- **`--metrics-only[=auto|legacy]` prints the report without its per-bar and per-trade arrays.**
150+
The document is the ordinary document without `trades`, `equity_curve`,
151+
`elapsed_seconds` and the two scratch-arena byte counters; every other key
152+
keeps its name, value and order. A bare option means `auto`. With `auto`, a
153+
native module that certifies itself takes the engine's no-curve metrics route
154+
(Sharpe, Sortino, the monthly ratios, the equity summary and the trade
155+
statistics are computed as the run goes), which saves at least 12 bytes of
156+
resident memory per bar (the bound the acceptance test asserts); the document
157+
is byte for byte the one `legacy` prints. A module certifies itself through one new optional export,
158+
`strategy_metrics_only_capability_v1`; a module without it, `legacy`, or a
159+
result the runner cannot accept keeps the actual equity curve, as before. The
160+
option is refused together with, among others, an explicit report sink, a
161+
phase stream, selected-window options, outputs, a benchmark run or symbol
162+
feeds. Without the option nothing changes.
163+
- **The reference-run scripts read the renamed range-provenance member.** They
164+
read a tape's `metrics.json` range-provenance member as `rangeProvenance`
165+
first; the member's previous name is still accepted until the next minor
166+
release, and the label the scripts print names the member they read.
167+
136168
### Native live report stream
137169

138170
- **Delta JSONL is opt-in; the default stream is unchanged.**

‎CMakeLists.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,7 @@ set(PINEFORGE_KERNEL_SOURCES src/array.cpp
127127
src/engine_execution.cpp
128128
src/engine_path_resolve.cpp
129129
src/engine_report.cpp
130+
src/report_replay.cpp
130131
src/reservation_expansion.cpp
131132
src/engine_run.cpp
132133
src/engine_security.cpp

‎CONTRIBUTING.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ Two consequences that are easy to miss:
6565
| Add a generic broker/matching capability | the kernel (`src/engine_*`, `src/native_*`) | make it opt-in, add its ADR 0001 ruling row, add a kernel-only test, and add an `examples/native/` host if the ruling is *native-only* |
6666
| Add or change a TA class | the right `ta_*.cpp` partition + its declaration in `<pineforge/ta.hpp>` | add a unit test against a hand-computed series |
6767
| Change what codegen may emit | the contract, not this repo's runtime | say so in the PR; the transpiler lives in `pineforge-codegen-oss` |
68-
| Add a runtime `PF_API` export | `src/c_abi.cpp` + `include/pineforge/pineforge.h` | update `EXPECTED_RUNTIME` check_c_abi_runtime.py:29 and its two declaration counts, the ctypes harnesses, the README symbol table and every page that states the export counts — all in the same commit |
68+
| Add a runtime `PF_API` export | `src/c_abi.cpp` + `include/pineforge/pineforge.h` | update `EXPECTED_RUNTIME` check_c_abi_runtime.py:35 and its two declaration counts, the ctypes harnesses, the README symbol table and every page that states the export counts — all in the same commit |
6969
| Add a C kernel-driving export | `src/native_c_host.cpp` + `include/pineforge/native_c_api.h` | add it to `EXPECTED_NATIVE_C_API` (and its two counts) in `scripts/check_c_abi_runtime.py`, update that header's COVERAGE block, and retire the 1.0 C boundary row the export closes; `scripts/check_native_c_api_surface.py` proves the block is exactly the host's public surface and fails on a closed gap |
7070
| Document something | `docs/pages/`, `README.md`, this file | cite the tree by `path:line`; the anchor guard checks that the line still holds the symbol |
7171

‎README.md‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ The separate PineForge compiler, [`pineforge-codegen`](https://github.com/pinefo
3333
- **Open runtime.** The engine and native live runner are Apache-2.0. The separately distributed [PineForge compiler](https://github.com/pineforge-4pass/pineforge-codegen-oss/blob/main/LICENSE) is source-available under the PineForge Source License 1.2 from its release 1.3.0 (release 1.2.0 keeps the PineForge Source License 1.1, and releases up to 1.1.0 keep the PolyForm Noncommercial terms they shipped with); personal trading is free, and commercial use requires a separate license; the capital in an account a proprietary-trading firm or funded-trader program provides or allocates, including a challenge, evaluation or simulated account, is investment capital. Public reference strategies, benchmarks and validation tooling are available in their respective repositories; the community-script test set is not redistributed.
3434
- **Fast.** In-process, no interpreter: median **36× faster than PyneCore** on the 200 strategies both engines time (100 corpus probes and 100 held-out private strategies), measured at engine `35db01c8` on 2026-09-29 on an AWS c7a.8xlarge with PyneCore timed as a subprocess, its interpreter start-up included (PineForge runs a median 807k bars/s per strategy over its 201 slots with the bar magnifier on; [method](benchmarks/results/speed.md)). Parameter sweeps re-run a loaded `.so` with new inputs — no recompile, no fork.
3535
- **Deterministic to the bit.** Two runs with the same inputs produce identical trade lists.
36-
- **Yours to embed.** 159 `PF_API` declarations across six installed headers — `pineforge.h`'s 103 (83 of the 94 runtime exports and up to 20 per-strategy functions a generated module defines), eleven more runtime declarations in `execution_observer.h`, `selected_window.h` and `report_sink.h` (three, three and five), `native_c_api.h`'s 43 native-host declarations and `selected_window_plan.h`'s two planner declarations — an ABI that is append-only within a major version from 1.0 on. Call it from C, Python, Rust, Go, Node, Julia — or let an AI agent drive it over MCP.
36+
- **Yours to embed.** 162 `PF_API` declarations across six installed headers — `pineforge.h`'s 106 (86 of the 97 runtime exports and up to 20 per-strategy functions a generated module defines), eleven more runtime declarations in `execution_observer.h`, `selected_window.h` and `report_sink.h` (three, three and five), `native_c_api.h`'s 43 native-host declarations and `selected_window_plan.h`'s two planner declarations — an ABI that is append-only within a major version from 1.0 on. Call it from C, Python, Rust, Go, Node, Julia — or let an AI agent drive it over MCP.
3737

3838
Recorded outputs and their additional API declarations described here are
3939
part of **1.5.0**.
@@ -542,11 +542,11 @@ is in [CHANGELOG.md](CHANGELOG.md).
542542

543543
The counts below describe the current API, including recorded outputs, the report sink and scratch-table selection; recorded outputs are not part of 1.4.0.
544544

545-
A newly generated strategy `.so` exposes 110 compiled-strategy `PF_API` declarations
546-
(94 runtime implementations plus sixteen generated exports) plus 43 native-host
547-
declarations: 153 `PF_API` exports in total; `use_bar_magnifier = true` adds
545+
A newly generated strategy `.so` exposes 113 compiled-strategy `PF_API` declarations
546+
(97 runtime implementations plus sixteen generated exports) plus 43 native-host
547+
declarations: 156 `PF_API` exports in total; `use_bar_magnifier = true` adds
548548
`strategy_declares_bar_magnifier` and a module that records [outputs](docs/outputs.md) its
549-
three outputs exports, bringing the inventories to at most 114 and 157. Older modules lack the six opt-in
549+
three outputs exports, bringing the inventories to at most 117 and 160. Older modules lack the six opt-in
550550
[checked-settings exports](docs/checked-settings.md). Older modules can also lack
551551
the two opt-in [execution-capability exports](docs/strategy-capabilities.md).
552552
In an optimized build `nm -gU` also shows libc++'s
@@ -592,6 +592,8 @@ inventories are pinned by `scripts/check_c_abi_runtime.py`:
592592
| `strategy_position_avg_price` / `strategy_position_cycle_seq` / `strategy_position_size` | ABI v4: the live position's volume-weighted average entry price, its cycle id, and its script-facing signed size |
593593
| `strategy_closed_trade_entry_id` / `_exit_id` / `_exit_comment` / `_close_cause` | ABI v4: per-closed-trade id/comment strings and a `close_cause` (`pf_close_cause_t`: `SCRIPT`, `BRACKET`, `LIQUIDATION` for a margin call, `RISK_LIMIT` for the intraday loss cap, `FILL_CAP` for the intraday fill cap, `RANGE_END`), indexed like `strategy_closed_trade_entry_incarnation` |
594594
| `strategy_initial_capital` | ABI v4: the engine's effective configured initial capital, after the native run specification or strategy override; `NaN` if `s` is `NULL` or the read raises |
595+
| `strategy_net_profit_sum` / `strategy_report_pointvalue` | Additive replay accessors, with ABI version unchanged at 5: raw whole-lot realized P&L and effective pointvalue; `NaN` on `NULL` |
596+
| `strategy_set_report_replay_v1` | Opt in to a separate [report replay companion](docs/pages/report-replay-rq.md) while idle; no existing report or sink layout changes |
595597
| `strategy_current_equity` / `strategy_script_bars_processed` | ABI v4: `initial_capital + netprofit` (not Pine's `strategy.equity`, which also adds open profit); total script bars dispatched by the most recent run |
596598

597599
### Driving the kernel from C

0 commit comments

Comments
 (0)