Skip to content

Commit 1bd075c

Browse files
luisleo526claude
andauthored
Trading metrics + equity curve in backtest report (ABI v2) (#31)
* feat(abi): metrics/equity-curve report fields, trade commission+bar-index fields, pf_abi_version (PF_ABI_VERSION=2) Co-Authored-By: Claude Fable 5 <[email protected]> * docs(abi): per-field metric doxygen with pinned units; test pf_abi_version C-side; document include-order Co-Authored-By: Claude Fable 5 <[email protected]> * feat(engine): capture per-trade commission at close; expose in TradeC Co-Authored-By: Claude Fable 5 <[email protected]> * feat(engine): per-script-bar equity curve + bars-in-market collection, reset-safe Co-Authored-By: Claude Fable 5 <[email protected]> * feat(metrics): compute_trade_stats with TV sign/NaN conventions + unit tests Co-Authored-By: Claude Fable 5 <[email protected]> * feat(metrics): compute_equity_stats (dd walk, TV monthly + per-bar sharpe/sortino, cagr/calmar) + unit tests Co-Authored-By: Claude Fable 5 <[email protected]> * test(metrics): engine-vs-walk dd invariant, per-bar + non-UTC sharpe oracles; document tz-lock contract Co-Authored-By: Claude Fable 5 <[email protected]> * feat(report): wire metrics + equity curve into fill_report/free_report Co-Authored-By: Claude Fable 5 <[email protected]> * feat(harness): mirror ABI v2 (metrics + equity curve), enforce pf_abi_version pf_report_t grew in ABI v2 (pf_metrics_t by value + equity_curve ptr + int64 len appended; pf_trade_t gained commission + entry/exit bar indices). pf_report_t is CALLER-allocated, so every ctypes harness that allocated it from the old field list would under-size the buffer and let the runtime write past it. Updated every ctypes mirror in the repo and added an ABI guard at every CDLL load site (pf_abi_version() must equal 2; a missing symbol means a pre-v2 .so and is rejected with a rebuild hint): - scripts/run_strategy.py (corpus harness; trade dicts now also carry commission + entry/exit bar_index) - tutorial/run.py (mirrors + reusable check_abi) - tutorial/run_mtf.py (guard via run.check_abi) - tutorial/run_advanced.py (guard via run.check_abi) - docker/run_json.py (codegen-mcp container harness) - benchmarks/throughput/grid_search_repro.py Mirror layouts verified against include/pineforge/pineforge.h: ctypes sizeof matches the C side exactly (trade 96, trade_stats 216, equity_stats 120, metrics 768, equity_point 24, report 944; metrics/ equity_curve/equity_curve_len offsets 160/928/936). Co-Authored-By: Claude Fable 5 <[email protected]> * polish(metrics): review nits — ordering comment, test deref guard, empty-curve coverage Co-Authored-By: Claude Fable 5 <[email protected]> * docs: sync FFI mirrors + report schema with ABI v2 (metrics, equity curve, pf_abi_version) Co-Authored-By: Claude Fable 5 <[email protected]> * docs(abi): truncation caveat on equity_curve len; drop gitignored-spec reference from source comment Co-Authored-By: Claude Fable 5 <[email protected]> * fix(metrics): TV conventions arbitrated vs real export — net return-on-cost pnl_pct, independent largest-%, inclusive avg bars Co-Authored-By: Claude Fable 5 <[email protected]> * test(metrics): cross-validation script vs quantstats/empyrical-reloaded Co-Authored-By: Claude Fable 5 <[email protected]> * ci: register pf_abi_version in check_c_abi_runtime EXPECTED_RUNTIME; SOP in CLAUDE.md Co-Authored-By: Claude Fable 5 <[email protected]> --------- Co-authored-by: Claude Fable 5 <[email protected]>
1 parent 3369343 commit 1bd075c

30 files changed

Lines changed: 2287 additions & 47 deletions

‎CLAUDE.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,28 @@ If `scripts/run_corpus.sh` reports any parity drift or failures, investigate the
3232
- Run all unit tests: `ctest --test-dir build --output-on-failure`
3333
- Run single test executable: `./build/bin/test_integration`
3434

35+
## SOP: adding a runtime `PF_API` export (CI gate — recurring failure)
36+
37+
CI runs `python3 scripts/check_c_abi_runtime.py` after build+test. It pins the
38+
exact set of `PF_API` symbols implemented in `src/c_abi.cpp` against the
39+
hardcoded `EXPECTED_RUNTIME` frozenset in that script. Adding (or removing) a
40+
runtime export WITHOUT updating that list fails ALL CI matrix jobs at the
41+
"C ABI runtime source check" step, even though build and ctest are green.
42+
43+
Checklist when touching runtime exports — update ALL of these together:
44+
45+
1. `src/c_abi.cpp` — the implementation (and its file-header symbol comment).
46+
2. `include/pineforge/pineforge.h` — the `PF_API` declaration (+ doxygen).
47+
3. `scripts/check_c_abi_runtime.py` — add the symbol to `EXPECTED_RUNTIME`.
48+
4. Python ctypes harnesses if consumers must call it
49+
(`scripts/run_strategy.py`, `tutorial/run*.py`, `docker/run_json.py`,
50+
`benchmarks/throughput/grid_search_repro.py`).
51+
5. README symbol table if it enumerates exports.
52+
53+
Before pushing: `python3 scripts/check_c_abi_runtime.py` (must exit 0).
54+
Per-strategy symbols (strategy_create, run_backtest, …) are NOT in this list —
55+
they are codegen-emitted; the checker enforces exactly that split.
56+
3557
## Code Style & Invariants
3658

3759
- Modern C++17. Use of `<cstdint>` fixed-width types.

‎CMakeLists.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,7 @@ add_library(pineforge STATIC
7878
src/c_abi.cpp
7979
src/engine_fills.cpp
8080
src/engine_lower_tf.cpp
81+
src/engine_metrics.cpp
8182
src/engine_orders.cpp
8283
src/engine_path_resolve.cpp
8384
src/engine_report.cpp

‎README.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -299,7 +299,7 @@ int main(void) {
299299
300300
## Public C ABI (the stability surface)
301301
302-
`<pineforge/pineforge.h>` is the **single canonical consumer header**. Every compiled PineForge strategy `.so` exports exactly the 10 symbols declared there:
302+
`<pineforge/pineforge.h>` is the **single canonical consumer header**. Every compiled PineForge strategy `.so` exports the symbols declared there:
303303
304304
305305
| Symbol | Role |
@@ -314,9 +314,10 @@ int main(void) {
314314
| `strategy_set_magnifier_volume_weighted` | Toggle volume-weighted magnifier |
315315
| `strategy_set_trace_enabled` | Toggle per-bar trace recording |
316316
| `pf_version_get` | Runtime version |
317+
| `pf_abi_version` | Struct-layout version (`PF_ABI_VERSION`) |
317318
318319
319-
POD types (`pf_bar_t`, `pf_trade_t`, `pf_report_t`, `pf_security_diag_t`, `pf_trace_entry_t`, `pf_version_t`) and the `pf_magnifier_distribution_t` enum complete the surface.
320+
POD types (`pf_bar_t`, `pf_trade_t`, `pf_report_t`, `pf_security_diag_t`, `pf_trace_entry_t`, `pf_version_t`, and — since ABI v2 — `pf_trade_stats_t`, `pf_equity_stats_t`, `pf_metrics_t`, `pf_equity_point_t`) and the `pf_magnifier_distribution_t` enum complete the surface. ABI v2 (`PF_ABI_VERSION == 2`, exported as `pf_abi_version()`) appends computed trading metrics and a per-script-bar equity curve to `pf_report_t`; callers must verify the version before running since the report struct is caller-allocated.
320321
321322
**Stability guarantee:** within the same `PINEFORGE_VERSION_MAJOR`, struct layouts and `extern "C"` signatures are append-only. New fields may be appended; existing fields are never reordered, removed, or retyped. New functions may be added; existing functions are never removed or signature-changed. Compile-time `static_assert`s in `src/c_abi.cpp` pin the layouts against drift.
322323

‎benchmarks/throughput/grid_search_repro.py‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,8 +26,53 @@ class pf_trade_t(ctypes.Structure):
2626
("max_runup", ctypes.c_double),
2727
("max_drawdown", ctypes.c_double),
2828
("qty", ctypes.c_double),
29+
("commission", ctypes.c_double),
30+
("entry_bar_index", ctypes.c_int32),
31+
("exit_bar_index", ctypes.c_int32),
2932
]
3033

34+
class pf_trade_stats_t(ctypes.Structure):
35+
_fields_ = [
36+
("num_trades", ctypes.c_int32), ("num_wins", ctypes.c_int32),
37+
("num_losses", ctypes.c_int32), ("num_even", ctypes.c_int32),
38+
("percent_profitable", ctypes.c_double),
39+
("net_profit", ctypes.c_double), ("net_profit_pct", ctypes.c_double),
40+
("gross_profit", ctypes.c_double), ("gross_profit_pct", ctypes.c_double),
41+
("gross_loss", ctypes.c_double), ("gross_loss_pct", ctypes.c_double),
42+
("profit_factor", ctypes.c_double),
43+
("avg_trade", ctypes.c_double), ("avg_trade_pct", ctypes.c_double),
44+
("avg_win", ctypes.c_double), ("avg_win_pct", ctypes.c_double),
45+
("avg_loss", ctypes.c_double), ("avg_loss_pct", ctypes.c_double),
46+
("ratio_avg_win_avg_loss", ctypes.c_double),
47+
("largest_win", ctypes.c_double), ("largest_win_pct", ctypes.c_double),
48+
("largest_loss", ctypes.c_double), ("largest_loss_pct", ctypes.c_double),
49+
("commission_paid", ctypes.c_double),
50+
("expectancy", ctypes.c_double),
51+
("max_consecutive_wins", ctypes.c_int32), ("max_consecutive_losses", ctypes.c_int32),
52+
("avg_bars_in_trade", ctypes.c_double), ("avg_bars_in_wins", ctypes.c_double),
53+
("avg_bars_in_losses", ctypes.c_double),
54+
]
55+
56+
class pf_equity_stats_t(ctypes.Structure):
57+
_fields_ = [
58+
("max_equity_drawdown", ctypes.c_double), ("max_equity_drawdown_pct", ctypes.c_double),
59+
("max_equity_runup", ctypes.c_double), ("max_equity_runup_pct", ctypes.c_double),
60+
("buy_hold_return", ctypes.c_double), ("buy_hold_return_pct", ctypes.c_double),
61+
("sharpe_tv", ctypes.c_double), ("sortino_tv", ctypes.c_double),
62+
("sharpe_bar", ctypes.c_double), ("sortino_bar", ctypes.c_double),
63+
("cagr", ctypes.c_double), ("calmar", ctypes.c_double),
64+
("recovery_factor", ctypes.c_double), ("time_in_market_pct", ctypes.c_double),
65+
("open_pl", ctypes.c_double),
66+
]
67+
68+
class pf_metrics_t(ctypes.Structure):
69+
_fields_ = [("all", pf_trade_stats_t), ("longs", pf_trade_stats_t),
70+
("shorts", pf_trade_stats_t), ("equity", pf_equity_stats_t)]
71+
72+
class pf_equity_point_t(ctypes.Structure):
73+
_fields_ = [("time_ms", ctypes.c_int64), ("equity", ctypes.c_double),
74+
("open_profit", ctypes.c_double)]
75+
3176
class pf_security_diag_t(ctypes.Structure):
3277
_fields_ = [
3378
("sec_id", ctypes.c_int),
@@ -74,8 +119,29 @@ class pf_report_t(ctypes.Structure):
74119
("trace_len", ctypes.c_int),
75120
("trace_names", ctypes.POINTER(ctypes.c_char_p)),
76121
("trace_names_len", ctypes.c_int),
122+
123+
("metrics", pf_metrics_t),
124+
("equity_curve", ctypes.POINTER(pf_equity_point_t)),
125+
("equity_curve_len", ctypes.c_int64), # int64, NOT c_int
77126
]
78127

128+
# pf_report_t is caller-allocated; a layout mismatch means the runtime
129+
# writes past this script's report buffer. Verify the ABI before running.
130+
EXPECTED_PF_ABI = 2
131+
132+
def check_abi(lib):
133+
try:
134+
lib.pf_abi_version.restype = ctypes.c_int
135+
abi = lib.pf_abi_version()
136+
except AttributeError:
137+
raise RuntimeError(
138+
"strategy library predates pf_abi_version (ABI v1); rebuild it "
139+
"against the current pineforge runtime (pf_report_t grew).")
140+
if abi != EXPECTED_PF_ABI:
141+
raise RuntimeError(
142+
f"pineforge ABI mismatch: library reports {abi}, harness expects "
143+
f"{EXPECTED_PF_ABI}; rebuild.")
144+
79145
def main():
80146
repo_root = Path(__file__).parent.parent.parent.resolve()
81147
dylib_path = repo_root / "benchmarks/assets/strategies/19-scalping-wunder-bots/strategy.dylib"
@@ -96,6 +162,7 @@ def main():
96162

97163
# Load shared library
98164
lib = ctypes.CDLL(str(dylib_path))
165+
check_abi(lib)
99166

100167
# Declare functions
101168
lib.strategy_create.argtypes = [ctypes.c_char_p]

‎docker/run_json.py‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,9 +81,62 @@ class TradeC(ctypes.Structure):
8181
("max_runup", ctypes.c_double),
8282
("max_drawdown", ctypes.c_double),
8383
("qty", ctypes.c_double),
84+
("commission", ctypes.c_double),
85+
("entry_bar_index", ctypes.c_int32),
86+
("exit_bar_index", ctypes.c_int32),
8487
]
8588

8689

90+
class TradeStatsC(ctypes.Structure):
91+
"""Mirror of pf_trade_stats_t (ABI v2)."""
92+
_fields_ = [
93+
("num_trades", ctypes.c_int32), ("num_wins", ctypes.c_int32),
94+
("num_losses", ctypes.c_int32), ("num_even", ctypes.c_int32),
95+
("percent_profitable", ctypes.c_double),
96+
("net_profit", ctypes.c_double), ("net_profit_pct", ctypes.c_double),
97+
("gross_profit", ctypes.c_double), ("gross_profit_pct", ctypes.c_double),
98+
("gross_loss", ctypes.c_double), ("gross_loss_pct", ctypes.c_double),
99+
("profit_factor", ctypes.c_double),
100+
("avg_trade", ctypes.c_double), ("avg_trade_pct", ctypes.c_double),
101+
("avg_win", ctypes.c_double), ("avg_win_pct", ctypes.c_double),
102+
("avg_loss", ctypes.c_double), ("avg_loss_pct", ctypes.c_double),
103+
("ratio_avg_win_avg_loss", ctypes.c_double),
104+
("largest_win", ctypes.c_double), ("largest_win_pct", ctypes.c_double),
105+
("largest_loss", ctypes.c_double), ("largest_loss_pct", ctypes.c_double),
106+
("commission_paid", ctypes.c_double),
107+
("expectancy", ctypes.c_double),
108+
("max_consecutive_wins", ctypes.c_int32), ("max_consecutive_losses", ctypes.c_int32),
109+
("avg_bars_in_trade", ctypes.c_double), ("avg_bars_in_wins", ctypes.c_double),
110+
("avg_bars_in_losses", ctypes.c_double),
111+
]
112+
113+
114+
class EquityStatsC(ctypes.Structure):
115+
"""Mirror of pf_equity_stats_t (ABI v2)."""
116+
_fields_ = [
117+
("max_equity_drawdown", ctypes.c_double), ("max_equity_drawdown_pct", ctypes.c_double),
118+
("max_equity_runup", ctypes.c_double), ("max_equity_runup_pct", ctypes.c_double),
119+
("buy_hold_return", ctypes.c_double), ("buy_hold_return_pct", ctypes.c_double),
120+
("sharpe_tv", ctypes.c_double), ("sortino_tv", ctypes.c_double),
121+
("sharpe_bar", ctypes.c_double), ("sortino_bar", ctypes.c_double),
122+
("cagr", ctypes.c_double), ("calmar", ctypes.c_double),
123+
("recovery_factor", ctypes.c_double), ("time_in_market_pct", ctypes.c_double),
124+
("open_pl", ctypes.c_double),
125+
]
126+
127+
128+
class MetricsC(ctypes.Structure):
129+
"""Mirror of pf_metrics_t (ABI v2)."""
130+
_fields_ = [("all", TradeStatsC), ("longs", TradeStatsC),
131+
("shorts", TradeStatsC), ("equity", EquityStatsC)]
132+
133+
134+
class EquityPointC(ctypes.Structure):
135+
"""Mirror of pf_equity_point_t (ABI v2)."""
136+
_fields_ = [("time_ms", ctypes.c_int64), ("equity", ctypes.c_double),
137+
("open_profit", ctypes.c_double)]
138+
139+
87140
class SecurityDiagC(ctypes.Structure):
88141
_fields_ = [
89142
("sec_id", ctypes.c_int),
@@ -126,9 +179,31 @@ class ReportC(ctypes.Structure):
126179
("trace_len", ctypes.c_int),
127180
("trace_names", ctypes.POINTER(ctypes.c_char_p)),
128181
("trace_names_len", ctypes.c_int),
182+
("metrics", MetricsC),
183+
("equity_curve", ctypes.POINTER(EquityPointC)),
184+
("equity_curve_len", ctypes.c_int64), # int64, NOT c_int
129185
]
130186

131187

188+
# pf_report_t is CALLER-allocated: a .so built against a different ABI
189+
# writes past (or short of) our ReportC buffer. Assert version up front.
190+
EXPECTED_PF_ABI = 2
191+
192+
193+
def check_abi(lib: ctypes.CDLL) -> None:
194+
try:
195+
lib.pf_abi_version.restype = ctypes.c_int
196+
abi = lib.pf_abi_version()
197+
except AttributeError:
198+
raise RuntimeError(
199+
"strategy .so predates pf_abi_version (ABI v1); rebuild it against "
200+
"the current pineforge runtime (pf_report_t grew).")
201+
if abi != EXPECTED_PF_ABI:
202+
raise RuntimeError(
203+
f"pineforge ABI mismatch: .so reports {abi}, harness expects "
204+
f"{EXPECTED_PF_ABI}; rebuild.")
205+
206+
132207
# --- helpers ----------------------------------------------------------
133208

134209
def load_bars(csv_path: Path) -> tuple[ctypes.Array, int]:
@@ -158,6 +233,7 @@ def load_bars(csv_path: Path) -> tuple[ctypes.Array, int]:
158233

159234
def load_strategy(so_path: Path) -> ctypes.CDLL:
160235
lib = ctypes.CDLL(str(so_path))
236+
check_abi(lib)
161237

162238
lib.strategy_create.argtypes = [ctypes.c_char_p]
163239
lib.strategy_create.restype = ctypes.c_void_p

‎docs/pages/examples-rust.md‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,59 @@ struct PfTrade {
5656
is_long: c_int,
5757
max_runup: f64, max_drawdown: f64,
5858
qty: f64,
59+
// ABI v2
60+
commission: f64,
61+
entry_bar_index: i32,
62+
exit_bar_index: i32,
63+
}
64+
65+
// ── ABI v2 metrics PODs ──────────────────────────────────────────────
66+
67+
#[repr(C)]
68+
#[derive(Clone, Copy, Default)]
69+
struct PfTradeStats {
70+
num_trades: i32, num_wins: i32, num_losses: i32, num_even: i32,
71+
percent_profitable: f64,
72+
net_profit: f64, net_profit_pct: f64,
73+
gross_profit: f64, gross_profit_pct: f64,
74+
gross_loss: f64, gross_loss_pct: f64,
75+
profit_factor: f64,
76+
avg_trade: f64, avg_trade_pct: f64,
77+
avg_win: f64, avg_win_pct: f64,
78+
avg_loss: f64, avg_loss_pct: f64,
79+
ratio_avg_win_avg_loss: f64,
80+
largest_win: f64, largest_win_pct: f64,
81+
largest_loss: f64, largest_loss_pct: f64,
82+
commission_paid: f64,
83+
expectancy: f64,
84+
max_consecutive_wins: i32, max_consecutive_losses: i32,
85+
avg_bars_in_trade: f64, avg_bars_in_wins: f64, avg_bars_in_losses: f64,
86+
}
87+
88+
#[repr(C)]
89+
#[derive(Clone, Copy, Default)]
90+
struct PfEquityStats {
91+
max_equity_drawdown: f64, max_equity_drawdown_pct: f64,
92+
max_equity_runup: f64, max_equity_runup_pct: f64,
93+
buy_hold_return: f64, buy_hold_return_pct: f64,
94+
sharpe_tv: f64, sortino_tv: f64,
95+
sharpe_bar: f64, sortino_bar: f64,
96+
cagr: f64, calmar: f64,
97+
recovery_factor: f64, time_in_market_pct: f64,
98+
open_pl: f64,
99+
}
100+
101+
#[repr(C)]
102+
#[derive(Clone, Copy, Default)]
103+
struct PfMetrics {
104+
all: PfTradeStats, longs: PfTradeStats, shorts: PfTradeStats,
105+
equity: PfEquityStats,
106+
}
107+
108+
#[repr(C)]
109+
#[derive(Clone, Copy, Default)]
110+
struct PfEquityPoint {
111+
time_ms: i64, equity: f64, open_profit: f64,
59112
}
60113

61114
#[repr(C)]
@@ -102,10 +155,19 @@ struct PfReport {
102155
trace_len: c_int,
103156
trace_names: *mut *const c_char,
104157
trace_names_len: c_int,
158+
159+
// ABI v2: computed metrics + per-script-bar equity curve
160+
metrics: PfMetrics,
161+
equity_curve: *mut PfEquityPoint,
162+
equity_curve_len: i64,
105163
}
106164

107165
const PF_MAGNIFIER_ENDPOINTS: c_int = 3;
108166

167+
// PfReport is CALLER-allocated: before any run, resolve `pf_abi_version`
168+
// via libloading and assert it returns 2 (PF_ABI_VERSION) — an old .so
169+
// writing into this larger struct (or vice versa) corrupts memory silently.
170+
109171
// ── Safe wrapper ──────────────────────────────────────────────────────
110172

111173
struct StrategyLib {

0 commit comments

Comments
 (0)