Skip to content

Commit 80e26e8

Browse files
luisleo526claude
andauthored
benchmarks: the PyneCore harness is kept outside this repository; its leg is optional (#383)
* benchmarks: the PyneCore harness is kept outside this repository; the… … PyneCore leg is optional (BIZ-11) The three files that drove PyneCore (benchmarks/runners/run_pynecore.py, benchmarks/speed/time_pynecore.py and the load trace benchmarks/results/raw/2026-09-29-35db01c8/loads/pynecore-load.tsv) are no longer in the public tree. A maintainer who has the harness points PYNECORE_HARNESS_DIR at it and the PyneCore leg runs exactly as before; without it benchmarks/run_all.sh prints "PyneCore harness not present (set PYNECORE_HARNESS_DIR to the private harness); skipping the PyneCore leg" and carries on with PineForge, PineTS and vectorbt. * benchmarks/run_all.sh: PYNECORE_HARNESS_DIR (unset = skip), a visible skip line for every PyneCore step (strategy runs, canonical indicators, speed timing, the PyneCore data conversion), PINEFORGE_REPO_ROOT exported so the harness finds this checkout, and the indicator comparison is skipped only when neither the harness nor a PyneCore indicator output exists. * benchmarks/speed/aggregate.py: --pynecore is optional. Given, the output is byte-identical to before; absent, the PyneCore columns, rows and the PyneCore method lines are omitted and a note goes to stderr. * benchmarks/compare.py: the docstring says the PyneCore runner is private and optional (no code change: an engine without output was already reported n/a). * benchmarks/tests/test_run_all.py: the existing scenarios get their PyneCore stub through PYNECORE_HARNESS_DIR, every assertion kept, and one new test covers the absent harness (the visible line, the run completes, no PyneCore output). * benchmarks/results/raw/manifest.json: the entry for the removed load trace is gone. * Published numbers, dates and commits are unchanged: the 36x median speedup stays measured at engine 35db01c on 2026-09-29; only its method wording changes. check_provenance.py, uv.lock and pyproject.toml are untouched. * Sentences reworded (named for the doc-revert gate, old first words): "runners/run_pynecore.py normalizes the output to TradingView's schema as pynecore_trades.csv." and "PyneCore is timed as the subprocess wall time of uv run python" (benchmarks/README.md); "PyneCore: subprocess wall time of uv run python runners/run_pynecore.py" (benchmarks/ results/speed.md); "The same-window A/B of 35db01c against 063e446 on all 201 slots and" (benchmarks/results/ raw/README.md); "Each case copies benchmarks/run_all.sh into a scratch tree whose engine runners are" (benchmarks/tests/test_run_all.py). * benchmarks: the 100 non-corpus speed strategies are described as held… …-out private strategies (BIZ-11) Where the PyneCore speedup is stated (README.md, benchmarks/README.md, results/speed.md and the aggregate that writes it), the 200 strategies are now described as 100 corpus probes and 100 held-out private strategies. The population paragraph of benchmarks/README.md says the same, with the market and timeframe facts kept. The provenance claims follow the new sentences exactly: the closed-half claim matches the new wording, and the speedup claim now also reads the two counts and checks them against the timed slot names (100 corpus probes, 100 closed), so it is stricter, not weaker. No number changes. Reworded sentences, named for the doc-revert gate: * the market and timeframe that both the bench feed and the TradingView tapes cover. * strategies both engines time (p5 15×, p95 131×), and 4.2× over vectorbt across its 13 ports. * strategies both engines time, measured at engine on an AWS c7a.8xlarge with PyneCore timed as a subprocess. * across the 200 strategies both engines time (p5 15×, p95 131×). * (across 200 commonly-timed strategies): 36× (p5 15×, p95 131×) * benchmarks: the optional PyneCore leg is tested in both worlds, the g… …enerator carries the new wording, and a run without it never rewrites the committed speed table (BIZ-11) A read of the change found the following; all are fixed here. * The generator still emitted the old sentence naming the private runner and its command line, so regenerating speed.md would have put the path back into a public file. The harness-present branch of speed/aggregate.py now emits exactly the text of the committed speed.md, and a new test, tests/test_aggregate.py, runs the real aggregate with and without --pynecore and checks that block against both its own output and the committed file. * A run without timings from PyneCore wrote its (different) table over the committed results/speed.md. It now writes _workdir/speed.md and says so; the committed file is not touched. Tested with an aggregate stand-in that writes to the path it is given: the committed file stays byte-identical in the absent cases and is the output in the present case. * SKIP_PYNE=1 now means what it meant before: it skips only the PyneCore strategy runs and the canonical indicator run, not the .ohlcv conversion and not the PyneCore timing. The "harness not present" line is printed only when the harness really is absent. A run without the harness leaves the committed PyneCore trade lists and canonical output as they are. * test_run_all.py: every case now starts from a clean slate (markers, outputs and committed PyneCore files re-seeded), so a result of one case cannot satisfy another; a new test runs with the harness present, with an absolute and a relative harness path, and checks that the runner sees PINEFORGE_REPO_ROOT, that the timings reach the aggregate as --pynecore and that the table goes to its usual place; a recording stand-in for pyne proves it is never called without the harness; the argument check also rejects --pynecore=... and any path ending pc_speed.json. * check_provenance.py: the 100/100 split in the speedup claim comes from the selection manifest (source == closed, the replacement slot included) instead of a slug suffix; probes + held must equal n. The claim pattern is unchanged. A test covers names that disagree with their manifest source. * README: the added pointer sentence no longer says "set PYNECORE_HARNESS_DIR to reproduce"; it says the harness is not distributed, when the leg runs, and that the published PyneCore figures are dated and not re-measured here. The manual timing command sets PINEFORGE_REPO_ROOT. Reworded base sentences, named for the doc-revert gate: * SKIP_BUILD, SKIP_PINEFORGE, SKIP_PYNE, SKIP_PINETS, SKIP_VECTORBT, SKIP_INDICATORS, SKIP_SPEED and SKIP_REPORTS each skip one step. * skip PyneCore strategy + indicator runs and timing Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
1 parent 353e72a commit 80e26e8

15 files changed

Lines changed: 442 additions & 722 deletions

File tree

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ The separate PineForge compiler, [`pineforge-codegen`](https://github.com/pinefo
3131

3232
- **Proven, not promised.** All <!-- pf:scoreboard.graded|int -->7,989<!-- /pf --> graded probes — <!-- pf:inventory.corpusScripts|int -->309<!-- /pf --> open reference strategies, <!-- pf:inventory.communityScripts|int -->680<!-- /pf --> community-shared scripts and <!-- pf:inventory.probeScripts|int -->61<!-- /pf --> probe scripts the maintainers wrote, on <!-- pf:scoreboard.lanes|int -->18<!-- /pf --> market/timeframe lanes — grade against TradingView's own trade lists: **<!-- pf:scoreboard.excellent|int -->7,983<!-- /pf --> excellent, <!-- pf:scoreboard.strong|int -->6<!-- /pf --> strong, <!-- pf:scoreboard.belowStrong|int -->0<!-- /pf --> below strong**. The graded probes' TradingView trade lists hold <!-- pf:inventory.tvTrades|int -->4,776,328<!-- /pf --> trades; <!-- pf:scoreboard.anomaliesExcluded|int -->17<!-- /pf --> probes with TradingView-side defects are excluded.
3333
- **Open runtime.** The engine and native live runner are Apache-2.0. The separately distributed [PineForge compiler](https://github.com/pineforge-4pass/pineforge-codegen-oss/blob/main/LICENSE) is source-available under the PineForge Source License 1.2 from its release 1.3.0 (release 1.2.0 keeps the PineForge Source License 1.1, and releases up to 1.1.0 keep the PolyForm Noncommercial terms they shipped with); personal trading is free, and commercial use requires a separate license; the capital in an account a proprietary-trading firm or funded-trader program provides or allocates, including a challenge, evaluation or simulated account, is investment capital. Public reference strategies, benchmarks and validation tooling are available in their respective repositories; the community-script test set is not redistributed.
34-
- **Fast.** In-process, no interpreter: median **36× faster than PyneCore** on the 200 strategies both engines time, measured at engine `35db01c8` on 2026-09-29 on an AWS c7a.8xlarge with PyneCore timed as a subprocess, its interpreter start-up included (PineForge runs a median 807k bars/s per strategy over its 201 slots with the bar magnifier on; [method](benchmarks/results/speed.md)). Parameter sweeps re-run a loaded `.so` with new inputs — no recompile, no fork.
34+
- **Fast.** In-process, no interpreter: median **36× faster than PyneCore** on the 200 strategies both engines time (100 corpus probes and 100 held-out private strategies), measured at engine `35db01c8` on 2026-09-29 on an AWS c7a.8xlarge with PyneCore timed as a subprocess, its interpreter start-up included (PineForge runs a median 807k bars/s per strategy over its 201 slots with the bar magnifier on; [method](benchmarks/results/speed.md)). Parameter sweeps re-run a loaded `.so` with new inputs — no recompile, no fork.
3535
- **Deterministic to the bit.** Two runs with the same inputs produce identical trade lists. Same on Linux and macOS.
3636
- **Yours to embed.** 145 `PF_API` declarations across five installed headers — `pineforge.h`'s 94 (74 of the 80 runtime exports, the other six being declared in `execution_observer.h` and `selected_window.h`, three each, and up to 20 per-strategy functions a generated module defines), `native_c_api.h`'s 43 native-host declarations and `selected_window_plan.h`'s two planner declarations — an ABI that is append-only within a major version from 1.0 on. Call it from C, Python, Rust, Go, Node, Julia — or let an AI agent drive it over MCP.
3737

@@ -380,7 +380,7 @@ Slot 192 has no PyneCore trade list: PyneSys rejects its source (`"Empty documen
380380
**Speed** was measured on an AWS c7a.8xlarge (AMD EPYC 9R14, 32 cores, SMT off) running Ubuntu 24.04, every engine in the same window at engine `35db01c8`, each timing process pinned to fixed cores ([`benchmarks/results/speed.md`](benchmarks/results/speed.md)). The 2026-09-22 table was timed on an Apple M4 Max, so its times are not compared with these; only ratios measured in one window are.
381381

382382
- **PineForge:** a median of 66.8 ms per strategy over the feed, in-process with the bar magnifier on (807k bars/s; quartiles 505k–1133k).
383-
- **PyneCore 6.10.3:** a median of 2,629 ms per subprocess (20.5k bars/s). The median per-strategy speedup is **36×** across the 200 strategies both engines time (p5 15×, p95 131×).
383+
- **PyneCore 6.10.3:** a median of 2,629 ms per subprocess (20.5k bars/s). The median per-strategy speedup is **36×** across the 200 strategies both engines time (100 corpus probes and 100 held-out private strategies; p5 15×, p95 131×).
384384
- **vectorbt:** a median of 209.3 ms for its 13 ports; PineForge's median per-strategy speedup on the same 13 is 4.2×, and vectorbt is faster on 2 of them.
385385
- **PineTS:** 1416.2 ms for the canonical 10-indicator script.
386386
- **Throughput package** ([`benchmarks/throughput/`](benchmarks/throughput/)): the magnifier-off hot loop runs at a median of **0.78 M bars/s** per strategy (N=201, median of five quiet runs).

‎benchmarks/README.md‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ This directory compares **PineForge** with two open-source PineScript runtimes a
66
- [**PineTS**](https://github.com/LuxAlgo/PineTS): a TypeScript transpiler and runtime that runs raw `.pine` source in Node.js or browsers. AGPL-3.0. It has shipped a `strategy.*` namespace since 0.9.17, but this harness runs only its indicator layer, through the canonical indicator script.
77
- [**vectorbt**](https://github.com/polakowo/vectorbt): a vectorized Pandas/NumPy/Numba backtester. It runs the hand-written `strategy_vbt.py` ports that some slots ship.
88

9+
The PyneCore harness is not part of this repository and is not distributed with it. `run_all.sh` runs the PyneCore leg only when `PYNECORE_HARNESS_DIR` names a directory holding `runners/run_pynecore.py` and `speed/time_pynecore.py`; otherwise it skips the leg with a message and the speed table is written without PyneCore. The PyneCore figures published here were measured on 2026-09-29 at engine `35db01c8` and are not re-measured by this repository.
10+
911
## Headline
1012

1113
Last refresh **2026-09-29**. Versions: engine `main` `35db01c8` running the committed `generated.cpp` (codegen `121b3e6a`), PyneCore 6.10.3, PineTS 0.9.34 and vectorbt 0.28.2, on an AWS c7a.8xlarge (AMD EPYC 9R14, 32 cores, SMT off) running Ubuntu 24.04.
@@ -41,7 +43,7 @@ Every engine was timed in the same window on this host, at engine `35db01c8`, wi
4143
| vectorbt | in-process, the 13 ports that load | 13 | 209.3 ms | — |
4244
| PineTS | subprocess wall time of the canonical 10-indicator script | 1 | 1416.2 ms | — |
4345

44-
- PineForge's median speedup is **36× over PyneCore**, per strategy across the 200 strategies both engines time (p5 15×, p95 131×), and 4.2× over vectorbt across its 13 ports.
46+
- PineForge's median speedup is **36× over PyneCore**, per strategy across the 200 strategies both engines time (100 corpus probes and 100 held-out private strategies; p5 15×, p95 131×), and 4.2× over vectorbt across its 13 ports.
4547
- The throughput package ([`throughput/`](throughput/)) measures the magnifier-off hot loop at a median of **0.78 M bars/s** per strategy. That figure is over all 201 slots, and is the median of five quiet runs.
4648
- The PineForge sweep ran at a 1-minute load of 4.68 (slots 001–100) and 1.32 (slots 101–201). Against engine `063e4460`, timed alternately with `35db01c8` in one window on 201 slots, `35db01c8` takes 0.35× the time per strategy at the median, each engine running its own `generated.cpp`.
4749
- On the same host and in the same window, PyneCore 6.10.3 takes 1.01× the time of 6.10.2 per strategy at the median, over 26 public slots timed alternately.
@@ -109,12 +111,12 @@ SKIP_BUILD=1 SKIP_PYNE=1 SKIP_PINETS=1 SKIP_SPEED=1 SKIP_REPORTS=1 JOBS=8 SLOTS=
109111
SKIP_BUILD=1 SKIP_PYNE=1 SKIP_PINETS=1 SKIP_SPEED=1 SKIP_REPORTS=1 SKIP_VECTORBT=1 SKIP_INDICATORS=1 JOBS=8 SLOTS=101-201 bash benchmarks/run_all.sh
110112
python3 benchmarks/compare.py && python3 benchmarks/compare_indicators.py # results/{summary,trade_comparison,indicator_comparison}.md
111113

112-
# speed on a quiet host (1-min load < 6, no cmake --build / ctest / ci_verify), all engines in one command
113-
QUIET_LOAD_MAX=6 SKIP_BUILD=1 SKIP_PINEFORGE=1 SKIP_PYNE=1 SKIP_PINETS=1 SKIP_REPORTS=1 bash benchmarks/run_all.sh
114+
# speed on a quiet host (1-min load < 6, no cmake --build / ctest / ci_verify), PyneCore when PYNECORE_HARNESS_DIR is set
115+
QUIET_LOAD_MAX=6 SKIP_BUILD=1 SKIP_PINEFORGE=1 SKIP_PINETS=1 SKIP_REPORTS=1 bash benchmarks/run_all.sh
114116
# or in chunks, re-checking the gate before each; the closed PyneCore slots run 1-72 s per backtest:
115117
./build/bin/pineforge_bench --benchmark_filter='/throughput/with_magnifier' --benchmark_format=json > benchmarks/_workdir/pf_speed.json
116118
# (or per slot range, e.g. --benchmark_filter='^(0[0-9][0-9]|100)-[^/]*/throughput/with_magnifier', then merge the "benchmarks" arrays)
117-
(cd benchmarks && uv run python speed/time_pynecore.py --n 20 --workers 8 \
119+
(cd benchmarks && PINEFORGE_REPO_ROOT="$(cd .. && pwd)" uv run python "${PYNECORE_HARNESS_DIR}/speed/time_pynecore.py" --n 20 --workers 8 \
118120
--first 178,162,164,163,165,132,114,171,181,168,189,105 --out _workdir/pc_speed.json)
119121
# (--first starts the longest slots first, so the pool stays 8 wide; one batch took an hour on the AWS host)
120122
(cd benchmarks && N=20 node speed/time_pinets.mjs > _workdir/pt_speed.json)
@@ -132,7 +134,7 @@ PyneSys is not needed to reproduce: the committed `strategy_pyne.py` files are t
132134

133135
**`run_all.sh` knobs:**
134136

135-
- `SKIP_BUILD`, `SKIP_PINEFORGE`, `SKIP_PYNE`, `SKIP_PINETS`, `SKIP_VECTORBT`, `SKIP_INDICATORS`, `SKIP_SPEED` and `SKIP_REPORTS` each skip one step.
137+
- `SKIP_BUILD`, `SKIP_PINEFORGE`, `SKIP_PINETS`, `SKIP_VECTORBT`, `SKIP_INDICATORS`, `SKIP_SPEED` and `SKIP_REPORTS` skip their named steps. `SKIP_PYNE=1` skips only PyneCore strategy runs and the canonical indicator run; `.ohlcv` conversion and PyneCore timing still run when the harness is present.
136138
- `JOBS` runs parallel parity jobs. It never affects timing.
137139
- `SLOTS` narrows the loops to slot ranges, for example `1-50,120`.
138140
- `QUIET_LOAD_MAX` holds each timing batch until the host is quiet. Each batch's load is recorded in `_workdir/speed_loads.tsv` and in `speed.md`.
@@ -173,11 +175,9 @@ benchmarks/
173175
├── assets-closed/ maintainers only (gitignored): the 101 closed slots, from the evidence store
174176
├── runners/
175177
│ ├── run_pineforge_canonical.cpp PineForge canonical indicator runner (links libpineforge)
176-
│ ├── run_pinets_canonical.mjs PineTS canonical indicator runner (Node)
177-
│ └── run_pynecore.py PyneCore strategy runner (wraps `pyne run` + TV-schema normalize)
178+
│ └── run_pinets_canonical.mjs PineTS canonical indicator runner (Node)
178179
├── speed/
179180
│ ├── pineforge_bench.cpp Google Benchmark harness for PineForge (in-process; both slot roots)
180-
│ ├── time_pynecore.py subprocess wall-time timer for PyneCore (--slots/--out for chunks)
181181
│ ├── time_pinets.mjs subprocess wall-time timer for PineTS canonical
182182
│ ├── time_vectorbt.py in-process timer (and trade writer) for the vectorbt ports
183183
│ ├── aggregate.py combine the timing JSONs + host loads -> results/speed.md
@@ -203,7 +203,7 @@ benchmarks/
203203
For each slot:
204204

205205
1. **PineForge:** CMake builds `strategy.dylib` from the committed `generated.cpp`. `scripts/run_strategy.py` loads it through ctypes, runs the feed with the slot's `inputs.json` and emits `pineforge_trades.csv` in TradingView's schema.
206-
2. **PyneCore:** `pyne run` executes the committed `strategy_pyne.py` against the same feed. Same-symbol daily, weekly and monthly `request.security` contexts are served from the chart feed. `runners/run_pynecore.py` normalizes the output to TradingView's schema as `pynecore_trades.csv`.
206+
2. **PyneCore:** `pyne run` executes the committed `strategy_pyne.py` against the same feed. Same-symbol daily, weekly and monthly `request.security` contexts are served from the chart feed. The external harness's `runners/run_pynecore.py` normalizes the output to TradingView's schema as `pynecore_trades.csv`.
207207
3. **vectorbt:** `speed/time_vectorbt.py --write-trades` runs every port that loads and writes `vectorbt_trades.csv`.
208208
4. **Ground truth:** `tv_trades.csv` is exported from TradingView's broker emulator running the same `.pine` source on the same market and timeframe.
209209

@@ -224,7 +224,7 @@ A single canonical script ([`assets/strategies/_indicators/canonical.pine`](http
224224
### Speed measurement
225225

226226
- **PineForge** is timed with Google Benchmark's in-process hot loop, with the bar magnifier on (1→4 ENDPOINTS sub-bar sampling), which is the engine's most expensive configuration. The strategy library is `dlopen`ed once, outside the timed region. Each timed iteration runs `strategy_create`, applies the slot's `inputs.json` `strategy_overrides` (the settings the graded run uses), and runs `run_backtest_full` over the whole feed; the figure is the per-iteration mean over 20 iterations, and each benchmark also reports its run's trade count.
227-
- **PyneCore** is timed as the subprocess wall time of `uv run python runners/run_pynecore.py <slot> --no-write`, which includes Python startup and framework import. The figures are the median and p95 over 20 invocations, with 8 slots timed concurrently.
227+
- **PyneCore** is timed with a harness kept outside this repository, as subprocess wall time including Python startup and framework import. The figures are the median and p95 over 20 invocations, with 8 slots timed concurrently.
228228
- **vectorbt** is timed in-process: the median over 20 iterations of each port.
229229
- **PineTS** is timed as the subprocess wall time of `node runners/run_pinets_canonical.mjs`. The harness has no PineTS strategy runner, so the canonical indicator script stands in for its indicator-layer cost.
230230
- **The quiet-host gate:** before every timing batch, the 1-minute load average must be below 6 with no `cmake --build`, `ctest` or `ci_verify` process running, counted by executable name. `speed.md` lists the load at every batch.

‎benchmarks/check_provenance.py‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -457,15 +457,23 @@ def pinets_row(tree, m):
457457

458458

459459
@claim(r"PineForge's median speedup is \*\*(?P<x>\d+)× over PyneCore\*\*, per strategy across the "
460-
r"(?P<n>\d+) strategies both engines time \(p5 (?P<p5>\d+)×, p95 (?P<p95>\d+)×\), and "
460+
r"(?P<n>\d+) strategies both engines time \((?P<probes>\d+) corpus probes and (?P<held>\d+) "
461+
r"held-out private strategies; p5 (?P<p5>\d+)×, p95 (?P<p95>\d+)×\), and "
461462
r"(?P<vbt>[\d.]+)× over vectorbt across its (?P<vbt_n>\d+) ports\.")
462463
def speedup(tree, m):
463464
pf = pineforge_ms(tree)
464465
pc = tree.json(f"{NOW}/pc_speed.json")
465466
vbt = tree.json(f"{NOW}/vbt_speed.json")
466467
over_pc = [pc[s]["median_ms"] / pf[s] for s in pf if s in pc]
467468
over_vbt = [vbt[s]["median_ms"] / pf[s] for s in pf if s in vbt]
469+
timed = [s for s in pf if s in pc]
470+
closed = {s["slot"] for s in tree.json(SELECTION)["slots"] if s["source"] == "closed"}
471+
held = sum(1 for s in timed if s in closed)
472+
probes = sum(1 for s in timed if s not in closed)
473+
if probes + held != len(over_pc):
474+
raise Mismatch("the timed corpus probes and held-out strategies do not sum to n")
468475
return {"x": Num(quantile(over_pc, .5)), "n": len(over_pc),
476+
"probes": probes, "held": held,
469477
"p5": Num(quantile(over_pc, .05)), "p95": Num(quantile(over_pc, .95)),
470478
"vbt": Num(quantile(over_vbt, .5)), "vbt_n": len(over_vbt)}
471479

‎benchmarks/compare.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515
1616
tv_trades.csv TradingView ground truth
1717
pineforge_trades.csv scripts/run_strategy.py output (run_all.sh)
18-
pynecore_trades.csv runners/run_pynecore.py output (run_all.sh)
18+
pynecore_trades.csv private runners/run_pynecore.py output (run_all.sh, optional)
1919
vectorbt_trades.csv speed/time_vectorbt.py --write-trades (optional)
2020
2121
An engine without output is reported ``n/a`` with its reason: the PyneSys

0 commit comments

Comments
 (0)