Skip to content

Commit a02160f

Browse files
luisleo526claude
andcommitted
Bench: parity re-run and PineForge re-timed at engine 063e446 -- all 201 trade lists unchanged, 89 ms / 603k bars/s, 15x vs PyneCore (lane BENCH3)
Parity: every dylib and the GBench harness rebuilt on main 063e446; all 201 pineforge_trades.csv are byte-identical to the e9ad37d refresh, so summary.md, trade_comparison.md and indicator_comparison.md regenerate unchanged (corpus 100/100 excellent, closed 100 excellent + 1 strong, slot 181 unchanged). Speed: PineForge re-timed in two GBench batches (slots 001-100 at load 5.41, 101-201 at 5.90): median 89.45 ms, 603k bars/s (Q1 425k, Q3 731k), median 1.02x the e9ad37d time per strategy. PyneCore, PineTS and vectorbt are not re-measured; their timing files reproduce the e9ad37d report byte for byte. The throughput package is not re-timed: the host was never quiet. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent ada3813 commit a02160f

4 files changed

Lines changed: 294 additions & 267 deletions

File tree

‎README.md‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ The separate PineForge compiler, [`pineforge-codegen`](https://github.com/pinefo
3333

3434
- **Proven, not promised.** All 4,190 probes — 312 open reference strategies plus 413 real community scripts on 15 markets and timeframes — grade *excellent* or *strong* against TradingView's own trade lists: **4,182 excellent, 8 strong, zero moderate**. The current full sweep evaluates 2,819,967 TradingView trades, with 2,818,237 matched by the verifier.
3535
- **Open runtime.** The engine and native live runner are Apache-2.0. The separately distributed [PineForge compiler](https://github.com/pineforge-4pass/pineforge-codegen-oss/blob/main/LICENSE) uses PolyForm Noncommercial terms with additional personal-trading permission; commercial use requires a separate license. Public reference strategies, benchmarks and validation tooling are available in their respective repositories; the community-script test set is not redistributed.
36-
- **Fast.** In-process, no interpreter: median **15× faster than PyneCore** on 196 timed strategies (a median 611k bars/s per strategy with the bar magnifier on). Parameter sweeps re-run a loaded `.so` with new inputs — no recompile, no fork.
36+
- **Fast.** In-process, no interpreter: median **15× faster than PyneCore** on 196 timed strategies (a median 603k bars/s per strategy with the bar magnifier on). Parameter sweeps re-run a loaded `.so` with new inputs — no recompile, no fork.
3737
- **Deterministic to the bit.** Two runs with the same inputs produce identical trade lists. Same on Linux and macOS.
3838
- **Yours to embed.** 97 `extern "C"` functions across two headers — 65 to run a compiled strategy, 32 to drive the kernel yourself — append-only ABI. Call it from C, Python, Rust, Go, Node, Julia — or let an AI agent drive it over MCP.
3939

@@ -311,20 +311,20 @@ Four slots have no PyneCore trade list:
311311
- **1 compile rejection:** PyneSys rejects slot 192 (`"Empty document."`).
312312
- **3 runtime errors:** PyneCore raises a `RuntimeError` in its `request.security` engine on slots 140, 166 and 197. It hits the same error intermittently on a fourth slot, 143, which could not be timed.
313313
314-
**Speed** was measured on a quiet host ([`benchmarks/results/speed.md`](benchmarks/results/speed.md)):
314+
**Speed** was measured on a quiet host ([`benchmarks/results/speed.md`](benchmarks/results/speed.md)). PineForge was re-timed at engine `063e4460`. PyneCore, vectorbt and PineTS were not re-measured; their figures come from the same host's earlier 2026-09-22 window, at engine `e9ad37dd`.
315315

316-
- **PineForge:** a median of 88 ms per strategy over the feed, in-process with the bar magnifier on (611k bars/s; quartiles 434k–726k).
317-
- **PyneCore:** a median of 1,475 ms per subprocess (36.6k bars/s). The median per-strategy speedup is **15×** across the 196 strategies both engines time (p5 6×, p95 70×).
316+
- **PineForge:** a median of 89 ms per strategy over the feed, in-process with the bar magnifier on (603k bars/s; quartiles 425k–731k).
317+
- **PyneCore:** a median of 1,475 ms per subprocess (36.6k bars/s). The median per-strategy speedup is **15×** across the 196 strategies both engines time (p5 6×, p95 68×).
318318
- **vectorbt:** a median of 102 ms for its 13 ports; PineForge is 1.3× faster on the same 13.
319319
- **PineTS:** 486 ms for the canonical 10-indicator script.
320-
- **Throughput package** ([`benchmarks/throughput/`](benchmarks/throughput/)): the magnifier-off hot loop runs at a median of **0.61 M bars/s** per strategy (N=201, median of five quiet runs).
320+
- **Throughput package** ([`benchmarks/throughput/`](benchmarks/throughput/)): the magnifier-off hot loop runs at a median of **0.61 M bars/s** per strategy (N=201, median of five quiet runs). Those runs were measured at `e9ad37dd`; they were not re-timed at `063e4460` because the host was never quiet.
321321

322322
**Not comparable with the 2026-06-11 table** (PineForge 100/100, PyneCore 85/100, 162×):
323323

324324
- **Tiers:** they now come from the canonical `scripts/verify_corpus.py::analyze_strategy` rubric. The old table graded a different 100-strategy population with `compare.py`'s own copy of the rubric, which had drifted from the canonical one and no longer parsed the current tape format.
325-
- **Speed:** the ratio fell because the engine is slower per bar, not because the host changed. The 2026-06-11 engine, rebuilt on the same host, reproduces its June timings, while the current engine is 12–20× slower on the probes both populations share ([provenance](benchmarks/results/speed.md#provenance)).
325+
- **Speed:** the ratio fell because the engine is slower per bar, not because the host changed. The 2026-06-11 engine, rebuilt on the same host, reproduced its June timings, while engine `e9ad37dd` was 12–20× slower on the probes both populations share. That A/B was measured at `e9ad37dd`; `063e4460` times those probes within 7 % of it ([provenance](benchmarks/results/speed.md#provenance)).
326326

327-
Last refresh **2026-09-22** (engine `e9ad37dd`, PyneCore 6.10.2, PineTS 0.9.34, vectorbt 0.28.2, Apple M4 Max). Per-strategy table: [`benchmarks/results/summary.md`](benchmarks/results/summary.md). Population manifest: [`benchmarks/results/selection.md`](benchmarks/results/selection.md). Method, fairness and the reproduction recipe: [`benchmarks/README.md`](benchmarks/README.md).
327+
Last refresh **2026-09-22** (engine `063e4460`, PyneCore 6.10.2, PineTS 0.9.34, vectorbt 0.28.2, Apple M4 Max). Per-strategy table: [`benchmarks/results/summary.md`](benchmarks/results/summary.md). Population manifest: [`benchmarks/results/selection.md`](benchmarks/results/selection.md). Method, fairness and the reproduction recipe: [`benchmarks/README.md`](benchmarks/README.md).
328328

329329
---
330330

‎benchmarks/README.md‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ This directory compares **PineForge** with two open-source PineScript runtimes a
88

99
## Headline
1010

11-
Last refresh **2026-09-22**. Versions: engine `main` `e9ad37dd` running the committed `generated.cpp` (codegen `89645d6`), PyneCore 6.10.2, PineTS 0.9.34 and vectorbt 0.28.2, on an Apple M4 Max.
11+
Last refresh **2026-09-22**. Versions: engine `main` `063e4460` running the committed `generated.cpp` (codegen `89645d6`), PyneCore 6.10.2, PineTS 0.9.34 and vectorbt 0.28.2, on an Apple M4 Max.
1212

1313
**Population: 200 strategies in 201 slots**, selected by [`select_population.py`](select_population.py) with seed 20260921. The manifest is [`results/selection.md`](results/selection.md).
1414

@@ -30,22 +30,23 @@ Each engine's trade list is graded against TradingView's own export (266,451 tra
3030

3131
Counting the 200 strategies (slot `201` in place of `192`), PineForge grades 199 excellent and 1 strong. PyneCore grades 133 excellent, 42 strong, 8 moderate, 12 weak and 2 minimal, and 3 slots have no trade list.
3232

33-
Speed was measured on a quiet host, re-checked before every timing batch; the details are in [`results/speed.md`](results/speed.md):
33+
Speed was measured on a quiet host, re-checked before every timing batch; the details are in [`results/speed.md`](results/speed.md). PineForge was re-timed at engine `063e4460`. PyneCore, vectorbt and PineTS were not re-measured: their rows come from the same host's earlier 2026-09-22 window, at engine `e9ad37dd`.
3434

3535
| Engine | How it is timed | Strategies | Median per strategy | Bars/s: Q1 · **median** · Q3 |
3636
|---|---|---:|---:|---|
37-
| PineForge | in-process Google Benchmark, bar magnifier on | 201 | 88.3 ms | 434k · **611k** · 726k |
37+
| PineForge | in-process Google Benchmark, bar magnifier on | 201 | 89.4 ms | 425k · **603k** · 731k |
3838
| PyneCore | subprocess wall time (interpreter start, import, backtest), 8 concurrent | 196 | 1,475 ms | 23.3k · **36.6k** · 58.5k |
3939
| vectorbt | in-process, the 13 ports that load | 13 | 102.3 ms | — |
4040
| PineTS | subprocess wall time of the canonical 10-indicator script | 1 | 485.8 ms | — |
4141

42-
- PineForge's median speedup is **15× over PyneCore**, per strategy across the 196 strategies both engines time (p5 6×, p95 70×), and 1.3× over vectorbt across its 13 ports.
43-
- The throughput package ([`throughput/`](throughput/)) measures the magnifier-off hot loop at a median of **0.61 M bars/s** per strategy. That figure is over all 201 slots, and is the median of five quiet runs.
42+
- PineForge's median speedup is **15× over PyneCore**, per strategy across the 196 strategies both engines time (p5 6×, p95 68×), and 1.3× over vectorbt across its 13 ports.
43+
- The throughput package ([`throughput/`](throughput/)) measures the magnifier-off hot loop at a median of **0.61 M bars/s** per strategy. That figure is over all 201 slots, and is the median of five quiet runs at engine `e9ad37dd`. It was not re-timed at `063e4460` because the host was never quiet.
44+
- Per strategy, the `063e4460` sweep takes 1.02× the `e9ad37dd` sweep's time at the median (p5 0.95×, p95 1.07×), at a higher host load (5.41–5.90 against 3.79).
4445

4546
**These numbers are not comparable with the 2026-06-11 table** (PineForge 100/100 excellent, PyneCore 85/100, 162×):
4647

4748
- **Tiers:** that table graded a different 100-strategy population with `compare.py`'s own copy of the rubric. The copy had drifted from the canonical rubric and no longer parsed the current tape format. `compare.py` now calls the canonical rubric directly, and PyneCore moved from 6.4.6 to 6.10.2 in between.
48-
- **Speed:** the ratio fell because the engine is slower per bar, not because the host changed. The 2026-06-11 engine, rebuilt on this host in the same window, reproduces its June timings. On the three probes both populations share, the current engine is 12–20× slower with the magnifier on; see [the provenance](results/speed.md#provenance).
49+
- **Speed:** the ratio fell because the engine is slower per bar, not because the host changed. The 2026-06-11 engine, rebuilt on this host in the `e9ad37dd` window, reproduced its June timings. On the three probes both populations share, engine `e9ad37dd` was 12–20× slower with the magnifier on. That A/B was measured at `e9ad37dd` and not re-run; `063e4460` times the same probes within 7 % of `e9ad37dd`. See [the provenance](results/speed.md#provenance).
4950

5051
### Where the non-excellent rows come from
5152

@@ -105,6 +106,7 @@ python3 benchmarks/compare.py && python3 benchmarks/compare_indicators.py # re
105106
QUIET_LOAD_MAX=6 SKIP_BUILD=1 SKIP_PINEFORGE=1 SKIP_PYNE=1 SKIP_PINETS=1 SKIP_REPORTS=1 bash benchmarks/run_all.sh
106107
# or in chunks, re-checking the gate before each; the closed PyneCore slots run 1-72 s per backtest:
107108
./build/bin/pineforge_bench --benchmark_filter='/throughput/with_magnifier' --benchmark_format=json > benchmarks/_workdir/pf_speed.json
109+
# (or per slot range, e.g. --benchmark_filter='^(0[0-9][0-9]|100)-[^/]*/throughput/with_magnifier', then merge the "benchmarks" arrays)
108110
(cd benchmarks && uv run python speed/time_pynecore.py --n 20 --workers 8 --slots 1-56 --out _workdir/pc_speed_c01.json)
109111
# ... one chunk per slot range up to 201, then merge them into _workdir/pc_speed.json
110112
(cd benchmarks && N=20 node speed/time_pinets.mjs > _workdir/pt_speed.json)

0 commit comments

Comments
 (0)