You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 4412a34
Browse filesBrowse the repository at this point in the historyBrowse files
Bench: throughput package re-timed at engine 063e446 -- 0.64 M bars/s over five quiet runs; June-engine A/B re-measured beside e9ad37d (lane BENCH3)
Throughput: five gate-checked reproduce.sh runs (run medians 0.621, 0.638,
0.611, 0.646, 0.640 M bars/s); the package publishes run 2, the median run
(e9ad37d measured 0.614). Two runs were discarded: one saw another project's
load spike to 16.8 before its benchmark began, one recompiled the bench tree
inside the run after a commit moved the version header.
A/B, one window, alternating: 063e446 takes 0.94x the time of e9ad37d per
strategy on 26 public slots (22 of 26 faster), and 10-18x the June engine
ac011d8 on the three probes both populations share (e9ad37d: 11-19x). The
published PineForge column stays the sweep at load 5.41/5.90; its cross-window
comparison with the e9ad37d sweep (1.02x) is the weaker one and is labelled so.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Copy file name to clipboardExpand all lines: README.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -317,12 +317,12 @@ Four slots have no PyneCore trade list:
317
317
-**PyneCore:** a median of 1,475 ms per subprocess (36.6k bars/s). The median per-strategy speedup is **15×** across the 196 strategies both engines time (p5 6×, p95 68×).
318
318
-**vectorbt:** a median of 102 ms for its 13 ports; PineForge is 1.3× faster on the same 13.
319
319
-**PineTS:** 486 ms for the canonical 10-indicator script.
320
-
-**Throughput package** ([`benchmarks/throughput/`](benchmarks/throughput/)): the magnifier-off hot loop runs at a median of **0.61 M bars/s** per strategy (N=201, median of five quiet runs). Those runs were measured at `e9ad37dd`; they were not re-timed at `063e4460` because the host was never quiet.
320
+
-**Throughput package** ([`benchmarks/throughput/`](benchmarks/throughput/)): the magnifier-off hot loop runs at a median of **0.64 M bars/s** per strategy (N=201, median of five quiet runs).
321
321
322
322
**Not comparable with the 2026-06-11 table** (PineForge 100/100, PyneCore 85/100, 162×):
323
323
324
324
-**Tiers:** they now come from the canonical `scripts/verify_corpus.py::analyze_strategy` rubric. The old table graded a different 100-strategy population with `compare.py`'s own copy of the rubric, which had drifted from the canonical one and no longer parsed the current tape format.
325
-
-**Speed:** the ratio fell because the engine is slower per bar, not because the host changed. The 2026-06-11 engine, rebuilt on the same host, reproduced its June timings, while engine `e9ad37dd` was 12–20× slower on the probes both populations share. That A/B was measured at `e9ad37dd`; `063e4460` times those probes within 7 % of it ([provenance](benchmarks/results/speed.md#provenance)).
325
+
-**Speed:** the ratio fell because the engine is slower per bar, not because the host changed. The 2026-06-11 engine, rebuilt and timed on the same host in the same window, still runs close to its June timings (5–16 % over them in the quieter pass), while the current engine is 10–18× slower on the probes both populations share ([provenance](benchmarks/results/speed.md#provenance)).
326
326
327
327
Last refresh **2026-09-22** (engine `063e4460`, PyneCore 6.10.2, PineTS 0.9.34, vectorbt 0.28.2, Apple M4 Max). Per-strategy table: [`benchmarks/results/summary.md`](benchmarks/results/summary.md). Population manifest: [`benchmarks/results/selection.md`](benchmarks/results/selection.md). Method, fairness and the reproduction recipe: [`benchmarks/README.md`](benchmarks/README.md).
Copy file name to clipboardExpand all lines: benchmarks/README.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -40,13 +40,13 @@ Speed was measured on a quiet host, re-checked before every timing batch; the de
40
40
| PineTS | subprocess wall time of the canonical 10-indicator script | 1 | 485.8 ms | — |
41
41
42
42
- PineForge's median speedup is **15× over PyneCore**, per strategy across the 196 strategies both engines time (p5 6×, p95 68×), and 1.3× over vectorbt across its 13 ports.
43
-
- The throughput package ([`throughput/`](throughput/)) measures the magnifier-off hot loop at a median of **0.61 M bars/s** per strategy. That figure is over all 201 slots, and is the median of five quiet runs at engine `e9ad37dd`. It was not re-timed at `063e4460` because the host was never quiet.
44
-
-Per strategy, the `063e4460` sweep takes 1.02× the `e9ad37dd` sweep's time at the median (p5 0.95×, p95 1.07×), at a higher host load (5.41–5.90 against 3.79).
43
+
- The throughput package ([`throughput/`](throughput/)) measures the magnifier-off hot loop at a median of **0.64 M bars/s** per strategy. That figure is over all 201 slots, and is the median of five quiet runs.
44
+
-The PineForge sweep ran at a 1-minute load of 5.41 (slots 001–100) and 5.90 (slots 101–201). Against engine `e9ad37dd`, timed alternately with `063e4460` in one window on 26 public slots, `063e4460` takes 0.94× the time per strategy at the median.
45
45
46
46
**These numbers are not comparable with the 2026-06-11 table** (PineForge 100/100 excellent, PyneCore 85/100, 162×):
47
47
48
48
-**Tiers:** that table graded a different 100-strategy population with `compare.py`'s own copy of the rubric. The copy had drifted from the canonical rubric and no longer parsed the current tape format. `compare.py` now calls the canonical rubric directly, and PyneCore moved from 6.4.6 to 6.10.2 in between.
49
-
-**Speed:** the ratio fell because the engine is slower per bar, not because the host changed. The 2026-06-11 engine, rebuilt on this host in the `e9ad37dd` window, reproduced its June timings. On the three probes both populations share, engine `e9ad37dd` was 12–20× slower with the magnifier on. That A/B was measured at `e9ad37dd` and not re-run; `063e4460` times the same probes within 7 % of `e9ad37dd`. See[the provenance](results/speed.md#provenance).
49
+
-**Speed:** the ratio fell because the engine is slower per bar, not because the host changed. The 2026-06-11 engine, rebuilt on this host and timed in the same window as the current one, still runs close to its June timings (5–16 % over them in the quieter pass). On the three probes both populations share, the current engine is 10–18× slower with the magnifier on; see[the provenance](results/speed.md#provenance).
*Note: the metric is GBench's `<slug>/throughput/no_magnifier` hot loop. Each strategy's shared library is `dlopen`ed once, outside the timed region. Each of the 20 timed iterations runs `strategy_create` plus `run_backtest` over the whole feed with the bar magnifier off, and the table reports the mean.*
26
26
27
-
*Last measured 2026-09-22 on an Apple M4 Max (16 cores) with engine `main``e9ad37dd`. This is the median of five quiet runs: the five run medians were 0.604, 0.615, 0.606, 0.620 and 0.614 M/s, and the table and chart come from the run with the median result (run 5). Every run started at a 1-minute load below 6 with no build or test process running; the loads are in [`../results/speed.md`](../results/speed.md). A public checkout only has the 100 public slots, whose median in the same run is 0.686 M/s.*
27
+
*Last measured 2026-09-22 on an Apple M4 Max (16 cores) with engine `main``063e4460`. This is the median of five quiet runs: the five run medians were 0.621, 0.638, 0.611, 0.646 and 0.640 M/s, and the table and chart come from the run with the median result (run 2). Every run started at a 1-minute load below 6 with no build or test process running; the loads are in [`../results/speed.md`](../results/speed.md). A public checkout only has the 100 public slots, whose median in the same run is 0.727 M/s. At engine `e9ad37dd` the same package measured 0.614 M/s.*
28
28
29
-
*These figures were measured at `e9ad37dd` and not re-timed at `063e4460`, because the host was never quiet during that refresh. The bar-magnifier sweep was re-timed at `063e4460`, and per strategy it takes 1.02× the `e9ad37dd` time at the median; see [`../results/speed.md`](../results/speed.md), Provenance.*
30
-
31
-
*The median fell from 17.05 M/s, measured 2026-05-29 on 100 strategies and a 41,307-bar feed. A same-host check at `e9ad37dd` against the engine of the 2026-06-11 speed table showed a per-bar cost 12–20× higher on the probes both populations share (see [`../results/speed.md`](../results/speed.md), Provenance).*
29
+
*The median fell from 17.05 M/s, measured 2026-05-29 on 100 strategies and a 41,307-bar feed. A same-host check against the engine of the 2026-06-11 speed table shows the current engine's per-bar cost is 10–18× higher on the probes both populations share (see [`../results/speed.md`](../results/speed.md), Provenance).*
32
30
33
31
### 🛠️ FFI Grid Search Optimization Result
34
32
@@ -78,7 +76,7 @@ This script will:
78
76
79
77
## 🔍 Auditing & Verifying Results (For Sceptics)
80
78
81
-
You can audit these figures (such as the median hot-loop throughput of 0.61 million bars per second) step by step:
79
+
You can audit these figures (such as the median hot-loop throughput of 0.64 million bars per second) step by step:
82
80
83
81
### A. Run a Single Strategy Individually
84
82
Instead of running every slot, you can compile and benchmark a single strategy of your choice (e.g. `001-analyzer-anvil-percent-costs-01`) using Google Benchmark directly to eliminate any script wrapper bias:
0 commit comments