Skip to content

Commit 4412a34

Browse files
luisleo526claude
andcommitted
Bench: throughput package re-timed at engine 063e446 -- 0.64 M bars/s over five quiet runs; June-engine A/B re-measured beside e9ad37d (lane BENCH3)
Throughput: five gate-checked reproduce.sh runs (run medians 0.621, 0.638, 0.611, 0.646, 0.640 M bars/s); the package publishes run 2, the median run (e9ad37d measured 0.614). Two runs were discarded: one saw another project's load spike to 16.8 before its benchmark began, one recompiled the bench tree inside the run after a commit moved the version header. A/B, one window, alternating: 063e446 takes 0.94x the time of e9ad37d per strategy on 26 public slots (22 of 26 faster), and 10-18x the June engine ac011d8 on the three probes both populations share (e9ad37d: 11-19x). The published PineForge column stays the sweep at load 5.41/5.90; its cross-window comparison with the e9ad37d sweep (1.02x) is the weaker one and is labelled so. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent 2cff809 commit 4412a34

6 files changed

Lines changed: 664 additions & 656 deletions

File tree

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -317,12 +317,12 @@ Four slots have no PyneCore trade list:
317317
- **PyneCore:** a median of 1,475 ms per subprocess (36.6k bars/s). The median per-strategy speedup is **15×** across the 196 strategies both engines time (p5 6×, p95 68×).
318318
- **vectorbt:** a median of 102 ms for its 13 ports; PineForge is 1.3× faster on the same 13.
319319
- **PineTS:** 486 ms for the canonical 10-indicator script.
320-
- **Throughput package** ([`benchmarks/throughput/`](benchmarks/throughput/)): the magnifier-off hot loop runs at a median of **0.61 M bars/s** per strategy (N=201, median of five quiet runs). Those runs were measured at `e9ad37dd`; they were not re-timed at `063e4460` because the host was never quiet.
320+
- **Throughput package** ([`benchmarks/throughput/`](benchmarks/throughput/)): the magnifier-off hot loop runs at a median of **0.64 M bars/s** per strategy (N=201, median of five quiet runs).
321321

322322
**Not comparable with the 2026-06-11 table** (PineForge 100/100, PyneCore 85/100, 162×):
323323

324324
- **Tiers:** they now come from the canonical `scripts/verify_corpus.py::analyze_strategy` rubric. The old table graded a different 100-strategy population with `compare.py`'s own copy of the rubric, which had drifted from the canonical one and no longer parsed the current tape format.
325-
- **Speed:** the ratio fell because the engine is slower per bar, not because the host changed. The 2026-06-11 engine, rebuilt on the same host, reproduced its June timings, while engine `e9ad37dd` was 12–20× slower on the probes both populations share. That A/B was measured at `e9ad37dd`; `063e4460` times those probes within 7 % of it ([provenance](benchmarks/results/speed.md#provenance)).
325+
- **Speed:** the ratio fell because the engine is slower per bar, not because the host changed. The 2026-06-11 engine, rebuilt and timed on the same host in the same window, still runs close to its June timings (5–16 % over them in the quieter pass), while the current engine is 10–18× slower on the probes both populations share ([provenance](benchmarks/results/speed.md#provenance)).
326326

327327
Last refresh **2026-09-22** (engine `063e4460`, PyneCore 6.10.2, PineTS 0.9.34, vectorbt 0.28.2, Apple M4 Max). Per-strategy table: [`benchmarks/results/summary.md`](benchmarks/results/summary.md). Population manifest: [`benchmarks/results/selection.md`](benchmarks/results/selection.md). Method, fairness and the reproduction recipe: [`benchmarks/README.md`](benchmarks/README.md).
328328

‎benchmarks/README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -40,13 +40,13 @@ Speed was measured on a quiet host, re-checked before every timing batch; the de
4040
| PineTS | subprocess wall time of the canonical 10-indicator script | 1 | 485.8 ms | — |
4141

4242
- PineForge's median speedup is **15× over PyneCore**, per strategy across the 196 strategies both engines time (p5 6×, p95 68×), and 1.3× over vectorbt across its 13 ports.
43-
- The throughput package ([`throughput/`](throughput/)) measures the magnifier-off hot loop at a median of **0.61 M bars/s** per strategy. That figure is over all 201 slots, and is the median of five quiet runs at engine `e9ad37dd`. It was not re-timed at `063e4460` because the host was never quiet.
44-
- Per strategy, the `063e4460` sweep takes 1.02× the `e9ad37dd` sweep's time at the median (p5 0.95×, p95 1.07×), at a higher host load (5.41–5.90 against 3.79).
43+
- The throughput package ([`throughput/`](throughput/)) measures the magnifier-off hot loop at a median of **0.64 M bars/s** per strategy. That figure is over all 201 slots, and is the median of five quiet runs.
44+
- The PineForge sweep ran at a 1-minute load of 5.41 (slots 001–100) and 5.90 (slots 101–201). Against engine `e9ad37dd`, timed alternately with `063e4460` in one window on 26 public slots, `063e4460` takes 0.94× the time per strategy at the median.
4545

4646
**These numbers are not comparable with the 2026-06-11 table** (PineForge 100/100 excellent, PyneCore 85/100, 162×):
4747

4848
- **Tiers:** that table graded a different 100-strategy population with `compare.py`'s own copy of the rubric. The copy had drifted from the canonical rubric and no longer parsed the current tape format. `compare.py` now calls the canonical rubric directly, and PyneCore moved from 6.4.6 to 6.10.2 in between.
49-
- **Speed:** the ratio fell because the engine is slower per bar, not because the host changed. The 2026-06-11 engine, rebuilt on this host in the `e9ad37dd` window, reproduced its June timings. On the three probes both populations share, engine `e9ad37dd` was 12–20× slower with the magnifier on. That A/B was measured at `e9ad37dd` and not re-run; `063e4460` times the same probes within 7 % of `e9ad37dd`. See [the provenance](results/speed.md#provenance).
49+
- **Speed:** the ratio fell because the engine is slower per bar, not because the host changed. The 2026-06-11 engine, rebuilt on this host and timed in the same window as the current one, still runs close to its June timings (5–16 % over them in the quieter pass). On the three probes both populations share, the current engine is 10–18× slower with the magnifier on; see [the provenance](results/speed.md#provenance).
5050

5151
### Where the non-excellent rows come from
5252

‎benchmarks/results/speed.md‎

Lines changed: 46 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -32,11 +32,12 @@ Timing is valid only on a quiet host: 1-minute load average below 6.0 and no `cm
3232
| pynecore-c10+178 (e9ad37dd window, not re-measured) | 2026-09-22T04:36:29Z | 5.77 | 0 |
3333
| pinets (e9ad37dd window, not re-measured) | 2026-09-22T04:44:08Z | 4.54 | 0 |
3434
| vectorbt (e9ad37dd window, not re-measured) | 2026-09-22T04:44:28Z | 4.31 | 0 |
35-
| throughput-r1 (e9ad37dd) | 2026-09-22T04:47:36Z | 3.98 | 0 |
36-
| throughput-r2 (e9ad37dd) | 2026-09-22T04:56:19Z | 5.90 | 0 |
37-
| throughput-r3 (e9ad37dd) | 2026-09-22T05:04:55Z | 5.93 | 0 |
38-
| throughput-r4 (e9ad37dd) | 2026-09-22T05:13:32Z | 5.45 | 0 |
39-
| throughput-r5 (e9ad37dd) | 2026-09-22T05:22:05Z | 4.93 | 0 |
35+
| throughput-r1 (063e4460) | 2026-09-22T13:29:51Z | 3.62 | 0 |
36+
| throughput-r2 (063e4460) | 2026-09-22T13:38:22Z | 4.95 | 0 |
37+
| throughput-r3 (063e4460) | 2026-09-22T13:46:30Z | 3.18 | 0 |
38+
| throughput-r4 (063e4460) | 2026-09-22T14:00:07Z | 2.97 | 0 |
39+
| throughput-r5 (063e4460) | 2026-09-22T14:08:13Z | 3.82 | 0 |
40+
| A/B 063e4460 vs e9ad37dd vs ac011d84 | 2026-09-22T14:17:12Z | 5.40 | 0 |
4041

4142
## Methodology
4243

@@ -295,31 +296,40 @@ realistic cost for engines whose entry point IS the process.
295296
- **PineForge re-timed at `063e4460`.** Lane BENCH3 re-ran the PineForge sweep on 2026-09-22 at
296297
11:57–12:07 UTC, on engine `main` `063e4460` running the committed `generated.cpp` (codegen
297298
`89645d6`). The sweep ran as two Google Benchmark batches, slots 001–100 and slots 101–201, on an
298-
Apple M4 Max (12 performance + 4 efficiency cores).
299+
Apple M4 Max (12 performance + 4 efficiency cores). The throughput package and a paired A/B
300+
followed, 13:29–14:22 UTC.
299301
- **The other engines were not re-measured.** The PyneCore 6.10.2, PineTS 0.9.34 and vectorbt
300302
0.28.2 columns are lane BENCH2's, measured 2026-09-22 03:03–05:30 UTC on the same host with the
301303
same versions and harness. They are regenerated from the same timing files, which reproduce
302304
BENCH2's report byte for byte.
303305
- **Quiet-host gate.** Before every batch, the 1-minute load average had to be below 6.0 with no
304306
`cmake --build`, `ctest` or `ci_verify` process, counted by executable name.
305-
- The two PineForge batches passed at 5.41 and 5.90, with zero such processes. Google
307+
- The two PineForge sweep batches passed at 5.41 and 5.90, with zero such processes. Google
306308
Benchmark's own start-of-batch reading was 5.62 / 7.09 / 8.21 (1/5/15-min) and
307-
5.90 / 6.62 / 7.71.
308-
- BENCH2's single PineForge batch had passed at 3.79.
309-
- The residual load came from processes outside this benchmark: orphaned test runs from
310-
another project, each holding one core, plus intermittent container and headless-browser work.
311-
- **`063e4460` against `e9ad37dd`.** Across all 201 slots, the median per-strategy ratio of this
312-
sweep's time to BENCH2's is 1.02 (p5 0.95, p95 1.07; min 0.89, max 1.13). The median moved from
313-
88.3 ms to 89.4 ms. This sweep ran at a higher host load than BENCH2's, and a paired A/B in one
314-
quiet window was not possible, so the 2 % is not attributed to the engine. The engine's
315-
runtime-budget gains between the two commits do not show up in this hot loop.
316-
- **The throughput package was not re-timed at `063e4460`; the host was never quiet.** The five
317-
`throughput-r*` rows above are BENCH2's runs at `e9ad37dd`, and the package's published figures
318-
come from them.
319-
- One run at `063e4460` passed the gate at 12:08:00 UTC (load 5.88). The 1-minute load was 16.8
320-
when its benchmark started, because another project's container work began. The run was
321-
discarded (median 0.551 M bars/s over 201 slots).
322-
- After that, the gate did not pass again during the lane's window.
309+
5.90 / 6.62 / 7.71. BENCH2's single PineForge batch had passed at 3.79.
310+
- The residual load came from processes outside this benchmark: orphaned test runs, container
311+
and headless-browser work, and a front-end build from other projects. From 12:49 UTC the gate
312+
was re-checked every 5 minutes while it failed; it passed at 13:09, 13:29 and 14:00 UTC, and
313+
the runs that followed each other directly re-checked it before starting.
314+
- **`063e4460` against `e9ad37dd`: 0.94× the time, paired.** Both engines were built on this host
315+
and timed in one window (14:17–14:22 UTC, gate 5.40), alternating new, old, old, new over 26
316+
public slots (every fourth slot, plus 004, 021 and 061).
317+
- Per strategy, `063e4460` takes 0.94× the time of `e9ad37dd` at the median (p10 0.91×,
318+
p90 1.01×), and is faster on 22 of the 26 slots. In the first pair, run at equal load, it is
319+
faster on all 26 (median 0.93×).
320+
- The sweep above, set against BENCH2's sweep, reads 1.02× at the median (p5 0.95×, p95 1.07×).
321+
The two sweeps ran at different host loads (5.41–5.90 against 3.79), so that comparison is
322+
the weaker one. The published PineForge column is this sweep.
323+
- **Throughput package: five quiet runs at `063e4460`.** `bash benchmarks/throughput/reproduce.sh`
324+
ran five times, gate-checked before each (`throughput-r1`…`r5` above). Their medians were 0.621,
325+
0.638, 0.611, 0.646 and 0.640 M bars/s. The package publishes run 2, the median run. The
326+
1-minute load sampled every 10 s during the runs had medians of 4.37, 3.54, 5.06, 3.09 and 4.08,
327+
and maxima of 5.98, 6.18, 7.82, 4.66 and 6.18.
328+
- Two earlier runs were discarded. The first passed the gate at 12:08:00 UTC (load 5.88), but
329+
another project's container work took the 1-minute load to 16.8 before its benchmark began.
330+
The second passed at 13:09:51 UTC (5.80), but the script's own build step recompiled the
331+
bench tree after a commit had changed the version header, and the run hit the tool's time
332+
bound.
323333
- **The 2026-09-21 attempt (lane BENCH1) timed nothing.** The same gate was polled every ~5 minutes
324334
for four hours (15:49–19:49 UTC, 100 probes), and no probe passed.
325335
- The 1-minute load was at least 7.74 (17:49:26 UTC, with 8 build/test processes), 97.0 at the
@@ -341,19 +351,19 @@ realistic cost for engines whose entry point IS the process.
341351
193–201 (7 workers plus 1).
342352
- **vectorbt: 13 of the 14 carried-over ports.** `061-pyramid-deferred-flip-close-all-01`'s port
343353
imports `speed.vbt_helpers`, a module that was never committed, so it does not load.
344-
- **Same-host check against the 2026-06-11 table.** BENCH2 rebuilt the engine that table measured
345-
(`ac011d84`) and timed it on the same host, feed and harness, in its own window at `e9ad37dd`. It
346-
was not re-run at `063e4460`. Three probes appear in both populations (magnifier-on hot loop, ms
347-
per 53,929-bar run):
354+
- **Same-host check against the 2026-06-11 table.** The engine that table measured (`ac011d84`)
355+
was rebuilt and timed in the paired window above, on the same host, feed and harness. Three
356+
probes appear in both populations. The figures are magnifier-on hot-loop ms per 53,929-bar run,
357+
as the mean of the window's two passes:
348358

349-
| Probe | 2026-06-11 table | `ac011d84`, BENCH2 window | `e9ad37dd`, BENCH2 window | `063e4460`, this sweep |
350-
|---|---:|---:|---:|---:|
351-
| `barstate-isconfirmed-magnifier-off-01b` | 4.80 | 5.34 | 69.63 | 70.50 |
352-
| `composite-scalping-integration-01` | 5.58 | 5.84 | 113.94 | 116.74 |
353-
| `pyramid-deferred-flip-close-all-01` | 8.72 | 8.40 | 98.31 | 104.47 |
359+
| Probe | 2026-06-11 table | `ac011d84` | `e9ad37dd` | `063e4460` | `063e4460` ÷ `ac011d84` |
360+
|---|---:|---:|---:|---:|---:|
361+
| `barstate-isconfirmed-magnifier-off-01b` | 4.80 | 6.11 | 75.68 | 70.85 | 11.6× |
362+
| `composite-scalping-integration-01` | 5.58 | 6.65 | 124.67 | 118.07 | 17.8× |
363+
| `pyramid-deferred-flip-close-all-01` | 8.72 | 9.85 | 108.56 | 99.57 | 10.1× |
354364

355-
The host reproduced the June figures. The per-bar cost at `e9ad37dd` was 12–20× the June
356-
engine's, and this sweep's `063e4460` times for the same probes are 12–20× those `ac011d84`
357-
figures too. The last column comes from this sweep, not from a paired run, so the June "162× vs
358-
PyneCore" still does not carry over.
365+
- In the window's first, quieter pass, the June engine ran at 5.56, 6.09 and 9.15 ms, which is
366+
5–16 % over its June figures. BENCH2's quieter window measured 5.34, 5.84 and 8.40.
367+
- The per-bar cost at `063e4460` is 10–18× the June engine's, against 11–19× at `e9ad37dd` in
368+
the same window. The June "162× vs PyneCore" does not carry over.
359369

‎benchmarks/throughput/README.md‎

Lines changed: 8 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -16,19 +16,17 @@ Below is the boxplot chart showing the distribution of backtest throughput acros
1616

1717
| Quartile Metric | Throughput (Millions of Bars per Second - M/s) | Equiv. Execution Time (per 10k bars) |
1818
| :--- | :---: | :---: |
19-
| **Minimum** | 0.089 M/s | 112.78 ms |
20-
| **Q1 (25th Percentile)** | 0.431 M/s | 23.21 ms |
21-
| **Median (50th Percentile)** | **0.614 M/s** | **16.29 ms** |
22-
| **Q3 (75th Percentile)** | 0.750 M/s | 13.34 ms |
23-
| **Maximum** | 0.928 M/s | 10.78 ms |
19+
| **Minimum** | 0.092 M/s | 109.15 ms |
20+
| **Q1 (25th Percentile)** | 0.454 M/s | 22.04 ms |
21+
| **Median (50th Percentile)** | **0.638 M/s** | **15.67 ms** |
22+
| **Q3 (75th Percentile)** | 0.792 M/s | 12.62 ms |
23+
| **Maximum** | 1.005 M/s | 9.95 ms |
2424

2525
*Note: the metric is GBench's `<slug>/throughput/no_magnifier` hot loop. Each strategy's shared library is `dlopen`ed once, outside the timed region. Each of the 20 timed iterations runs `strategy_create` plus `run_backtest` over the whole feed with the bar magnifier off, and the table reports the mean.*
2626

27-
*Last measured 2026-09-22 on an Apple M4 Max (16 cores) with engine `main` `e9ad37dd`. This is the median of five quiet runs: the five run medians were 0.604, 0.615, 0.606, 0.620 and 0.614 M/s, and the table and chart come from the run with the median result (run 5). Every run started at a 1-minute load below 6 with no build or test process running; the loads are in [`../results/speed.md`](../results/speed.md). A public checkout only has the 100 public slots, whose median in the same run is 0.686 M/s.*
27+
*Last measured 2026-09-22 on an Apple M4 Max (16 cores) with engine `main` `063e4460`. This is the median of five quiet runs: the five run medians were 0.621, 0.638, 0.611, 0.646 and 0.640 M/s, and the table and chart come from the run with the median result (run 2). Every run started at a 1-minute load below 6 with no build or test process running; the loads are in [`../results/speed.md`](../results/speed.md). A public checkout only has the 100 public slots, whose median in the same run is 0.727 M/s. At engine `e9ad37dd` the same package measured 0.614 M/s.*
2828

29-
*These figures were measured at `e9ad37dd` and not re-timed at `063e4460`, because the host was never quiet during that refresh. The bar-magnifier sweep was re-timed at `063e4460`, and per strategy it takes 1.02× the `e9ad37dd` time at the median; see [`../results/speed.md`](../results/speed.md), Provenance.*
30-
31-
*The median fell from 17.05 M/s, measured 2026-05-29 on 100 strategies and a 41,307-bar feed. A same-host check at `e9ad37dd` against the engine of the 2026-06-11 speed table showed a per-bar cost 12–20× higher on the probes both populations share (see [`../results/speed.md`](../results/speed.md), Provenance).*
29+
*The median fell from 17.05 M/s, measured 2026-05-29 on 100 strategies and a 41,307-bar feed. A same-host check against the engine of the 2026-06-11 speed table shows the current engine's per-bar cost is 10–18× higher on the probes both populations share (see [`../results/speed.md`](../results/speed.md), Provenance).*
3230

3331
### 🛠️ FFI Grid Search Optimization Result
3432

@@ -78,7 +76,7 @@ This script will:
7876

7977
## 🔍 Auditing & Verifying Results (For Sceptics)
8078

81-
You can audit these figures (such as the median hot-loop throughput of 0.61 million bars per second) step by step:
79+
You can audit these figures (such as the median hot-loop throughput of 0.64 million bars per second) step by step:
8280

8381
### A. Run a Single Strategy Individually
8482
Instead of running every slot, you can compile and benchmark a single strategy of your choice (e.g. `001-analyzer-anvil-percent-costs-01`) using Google Benchmark directly to eliminate any script wrapper bias:

0 commit comments

Comments
 (0)