Skip to content

Commit 2d4d43b

Browse files
ryansolidClaude via Cursor
andcommitted
docs(perf): Store / List Delivery Lane — the Octane board campaign (2026-08-16 → 09-18) and the board workflow
Hand-off for the store/list performance campaign, so it can continue in a fresh session. performance-experiments.md gains the lane: goal and rulings, instruments (Octane fork, local dbmon harness, results dirs), the seven experiments in order with PR/branch, measurement and verdict (patch channel + driver #3079/#3091/#3229, regions branch, unified For slot #3281 / engine #3308, small-move #3227, coarse reads #3275, create-floor diets #3270/#3367/#3368, props/spread/view work), the fixture authoring rules and what the two passes covered, the uibench findings (endpoint cloning 7.7x -> 2.5x; signal shape 0.46x the reconcile shape on 96/96 cases, ruled idiomatic), the todomvc add100 4x that was measurement noise, the Sep 9 -> Sep 17 board with its caveats, standing lessons, and the ordered open targets (disposal first). benchmarking-strategy.md gains the Octane board workflow: fork/branch, linking next, running, comparing, the gotchas each learned the hard way, and the single-op profiling method. Co-authored-by: Claude via Cursor <[email protected]>
1 parent 5fe2686 commit 2d4d43b

2 files changed

Lines changed: 434 additions & 0 deletions

File tree

‎documentation/benchmarking-strategy.md‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -378,6 +378,97 @@ Some gaps are not optimization targets. From the 2026-05 investigation:
378378
- **σ-band, not point estimates.** Single-run "improvements" that fall
379379
inside the σ band of the baseline are not improvements.
380380

381+
## Octane board workflow
382+
383+
The cross-framework source of truth since 2026-08 is Octane's benchmark
384+
board run from our fork, with Solid linked to a local `next` build. The
385+
campaign that built this and what it found is the
386+
[Store / List Delivery Lane](./performance-experiments.md#store--list-delivery-lane-2026-08-16--2026-09-18-the-octane-board)
387+
in the journal. The mechanics:
388+
389+
**Where.** Fork `ryansolid/octane`, branch `solid-rc8-fixtures-rebase`
390+
(upstream `main` + our Solid fixture commits; rebase it when upstream
391+
moves — the Solid fixtures are ours alone, upstream still pins
392+
2.0-beta.20). Local checkout `~/Development/octane-fork`. A second
393+
checkout, `~/Development/octane`, holds the pre-September experiment
394+
scaffolding and the older results directories.
395+
396+
**Linking `next`.** Keep a worktree of this repo on `origin/next`
397+
(`~/Development/solid-next-bench`), `pnpm install`, `pnpm build` in
398+
`packages/compiler` (native binary, needs cargo), then
399+
`pnpm turbo run build --filter=@solidjs/signals --filter=solid-js
400+
--filter=@solidjs/web --filter=@solidjs/babel-plugin --filter=@solidjs/compiler`.
401+
In the fork's `pnpm-workspace.yaml` `overrides:` block, point `solid-js`,
402+
`@solidjs/web`, `@solidjs/signals`, `@solidjs/compiler` and
403+
`@solidjs/babel-plugin` at `link:../solid-next-bench/packages/<name>`,
404+
then `pnpm install --no-frozen-lockfile`. `@solidjs/vite-plugin` stays
405+
npm and picks the linked compiler up through the override — verify with
406+
`readlink -f node_modules/.pnpm/@solidjs+vite-plugin*/node_modules/@solidjs/compiler`.
407+
**Keep the override and the lockfile it produces uncommitted**; the
408+
fixture branch must install from the published rc for anyone else.
409+
410+
**Running.** From the fork root:
411+
412+
```bash
413+
env -u PLAYWRIGHT_BROWSERS_PATH node benchmarks/bench.mjs \
414+
--results-dir=benchmarks/results-<label> <suites…>
415+
```
416+
417+
The 26-suite board used on 2026-09-17: js-framework js-framework-reorder
418+
dbmon todomvc chat-stream svg-dashboard signal-favoring effectful-list
419+
list-clear memo-wall store-selector-fanout scaling-curves spa-navigation
420+
recursive-context external-store-fanout controlled-form event-delegation
421+
application-composition suspense-recovery scheduler-responsiveness
422+
hydration-interactivity lifecycle-memory ssr-throughput streaming-ssr
423+
bundle-size uibench. ~45–60 minutes on an idle machine. `bench.mjs`
424+
builds every fixture and boots the preview servers itself; each suite
425+
runs its correctness gates before timing, so a fixture change that breaks
426+
the DOM contract fails the suite rather than posting numbers. Exit code 1
427+
means _some_ gate failed — read the tail of the log; on 2026-09-17 the
428+
failures were Octane's own targets.
429+
430+
Then `node benchmarks/board-compare.mjs <dirA> <dirB> [labelA labelB]`
431+
for the two-sided geomean board (timed ops only). For a single suite,
432+
`node benchmarks/<suite>/run.mjs <iterations>` with
433+
`TARGETS='[{"name":"solid","url":"http://localhost:<port>/"}]'` against a
434+
`pnpm --filter <fixture> preview` you started yourself.
435+
436+
**Gotchas, each learned the hard way:**
437+
438+
- `PLAYWRIGHT_BROWSERS_PATH` is set in the agent environment to a sandbox
439+
cache; unset it or Chromium "doesn't exist".
440+
- Long runs and preview servers must be tool-managed background tasks
441+
(block-until 0). `nohup … &` from a tool shell dies with the shell —
442+
silently, a minute in, leaving orphaned preview servers on their ports.
443+
- Eight samples is the default; on sub-5 ms ops the median is bimodal
444+
under the harness's `gc()` + 40 ms idle protocol. Read `min` beside
445+
the median. Any single-op ratio over 2× is "rerun in isolation" until
446+
it reproduces — todomvc `add100` read 4.3× on the board and 1.2× on a
447+
quiet machine.
448+
- `board-compare` uses timed ops. Do not hand-roll a geomean over "all
449+
ops": the result JSONs also carry structural gate counters
450+
(`nodes_deep`, `comments_*`, `text_*`) that look like numbers.
451+
- Octane's own fixtures can regress or fail their gates between upstream
452+
releases (0.2.13 did both). When the Solid/Octane ratio moves, check
453+
Octane's _absolute_ times against the previous run before believing it.
454+
- The bundle-size suite's app/framework split is meaningless under local
455+
links (framework shows ~700 B); totals are fine.
456+
- `signal-favoring` swings between runs by design (one text node);
457+
ignore single-run ratios.
458+
- Fixture authoring rules (textContent, read-once structural props,
459+
ternary over per-row `<Show>`, `onSettled`, `dynamic()`, signal shape for
460+
snapshot ingestion) are in the journal section; a "gap" is authoring
461+
until the fixture has been read.
462+
463+
**Profiling a single op.** Build the fixture unminified
464+
(`npx vite build --minify false --outDir dist-prof`), serve it, and drive
465+
the op from a Playwright script with CDP `Profiler` (self-time by
466+
function) and `HeapProfiler.startSampling` (allocation by function);
467+
interleave the A and B pages in one browser for paired comparison. The
468+
2026-09-18 todomvc scripts (`todo-prof.mjs`, `todo-var.mjs`,
469+
`todo-alloc.mjs`, `todo-ab.mjs`) are the template; they lived in `/tmp`
470+
and should be re-created next to the harness if this becomes routine.
471+
381472
## Cross-references
382473

383474
- Running journal of probes, findings, and reverts:

0 commit comments

Comments
 (0)