Skip to content

Commit c178197

Browse files
feat: add multi-scenario startup benchmark suite
## Problem Investigating startup performance (e.g. #809) meant ad-hoc scripts, and a single blended timing number conflates Node's floor, tsx's hook-registration tax, resolution, and transform. There was no reusable way to isolate those or compare across tsx/Node versions at scale. ## Changes Adds `scripts/benchmark/` (`pnpm benchmark`) — a scenario-based startup benchmark. - **Scenarios** isolate distinct costs: `node-baseline` (floor), `hooks-passthrough` (registration + pass-through, zero transforms), `esm-ts` (transform + resolution), `cjs-require`, `cjs-interop` (the #809 surface), `native-ts` (Node's type-stripping floor). - **Axes**: `--compare` (npm version or path to a tsx checkout), `--node` (downloaded via get-node), `--scale` (module sweep → per-module cost + fixed startup tax), `--cold`, `--json`. - **Metrics**: wall (mean±stdev + min headline); RSS and a load/eval split recorded in `--json`. - The default set (`node-baseline`, `hooks-passthrough`, `esm-ts`, `native-ts`) is a data-driven high-signal subset — a full sweep found `cjs-require` low-signal and `cjs-interop` ~1.0 correlated with `hooks-passthrough` at fixed size, so both are opt-in. Rationale in `scripts/benchmark/README.md`. The `--node` axis cleanly A/Bs async `module.register()` (worker thread) vs sync `module.registerHooks()`; the boundary is documented in `notes/node/module-hooks.md` (privatenumber/tsx#145). Example (esm-ts, 1000 modules, Node 24.15.0, M5 Pro): ``` [email protected] 600ms [email protected] 1906ms (3.18x — the #809 regression, sync-hooks path) ```
1 parent 51800ac commit c178197

9 files changed

Lines changed: 813 additions & 0 deletions

File tree

‎package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,7 @@
5858
"homepage": "https://tsx.hirok.io",
5959
"scripts": {
6060
"build": "pkgroll --minify",
61+
"benchmark": "pnpm build && node ./dist/cli.mjs scripts/benchmark/index.ts",
6162
"lint": "lintroll --node --cache --ignore-pattern 'docs/*.md' .",
6263
"type-check": "tsc",
6364
"test": "pnpm build && node ./dist/cli.mjs tests/index.ts",

‎scripts/benchmark/README.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Benchmark
2+
3+
Measures tsx startup across **scenarios**, **Node versions**, and **project sizes**. Built to isolate where startup time goes — Node's floor, tsx's hook-registration tax, resolution, and transform — rather than one blended number. Motivated by [#809](https://github.com/privatenumber/tsx/issues/809).
4+
5+
## Usage
6+
7+
```sh
8+
pnpm benchmark # all scenarios, current Node, 1000 modules
9+
pnpm benchmark esm-ts --compare 4.21.0 # one scenario vs a released version
10+
pnpm benchmark hooks-passthrough --node 24.10.0 # async (worker) vs sync hooks (see below)
11+
pnpm benchmark esm-ts --scale # per-module cost + fixed startup tax
12+
pnpm --silent benchmark --json > results.json # raw per-run data
13+
```
14+
15+
Positional arguments select scenarios (default: all).
16+
17+
For `--json`, use `pnpm --silent` so pnpm's script banner stays out of stdout and the redirected file is valid JSON (progress still goes to stderr).
18+
19+
## Scenarios
20+
21+
| Scenario | Default | Runs via | Isolates |
22+
| --- | --- | --- | --- |
23+
| `node-baseline` | ✅ | `node` | Absolute startup floor (ignores `--compare`) |
24+
| `hooks-passthrough` | ✅ | tsx CLI | Hook registration + pass-through resolve/load, **zero transforms** |
25+
| `esm-ts` | ✅ | tsx CLI | Transform + resolution hot path (`--specifier` applies) |
26+
| `native-ts` | ✅ | `node` | Node's native type stripping — reference floor (ignores `--compare`) |
27+
| `cjs-require` | | tsx CLI | The `tsx/cjs` require path |
28+
| `cjs-interop` | | tsx CLI | ESM importing N CommonJS `node_modules` packages (the #809 surface) |
29+
30+
Running with no scenario names runs the **default set**; name any scenario (e.g. `pnpm benchmark cjs-interop`) to run it explicitly. All tsx scenarios run through the **tsx CLI** (not `--import`) for consistency with real usage; the CLI's extra Node spawn is constant across rows and visible against `node-baseline`.
31+
32+
### Why this default set
33+
34+
Chosen from a full sweep. The four defaults cover distinct, high-signal axes with no redundancy:
35+
36+
- `node-baseline` and `native-ts` are the Node floors needed to interpret the rest (native-ts also exposes tsx's transform surface as the gap above it).
37+
- `hooks-passthrough` uniquely isolates the hook-registration / worker-thread tax (zero transforms).
38+
- `esm-ts` carries the transform + resolution signal (and reproduces #809).
39+
40+
`cjs-require` barely varied across tsx versions (low discriminating signal) and `cjs-interop` correlated ~1.0 with `hooks-passthrough` at fixed size, so both are opt-in rather than default. They still exercise distinct code paths worth running deliberately.
41+
42+
## Flags
43+
44+
| Flag | Description | Default |
45+
| --- | --- | --- |
46+
| `-c, --compare` | tsx to compare: npm version or path to a tsx checkout (repeatable) | |
47+
| `-n, --node` | Additional Node version to test, downloaded via get-node (repeatable) | current |
48+
| `-m, --modules` | Module count (ignored with `--scale`) | `1000` |
49+
| `-s, --specifier` | `esm-ts` import style: `ts`, `js`, `extensionless` | `ts` |
50+
| `-r, --runs` | Timed runs per cell | `5` |
51+
| `--cold` | Clear the tsx transform cache before every run | `false` |
52+
| `--scale` | Sweep module counts 10/100/300/1000; report per-module cost + fixed tax | `false` |
53+
| `--json` | Emit raw per-run results as JSON (stdout) | `false` |
54+
55+
Progress goes to stderr; result tables / JSON to stdout.
56+
57+
## Metrics
58+
59+
- **wall** — parent-measured spawn→exit; the headline, reported as mean ± stdev and **min**. Prefer min when comparing: run-to-run CV was ~6% median in the sweep, and min is the more stable statistic for startup timing.
60+
- **rss** (`--json` only) — peak RSS (`resourceUsage().maxRSS`). Small spread across implementations; the one independent signal is the async loader worker's ~12MB overhead.
61+
- **load / eval split** (`--json` only) — each module timestamps its first evaluation; since ESM evaluates post-order, the earliest ≈ "graph loaded/transformed" and the entry's last ≈ "graph evaluated". Eval was a negligible share on synthetic trees, so it's recorded in JSON but kept out of the table.
62+
63+
## Node version axis (async vs sync hooks)
64+
65+
tsx uses async `module.register()` (a loader **worker thread**) below the CJS-reload-safe boundary, and sync `module.registerHooks()` (in-thread) at or above it. The boundary and its verification are documented in [`notes/node/module-hooks.md`](../../notes/node/module-hooks.md). The cleanest same-major A/B is:
66+
67+
```sh
68+
pnpm benchmark hooks-passthrough --node 24.10.0 # 24.10.0 = async worker, current (>=24.11.1) = sync
69+
```
70+
71+
Scenarios below their `minNodeVersion` (e.g. `native-ts` under 22.18.0) are skipped, not failed.
72+
73+
## Reproducibility
74+
75+
The tsx transform cache is reset before each cell's warmup to avoid stale-file skew (a large `$TMPDIR/tsx-<uid>` degrades warm runs via a linear cache scan). Runs are interleaved after warmup.

‎scripts/benchmark/index.ts‎

Lines changed: 297 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,297 @@
1+
import path from 'node:path';
2+
import fs from 'node:fs/promises';
3+
import { fileURLToPath } from 'node:url';
4+
import { cli } from 'cleye';
5+
import { createFixture } from 'fs-fixture';
6+
import { tmpdir as tsxCacheDirectory } from '../../src/utils/temporary-directory.js';
7+
import { isFeatureSupported, type Version } from '../../src/utils/node-features.js';
8+
import type { SpecifierStyle } from './utils/generate-fixture.js';
9+
import { scenarios, type Scenario } from './utils/scenarios.js';
10+
import { resolveTsx, type TsxImplementation } from './utils/resolve-tsx.js';
11+
import { resolveNode, type NodeBinary } from './utils/resolve-node.js';
12+
import { runOnce, type RunResult } from './utils/run.js';
13+
import { mean, standardDeviation, linearFit } from './utils/stats.js';
14+
15+
const specifierStyles: SpecifierStyle[] = ['ts', 'js', 'extensionless'];
16+
const scaleCounts = [10, 100, 300, 1000];
17+
18+
const argv = cli({
19+
name: 'benchmark',
20+
parameters: ['[scenarios...]'],
21+
help: {
22+
description: 'Measures tsx startup across scenarios, Node versions, and project sizes',
23+
},
24+
flags: {
25+
compare: {
26+
type: [String],
27+
alias: 'c',
28+
description: 'tsx to compare: npm version (e.g. 4.21.0) or path to a tsx checkout',
29+
default: () => [] as string[],
30+
},
31+
node: {
32+
type: [String],
33+
alias: 'n',
34+
description: 'Additional Node version to test (repeatable); downloaded via get-node',
35+
default: () => [] as string[],
36+
},
37+
modules: {
38+
type: Number,
39+
alias: 'm',
40+
description: 'Module count (ignored with --scale)',
41+
default: 1000,
42+
},
43+
specifier: {
44+
type: (style: SpecifierStyle) => {
45+
if (!specifierStyles.includes(style)) {
46+
throw new Error(`Invalid specifier style: ${style}`);
47+
}
48+
return style;
49+
},
50+
alias: 's',
51+
description: `Import specifier style for esm-ts: ${specifierStyles.join(', ')}`,
52+
default: 'ts' as SpecifierStyle,
53+
},
54+
runs: {
55+
type: Number,
56+
alias: 'r',
57+
description: 'Timed runs per cell',
58+
default: 5,
59+
},
60+
cold: {
61+
type: Boolean,
62+
description: 'Clear the tsx transform cache before every run',
63+
default: false,
64+
},
65+
scale: {
66+
type: Boolean,
67+
description: `Sweep module counts ${scaleCounts.join('/')} and report per-module cost + fixed tax`,
68+
default: false,
69+
},
70+
json: {
71+
type: Boolean,
72+
description: 'Emit raw per-run results as JSON',
73+
default: false,
74+
},
75+
},
76+
});
77+
78+
const {
79+
compare, node: nodeVersions, modules, specifier, runs, cold, scale, json,
80+
} = argv.flags;
81+
82+
const selectedScenarios = (
83+
argv._.scenarios.length > 0
84+
? argv._.scenarios.map((name) => {
85+
const scenario = scenarios.find(s => s.name === name);
86+
if (!scenario) {
87+
throw new Error(`Unknown scenario "${name}". Available: ${scenarios.map(s => s.name).join(', ')}`);
88+
}
89+
return scenario;
90+
})
91+
: scenarios.filter(scenario => scenario.default)
92+
);
93+
94+
const moduleCounts = scale ? scaleCounts : [modules];
95+
const log = (message = '') => process.stderr.write(`${message}\n`);
96+
const out = (message = '') => process.stdout.write(`${message}\n`);
97+
98+
const clearTransformCache = () => fs.rm(tsxCacheDirectory, {
99+
recursive: true,
100+
force: true,
101+
});
102+
103+
type Row = {
104+
scenario: string;
105+
nodeVersion: string;
106+
impl: string;
107+
moduleCount: number;
108+
results: RunResult[];
109+
};
110+
111+
const measureCell = async (
112+
scenario: Scenario,
113+
node: NodeBinary,
114+
cliPath: string,
115+
fixturePath: string,
116+
): Promise<RunResult[]> => {
117+
const entryPath = path.join(fixturePath, scenario.entry);
118+
const args = scenario.runner === 'tsx' ? [cliPath, entryPath] : [entryPath];
119+
const usesCache = scenario.runner === 'tsx';
120+
121+
// Reset cache before warmup: avoids stale-file skew and cross-cell pollution
122+
if (usesCache) {
123+
await clearTransformCache();
124+
}
125+
await runOnce(node.path, args, fixturePath);
126+
127+
const results: RunResult[] = [];
128+
for (let run = 0; run < runs; run += 1) {
129+
if (usesCache && cold) {
130+
await clearTransformCache();
131+
}
132+
results.push(await runOnce(node.path, args, fixturePath));
133+
}
134+
return results;
135+
};
136+
137+
// Resolve Node binaries (current + any --node), de-duplicated by version
138+
const nodeBinaries: NodeBinary[] = [];
139+
const resolvedBinaries = await Promise.all([
140+
resolveNode(),
141+
...nodeVersions.map(version => resolveNode(version)),
142+
]);
143+
for (const binary of resolvedBinaries) {
144+
if (!nodeBinaries.some(existing => existing.version === binary.version)) {
145+
nodeBinaries.push(binary);
146+
}
147+
}
148+
149+
// tsx implementations: local + comparisons (npm/path)
150+
await using installRoot = await createFixture();
151+
const implementations: TsxImplementation[] = [
152+
{
153+
name: 'local',
154+
cliPath: fileURLToPath(new URL('../../dist/cli.mjs', import.meta.url)),
155+
},
156+
...await Promise.all(
157+
compare.map(comparison => resolveTsx(comparison, installRoot.path)),
158+
),
159+
];
160+
161+
const rows: Row[] = [];
162+
const skipped: string[] = [];
163+
164+
// Measurements are intentionally sequential: concurrent child processes
165+
// would contend for CPU and corrupt the timings.
166+
for (const scenario of selectedScenarios) {
167+
// Build each fixture once per module count; reuse across nodes and cells
168+
const fixtures = await Promise.all(moduleCounts.map(
169+
moduleCount => createFixture(scenario.build(moduleCount, specifier)),
170+
));
171+
await using _scenarioFixtures = {
172+
[Symbol.asyncDispose]: async () => {
173+
await Promise.all(fixtures.map(fixture => fixture.rm()));
174+
},
175+
};
176+
177+
for (const node of nodeBinaries) {
178+
if (
179+
scenario.supportedNodeVersions
180+
&& !isFeatureSupported(
181+
scenario.supportedNodeVersions,
182+
node.version.split('.').map(Number) as Version,
183+
)
184+
) {
185+
skipped.push(`${scenario.name} on Node ${node.version} (unsupported)`);
186+
continue;
187+
}
188+
189+
// node-runner scenarios (baseline/native) ignore --compare
190+
const cells = (
191+
scenario.runner === 'node'
192+
? [{
193+
name: 'node',
194+
cliPath: '',
195+
}]
196+
: implementations
197+
);
198+
for (const { name, cliPath } of cells) {
199+
for (const [index, moduleCount] of moduleCounts.entries()) {
200+
log(` ${scenario.name} · Node ${node.version} · ${name} · ${moduleCount} modules`);
201+
const results = await measureCell(scenario, node, cliPath, fixtures[index].path);
202+
rows.push({
203+
scenario: scenario.name,
204+
nodeVersion: node.version,
205+
impl: name,
206+
moduleCount,
207+
results,
208+
});
209+
}
210+
}
211+
}
212+
}
213+
214+
if (json) {
215+
process.stdout.write(`${JSON.stringify({
216+
meta: {
217+
runs,
218+
cold,
219+
scale,
220+
specifier,
221+
},
222+
rows,
223+
}, null, 2)}\n`);
224+
} else {
225+
renderTables();
226+
}
227+
228+
function renderTables() {
229+
out();
230+
out(`${runs} runs/cell · ${cold ? 'cold' : 'warm'} cache · specifier=${specifier}`);
231+
232+
for (const scenario of selectedScenarios) {
233+
const scenarioRows = rows.filter(row => row.scenario === scenario.name);
234+
if (scenarioRows.length === 0) {
235+
continue;
236+
}
237+
out(`\n## ${scenario.name} — ${scenario.description}`);
238+
239+
const byNode = new Map<string, Row[]>();
240+
for (const row of scenarioRows) {
241+
const list = byNode.get(row.nodeVersion) ?? [];
242+
list.push(row);
243+
byNode.set(row.nodeVersion, list);
244+
}
245+
246+
for (const [nodeVersion, nodeRows] of byNode) {
247+
if (scale) {
248+
out(`Node ${nodeVersion} — scale ${scaleCounts.join('/')}:`);
249+
const byImpl = new Map<string, Row[]>();
250+
for (const row of nodeRows) {
251+
const list = byImpl.get(row.impl) ?? [];
252+
list.push(row);
253+
byImpl.set(row.impl, list);
254+
}
255+
for (const [impl, implRows] of byImpl) {
256+
const points = implRows.map(row => ({
257+
x: row.moduleCount,
258+
y: mean(row.results.map(result => result.wallMs)),
259+
}));
260+
const { slope, intercept } = linearFit(points);
261+
const detail = points
262+
.slice()
263+
.sort((a, b) => a.x - b.x)
264+
.map(point => `${point.x}→${point.y.toFixed(0)}ms`)
265+
.join(' ');
266+
out(` ${impl.padEnd(20)} ${(slope * 1000).toFixed(1)}µs/module +${intercept.toFixed(0)}ms fixed [${detail}]`);
267+
}
268+
} else {
269+
const [count] = moduleCounts;
270+
out(`Node ${nodeVersion} — ${count} modules:`);
271+
const localMean = mean(
272+
(nodeRows.find(row => row.impl === 'local') ?? nodeRows[0])
273+
.results.map(result => result.wallMs),
274+
);
275+
for (const row of nodeRows) {
276+
// wall is the headline signal; RSS + load/eval split carry little
277+
// independent signal on synthetic trees (see README) — JSON only.
278+
const wall = row.results.map(result => result.wallMs);
279+
const wallMean = mean(wall);
280+
const relative = (
281+
row.impl === 'local' || scenario.runner === 'node'
282+
? ''
283+
: ` (${(wallMean / localMean).toFixed(2)}x)`
284+
);
285+
const name = row.impl.padEnd(20);
286+
const meanColumn = `mean ${wallMean.toFixed(0)}±${standardDeviation(wall).toFixed(0)}ms`.padEnd(18);
287+
const minColumn = `min ${Math.min(...wall).toFixed(0)}ms`;
288+
out(` ${name}${meanColumn}${minColumn}${relative}`);
289+
}
290+
}
291+
}
292+
}
293+
294+
if (skipped.length > 0) {
295+
out(`\nSkipped:\n${skipped.map(entry => ` ${entry}`).join('\n')}`);
296+
}
297+
}

0 commit comments

Comments
 (0)