Skip to content

Commit c20c1a6

Browse files
authored
test: benchmark the runtime performance of emitted bundles (#21544)
Add a `-runtime` benchmark type that compiles the case outside every measured region and then times only the generated code: instantiating the emitted bundle and calling its exported `run`. The first case, `many-modules-interop-runtime`, is a 600-module graph mixing CommonJS and ESM in both interop directions.
1 parent f78b353 commit c20c1a6

6 files changed

Lines changed: 401 additions & 7 deletions

File tree

‎TESTING_DOCS.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,13 @@ This document explains the structure of the `test/` directory in the Webpack pro
1313

1414
- **Purpose**: Contains test cases for benchmarking Webpack's performance.
1515
- **Usage**: Measures build times, memory usage, and optimization impact.
16+
- **Kinds of case** (picked from the directory name):
17+
- _build_ (default, e.g. `many-modules-esm/`) — a `webpack.config.mjs` plus an entry; measures one build per scenario (`mode-development`, `mode-development-rebuild`, `mode-production`).
18+
- `*-unit` (e.g. `js-parser-unit/`) — an `index.bench.mjs` exporting `default (bench) => {…}`; measures a piece of `lib/` directly, with no scenarios.
19+
- `*-runtime` (e.g. `many-modules-interop-runtime/`) — measures **the emitted bundle**, not the build. The case is compiled once per scenario outside any measured region; the `exec` task then instantiates the output (runtime bootstrap plus every module factory that runs at import time) and calls the entry's exported `run` on it. The rebuild scenario is skipped, since it emits the same output.
20+
- **Writing a `*-runtime` case**: the entry must export `run(seed)`, do its work with that opaque seed, and return the result — otherwise the compiler folds the workload away and the bench measures nothing (both are checked and fail the run). The harness forces `target: "node"` and a `commonjs2` library so the output can be instantiated in-process; generate large fixtures from `options.mjs` (`setup()`), as the build cases do.
21+
22+
`exec` is reported per scenario, so a `development`/`production` pair shows what scope hoisting and minification are worth at runtime.
1623

1724
### 3. `cases/`
1825

‎test/BenchmarkTestCases.benchmark.mjs‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -334,7 +334,9 @@ class BenchmarkRunner {
334334
/**
335335
* Build the benchmark tasks for this shard. Each non-unit benchmark expands
336336
* to one task per scenario (all baselines measured within the task); `-unit`
337-
* benchmarks have no scenarios and become a single task.
337+
* benchmarks have no scenarios and become a single task; `-runtime`
338+
* benchmarks skip the rebuild scenario, which emits the same output as the
339+
* initial build.
338340
* @param {string[]} benchmarks discovered benchmarks
339341
* @param {[number, number]} shard shard [part, count]
340342
* @param {Baseline[]} baselines baselines
@@ -362,7 +364,11 @@ class BenchmarkRunner {
362364
continue;
363365
}
364366

365-
for (const scenario of this.scenarios) {
367+
const scenarios = benchmark.includes("-runtime")
368+
? this.scenarios.filter((scenario) => !scenario.watch)
369+
: this.scenarios;
370+
371+
for (const scenario of scenarios) {
366372
benchmarkTasks.push({
367373
id: `${benchmark}-${scenario.name}`,
368374
benchmark,
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
export { run } from "./generated/module.js";
Lines changed: 234 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,234 @@
1+
import fs from "fs/promises";
2+
import { dirname, resolve } from "path";
3+
import { fileURLToPath } from "url";
4+
import memoryScaledCount from "../../harness/benchmark/scale.mjs";
5+
6+
// Five modules per group — an ESM leaf, two CommonJS leaves, and one consumer
7+
// of each kind reaching across the other module system — so the bundle mixes
8+
// both systems at every level instead of only at the entry. 2042 modules total.
9+
const GROUP_COUNT = 400;
10+
const GROUPS_PER_BARREL = 10;
11+
// Memory mode measures allocations and the workload loop allocates nothing, so
12+
// instantiating the graph is the whole signal there; a couple of rounds suffice.
13+
const ROUNDS = memoryScaledCount(30, 2);
14+
const TABLE_SIZE = 8;
15+
16+
/**
17+
* @param {number} index group index
18+
* @returns {string} generated module
19+
*/
20+
function generateEsm(index) {
21+
return `const table = [];
22+
for (let i = 0; i < ${TABLE_SIZE}; i++) {
23+
table.push((i * ${index} + 1) | 0);
24+
}
25+
26+
export const value = table[3];
27+
28+
export function scale(x) {
29+
return (x + table[x & ${TABLE_SIZE - 1}]) | 0;
30+
}
31+
32+
export default function identity(x) {
33+
return (x + ${index}) | 0;
34+
}
35+
`;
36+
}
37+
38+
/**
39+
* @param {number} index group index
40+
* @returns {string} generated module
41+
*/
42+
function generateCommonJs(index) {
43+
return `const table = [];
44+
for (let i = 0; i < ${TABLE_SIZE}; i++) {
45+
table.push((i * ${index} + 3) | 0);
46+
}
47+
48+
exports.value = table[5];
49+
50+
exports.scale = function scale(x) {
51+
return (x + table[x & ${TABLE_SIZE - 1}]) | 0;
52+
};
53+
54+
module.exports.identity = function identity(x) {
55+
return (x + ${index}) | 0;
56+
};
57+
`;
58+
}
59+
60+
/**
61+
* `module.exports` assigned a function, the shape that makes an ESM default
62+
* import go through the runtime's compat getter instead of a direct property
63+
* read on the exports object.
64+
* @param {number} index group index
65+
* @returns {string} generated module
66+
*/
67+
function generateCommonJsCallable(index) {
68+
return `module.exports = function callable(x) {
69+
return (x + ${index}) | 0;
70+
};
71+
72+
module.exports.extra = function extra(x) {
73+
return (x * 2 + ${index}) | 0;
74+
};
75+
`;
76+
}
77+
78+
/**
79+
* CommonJS reading ESM: `require()` hands back a namespace object the runtime
80+
* builds, `default` binding included.
81+
* @param {number} index group index
82+
* @returns {string} generated module
83+
*/
84+
function generateCommonJsConsumer(index) {
85+
return `const esm = require("./esm-${index}.js");
86+
const commonjs = require("./commonjs-${index}.js");
87+
88+
module.exports.call = function call(x) {
89+
return (esm.default(x) + esm.scale(x) + commonjs.identity(x)) | 0;
90+
};
91+
`;
92+
}
93+
94+
/**
95+
* ESM reading CommonJS: the default import goes through the compat getter, the
96+
* named one through webpack's analysis of the `exports` assignments. Export
97+
* names carry the index so the barrels can `export *` without conflicts.
98+
* @param {number} index group index
99+
* @returns {string} generated module
100+
*/
101+
function generateEsmConsumer(index) {
102+
return `import commonjsDefault from "./commonjs-${index}.js";
103+
import { scale } from "./commonjs-${index}.js";
104+
105+
export const total${index} = commonjsDefault.value;
106+
107+
export function mix${index}(x) {
108+
return (commonjsDefault.identity(x) + scale(x)) | 0;
109+
}
110+
`;
111+
}
112+
113+
/**
114+
* @param {string[]} requests re-exported requests
115+
* @returns {string} generated module
116+
*/
117+
function generateBarrel(requests) {
118+
return requests
119+
.map((request) => `export * from ${JSON.stringify(request)};\n`)
120+
.join("");
121+
}
122+
123+
/**
124+
* Entry aggregator. Every round walks each group through all the ways the two
125+
* module systems reach each other: default, namespace and named import of a
126+
* CommonJS module, a callable one imported as default, a CommonJS module that
127+
* consumed ESM, and an ESM module that consumed CommonJS — the last one behind
128+
* a two-level barrel, read once as a named import and once off the namespace
129+
* object.
130+
* @param {number} groupCount count of groups
131+
* @returns {string} generated module
132+
*/
133+
function generateAggregator(groupCount) {
134+
const imports = ['import * as barrel from "./barrel.js";'];
135+
const barrelNames = [];
136+
const body = [];
137+
138+
for (let index = 0; index < groupCount; index++) {
139+
imports.push(
140+
`import commonjsDefault${index} from "./commonjs-${index}.js";
141+
import * as commonjsNamespace${index} from "./commonjs-${index}.js";
142+
import { scale as commonjsNamed${index} } from "./commonjs-${index}.js";
143+
import callable${index} from "./commonjs-callable-${index}.js";
144+
import consumer${index} from "./commonjs-consumer-${index}.js";`
145+
);
146+
barrelNames.push(`mix${index}`);
147+
body.push(
148+
` total = commonjsDefault${index}.scale(total);
149+
total = commonjsNamespace${index}.identity(total);
150+
total = commonjsNamed${index}(total);
151+
total = callable${index}(total);
152+
total = callable${index}.extra(total);
153+
total = consumer${index}.call(total);
154+
total = mix${index}(total);
155+
total = (total + barrel.total${index}) | 0;`
156+
);
157+
}
158+
159+
imports.push(`import { ${barrelNames.join(", ")} } from "./barrel.js";`);
160+
161+
return `${imports.join("\n")}
162+
163+
export function run(seed) {
164+
let total = seed | 0;
165+
166+
for (let round = 0; round < ${ROUNDS}; round++) {
167+
${body.join("\n")}
168+
}
169+
170+
return total;
171+
}
172+
`;
173+
}
174+
175+
export async function setup() {
176+
const __dirname = dirname(fileURLToPath(import.meta.url));
177+
const generated = resolve(__dirname, "./generated");
178+
179+
await fs.rm(generated, { recursive: true, force: true });
180+
await fs.mkdir(generated, { recursive: true });
181+
182+
/** @type {string[]} */
183+
const barrels = [];
184+
185+
for (let start = 0; start < GROUP_COUNT; start += GROUPS_PER_BARREL) {
186+
/** @type {string[]} */
187+
const esmConsumers = [];
188+
189+
for (
190+
let index = start;
191+
index < Math.min(start + GROUPS_PER_BARREL, GROUP_COUNT);
192+
index++
193+
) {
194+
await fs.writeFile(
195+
resolve(generated, `./esm-${index}.js`),
196+
generateEsm(index)
197+
);
198+
await fs.writeFile(
199+
resolve(generated, `./commonjs-${index}.js`),
200+
generateCommonJs(index)
201+
);
202+
await fs.writeFile(
203+
resolve(generated, `./commonjs-callable-${index}.js`),
204+
generateCommonJsCallable(index)
205+
);
206+
await fs.writeFile(
207+
resolve(generated, `./commonjs-consumer-${index}.js`),
208+
generateCommonJsConsumer(index)
209+
);
210+
await fs.writeFile(
211+
resolve(generated, `./esm-consumer-${index}.js`),
212+
generateEsmConsumer(index)
213+
);
214+
esmConsumers.push(`./esm-consumer-${index}.js`);
215+
}
216+
217+
const barrel = `./barrel-${start / GROUPS_PER_BARREL}.js`;
218+
219+
await fs.writeFile(
220+
resolve(generated, barrel),
221+
generateBarrel(esmConsumers)
222+
);
223+
barrels.push(barrel);
224+
}
225+
226+
await fs.writeFile(
227+
resolve(generated, "./barrel.js"),
228+
generateBarrel(barrels)
229+
);
230+
await fs.writeFile(
231+
resolve(generated, "./module.js"),
232+
generateAggregator(GROUP_COUNT)
233+
);
234+
}
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
/** @type {import("../../..").Configuration} */
2+
export default {
3+
entry: "./index"
4+
};

0 commit comments

Comments
 (0)