Skip to content

Commit 556a35f

Browse files
test(benchmark): port the webpack/benchmark scenarios (#21935)
* test(benchmark): cover the loader pipeline, a cold cache and HMR The benchmark suite measured webpack over synthetic graphs with no loader in the pipeline, a filesystem cache only on its restore path, and no HMR at all — the scenarios webpack/benchmark carried outside this repository. Three cases close that: `loader-babel` runs babel-loader over generated JSX so the loader-runner and the source round trip are measured; `cache-filesystem-cold` drops the pack each iteration so the store path is measured beside `cache-filesystem`'s restore; `hmr` builds with HotModuleReplacementPlugin, which the rebuild scenario turns into an update round trip. Two of them needed the harness to grow a hook. A config's `plugins` did not survive `structuredClone` — it strips the prototype, and with it `apply` — so a case declares them through `createPlugins(webpack)` instead, bound to the baseline being measured; declaring them directly now throws rather than silently building without them. A case's `options.mjs` may export `beforeEach`, run before every iteration and outside the measured region, which is what lets the cold case reset its pack. `-on-schedule` cases ran nowhere: the workflow filtered them out and had no schedule to run them on. It gets a weekly one, and an empty NEGATIVE_FILTER now means unset instead of excluding every case. A failed task also reports the rejection it failed with, not just its id. * fix(benchmark): let the scheduled run actually lift the case filter `event_name == 'schedule' && '' || 'on-schedule'` returns `on-schedule` on a scheduled run too: an empty string is falsy in a GitHub Actions expression, so the `||` branch always wins and the cases the schedule exists for stayed excluded. Inverted so the truthy branch carries the non-empty value. A case's per-iteration hook is renamed `beforeEachIteration`, matching the parameter it feeds. Under the old name it read as tinybench's own `beforeEach`, which is called with a mode string rather than the configuration. Also shortens three comments to the two-line cap. * ci: bound the benchmark jobs with a timeout Neither job set timeout-minutes, so both inherited the 6-hour default, and the harness has no deadline of its own: a watch bench awaits a rebuild callback that nothing cancels. main's slowest shard runs ~55 minutes, so 120 leaves better than twice the headroom and only trips on a genuine hang. * test(benchmark): port the remaining webpack/benchmark scenarios webpack/benchmark measured 21 addon scenarios; this repository covered eight of them. Seven more land here as cases, leaving only the ones that need a toolchain webpack does not depend on. Four turn off an optimization webpack enables by default, so what each one costs is measurable: no-minimize, no-concatenation, no-exports-analysis (which follows the original in also dropping sideEffects, mangleExports and innerGraph) and, in the other direction, module-unsafe-cache, resolve-unsafe-cache and unsafe-cache, which widen caching past the node_modules-only default. common-libs replaces the fixture of the same name, which pinned Material-UI 4, Vue 2, moment and jQuery as of 2021. It builds a graph out of react, react-dom, lodash-es and date-fns at the versions this repository already carries, so the scenario survives without the dead packages. Not ported: swc, thread-loader and babel preset-env each need a dependency webpack does not have; Yarn PnP needs an install layout the harness cannot produce; and no-cache is inexpressible here, since the watch scenario overrides `cache` and every iteration builds a fresh compiler, so a memory cache never carries over. * chore: format the regenerated changelog * fix(benchmark): fail the run when a case's setup does The options.mjs load ignored every rejection as "benchmark has no options.mjs", so a setup() that threw left the fixture unbuilt and the case was still measured. Only a missing file is optional now.
1 parent 7cb6dd4 commit 556a35f

33 files changed

Lines changed: 439 additions & 27 deletions

‎.github/workflows/benchmarks.yml‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,10 @@ on:
55
branches: [main]
66
pull_request:
77
branches: [main]
8+
# The only trigger that measures the `-on-schedule` cases; every other one
9+
# filters them out as too expensive to run per pull request.
10+
schedule:
11+
- cron: "0 4 * * 1"
812
workflow_dispatch:
913

1014
concurrency:
@@ -26,6 +30,9 @@ jobs:
2630
matrix:
2731
shard: [1/4, 2/4, 3/4, 4/4]
2832
runs-on: ubuntu-latest
33+
# The harness has no deadline of its own and a watch bench waits on a
34+
# rebuild callback; main's slowest shard is ~55m, so this only trips on a hang.
35+
timeout-minutes: 120
2936
permissions:
3037
issues: write
3138
pull-requests: write
@@ -60,7 +67,7 @@ jobs:
6067
mode: "simulation"
6168
env:
6269
LAST_COMMIT: 1
63-
NEGATIVE_FILTER: on-schedule
70+
NEGATIVE_FILTER: ${{ github.event_name != 'schedule' && 'on-schedule' || '' }}
6471
SHARD: ${{ matrix.shard }}
6572
# Single libuv thread → deterministic fs-completion order, removing the
6673
# last I/O race that perturbs allocation counts between runs.
@@ -71,6 +78,9 @@ jobs:
7178
matrix:
7279
shard: [1/4, 2/4, 3/4, 4/4]
7380
runs-on: ubuntu-latest
81+
# The harness has no deadline of its own and a watch bench waits on a
82+
# rebuild callback; main's slowest shard is ~55m, so this only trips on a hang.
83+
timeout-minutes: 120
7484
permissions:
7585
issues: write
7686
pull-requests: write
@@ -106,7 +116,7 @@ jobs:
106116
token: ${{ secrets.CODSPEED_TOKEN }}
107117
env:
108118
LAST_COMMIT: 1
109-
NEGATIVE_FILTER: on-schedule
119+
NEGATIVE_FILTER: ${{ github.event_name != 'schedule' && 'on-schedule' || '' }}
110120
SHARD: ${{ matrix.shard }}
111121
# Single libuv thread → deterministic fs-completion order, removing the
112122
# last I/O race that perturbs allocation counts between runs.

‎CHANGELOG.md‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -757,7 +757,6 @@
757757
- Fix `RangeError: Maximum call stack size exceeded` thrown from `HarmonyImportSideEffectDependency.getModuleEvaluationSideEffectsState` on long linear chains of side-effect-free imports. `NormalModule.getSideEffectsConnectionState` previously descended through `HarmonyImportSideEffectDependency.getModuleEvaluationSideEffectsState` recursively, adding two stack frames per module, which overflowed V8's stack at a few thousand modules deep. The traversal is now iterative. (by [@alexander-akait](https://github.com/alexander-akait) in [#20993](https://github.com/webpack/webpack/pull/20993))
758758

759759
- Fix `NormalModuleFactory` parser/generator types: (by [@alexander-akait](https://github.com/alexander-akait) in [#20999](https://github.com/webpack/webpack/pull/20999))
760-
761760
- `module.generator.html` now uses `HtmlGeneratorOptions` instead of `EmptyGeneratorOptions` (the `extract` option was hidden from the `createGenerator` / `generator` hook types).
762761
- WebAssembly (`webassembly/async`, `webassembly/sync`) generator hooks now use `EmptyGeneratorOptions` instead of `EmptyParserOptions`.
763762
- `NormalModuleFactory#getParser` / `createParser` / `getGenerator` / `createGenerator` are now generic over the module-type string, returning the specific parser/generator class for known types (e.g. `JavascriptParser` for `"javascript/auto"`, `CssGenerator` for `"css"`, etc.) instead of always returning the base `Parser` / `Generator`.
@@ -900,7 +899,6 @@
900899
- Add `exportType: "style"` for CSS modules to inject styles into DOM via HTMLStyleElement, similar to style-loader functionality. (by [@xiaoxiaojx](https://github.com/xiaoxiaojx) in [#20579](https://github.com/webpack/webpack/pull/20579))
901900

902901
- Add `context` option support for VirtualUrlPlugin (by [@xiaoxiaojx](https://github.com/xiaoxiaojx) in [#20449](https://github.com/webpack/webpack/pull/20449))
903-
904902
- The context for the virtual module. A string path. Defaults to 'auto', which will try to resolve the context from the module id.
905903
- Support custom context path for resolving relative imports in virtual modules
906904
- Add examples demonstrating context usage and filename customization

‎test/BenchmarkTestCases.benchmark.mjs‎

Lines changed: 32 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -294,15 +294,15 @@ class BenchmarkRunner {
294294
* @returns {Promise<string[]>} benchmark names
295295
*/
296296
async discoverBenchmarks() {
297-
const FILTER =
298-
typeof process.env.FILTER !== "undefined"
299-
? new RegExp(process.env.FILTER)
300-
: undefined;
297+
// Empty means unset: `new RegExp("")` matches every case, so an empty
298+
// NEGATIVE_FILTER would exclude all of them.
299+
const FILTER = process.env.FILTER
300+
? new RegExp(process.env.FILTER)
301+
: undefined;
301302

302-
const NEGATIVE_FILTER =
303-
typeof process.env.NEGATIVE_FILTER !== "undefined"
304-
? new RegExp(process.env.NEGATIVE_FILTER)
305-
: undefined;
303+
const NEGATIVE_FILTER = process.env.NEGATIVE_FILTER
304+
? new RegExp(process.env.NEGATIVE_FILTER)
305+
: undefined;
306306

307307
/** @type {string[]} */
308308
const allBenchmarks = (await fs.readdir(this.casesPath))
@@ -406,12 +406,20 @@ class BenchmarkRunner {
406406
);
407407

408408
try {
409-
const options = await import(`${pathToFileURL(optionsPath)}`);
410-
if (typeof options.setup !== "undefined") {
411-
await options.setup();
409+
await fs.stat(optionsPath);
410+
} catch (err) {
411+
// Only a missing options.mjs is optional; anything else would
412+
// otherwise measure a fixture that never got built.
413+
if (/** @type {NodeJS.ErrnoException} */ (err).code !== "ENOENT") {
414+
throw err;
412415
}
413-
} catch (_err) {
414-
// Ignore — benchmark has no options.mjs
416+
continue;
417+
}
418+
419+
const options = await import(`${pathToFileURL(optionsPath)}`);
420+
421+
if (typeof options.setup !== "undefined") {
422+
await options.setup();
415423
}
416424
}
417425
}
@@ -463,12 +471,20 @@ class BenchmarkRunner {
463471
const benchmarkResults = [];
464472
/** @type {string[]} */
465473
const failedTasks = [];
474+
/** @type {unknown} */
475+
let firstFailure;
466476

467477
for (const [index, settled] of settledResults.entries()) {
468478
if (settled.status === "fulfilled") {
469479
benchmarkResults.push(settled.value);
470480
} else {
471-
failedTasks.push(benchmarkTasks[index].id);
481+
const { id } = benchmarkTasks[index];
482+
483+
failedTasks.push(id);
484+
// The rejection is the only description of what broke; without it a
485+
// failing case reports a task id and nothing else.
486+
console.error(`Failed: ${id}`, settled.reason);
487+
if (failedTasks.length === 1) firstFailure = settled.reason;
472488
}
473489
}
474490

@@ -478,7 +494,8 @@ class BenchmarkRunner {
478494

479495
if (failedTasks.length > 0) {
480496
throw new Error(
481-
`${failedTasks.length} benchmark task(s) failed: ${failedTasks.join(", ")}`
497+
`${failedTasks.length} benchmark task(s) failed: ${failedTasks.join(", ")}`,
498+
{ cause: firstFailure }
482499
);
483500
}
484501
}
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
import * as mod from "./generated/module.js";
2+
3+
export { mod };
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
import fs from "fs/promises";
2+
import { dirname, resolve } from "path";
3+
import { fileURLToPath } from "url";
4+
import createTree from "../../harness/benchmark/create-tree.mjs";
5+
import memoryScaledCount from "../../harness/benchmark/scale.mjs";
6+
7+
export async function setup() {
8+
const __dirname = dirname(fileURLToPath(import.meta.url));
9+
const generated = resolve(__dirname, "./generated");
10+
11+
await fs.rm(generated, { recursive: true, force: true });
12+
await createTree(generated, false, memoryScaledCount(75, 200));
13+
}
14+
15+
/**
16+
* Drops the pack so each iteration measures the store path, not the restore one.
17+
* @param {import("../../..").Configuration} config built configuration
18+
* @returns {Promise<void>}
19+
*/
20+
export async function beforeEachIteration(config) {
21+
const { cache } = config;
22+
23+
if (
24+
!cache ||
25+
typeof cache === "boolean" ||
26+
cache.type !== "filesystem" ||
27+
!cache.cacheDirectory
28+
) {
29+
throw new Error("Expected a filesystem cache with a resolved directory");
30+
}
31+
32+
await fs.rm(cache.cacheDirectory, { recursive: true, force: true });
33+
}
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
/** @type {import("../../..").Configuration} */
2+
export default {
3+
entry: "./index",
4+
cache: {
5+
type: "filesystem",
6+
// For benchmark stability
7+
maxMemoryGenerations: 0,
8+
idleTimeoutForInitialStore: 0
9+
}
10+
};
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
import * as mod1 from "react";
2+
export { mod1 };
3+
import * as mod2 from "react-dom";
4+
export { mod2 };
5+
import * as mod3 from "lodash-es";
6+
export { mod3 };
7+
import * as mod4 from "date-fns";
8+
export { mod4 };
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
/** @type {import("../../..").Configuration} */
2+
export default {
3+
entry: "./index"
4+
};

‎test/benchmarkCases/hmr/index.js‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
import * as mod from "./generated/module.js";
2+
3+
export { mod };
4+
5+
if (import.meta.webpackHot) {
6+
import.meta.webpackHot.accept();
7+
}
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
import fs from "fs/promises";
2+
import { dirname, resolve } from "path";
3+
import { fileURLToPath } from "url";
4+
import createTree from "../../harness/benchmark/create-tree.mjs";
5+
import memoryScaledCount from "../../harness/benchmark/scale.mjs";
6+
7+
export async function setup() {
8+
const __dirname = dirname(fileURLToPath(import.meta.url));
9+
const generated = resolve(__dirname, "./generated");
10+
11+
await fs.rm(generated, { recursive: true, force: true });
12+
await createTree(generated, false, memoryScaledCount(75, 200));
13+
}

0 commit comments

Comments
 (0)