Skip to content

Commit 046a004

Browse files
test(size): report whole bundles of sample apps for every target and library type (#22997)
* test(size): report whole bundles of generated React apps Build three apps from the realistic-app generator (100, 1k and 5k components) in yarn test:size and add a Sample apps table with each app's total gzip and raw size. * test(size): read sample apps from test/sizeCoreReports Each app is a directory with a webpack config and an optional options.mjs setup(), like benchmarkCases, so new apps need no harness change. * test(size): add node, webworker, ESM and per-library-type sample apps Cover web, node (React SSR) and webworker targets with script and ES module output, and build one library per output.library.type read from the schema; a multi-compiler app reports one row per compiler.
1 parent 445debb commit 046a004

29 files changed

Lines changed: 774 additions & 18 deletions

‎TESTING_DOCS.md‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,16 @@ Directories come first, in alphabetical order, then the individual files worth t
109109
- **Purpose**: Contains test cases related to memory limits.
110110
- **Usage**: Ensures Webpack doesn’t exceed memory constraints.
111111

112+
### `sizeCoreReports/`
113+
114+
- **Purpose**: Whole applications for `yarn test:size` ([below](#codesizetestcasessizejs)), whose report gives each one's total download in a **Sample apps** table.
115+
- **A case** is a directory with a `webpack.config.mjs` (or `.js`; an array builds one compiler per entry, each reported on its own row) and, optionally, an `options.mjs` whose exported `setup()` runs first to generate or install its sources (into `generated/`, which git ignores). The harness sets only `context`, `output.path`, `mode: "production"` when unset and quiet logging; everything else is the application's own configuration.
116+
- **Today**, each target with script (or CommonJS) and ES module output (`-esm`):
117+
- `web`: `react-100`, `react-1k`, `react-5k` and `react-1k-esm`, React apps of that many components (lazy routes, CSS modules, JSON, SVG) from the `realistic-app` benchmark's generator (`test/harness/benchmark/create-app.mjs`).
118+
- `node`: `node-ssr` and `node-ssr-esm`, that app rendered on the server by an `http` server with `react-dom/server` (`test/harness/size/create-server.mjs`).
119+
- `webworker`: `webworker` and `webworker-esm`, a message loop lazily loading 40 data tasks built on `lodash-es` and `date-fns` (`test/harness/size/create-worker.mjs`).
120+
- `library`: one small library built once per `output.library.type`, the types read from the schema.
121+
112122
### `specCases/`
113123

114124
- **Purpose**: Holds the runners for specification-conformance suites.
@@ -153,8 +163,8 @@ Directories come first, in alphabetical order, then the individual files worth t
153163
### `CodeSizeTestCases.size.js`
154164

155165
- **Purpose**: Measures how large the code webpack generates is, so a change to `lib/` that grows (or shrinks) every bundle is visible.
156-
- **Usage**: `yarn test:size` builds every `configCases/` case — one plain Node.js process, outside jest, no worker pool — and writes a JSON report of what each case emitted: the raw, gzip, brotli and zstd size of every asset, plus a per-runtime-module breakdown (total bytes over the suite, how many cases emit it, the biggest single instance) — which is what shows _which_ runtime grew, which is no longer emitted at all, and which one is simply large. The CI job (`.github/workflows/code-size.yml`) compares the report against the one `main` last uploaded, posts it as a pull request comment (updated in place on every push) and repeats it in the job summary.
157-
- **What the comparison shows**: how many cases, runtime modules and assets are changed / new / deleted / unchanged, then **one row per changed asset** — raw before → after, and the percentage each of gzip, brotli and zstd moved. A suite-wide total is deliberately not reported: a single number over 1400 cases hides which file moved, which is the only actionable part. With no baseline to compare against, the same table ranks the biggest assets instead.
166+
- **Usage**: `yarn test:size` builds every `configCases/` case, plus every application in [`sizeCoreReports/`](#sizecorereports) (reported as `apps/<case>`) — one plain Node.js process, outside jest, no worker pool — and writes a JSON report of what each case emitted: the raw, gzip, brotli and zstd size of every asset, plus a per-runtime-module breakdown (total bytes over the suite, how many cases emit it, the biggest single instance) — which is what shows _which_ runtime grew, which is no longer emitted at all, and which one is simply large. The CI job (`.github/workflows/code-size.yml`) compares the report against the one `main` last uploaded, posts it as a pull request comment (updated in place on every push) and repeats it in the job summary.
167+
- **What the comparison shows**: how many cases, runtime modules and assets are changed / new / deleted / unchanged, then **one row per changed asset** — raw before → after, and the percentage each of gzip, brotli and zstd moved. A suite-wide total is deliberately not reported: a single number over 1400 cases hides which file moved, which is the only actionable part. The applications are the exception: a **Sample apps** table gives each one's whole download (gzip and raw, summed over its assets), which per-asset rows hide once code moves between chunks. With no baseline to compare against, the same table ranks the biggest assets instead.
158168
- **It never fails.** There is nothing to assert here and no size budget to breach: a growing bundle is information, not a defect. Cases whose expected result _is_ a build error emit nothing and are reported as such, with the reason, rather than counted as failures.
159169
- **Options**: `--output <file>` (report path), `--baseline <file>` (report to compare against), `--summary <file>` (append the markdown comparison, e.g. `$GITHUB_STEP_SUMMARY`), `--filter` / `--negative-filter` (regexps matched against `<category>/<case>`, also read from `FILTER` / `NEGATIVE_FILTER`).
160170
- **Note**: the cases are built with the defaults a user gets — minification on, no `output.pathinfo` — not with the `ConfigTestCases` ones. Needs Node.js >= 22.15 for zstd.

‎docs/architecture.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ Every schema is derived from the module declaring its options by `generate-schem
8282

8383
**Tests** — structure, naming and running one case: [TESTING_DOCS.md](../TESTING_DOCS.md).
8484

85-
- `test/` — all suites (`unitCases/`, `cases/`, `configCases/`, `specCases/`, `watchCases/`, `hotCases/`, `statsCases/`, `typesCases/`, `benchmarkCases/`, `memoryLimitCases/`, …).
85+
- `test/` — all suites (`unitCases/`, `cases/`, `configCases/`, `specCases/`, `watchCases/`, `hotCases/`, `statsCases/`, `typesCases/`, `benchmarkCases/`, `sizeCoreReports/` (whole applications for `yarn test:size`), `memoryLimitCases/`, …).
8686
- `templates/` — suite drivers (`TestCases.js`, `ConfigTestCases.js`, `HotTestCases.js`, `WatchTestCases.js`); the top-level `*.test.js` / `*.basictest.js` / `*.longtest.js` are thin shims over them. Each exports `describeCases(config)` and resolves case directories against `test/`, never its own. The shims' option sets live together in `templates/variants.js` (re-exported as `variants`) so suite differences read side by side; there's one shim per entry because jest parallelizes per file, not per describe.
8787
- `harness/` — what runs the suites: jest lifecycle (`globalSetup.js`, `globalTeardown.js`, `setupTestFramework.js`), the `patch-node-env.js` environment, the crash reporter, the case `runner/`, the `snapshot/` resolver, and `runtimes/` (preload/setup files for Bun and Deno). Reusable assertions and fixtures go in `helpers/`.
8888
- `RoundTripConfigCases` re-bundles the output of `configCases` that have a `roundTrip.js`.

‎docs/performance.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@ Pitfalls that produced wrong conclusions here:
5050
- **Which way?** 🔴 ↑ grew, 🟢 ↓ shrank.
5151
- **Change or new?** Assets both runs emit are changes; ones only one run emits are new or deleted files, not deltas. They're in separate tables — changed first and unfolded, new/deleted folded, each with its own row budget — and separate verdict rows (`Changed …` vs `New` / `Deleted`), so new test cases' bundles don't bury real changes.
5252
- **webpack or the case?** A bundle is a function of its case's source, so adding assertions to a `configCases/` case grows its bundle without touching `lib/`. The report measures each case's module source and splits changed assets: `Changed, test untouched` is webpack's doing and the row size claims are read from; `Changed, test edited` has a `Test edit` column with bytes of source gained — a bundle growing less than its case isn't a regression. Cite the first row, never suite-wide numbers, when a PR touches both `lib/` and tests.
53+
- **What does a whole app download?** The **Sample apps** table sums every asset of each application in `test/sizeCoreReports/` (web, node and webworker apps with script and ES module output, and a library per `output.library.type`): the one place the report reads like a real bundle, since a config case isolates a single feature.
5354
- **Did a runtime gain or lose a runtime module?** A second table counts runtime modules per runtime and names those that came or went, split the same way (a runtime a new case brought gained nothing). Deliberately no per-runtime-module bytes (not what anyone downloads); the count catches a runtime module added for one target and forgotten for another.
5455

5556
Read the "emitted nothing" note first: a case whose build now errors contributes no bytes, which otherwise looks like an improvement. When the numbers moved, say what it reported in the PR.

‎test/CodeSizeTestCases.size.js‎

Lines changed: 172 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
11
"use strict";
22

3-
// Builds every `test/configCases` case with the defaults a user gets and reports
4-
// how large what webpack generated is. Run through `yarn test:size`; CI compares
5-
// the report against the one `main` last uploaded. Nothing here asserts and
6-
// nothing here fails: a size that moved is information, not a defect.
3+
// Builds every `test/configCases` case with the defaults a user gets, plus the
4+
// whole applications in `test/sizeCoreReports`, and reports how large what
5+
// webpack generated is. Run through `yarn test:size`; CI compares the report
6+
// against the one `main` last uploaded. Nothing here asserts and nothing here
7+
// fails: a size that moved is information, not a defect.
78

89
const { execFileSync } = require("child_process");
910
const { createHash } = require("crypto");
@@ -51,6 +52,9 @@ const prepareOptions = require("./helpers/prepareOptions");
5152
* @property {Record<string, string[]>} runtimes runtime module names per runtime,
5253
* sorted — how many a runtime carries, and which ones, is the deterministic half
5354
* of what this measures; their bytes in isolation are not what anyone downloads
55+
* @property {Record<string, Metrics>} compilers metrics per compiler of a
56+
* multi-compiler case, keyed by its report prefix — what a case building one
57+
* library per `output.library.type` is read by
5458
* @property {number} errors number of compilation errors
5559
* @property {string=} noOutput why the case emitted nothing — a build a case
5660
* expects to error (its own snapshot records it) counts as measured, not failed:
@@ -99,11 +103,16 @@ const prepareOptions = require("./helpers/prepareOptions");
99103
* @property {string} category case category
100104
* @property {string} name case name
101105
* @property {string} id `<category>/<case>`
106+
* @property {string} directory directory the case is built from
102107
*/
103108

104109
const rootPath = path.resolve(__dirname, "..");
105110
const casesPath = path.join(__dirname, "configCases");
106111
const outputBaseDir = path.join(__dirname, "js", "size");
112+
// Whole applications, one directory each, reported as `apps/<case>`: a config
113+
// case isolates one feature, so only these show what a real bundle pays.
114+
const appsPath = path.join(__dirname, "sizeCoreReports");
115+
const APP_CATEGORY = "apps";
107116

108117
// Bumped whenever a compression setting or the report shape changes, so a stale
109118
// baseline is reported as incomparable instead of compared against silently.
@@ -296,6 +305,46 @@ const applyDefaults = (options, index, testDirectory, outputDirectory) => {
296305
};
297306
};
298307

308+
/**
309+
* Loads an application's configuration, first running the `setup()` its
310+
* `options.mjs` may export to generate or install its sources. Only what a
311+
* build needs to run here is filled in; the rest is as the application says.
312+
* @param {string} directory application directory
313+
* @param {string} outputDirectory directory to build into
314+
* @returns {Promise<Configuration[]>} its configuration, one per compiler
315+
*/
316+
const loadAppOptions = async (directory, outputDirectory) => {
317+
// `require` of an ES module needs the Node.js this script already demands.
318+
const optionsPath = path.join(directory, "options.mjs");
319+
if (fs.existsSync(optionsPath)) {
320+
const { setup } = require(optionsPath);
321+
322+
if (setup) await setup();
323+
}
324+
const configName = fs.existsSync(path.join(directory, "webpack.config.mjs"))
325+
? "webpack.config.mjs"
326+
: "webpack.config.js";
327+
328+
const config = require(path.join(directory, configName));
329+
330+
/** @type {Configuration | Configuration[]} */
331+
const loaded = config.default || config;
332+
const configs = Array.isArray(loaded) ? loaded : [loaded];
333+
return configs.map((options, index) => ({
334+
...options,
335+
context: directory,
336+
mode: options.mode || "production",
337+
output: {
338+
...options.output,
339+
path:
340+
configs.length === 1
341+
? outputDirectory
342+
: path.join(outputDirectory, options.name || String(index))
343+
},
344+
infrastructureLogging: { level: "error", progress: false }
345+
}));
346+
};
347+
299348
/**
300349
* Which runtime modules each runtime carries. A count is the deterministic thing
301350
* to compare — it catches a runtime module added for one target and forgotten for
@@ -400,8 +449,8 @@ const closeCompiler = (compiler) =>
400449
* @param {Case} testCase case to measure
401450
* @returns {Promise<CaseResult>} measured metrics
402451
*/
403-
const measureCase = async ({ category, name }) => {
404-
const testDirectory = path.join(casesPath, category, name);
452+
const measureCase = async ({ category, name, directory: testDirectory }) => {
453+
const app = category === APP_CATEGORY;
405454
const outputDirectory = path.join(outputBaseDir, category, name);
406455

407456
fs.rmSync(outputDirectory, { recursive: true, force: true });
@@ -420,6 +469,7 @@ const measureCase = async ({ category, name }) => {
420469
assets: {},
421470
input: createInput(),
422471
runtimes: {},
472+
compilers: {},
423473
errors: 0
424474
};
425475

@@ -437,13 +487,17 @@ const measureCase = async ({ category, name }) => {
437487
/** @type {Configuration[]} */
438488
let optionsArr;
439489
try {
440-
const options = await prepareOptions(
441-
require(path.join(testDirectory, "webpack.config.js")),
442-
{ testPath: outputDirectory }
443-
);
444-
optionsArr = Array.isArray(options) ? [...options] : [options];
445-
for (const [index, options] of optionsArr.entries()) {
446-
applyDefaults(options, index, testDirectory, outputDirectory);
490+
if (app) {
491+
optionsArr = await loadAppOptions(testDirectory, outputDirectory);
492+
} else {
493+
const options = await prepareOptions(
494+
require(path.join(testDirectory, "webpack.config.js")),
495+
{ testPath: outputDirectory }
496+
);
497+
optionsArr = Array.isArray(options) ? [...options] : [options];
498+
for (const [index, options] of optionsArr.entries()) {
499+
applyDefaults(options, index, testDirectory, outputDirectory);
500+
}
447501
}
448502
} catch (err) {
449503
result.noOutput = `config: ${/** @type {Error} */ (err).message}`;
@@ -490,6 +544,14 @@ const measureCase = async ({ category, name }) => {
490544
? addMetrics(existing, metrics)
491545
: metrics;
492546
addMetrics(result.metrics, metrics);
547+
if (compilers.length > 1) {
548+
const key = prefix.slice(0, -1);
549+
addMetrics(
550+
result.compilers[key] ||
551+
(result.compilers[key] = createMetrics()),
552+
metrics
553+
);
554+
}
493555
}
494556
});
495557
}
@@ -537,6 +599,7 @@ const measureCase = async ({ category, name }) => {
537599
result.assets = {};
538600
result.input = createInput();
539601
result.runtimes = {};
602+
result.compilers = {};
540603
result.metrics = createMetrics();
541604
} finally {
542605
DEFAULTS.HASH_FUNCTION = hashFunction;
@@ -1067,6 +1130,85 @@ const formatBiggestAssets = (report) => {
10671130
});
10681131
};
10691132

1133+
/**
1134+
* One row per sample application, summed over every asset it emits: the whole
1135+
* download, which per-asset rows hide once code moves between chunks.
1136+
* @param {Report} report current report
1137+
* @param {Report=} baseline baseline report
1138+
* @returns {string[]} markdown lines
1139+
*/
1140+
const formatApps = (report, baseline) => {
1141+
const names = Object.keys(report.cases).filter((name) =>
1142+
name.startsWith(`${APP_CATEGORY}/`)
1143+
);
1144+
if (names.length === 0) return [];
1145+
1146+
/**
1147+
* @param {number | undefined} before baseline bytes, none when the app is new
1148+
* @param {number} after current bytes
1149+
* @returns {string} the change cell
1150+
*/
1151+
const change = (before, after) =>
1152+
before === undefined
1153+
? ADDED_MARKER
1154+
: after === before
1155+
? "—"
1156+
: `${changeMarker(after - before)} ${formatDelta(
1157+
after - before
1158+
)} (${formatPercent(before, after)})`;
1159+
1160+
const lines = [
1161+
"### Sample apps",
1162+
"",
1163+
baseline
1164+
? "| App | Gzip | Gzip change | Raw | Raw change |"
1165+
: "| App | Gzip | Raw |",
1166+
baseline ? "| :-- | --: | --: | --: | --: |" : "| :-- | --: | --: |"
1167+
];
1168+
/**
1169+
* @param {string} label app, and compiler of a multi-compiler app
1170+
* @param {Metrics} metrics current metrics
1171+
* @param {Metrics=} before baseline metrics, none when the row is new
1172+
* @returns {string} the row
1173+
*/
1174+
const row = (label, { gzip, raw }, before) =>
1175+
baseline
1176+
? `| ${label} | ${formatBytes(gzip)} | ${change(
1177+
before && before.gzip,
1178+
gzip
1179+
)} | ${formatBytes(raw)} | ${change(before && before.raw, raw)} |`
1180+
: `| ${label} | ${formatBytes(gzip)} | ${formatBytes(raw)} |`;
1181+
1182+
for (const name of names) {
1183+
const result = report.cases[name];
1184+
const app = name.slice(APP_CATEGORY.length + 1);
1185+
if (result.noOutput) {
1186+
lines.push(`| \`${app}\` | emitted nothing |`);
1187+
continue;
1188+
}
1189+
const previous = baseline && baseline.cases[name];
1190+
// A multi-compiler app is one row per compiler: its total would add up
1191+
// bundles no user downloads together.
1192+
const compilers = Object.keys(result.compilers || {}).sort();
1193+
if (compilers.length === 0) {
1194+
lines.push(
1195+
row(`\`${app}\``, result.metrics, previous && previous.metrics)
1196+
);
1197+
}
1198+
for (const compiler of compilers) {
1199+
lines.push(
1200+
row(
1201+
`\`${app}\` ${compiler}`,
1202+
result.compilers[compiler],
1203+
previous && previous.compilers && previous.compilers[compiler]
1204+
)
1205+
);
1206+
}
1207+
}
1208+
lines.push("");
1209+
return lines;
1210+
};
1211+
10701212
/**
10711213
* The verdict sentence: what a reviewer reads before anything else, so it leads
10721214
* with the assets webpack builds from unchanged source — the only ones whose
@@ -1109,7 +1251,7 @@ const formatMarkdown = (report, baseline, noBaselineReason) => {
11091251
""
11101252
];
11111253

1112-
const built = `Built \`test/configCases\` with the defaults a user gets: ${
1254+
const built = `Built \`test/configCases\` with the defaults a user gets, and the apps in \`test/sizeCoreReports\` (\`apps/\`): ${
11131255
report.meta.cases
11141256
} test(s), ${report.meta.assets} asset(s)${
11151257
report.meta.withoutOutput > 0
@@ -1123,6 +1265,7 @@ const formatMarkdown = (report, baseline, noBaselineReason) => {
11231265
"",
11241266
`${noBaselineReason}, so there is nothing to compare against yet.`,
11251267
"",
1268+
...formatApps(report),
11261269
...formatBiggestAssets(report)
11271270
);
11281271
return `${lines.join("\n")}\n`;
@@ -1262,6 +1405,8 @@ const formatMarkdown = (report, baseline, noBaselineReason) => {
12621405
""
12631406
);
12641407

1408+
lines.push(...formatApps(report, baseline));
1409+
12651410
// One row per emitted file, so a minifier change reads as the files it shrank.
12661411
// Unchanged-source assets first, then rebuilt ones, then new and deleted.
12671412
if (changes.length === 0) {
@@ -1379,8 +1524,19 @@ const discoverCases = (filter, negativeFilter) => {
13791524
continue;
13801525
}
13811526

1382-
cases.push({ category, name, id });
1527+
cases.push({ category, name, id, directory: testDirectory });
1528+
}
1529+
}
1530+
1531+
for (const name of fs.readdirSync(appsPath).sort()) {
1532+
const directory = path.join(appsPath, name);
1533+
if (name.startsWith("_") || !fs.statSync(directory).isDirectory()) {
1534+
continue;
13831535
}
1536+
const id = `${APP_CATEGORY}/${name}`;
1537+
if (filter && !filter.test(id)) continue;
1538+
if (negativeFilter && negativeFilter.test(id)) continue;
1539+
cases.push({ category: APP_CATEGORY, name, id, directory });
13841540
}
13851541

13861542
return cases;
@@ -1445,6 +1601,7 @@ const run = async () => {
14451601
assets: {},
14461602
input: createInput(),
14471603
runtimes: {},
1604+
compilers: {},
14481605
errors: 0,
14491606
noOutput: `harness: ${/** @type {Error} */ (err).message}`
14501607
};

0 commit comments

Comments
 (0)