Skip to content

Commit 9fd4df5

Browse files
refactor(javascript): minify with webpack's printer, without terser (#22397)
* perf(javascript): minify from the printer's own parser, not a converted tree `createModules` builds the parser, so `minify` reads a source into the printer's nodes by itself. The `parse` phase still parsed every source into webpack's ESTree tree first and converted that, so a minify built two trees and the converter's offset, token, value and comment-gap tables, then dropped the first tree. Dropping the phase takes `createTerserTree` and its tables with it. On an 8.8 MiB asset, minify peaks 1443.8 -> 1193.0 MiB with the default options and 745.7 -> 606.7 with compress off; a production build of that asset goes 1662 -> 1444 MiB. Time is unchanged, except on a source the conversion refused, which paid for a parse it threw away: 15.9 -> 8.6 s there. Output is byte-identical, and `estree` still converts a tree given or asked for. * perf(javascript): count nodes to converge only when a pass can follow terser walks the whole tree after the last compress pass too, for a convergence verdict it then drops. The output is byte for byte the same. * perf(javascript): minify with webpack's own parser, not terser's The printer's tree is built by webpack's parser as it finishes each statement, with that parser's tokens and comments, so an asset is read by one parser rather than two. terser's parser stays behind it for what it reads its own way: an expression, `strict`, TypeScript, a line opening an HTML comment, and a source webpack's parser refuses. A token is built only when a node is first given one. The parser reports every token at its deepest, where frames are what it has least of, so nothing is built there: the offsets, kinds and line breaks go into arrays, and the comments of a gap into the list the tokens on both sides share, which is one empty list wherever a gap holds none. Output is byte for byte terser's over the 180 corpus cases and the 288 printer tests. Peak on a real bundle, minified with compress and mangle: 1121 -> 1048 MiB for 8.8 MiB of input, 1591 -> 1529 MiB for 17.3 MiB, with the time unchanged either way. * perf(javascript): keep a token's position, and read the rest off the parse A token terser builds holds nine fields; a tree of a million nodes holds one each, where only its position is ever read as often. webpack's parse now hands the tree a token of two fields — the parse and which token it is — and reads a line, a kind, a value or the comments beside it off narrow tables the parse keeps: the offsets and kinds in typed arrays, a value only where the source does not spell it out, and the comments of a gap in a map of the gaps that hold any. The parsed tree of an 8.8 MiB bundle retains 129.1 MiB where terser's parser leaves 237.1 MiB, and 247.6 against 457.6 MiB for 17.3 MiB of input — 159 bytes a node against 292. Minifying the first peaks at 872 MiB against 1203; output stays byte for byte terser's. * fix(javascript): find an HTML comment's line by scanning, not by pattern A pattern spelling `<!--` and `-->` reads as a filter over HTML, which CodeQL reports as an incomplete one; this decides which parser reads a source, and filters nothing. A scan says the same thing without a pattern, and skips the line it would have walked where neither spelling occurs. Also say in docs/syntax.md what the printer's `parse` phase now does, and what `parse.webpackParser` turns off. * fix(javascript): read a comment's line break as terser's tokenizer does terser steps over a comment's own characters before it makes the token, so a block comment holding a line break reads as having one before it; webpack's parse read only the gap ahead of it, and wrote `!/*!…` where terser puts the `!` on a line of its own. A parenthesis hands its break to the first comment inside it only where it holds no comment of its own, as terser's parser has it, and a block comment is read with its breaks spelled `\n`. The swc corpus differed over `projects/files` and `benches-full`, where each library's banner comment is followed by `(function (…) {`; all nine print byte for byte terser's again. * fix(javascript): mark every link of an annotated chain, and end a sequence where terser does terser's parser leaves the annotation of a chain on each of its links, where webpack's parse took it off the one an outer link was built over, so a compressor inlined an arrow call terser keeps. A sequence ends at `peek()`, the token past the one that follows it — past the parenthesis holding it where there is one — which decides whether a comment after the statement is written inside the brace that closes the function or after it. The parse has not read that token when the statement is converted, so the pool hands out the one it will read, and `finish` holds it back to the token past the source where the parse ended first. * fix(javascript): annotate a chain's member link only where the chain ends there Marking every link, as the commit before this did, mangles a property terser keeps: `/*@__MANGLE_PROP__*/ this.keepthis.manglethis` is one mark on the last link, because terser's parser annotates a `.` or `[…]` link by annotating whatever the rest of the chain returns. A call annotates itself, so it keeps its own mark, and a link built over a member takes that member's. terser also marks the chain wrapper of `this.a?.b.c`, which is left as it is: nothing reads an annotation there, and the output of each optional-chain shape is byte for byte terser's either way. * perf(javascript): read a parse's tokens into the room they need, once Collecting a token cost more than reading one: four growable arrays to push into, two map lookups for its kind, a sealed object per token, and a token object the parser built only for the reader to copy from. The offsets, kinds and breaks now go into typed arrays sized for the source and doubled where it holds more, `finish` trims them instead of converting them, a kind is a code the table already holds, and `onTokenRead` hands the reader the parser itself, which also keeps the parser's token fast path. Parsing an 8.8 MiB bundle into the printer's tree: 2217 -> 1642 ms and a peak of 483 -> 351 MiB, against terser's parser at 1520 ms and 366 MiB. What the tree retains is unchanged at 129.0 MiB against terser's 237.1. Output stays byte for byte terser's over the 232 corpus cases and the 301 printer tests. * refactor(javascript): minify with webpack's printer, and type it ourselves The printer wrote what terser writes over the whole corpus, so it is what `jsMinify` loads: the branch that read the user's own terser goes, with the `printer` option that chose between them and the `futureDefaults` default that turned it on. Output is cached against webpack's version now, since the minifier is webpack's. Its options, its answer and its `minify` are typed here rather than imported from terser's package, so `types.d.ts` no longer reads that package at all. The keys the minifier reads are named and the rest stay open, as the configuration schema already had them. test262 built every case twice, once per minifier; one minifier needs one production mode, and the three groups of `MINIFIED_FAILURES` that only terser failed — 28 cases it drops, miswrites or changes the meaning of — go with it. * test(javascript): report every stale MINIFIED_FAILURES entry at once A case listed as failing that passes was its own jest failure, and five hundred of those with a code frame each overflow what a run's log holds, so the list the suite asks you to edit could not be read back from it. They are collected and reported together now, one line each. * refactor(javascript): name the printer's classes and types webpack's own A node class was `AST_Call`, which named terser rather than what it is. Each one is now its type with `Node` after it — `CallNode`, `BinaryNode` — the root is `SyntaxNode` and the token class `ParsedToken`, which `classNameOf` decides for the classes the generated descriptors build. The suffix is what keeps `Object`, `String`, `Number` and `Function` the globals they are. The types and the helpers lose the name too: `TerserModules` is `MinifierModules`, `TerserNode` `NodeShape`, `TerserCompressor` `CompressorShape`, `TerserOutputStream` `OutputStream`, `createTerserParser` `createLenientParser`, `terserParse` `parseLeniently`, and the token and word kinds `tokenKindOf` and `wordKindOf`. A bigint kept the separators it was written with, where the tokenizer drops them: `123_456_789n` is printed `123456789n` again, which is five groups of the test262 corpus back to byte for byte. Where a test holds webpack's port to terser's own modules, each side answers to its own names: terser's classes gain webpack's as aliases, and a tree is compared by what a node holds rather than by what its class is called. * docs(javascript): say that the printer minifies every asset The map still described the switch that chose between webpack's printer and terser, and the parser factory under its old name. One fixture is no longer bigger than swc's output, so it leaves `SWC_SMALLER`. * test(javascript): drop the test262 cases the minifier no longer fails Minifying with webpack's printer — a bigint written without the separators it was read with, a comment's line break read as terser's tokenizer reads it, an annotated chain marked link by link — passes 527 of the cases `MINIFIED_FAILURES` listed as failing, so they leave it. 638 remain, which is what the minifier has left to fix. * fix(javascript): refuse a private field with no class around it The printer asked webpack's parser not to check private fields, so `foo.#bar` and `#foo in bar` outside a class minified where terser and oxc both refuse them — 76 differences over the comparison corpora, and a name the mangler then renamed. The parser's own default checks it, so this drops the option. * test(javascript): record where terser refuses or misreads what the printer reads Reading a source with webpack's parser rather than terser's ported one leaves 757 differences over the comparison corpora, none of them webpack's. 704 are a source the spec allows and terser refuses: an escape anywhere in an identifier name, a trailing comma in an argument list, a regular expression after the ogham space mark, `let` as a name outside strict mode, `yield` naming a function inside a generator, `export * as default from`, a field named `get` before a generator, and an escape no cooked template value can read. The other 53 are bytes where terser reads or prints the program differently: a `let` or `async` before a line break, which the semicolon inserted after it leaves a name; a line continuation's own break written into the string it holds nothing of; and a binary, octal or hex property key quoted as it is written, naming the property `0b1` where it is `1`. `REFUSED_BY_REFERENCE` lists the parse errors with what makes each source legal and `REFERENCE_MISPRINTS` the cases by name. Both fail where the reference stops differing, and an accepted output still runs against what its test prints. * test(javascript): retire three outputs no longer bigger than oxc's Writing an import of nothing without its empty list makes two of oxc's own cases and one of swc's no bigger than what oxc writes, so `OXC_SMALLER` stops listing them. * refactor(javascript): keep terser the minifier, the printer a future default Minifying every asset with the printer waits for the measurements to settle, so `jsMinify` reads the user's terser again unless `printer: true` asks for the printer, which `optimization.minimizeOptions.javascript` still defaults to under `experiments.futureDefaults`. A cache entry follows terser's version again. The test262 suite minifies through the printer whatever a build defaults to: it holds webpack's minifier to the spec, not the one a user brings, so the cases it no longer fails stay out of `MINIFIED_FAILURES`. * test(javascript): build test262 for terser and for the printer again Collapsing the two minified modes into one dropped terser's coverage and the per-minifier `minifiers` keying of `MINIFIED_FAILURES`, so a case failing under only one of them could not be recorded. Production is built once per minimizer again, and every entry the earlier collapse removed is back. What stays is the report: a case listed as failing that now passes is collected rather than thrown, so one run names every entry to retire. * refactor(javascript): minify with terser by default, the printer under futureDefaults The default minimizer dispatches JavaScript to `terserMinify` as `minimizer-webpack-plugin` publishes it, and to `jsMinify` — webpack's own printer — only under `experiments.futureDefaults`. Which one runs is the wiring's to decide, so `jsMinify` no longer takes a `printer` flag and goes through the printer alone. `getMinimizerVersion` reports webpack's version rather than terser's, since the printer ships inside webpack and an upgrade of it is what moves the output a cache entry holds. * test(javascript): run test262's production mode through each minifier Each minified mode names the function it builds with rather than an option: `terserMinify` for the mode a plain production build runs, `jsMinify` for the one `futureDefaults` switches to. Both read the same options, so a difference between the modes is the minifier's own. Splitting what terser's parser refuses follows: webpack's parser reads 515 of those programs, so only terser is listed for them, while the 14 terser rejects as a redeclaration are listed for both, the printer porting that refusal. * test(javascript): list each minifier's test262 failures under its own cause Both shards reported, and `printer` leaves fourteen entries it passes now, each a program terser miscompiles and the `correct` phase gets right: `let` after a newline in a `for-in`, `for-of` or `for-await-of` head, `async` with no line terminator before a function declaration or an arrow, an escaped `async` before one, `await` as the second argument of a dynamic import, and a line continuation inside a single- or double-quoted string. Each moves to the terser-only group its reason already has. Two entries keep the printer, under what actually fails them rather than terser's reason. `import/import-attributes/text-javascript.js` imports a fixture that is not parsable JavaScript, so neither minifier can read the asset. `expressions/await/await-in-nested-function.js` is a printer defect: compressing inlines a function naming `await` into the async function calling it, where the name is reserved, so the output does not parse — reached only now that the printer is held to this case.
1 parent cb47bb9 commit 9fd4df5

23 files changed

Lines changed: 5691 additions & 6972 deletions

‎.changeset/036-js-minify-entry-point.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,4 @@
22
"webpack": patch
33
---
44

5-
Minify JavaScript with webpack's printer under `futureDefaults`; drop `terser`.
5+
Minify JavaScript with webpack's printer under `futureDefaults`, `terser` otherwise.
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"webpack": patch
3+
---
4+
5+
Minify by parsing with webpack's own parser rather than terser's.
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"webpack": patch
3+
---
4+
5+
Refuse a private field with no class around it when minifying with the printer.

‎docs/syntax.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ Read this before touching the CSS, HTML or JavaScript parser, printer or minifie
3636

3737
## JavaScript
3838

39-
- `syntax.js` reaches `parser` and `printer` through getters: parsing never loads the printer, and the printer loads the parser only for its `parse` phase.
39+
- `syntax.js` reaches `parser` and `printer` through getters: parsing never loads the printer, and the printer loads the parser for its `parse` phase, which reads an asset with it — building the printer's tree as each statement finishes, rather than converting a tree. `parse.webpackParser` defaults to true and turns that off; terser's own parser reads what it reads its own way (`parse.expression`, `parse.strict`, TypeScript, a line opening an HTML comment) and whatever webpack's parser refuses.
4040
- `syntax-parser.js` — tokenizer, acorn-derived core and every production in one file (any parse reaches the productions, and the coming struct-of-arrays rewrite moves node creation through them). `regexp.js` (pattern validator) is the one on-demand piece, reached only for a pattern the host engine rejected — never `require` it from a build path.
41-
- `syntax-printer.js` — webpack's JavaScript minifier, terser's design ported to write byte for byte what terser writes; `jsMinify.js` (the `minify` the default minimizer dispatches JS to) goes through it when given `printer: true` — which `optimization.minimizeOptions.javascript` defaults to under `experiments.futureDefaults` — and through terser as published otherwise — the user's own, as `terser-webpack-plugin` brings it, since webpack lists terser only as a devDependency; the printer itself never loads terser. `createModules` builds what terser's modules exported — the node classes from `nodeClasses()` (`createAst`), the parser (`createTerserParser`), the compressor and the output stream — and each entry of `PHASES` installs one part onto them: add phases there and nowhere else. `correct` wraps the methods with which terser writes a wrong program and changes only that output; `corrections.enabled` switches it off. `improve` writes less where terser's longer program provably runs the same — after compressing, a statement calling a function in place with nothing passed runs its body there — and `improvements.enabled` switches it off. The `parse` phase builds the tree from `syntax-parser.js` (`createTerserTree`), leaving to the ported parser a source terser reads unlike the spec; `syntax-printer-parse.unittest.js` holds that to terser's parse over acorn's corpus, `syntax-printer-terser-parser.unittest.js` the ported parser and `syntax-printer-estree.unittest.js` the ESTree conversion. `test/specCases/minify-corpora.spectest.js` holds the whole printer to terser's output, errors included, over each corpus in its `CORPORA` list (add a new one there) under the option sets in `OPTION_SETS`, and runs each output whose test states what its input prints (terser's `expect_stdout`, swc's `expected.stdout` and `exec.rs`) in terser's sandbox. A byte difference is accepted only where switching the `correct` and `improve` phases off gives terser's bytes back, and fails where `improve` writes more than without it; what it fixes is listed in `CORRECTED` (printing right with the phase, as terser without), and an output misprinting as terser's by an option's own contract in `INHERITED`, each with the reason — an entry that stops matching fails until retired. Each output is also sized against swc's recorded one, a fixture's `output.js` or, for one of terser's cases, what swc's port of it records (gzip decides, raw breaks a tie): one bigger fails unless `test/helpers/swcSmaller.js` lists it under what swc does that webpack does not yet, and a listed one no longer bigger fails until retired. Under the default minimizer's options each source of every corpus but test262 is also minified by `oxc-minify`, held the same way to `test/helpers/oxcSmaller.js`; an oxc output that misprints is no lead. `JS_MINIFY_REPORT=<file>` tallies where webpack's is worse, the same and better. terser's corpus submodule is pinned to the `terser` devDependency — bump both together, and regenerate `data.js`.
41+
- `syntax-printer.js` — webpack's JavaScript minifier, terser's design ported to write byte for byte what terser writes; `jsMinify.js` is the `minify` the default minimizer dispatches JS to under `experiments.futureDefaults`, and goes through it alone; without that flag the minimizer dispatches to `terserMinify` as `minimizer-webpack-plugin` publishes it, running the user's own terser since webpack lists it only as a devDependency. Neither needs an option to say which, and the printer itself never loads terser. `createModules` builds what terser's modules exported — the node classes from `nodeClasses()` (`createAst`), the parser (`createLenientParser`), the compressor and the output stream — and each entry of `PHASES` installs one part onto them: add phases there and nowhere else. `correct` wraps the methods with which terser writes a wrong program and changes only that output; `corrections.enabled` switches it off. `improve` writes less where terser's longer program provably runs the same — after compressing, a statement calling a function in place with nothing passed runs its body there — and `improvements.enabled` switches it off. The `parse` phase builds the tree from `syntax-parser.js`, leaving to the lenient parser a source read unlike the spec; `syntax-printer-terser-parser.unittest.js` holds the ported parser to terser's own and `syntax-printer-estree.unittest.js` the ESTree conversion. `test/specCases/minify-corpora.spectest.js` holds the whole printer to terser's output, errors included, over each corpus in its `CORPORA` list (add a new one there) under the option sets in `OPTION_SETS`, and runs each output whose test states what its input prints (terser's `expect_stdout`, swc's `expected.stdout` and `exec.rs`) in terser's sandbox. A byte difference is accepted only where switching the `correct` and `improve` phases off gives terser's bytes back, and fails where `improve` writes more than without it; what it fixes is listed in `CORRECTED` (printing right with the phase, as terser without), and an output misprinting as terser's by an option's own contract in `INHERITED`, each with the reason — an entry that stops matching fails until retired. Where the reference instead refuses a source the spec allows, the parse error it answers with is listed in `REFUSED_BY_REFERENCE` with what makes the source legal; where it reads or prints one differently, the case is listed in `REFERENCE_MISPRINTS` with what it gets wrong — a listed one webpack must still run correctly, and one the reference stops differing on fails until retired. Each output is also sized against swc's recorded one, a fixture's `output.js` or, for one of terser's cases, what swc's port of it records (gzip decides, raw breaks a tie): one bigger fails unless `test/helpers/swcSmaller.js` lists it under what swc does that webpack does not yet, and a listed one no longer bigger fails until retired. Under the default minimizer's options each source of every corpus but test262 is also minified by `oxc-minify`, held the same way to `test/helpers/oxcSmaller.js`; an oxc output that misprints is no lead. `JS_MINIFY_REPORT=<file>` tallies where webpack's is worse, the same and better. terser's corpus submodule is pinned to the `terser` devDependency — bump both together, and regenerate `data.js`.
4242
- `yarn test:minify-corpora` — webpack's JS minifier vs the published one it replaces, byte for byte (errors included), over every JS corpus: terser's own `test/compress` and `test/input`, test262, swc's minifier tests and oxc's; where a test states what its input prints, the output is also run and must print it. Each output must also be no bigger than swc's recorded one unless `SWC_SMALLER` lists why, and every output outside test262 no bigger than `oxc-minify`'s unless `OXC_SMALLER` does; `JS_MINIFY_REPORT=<file>` tallies worse, the same and better.

‎lib/config/defaults.js‎

Lines changed: 17 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -2519,15 +2519,6 @@ const applyOptimizationDefaults = (
25192519
}
25202520
: { compress: { passes: 2 } }
25212521
);
2522-
// webpack's printer, held to what terser writes, replaces terser; until
2523-
// proven on real builds it is a future default.
2524-
// A copy, so the user's object is not written to; their own value wins.
2525-
if (futureDefaults && minimizeOptions.javascript) {
2526-
minimizeOptions.javascript = {
2527-
printer: true,
2528-
...minimizeOptions.javascript
2529-
};
2530-
}
25312522
F(minimizeOptions, "css", () => ({}));
25322523
F(minimizeOptions, "html", () => ({}));
25332524
D(minimizeOptions, "json", futureDefaults);
@@ -2536,12 +2527,12 @@ const applyOptimizationDefaults = (
25362527
{
25372528
apply: (compiler) => {
25382529
// Lazy load the minimizer plugin
2539-
const TerserPlugin = require("minimizer-webpack-plugin");
2530+
const MinimizerPlugin = require("minimizer-webpack-plugin");
25402531

25412532
// WHY: one minimizer instance, so one worker-thread pool, handles JS and
25422533
// — where native CSS and HTML are enabled — those assets too, as an array
2543-
// of minify functions each dispatched by its own `filter`: `jsMinify`
2544-
// (terser) to JS, `cssMinify` to CSS, `htmlMinify` to HTML. The CSS and
2534+
// of minify functions each dispatched by its own `filter`: the JS one
2535+
// to JS, `cssMinify` to CSS, `htmlMinify` to HTML. The CSS and
25452536
// HTML serializers stay worker-safe by resolving their parser with
25462537
// `require("webpack/…")` in the worker, so all three share the pool, and
25472538
// `test` is widened to let those assets reach the dispatcher.
@@ -2557,9 +2548,15 @@ const applyOptimizationDefaults = (
25572548
const minimizeByType = /** @type {OptimizationMinimizeOptions} */ (
25582549
compiler.options.optimization.minimizeOptions
25592550
);
2560-
const terserOptions = minimizeByType.javascript || {
2551+
const javascriptOptions = minimizeByType.javascript || {
25612552
compress: { passes: 2 }
25622553
};
2554+
// webpack's printer, held to what terser writes, replaces terser; until
2555+
// proven on real builds it is a future default, and terser as the plugin
2556+
// publishes it minifies JS everywhere else.
2557+
const javascriptMinify = futureDefaults
2558+
? require("../javascript/jsMinify")
2559+
: MinimizerPlugin.terserMinify;
25632560

25642561
const { css, html } = compiler.options.experiments;
25652562
const minifyJs = minimizeByType.javascript !== false;
@@ -2584,14 +2581,14 @@ const applyOptimizationDefaults = (
25842581
const assetNameTest = (extensions) =>
25852582
new RegExp(`^[^?#]*\\.(?:${extensions.join("|")})(?:[?#].*)?$`, "i");
25862583
if (minifyJs && !minifyCss && !minifyHtml && !minifyJson) {
2587-
new TerserPlugin({
2584+
new MinimizerPlugin({
25882585
test: assetNameTest(["[cm]?js"]),
2589-
minify: require("../javascript/jsMinify"),
2590-
minimizerOptions: terserOptions
2586+
minify: javascriptMinify,
2587+
minimizerOptions: javascriptOptions
25912588
}).apply(/** @type {EXPECTED_ANY} */ (compiler));
25922589
return;
25932590
}
2594-
/** @type {(typeof import("../javascript/jsMinify") | typeof import("../css/cssMinify") | typeof import("../html/htmlMinify") | typeof import("../json/jsonMinify"))[]} */
2591+
/** @type {(typeof javascriptMinify | typeof import("../css/cssMinify") | typeof import("../html/htmlMinify") | typeof import("../json/jsonMinify"))[]} */
25952592
const minify = [];
25962593
/** @type {EXPECTED_OBJECT[]} */
25972594
const minimizerOptions = [];
@@ -2623,8 +2620,8 @@ const applyOptimizationDefaults = (
26232620
? minimizeByType.html
26242621
: undefined;
26252622
if (minifyJs) {
2626-
minify.push(require("../javascript/jsMinify"));
2627-
minimizerOptions.push(terserOptions);
2623+
minify.push(javascriptMinify);
2624+
minimizerOptions.push(javascriptOptions);
26282625
extensions.push("[cm]?js");
26292626
}
26302627
if (minifyCss) {
@@ -2653,7 +2650,7 @@ const applyOptimizationDefaults = (
26532650
// A web app manifest is JSON under its own extension.
26542651
extensions.push("json", "webmanifest");
26552652
}
2656-
new TerserPlugin({
2653+
new MinimizerPlugin({
26572654
test: assetNameTest(extensions),
26582655
minify,
26592656
minimizerOptions

0 commit comments

Comments
 (0)