Repository navigation
Commit 9fd4df5
authored
refactor(javascript): minify with webpack's printer, without terser (#22397)
* perf(javascript): minify from the printer's own parser, not a converted tree
`createModules` builds the parser, so `minify` reads a source into the
printer's nodes by itself. The `parse` phase still parsed every source into
webpack's ESTree tree first and converted that, so a minify built two trees
and the converter's offset, token, value and comment-gap tables, then
dropped the first tree.
Dropping the phase takes `createTerserTree` and its tables with it. On an
8.8 MiB asset, minify peaks 1443.8 -> 1193.0 MiB with the default options
and 745.7 -> 606.7 with compress off; a production build of that asset goes
1662 -> 1444 MiB. Time is unchanged, except on a source the conversion
refused, which paid for a parse it threw away: 15.9 -> 8.6 s there. Output
is byte-identical, and `estree` still converts a tree given or asked for.
* perf(javascript): count nodes to converge only when a pass can follow
terser walks the whole tree after the last compress pass too, for a
convergence verdict it then drops. The output is byte for byte the same.
* perf(javascript): minify with webpack's own parser, not terser's
The printer's tree is built by webpack's parser as it finishes each
statement, with that parser's tokens and comments, so an asset is read by
one parser rather than two. terser's parser stays behind it for what it
reads its own way: an expression, `strict`, TypeScript, a line opening an
HTML comment, and a source webpack's parser refuses.
A token is built only when a node is first given one. The parser reports
every token at its deepest, where frames are what it has least of, so
nothing is built there: the offsets, kinds and line breaks go into arrays,
and the comments of a gap into the list the tokens on both sides share,
which is one empty list wherever a gap holds none.
Output is byte for byte terser's over the 180 corpus cases and the 288
printer tests. Peak on a real bundle, minified with compress and mangle:
1121 -> 1048 MiB for 8.8 MiB of input, 1591 -> 1529 MiB for 17.3 MiB,
with the time unchanged either way.
* perf(javascript): keep a token's position, and read the rest off the parse
A token terser builds holds nine fields; a tree of a million nodes holds
one each, where only its position is ever read as often. webpack's parse
now hands the tree a token of two fields — the parse and which token it is
— and reads a line, a kind, a value or the comments beside it off narrow
tables the parse keeps: the offsets and kinds in typed arrays, a value only
where the source does not spell it out, and the comments of a gap in a map
of the gaps that hold any.
The parsed tree of an 8.8 MiB bundle retains 129.1 MiB where terser's
parser leaves 237.1 MiB, and 247.6 against 457.6 MiB for 17.3 MiB of input
— 159 bytes a node against 292. Minifying the first peaks at 872 MiB
against 1203; output stays byte for byte terser's.
* fix(javascript): find an HTML comment's line by scanning, not by pattern
A pattern spelling `<!--` and `-->` reads as a filter over HTML, which
CodeQL reports as an incomplete one; this decides which parser reads a
source, and filters nothing. A scan says the same thing without a pattern,
and skips the line it would have walked where neither spelling occurs.
Also say in docs/syntax.md what the printer's `parse` phase now does, and
what `parse.webpackParser` turns off.
* fix(javascript): read a comment's line break as terser's tokenizer does
terser steps over a comment's own characters before it makes the token, so a
block comment holding a line break reads as having one before it; webpack's
parse read only the gap ahead of it, and wrote `!/*!…` where terser puts the
`!` on a line of its own. A parenthesis hands its break to the first comment
inside it only where it holds no comment of its own, as terser's parser has
it, and a block comment is read with its breaks spelled `\n`.
The swc corpus differed over `projects/files` and `benches-full`, where each
library's banner comment is followed by `(function (…) {`; all nine print
byte for byte terser's again.
* fix(javascript): mark every link of an annotated chain, and end a sequence where terser does
terser's parser leaves the annotation of a chain on each of its links, where
webpack's parse took it off the one an outer link was built over, so a
compressor inlined an arrow call terser keeps.
A sequence ends at `peek()`, the token past the one that follows it — past
the parenthesis holding it where there is one — which decides whether a
comment after the statement is written inside the brace that closes the
function or after it. The parse has not read that token when the statement is
converted, so the pool hands out the one it will read, and `finish` holds it
back to the token past the source where the parse ended first.
* fix(javascript): annotate a chain's member link only where the chain ends there
Marking every link, as the commit before this did, mangles a property terser
keeps: `/*@__MANGLE_PROP__*/ this.keepthis.manglethis` is one mark on the last
link, because terser's parser annotates a `.` or `[…]` link by annotating
whatever the rest of the chain returns. A call annotates itself, so it keeps
its own mark, and a link built over a member takes that member's.
terser also marks the chain wrapper of `this.a?.b.c`, which is left as it is:
nothing reads an annotation there, and the output of each optional-chain shape
is byte for byte terser's either way.
* perf(javascript): read a parse's tokens into the room they need, once
Collecting a token cost more than reading one: four growable arrays to push
into, two map lookups for its kind, a sealed object per token, and a token
object the parser built only for the reader to copy from. The offsets, kinds
and breaks now go into typed arrays sized for the source and doubled where it
holds more, `finish` trims them instead of converting them, a kind is a code
the table already holds, and `onTokenRead` hands the reader the parser itself,
which also keeps the parser's token fast path.
Parsing an 8.8 MiB bundle into the printer's tree: 2217 -> 1642 ms and a peak
of 483 -> 351 MiB, against terser's parser at 1520 ms and 366 MiB. What the
tree retains is unchanged at 129.0 MiB against terser's 237.1. Output stays
byte for byte terser's over the 232 corpus cases and the 301 printer tests.
* refactor(javascript): minify with webpack's printer, and type it ourselves
The printer wrote what terser writes over the whole corpus, so it is what
`jsMinify` loads: the branch that read the user's own terser goes, with the
`printer` option that chose between them and the `futureDefaults` default that
turned it on. Output is cached against webpack's version now, since the
minifier is webpack's.
Its options, its answer and its `minify` are typed here rather than imported
from terser's package, so `types.d.ts` no longer reads that package at all. The
keys the minifier reads are named and the rest stay open, as the configuration
schema already had them.
test262 built every case twice, once per minifier; one minifier needs one
production mode, and the three groups of `MINIFIED_FAILURES` that only terser
failed — 28 cases it drops, miswrites or changes the meaning of — go with it.
* test(javascript): report every stale MINIFIED_FAILURES entry at once
A case listed as failing that passes was its own jest failure, and five
hundred of those with a code frame each overflow what a run's log holds, so
the list the suite asks you to edit could not be read back from it. They are
collected and reported together now, one line each.
* refactor(javascript): name the printer's classes and types webpack's own
A node class was `AST_Call`, which named terser rather than what it is. Each
one is now its type with `Node` after it — `CallNode`, `BinaryNode` — the root
is `SyntaxNode` and the token class `ParsedToken`, which `classNameOf` decides
for the classes the generated descriptors build. The suffix is what keeps
`Object`, `String`, `Number` and `Function` the globals they are.
The types and the helpers lose the name too: `TerserModules` is
`MinifierModules`, `TerserNode` `NodeShape`, `TerserCompressor`
`CompressorShape`, `TerserOutputStream` `OutputStream`, `createTerserParser`
`createLenientParser`, `terserParse` `parseLeniently`, and the token and word
kinds `tokenKindOf` and `wordKindOf`.
A bigint kept the separators it was written with, where the tokenizer drops
them: `123_456_789n` is printed `123456789n` again, which is five groups of the
test262 corpus back to byte for byte.
Where a test holds webpack's port to terser's own modules, each side answers to
its own names: terser's classes gain webpack's as aliases, and a tree is
compared by what a node holds rather than by what its class is called.
* docs(javascript): say that the printer minifies every asset
The map still described the switch that chose between webpack's printer and
terser, and the parser factory under its old name. One fixture is no longer
bigger than swc's output, so it leaves `SWC_SMALLER`.
* test(javascript): drop the test262 cases the minifier no longer fails
Minifying with webpack's printer — a bigint written without the separators it
was read with, a comment's line break read as terser's tokenizer reads it, an
annotated chain marked link by link — passes 527 of the cases
`MINIFIED_FAILURES` listed as failing, so they leave it. 638 remain, which is
what the minifier has left to fix.
* fix(javascript): refuse a private field with no class around it
The printer asked webpack's parser not to check private fields, so `foo.#bar`
and `#foo in bar` outside a class minified where terser and oxc both refuse
them — 76 differences over the comparison corpora, and a name the mangler then
renamed. The parser's own default checks it, so this drops the option.
* test(javascript): record where terser refuses or misreads what the printer reads
Reading a source with webpack's parser rather than terser's ported one leaves
757 differences over the comparison corpora, none of them webpack's. 704 are a
source the spec allows and terser refuses: an escape anywhere in an identifier
name, a trailing comma in an argument list, a regular expression after the ogham
space mark, `let` as a name outside strict mode, `yield` naming a function
inside a generator, `export * as default from`, a field named `get` before a
generator, and an escape no cooked template value can read.
The other 53 are bytes where terser reads or prints the program differently: a
`let` or `async` before a line break, which the semicolon inserted after it
leaves a name; a line continuation's own break written into the string it holds
nothing of; and a binary, octal or hex property key quoted as it is written,
naming the property `0b1` where it is `1`.
`REFUSED_BY_REFERENCE` lists the parse errors with what makes each source legal
and `REFERENCE_MISPRINTS` the cases by name. Both fail where the reference stops
differing, and an accepted output still runs against what its test prints.
* test(javascript): retire three outputs no longer bigger than oxc's
Writing an import of nothing without its empty list makes two of oxc's own
cases and one of swc's no bigger than what oxc writes, so `OXC_SMALLER` stops
listing them.
* refactor(javascript): keep terser the minifier, the printer a future default
Minifying every asset with the printer waits for the measurements to settle, so
`jsMinify` reads the user's terser again unless `printer: true` asks for the
printer, which `optimization.minimizeOptions.javascript` still defaults to under
`experiments.futureDefaults`. A cache entry follows terser's version again.
The test262 suite minifies through the printer whatever a build defaults to: it
holds webpack's minifier to the spec, not the one a user brings, so the cases it
no longer fails stay out of `MINIFIED_FAILURES`.
* test(javascript): build test262 for terser and for the printer again
Collapsing the two minified modes into one dropped terser's coverage and the
per-minifier `minifiers` keying of `MINIFIED_FAILURES`, so a case failing under
only one of them could not be recorded. Production is built once per minimizer
again, and every entry the earlier collapse removed is back.
What stays is the report: a case listed as failing that now passes is collected
rather than thrown, so one run names every entry to retire.
* refactor(javascript): minify with terser by default, the printer under futureDefaults
The default minimizer dispatches JavaScript to `terserMinify` as
`minimizer-webpack-plugin` publishes it, and to `jsMinify` — webpack's own
printer — only under `experiments.futureDefaults`. Which one runs is the
wiring's to decide, so `jsMinify` no longer takes a `printer` flag and goes
through the printer alone.
`getMinimizerVersion` reports webpack's version rather than terser's, since
the printer ships inside webpack and an upgrade of it is what moves the
output a cache entry holds.
* test(javascript): run test262's production mode through each minifier
Each minified mode names the function it builds with rather than an option:
`terserMinify` for the mode a plain production build runs, `jsMinify` for the
one `futureDefaults` switches to. Both read the same options, so a difference
between the modes is the minifier's own.
Splitting what terser's parser refuses follows: webpack's parser reads 515 of
those programs, so only terser is listed for them, while the 14 terser rejects
as a redeclaration are listed for both, the printer porting that refusal.
* test(javascript): list each minifier's test262 failures under its own cause
Both shards reported, and `printer` leaves fourteen entries it passes now,
each a program terser miscompiles and the `correct` phase gets right: `let`
after a newline in a `for-in`, `for-of` or `for-await-of` head, `async` with
no line terminator before a function declaration or an arrow, an escaped
`async` before one, `await` as the second argument of a dynamic import, and a
line continuation inside a single- or double-quoted string. Each moves to the
terser-only group its reason already has.
Two entries keep the printer, under what actually fails them rather than
terser's reason. `import/import-attributes/text-javascript.js` imports a
fixture that is not parsable JavaScript, so neither minifier can read the
asset. `expressions/await/await-in-nested-function.js` is a printer defect:
compressing inlines a function naming `await` into the async function calling
it, where the name is reserved, so the output does not parse — reached only
now that the printer is held to this case.1 parent cb47bb9 commit 9fd4df5
23 files changed
Lines changed: 5691 additions & 6972 deletions
File tree
- .changeset
- docs
- lib
- config
- javascript
- test
- benchmarkCases/js-minifier-unit
- helpers
- specCases
- unitCases
- tooling
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
39 | | - | |
| 39 | + | |
40 | 40 | | |
41 | | - | |
| 41 | + | |
42 | 42 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2519 | 2519 | | |
2520 | 2520 | | |
2521 | 2521 | | |
2522 | | - | |
2523 | | - | |
2524 | | - | |
2525 | | - | |
2526 | | - | |
2527 | | - | |
2528 | | - | |
2529 | | - | |
2530 | | - | |
2531 | 2522 | | |
2532 | 2523 | | |
2533 | 2524 | | |
| |||
2536 | 2527 | | |
2537 | 2528 | | |
2538 | 2529 | | |
2539 | | - | |
| 2530 | + | |
2540 | 2531 | | |
2541 | 2532 | | |
2542 | 2533 | | |
2543 | | - | |
2544 | | - | |
| 2534 | + | |
| 2535 | + | |
2545 | 2536 | | |
2546 | 2537 | | |
2547 | 2538 | | |
| |||
2557 | 2548 | | |
2558 | 2549 | | |
2559 | 2550 | | |
2560 | | - | |
| 2551 | + | |
2561 | 2552 | | |
2562 | 2553 | | |
| 2554 | + | |
| 2555 | + | |
| 2556 | + | |
| 2557 | + | |
| 2558 | + | |
| 2559 | + | |
2563 | 2560 | | |
2564 | 2561 | | |
2565 | 2562 | | |
| |||
2584 | 2581 | | |
2585 | 2582 | | |
2586 | 2583 | | |
2587 | | - | |
| 2584 | + | |
2588 | 2585 | | |
2589 | | - | |
2590 | | - | |
| 2586 | + | |
| 2587 | + | |
2591 | 2588 | | |
2592 | 2589 | | |
2593 | 2590 | | |
2594 | | - | |
| 2591 | + | |
2595 | 2592 | | |
2596 | 2593 | | |
2597 | 2594 | | |
| |||
2623 | 2620 | | |
2624 | 2621 | | |
2625 | 2622 | | |
2626 | | - | |
2627 | | - | |
| 2623 | + | |
| 2624 | + | |
2628 | 2625 | | |
2629 | 2626 | | |
2630 | 2627 | | |
| |||
2653 | 2650 | | |
2654 | 2651 | | |
2655 | 2652 | | |
2656 | | - | |
| 2653 | + | |
2657 | 2654 | | |
2658 | 2655 | | |
2659 | 2656 | | |
| |||
0 commit comments