Skip to content

Commit 21a386a

Browse files
perf(javascript): read a parse's tokens into the room they need, once
Collecting a token cost more than reading one: four growable arrays to push into, two map lookups for its kind, a sealed object per token, and a token object the parser built only for the reader to copy from. The offsets, kinds and breaks now go into typed arrays sized for the source and doubled where it holds more, `finish` trims them instead of converting them, a kind is a code the table already holds, and `onTokenRead` hands the reader the parser itself, which also keeps the parser's token fast path. Parsing an 8.8 MiB bundle into the printer's tree: 2217 -> 1642 ms and a peak of 483 -> 351 MiB, against terser's parser at 1520 ms and 366 MiB. What the tree retains is unchanged at 129.0 MiB against terser's 237.1. Output stays byte for byte terser's over the 232 corpus cases and the 301 printer tests.
1 parent f634529 commit 21a386a

3 files changed

Lines changed: 88 additions & 41 deletions

File tree

‎lib/javascript/syntax-parser.js‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ const {
4747
* @property {boolean=} locations whether nodes carry `loc`
4848
* @property {{ line: number, column: number }=} startLocation where the first offset falls
4949
* @property {(((token: EXPECTED_ANY) => void) | EXPECTED_ANY[])=} onToken where tokens are reported
50+
* @property {((parser: EXPECTED_ANY) => void)=} onTokenRead the parser itself at each token, for a reader that copies what it needs rather than taking a token of its own
5051
* @property {(OnComment | EXPECTED_ANY[])=} onComment where comments are reported
5152
* @property {boolean=} ranges whether nodes carry `range`
5253
* @property {NodeLike=} program a program node to append to
@@ -71,6 +72,7 @@ const {
7172
* @property {boolean} locations whether nodes carry `loc`
7273
* @property {{ line: number, column: number } | null} startLocation where the first offset falls
7374
* @property {((token: EXPECTED_ANY) => void) | null} onToken where tokens are reported
75+
* @property {((parser: EXPECTED_ANY) => void) | null} onTokenRead the parser itself at each token
7476
* @property {OnComment | null} onComment where comments are reported
7577
* @property {boolean} ranges whether nodes carry `range`
7678
* @property {NodeLike | null} program a program node to append to
@@ -520,6 +522,7 @@ const defaultOptions = {
520522
locations: false,
521523
startLocation: null,
522524
onToken: null,
525+
onTokenRead: null,
523526
onComment: null,
524527
ranges: false,
525528
program: null,
@@ -5500,6 +5503,7 @@ class Parser {
55005503
);
55015504
}
55025505
if (this.options.onToken) this.options.onToken(new Token(this));
5506+
if (this.options.onTokenRead) this.options.onTokenRead(this);
55035507

55045508
this.lastTokEnd = this.end;
55055509
this.lastTokStart = this.start;
@@ -10684,6 +10688,7 @@ class WebpackParser extends BaseParser {
1068410688
`Escape sequence in keyword ${type.keyword}`
1068510689
);
1068610690
}
10691+
if (this.options.onTokenRead) this.options.onTokenRead(this);
1068710692
this.lastTokEnd = this.end;
1068810693
this.lastTokStart = this.start;
1068910694
this.nextToken();

‎lib/javascript/syntax-printer.js‎

Lines changed: 72 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -16350,7 +16350,6 @@ class SourceToken {
1635016350
constructor(tokens, index) {
1635116351
this.tokens = tokens;
1635216352
this.index = index;
16353-
Object.seal(this);
1635416353
}
1635516354

1635616355
/** @returns {number} where it starts */
@@ -16562,11 +16561,25 @@ const installFusedParse = (modules) => {
1656216561
// into it: a gap gaining a comment takes a list of its own first.
1656316562
/** @type {EXPECTED_ANY[]} */
1656416563
const NO_COMMENTS = [];
16565-
/** @type {Map<EXPECTED_ANY, string | null>} */
16564+
// A token type's kind as a code, so reporting one costs a single lookup; null
16565+
// where the kind follows from the word the token spells.
16566+
/** @type {Map<EXPECTED_ANY, number | null>} */
1656616567
const TOKEN_TYPES = new Map(
16567-
Object.values(tokTypes).map((type) => [type, terserTokenType(type)])
16568+
Object.values(tokTypes).map((type) => {
16569+
const named = terserTokenType(type);
16570+
return [
16571+
type,
16572+
named === null ? null : /** @type {number} */ (KIND_CODES.get(named))
16573+
];
16574+
})
1656816575
);
1656916576

16577+
const STRING_CODE = /** @type {number} */ (KIND_CODES.get("string"));
16578+
const PRIVATENAME_CODE = /** @type {number} */ (
16579+
KIND_CODES.get("privatename")
16580+
);
16581+
const EOF_CODE = /** @type {number} */ (KIND_CODES.get("eof"));
16582+
1657016583
/**
1657116584
* The tokens of a parse, as terser's tokenizer leaves them: each one holding
1657216585
* the comments read since the token before it, in a list it shares with that
@@ -16579,16 +16592,37 @@ const installFusedParse = (modules) => {
1657916592
*/
1658016593
const collectTokens = (source, filename) => {
1658116594
// Where each token sits, its kind as an index into `KINDS`, and whether a
16582-
// line break came before it. Grown while the parse runs, then narrowed by
16583-
// `finish` into arrays a token reads for as long as the tree lives.
16595+
// line break came before it: sized for the tokens a source this long holds,
16596+
// doubled where it holds more, and trimmed by `finish` to what it read.
16597+
let capacity = 64 + (source.length >> 2);
16598+
let count = 0;
1658416599
/** @type {EXPECTED_ANY} */
16585-
let starts = [];
16600+
let starts = new Int32Array(capacity);
1658616601
/** @type {EXPECTED_ANY} */
16587-
let ends = [];
16602+
let ends = new Int32Array(capacity);
1658816603
/** @type {EXPECTED_ANY} */
16589-
let kinds = [];
16604+
let kinds = new Uint8Array(capacity);
1659016605
/** @type {EXPECTED_ANY} */
16591-
let breaks = [];
16606+
let breaks = new Uint8Array(capacity);
16607+
16608+
/**
16609+
* @returns {void}
16610+
*/
16611+
const grow = () => {
16612+
capacity *= 2;
16613+
const widerStarts = new Int32Array(capacity);
16614+
widerStarts.set(starts);
16615+
starts = widerStarts;
16616+
const widerEnds = new Int32Array(capacity);
16617+
widerEnds.set(ends);
16618+
ends = widerEnds;
16619+
const widerKinds = new Uint8Array(capacity);
16620+
widerKinds.set(kinds);
16621+
kinds = widerKinds;
16622+
const widerBreaks = new Uint8Array(capacity);
16623+
widerBreaks.set(breaks);
16624+
breaks = widerBreaks;
16625+
};
1659216626
// The value of a token the source does not spell out: a string holds what
1659316627
// its escapes mean, and a private name is read without its `#`.
1659416628
/** @type {Map<number, EXPECTED_ANY> | null} */
@@ -16625,7 +16659,7 @@ const installFusedParse = (modules) => {
1662516659
// A gap the parser has not read past gains its comments after this, so it
1662616660
// is given a list of its own to gain them in; one already read and holding
1662716661
// none shares the empty list with every other.
16628-
const list = index < starts.length ? NO_COMMENTS : [];
16662+
const list = index < count ? NO_COMMENTS : [];
1662916663
lists[index] = list;
1663016664
return list;
1663116665
};
@@ -16760,7 +16794,7 @@ const installFusedParse = (modules) => {
1676016794
// break before it as well as breaking the line for what follows.
1676116795
brokenLine = isBlock && value.includes("\n");
1676216796
read(
16763-
starts.length,
16797+
count,
1676416798
isBlock ? "comment2" : "comment1",
1676516799
value,
1676616800
start,
@@ -16777,45 +16811,39 @@ const installFusedParse = (modules) => {
1677716811
code === 10 || code === 13 || code === 0x2028 || code === 0x2029;
1677816812
}
1677916813
brokenLine = false;
16780-
const named = TOKEN_TYPES.get(token.type);
16814+
const type = token.type;
16815+
const named = TOKEN_TYPES.get(type);
1678116816
const value = token.value === undefined ? "" : token.value;
1678216817
const kind =
1678316818
named === undefined || named === null
16784-
? terserWordType(String(value), afterDot)
16819+
? /** @type {number} */ (
16820+
KIND_CODES.get(terserWordType(String(value), afterDot))
16821+
)
1678516822
: named;
16786-
afterDot =
16787-
token.type === tokTypes.dot || token.type === tokTypes.questionDot;
16788-
if (kind === "string" || kind === "privatename") {
16789-
/** @type {Map<number, EXPECTED_ANY>} */ (cooked).set(
16790-
starts.length,
16791-
value
16792-
);
16823+
afterDot = type === tokTypes.dot || type === tokTypes.questionDot;
16824+
if (kind === STRING_CODE || kind === PRIVATENAME_CODE) {
16825+
/** @type {Map<number, EXPECTED_ANY>} */ (cooked).set(count, value);
1679316826
}
16794-
starts.push(from);
16827+
if (count === capacity) grow();
16828+
starts[count] = from;
1679516829
// A node ends at the last token it holds, which is never the one past
1679616830
// the source: terser leaves the comments after it on the token before.
16797-
ends.push(kind === "eof" ? NO_END : token.end);
16798-
kinds.push(KIND_CODES.get(kind));
16799-
breaks.push(nlb ? 1 : 0);
16831+
ends[count] = kind === EOF_CODE ? NO_END : token.end;
16832+
kinds[count] = kind;
16833+
breaks[count] = nlb ? 1 : 0;
16834+
count++;
1680016835
previousEnd = token.end;
1680116836
},
1680216837
shebang: (/** @type {string} */ text) => {
1680316838
read(0, "comment5", text, 0, false);
1680416839
},
1680516840
finish: () => {
16806-
// The parse is over: what it grew becomes what a token reads, in a
16807-
// quarter of the room, and what only the parse needed goes.
16808-
const count = starts.length;
16809-
const narrowStarts = new Int32Array(count);
16810-
const narrowEnds = new Int32Array(count);
16811-
const narrowKinds = new Uint8Array(count);
16812-
const narrowBreaks = new Uint8Array(count);
16813-
for (let i = 0; i < count; i++) {
16814-
narrowStarts[i] = starts[i];
16815-
narrowEnds[i] = ends[i];
16816-
narrowKinds[i] = kinds[i];
16817-
narrowBreaks[i] = breaks[i];
16818-
}
16841+
// The parse is over: the room it was given beyond what it read goes,
16842+
// and so does everything only the parse needed.
16843+
const narrowStarts = starts.slice(0, count);
16844+
const narrowEnds = ends.slice(0, count);
16845+
const narrowKinds = kinds.slice(0, count);
16846+
const narrowBreaks = breaks.slice(0, count);
1681916847
/** @type {Map<number, EXPECTED_ANY[]>} */
1682016848
const kept = new Map();
1682116849
for (let i = 0; i < lists.length; i++) {
@@ -16843,7 +16871,8 @@ const installFusedParse = (modules) => {
1684316871
) => {
1684416872
const offsets = ending ? ends : starts;
1684516873
let low = 0;
16846-
let high = offsets.length - 1;
16874+
// What the parse has read, which is short of the room it was given.
16875+
let high = count - 1;
1684716876
// The last token ending by `pos`, or the first one starting from it.
1684816877
while (low <= high) {
1684916878
const middle = (low + high) >> 1;
@@ -16859,13 +16888,13 @@ const installFusedParse = (modules) => {
1685916888
// when a node is given it, and the one past the source where the
1686016889
// parse ends before reaching that far.
1686116890
index += forward;
16862-
if (index >= starts.length) {
16891+
if (index >= count) {
1686316892
const ahead = build(index);
1686416893
beyond.push(ahead);
1686516894
return ahead;
1686616895
}
1686716896
}
16868-
return index < 0 || index >= starts.length ? undefined : build(index);
16897+
return index < 0 || index >= count ? undefined : build(index);
1686916898
}
1687016899
};
1687116900
};
@@ -16992,7 +17021,9 @@ const installFusedParse = (modules) => {
1699217021
{
1699317022
...parserOptions(text, options),
1699417023
onComment: tokens.onComment,
16995-
onToken: tokens.onToken
17024+
// The parser itself at each token, which keeps its token fast path
17025+
// and spares a token object a reader would only copy from.
17026+
onTokenRead: tokens.onToken
1699617027
},
1699717028
text,
1699817029
tokens.at

‎types.d.ts‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22565,6 +22565,11 @@ declare interface OptionsSyntaxParser {
2256522565
*/
2256622566
onToken?: any[] | ((token?: any) => void);
2256722567

22568+
/**
22569+
* the parser itself at each token, for a reader that copies what it needs rather than taking a token of its own
22570+
*/
22571+
onTokenRead?: (parser?: any) => void;
22572+
2256822573
/**
2256922574
* where comments are reported
2257022575
*/
@@ -26898,6 +26903,11 @@ declare interface ResolvedOptionsSyntaxParser {
2689826903
*/
2689926904
onToken: null | ((token?: any) => void);
2690026905

26906+
/**
26907+
* the parser itself at each token
26908+
*/
26909+
onTokenRead: null | ((parser?: any) => void);
26910+
2690126911
/**
2690226912
* where comments are reported
2690326913
*/
@@ -33142,6 +33152,7 @@ declare namespace exports {
3314233152
export let locations: boolean;
3314333153
export let startLocation: null;
3314433154
export let onToken: null;
33155+
export let onTokenRead: null;
3314533156
export let onComment: null;
3314633157
export let ranges: boolean;
3314733158
export let program: null;

0 commit comments

Comments
 (0)