Skip to content

Commit c150225

Browse files
fix(cache): accept a managedPaths RegExp without a capture group, and speed up pack writes (#21843)
* perf(serialization): dispatch cache items on their type `ObjectMiddleware`'s write loop is the hottest function webpack runs while storing the persistent cache. It opened with `Buffer.isBuffer(item)` and only reached numbers, booleans and `undefined` after four more failed comparisons — and counted over a 1,500-module build, those three are 1,066,959 of the 1,649,896 items it sees (64.6%), against 46,549 buffers (2.8%). Dispatch on `typeof` instead, so each item reaches its branch directly. The object branch also keys its type lookup on the serializer registration rather than on a `${request}/${name}` string, dropping another 254,319 string allocations per build. The wire format is unchanged: one registration is one `(request, name)` pair, so the same entries are emitted in the same order. Self time of that function drops from 296-321 ms to 201-230 ms over a 1,897-module fixture (`--cpu-prof`, two runs each, interleaved), and the cache store measured through `compiler.close()` from 1,179 ms to 1,056 ms (median of 6 interleaved runs). Emitted output is byte-identical. * fix: accept a snapshot.managedPaths RegExp without a capture group The schema documents a `snapshot.managedPaths` entry as "a RegExp matching a managed directory", but `FileSystemInfo` read the managed directory out of `match[1]`. A pattern written exactly as documented — with no capture group — therefore reached `getManagedItem(undefined, path)` and failed the build with `TypeError: Cannot read properties of undefined (reading 'length')` from inside the snapshot layer, naming nothing the user wrote. Fall back to the whole match, which is the directory the schema describes. * test: type the ObjectMiddleware round-trip context `ObjectMiddleware`'s serialize/deserialize take a context carrying a logger, and deserialize returns a value that may be a promise, so `{}` and a bare `[0]` failed `lint:types-test` on CI. * chore: shorten the comments the review flagged Both new comments ran past the two-line limit AGENTS.md sets for `lib/`, and one narrated the code it replaced. The changeset entry now fits the 80-character limit while still naming the option. * style: run prettier over CHANGELOG.md The entry for #20889 padded a double-backtick code span with spaces, which `fmt:check` rejects, so `lint` is red on main and on every branch cut from it. * test: pin the round-trip with strict equality `toEqual` ignores an undefined property and a swapped prototype, so the two cases the dispatch table exists for — `d: undefined` and the null-prototype object — would have passed even if the format dropped them. * test: use strict equality for the rest of the round-trip assertions The buffer case is the one that mattered: `toEqual` would have accepted a plain Uint8Array where the format restores a Buffer.
1 parent 3a7c0e6 commit c150225

12 files changed

Lines changed: 318 additions & 111 deletions

File tree

‎.changeset/010-persistent-cache.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"webpack": patch
3+
---
4+
5+
Fix a capture-group-less `snapshot.managedPaths` RegExp; speed up cache writes.

‎CHANGELOG.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -789,7 +789,7 @@
789789

790790
- Fix snapshot validity check for context dependencies in watch mode by treating watchpack's existence-only entries (`{}`) as cache misses. (by [@alexander-akait](https://github.com/alexander-akait) in [#20916](https://github.com/webpack/webpack/pull/20916))
791791

792-
- Support no-expression template literals in computed member access (e.g. `` import.meta[`url`] ``). (by [@alexander-akait](https://github.com/alexander-akait) in [#20889](https://github.com/webpack/webpack/pull/20889))
792+
- Support no-expression template literals in computed member access (e.g. ``import.meta[`url`]``). (by [@alexander-akait](https://github.com/alexander-akait) in [#20889](https://github.com/webpack/webpack/pull/20889))
793793

794794
- Improve tree-shaking in `isPure`: handle more expression types (`ArrayExpression`, `ObjectExpression`, `NewExpression`, `ChainExpression`, `UnaryExpression` (safe operators), `MetaProperty`, `TaggedTemplateExpression`, `BinaryExpression` (strict equality)), prevent `/*#__PURE__*/` comments from leaking across `ObjectExpression` properties, and detect PURE comments inside `TemplateLiteral` interpolations. (by [@alexander-akait](https://github.com/alexander-akait) in [#20723](https://github.com/webpack/webpack/pull/20723))
795795

‎lib/FileSystemInfo.js‎

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1207,6 +1207,15 @@ const getManagedItem = (managedPath, path) => {
12071207
return path.slice(0, i);
12081208
};
12091209

1210+
/**
1211+
* The managed directory a `snapshot.managedPaths` RegExp matched. The schema
1212+
* documents it as matching that directory, so a capture-less pattern is it.
1213+
* @param {RegExpExecArray} match a match of a managed path RegExp
1214+
* @returns {string} the managed directory
1215+
*/
1216+
const getManagedPathFromMatch = (match) =>
1217+
match[1] === undefined ? match[0] : match[1];
1218+
12101219
/**
12111220
* Gets resolved timestamp.
12121221
* @template {ContextFileSystemInfoEntry | ContextTimestampAndHash} T
@@ -2663,7 +2672,10 @@ class FileSystemInfo {
26632672
for (const managedPath of this.managedPathsRegExps) {
26642673
const match = managedPath.exec(path);
26652674
if (match) {
2666-
const managedItem = getManagedItem(match[1], path);
2675+
const managedItem = getManagedItem(
2676+
getManagedPathFromMatch(match),
2677+
path
2678+
);
26672679
if (managedItem) {
26682680
classificationCache.set(path, managedItem);
26692681
managedItems.add(managedItem);
@@ -3921,7 +3933,10 @@ class FileSystemInfo {
39213933
for (const managedPath of this.managedPathsRegExps) {
39223934
const match = managedPath.exec(path);
39233935
if (match) {
3924-
const managedItem = getManagedItem(match[1], path);
3936+
const managedItem = getManagedItem(
3937+
getManagedPathFromMatch(match),
3938+
path
3939+
);
39253940
if (managedItem) {
39263941
// construct timestampHash from managed info
39273942
return this.managedItemQueue.add(managedItem, (err, info) => {

‎lib/serialization/ObjectMiddleware.js‎

Lines changed: 128 additions & 108 deletions
Original file line numberDiff line numberDiff line change
@@ -461,7 +461,7 @@ class ObjectMiddleware extends SerializerMiddleware {
461461
return buf;
462462
};
463463
let currentPosTypeLookup = 0;
464-
/** @type {Map<ComplexSerializableType, number>} */
464+
/** @type {Map<SerializerConfigWithSerializer, number>} */
465465
let objectTypeLookup = new Map();
466466
/** @type {Set<ComplexSerializableType>} */
467467
const cycleStack = new Set();
@@ -573,140 +573,160 @@ class ObjectMiddleware extends SerializerMiddleware {
573573
...context
574574
};
575575
this.extendContext(ctx);
576+
// Dispatched on `typeof` first: two thirds of the items are numbers,
577+
// booleans and `undefined`, and only 3% are buffers.
576578
/**
577579
* Processes the provided item.
578580
* @param {ComplexSerializableType} item item to serialize
579581
*/
580582
const process = (item) => {
581-
if (Buffer.isBuffer(item)) {
582-
// check if we can emit a reference
583-
const ref = referenceable.get(item);
584-
if (ref !== undefined) {
585-
result.push(ESCAPE, ref - currentPos);
583+
switch (typeof item) {
584+
case "number":
585+
case "boolean":
586+
result.push(item);
586587
return;
587-
}
588-
const alreadyUsedBuffer = dedupeBuffer(item);
589-
if (alreadyUsedBuffer !== item) {
590-
const ref = referenceable.get(alreadyUsedBuffer);
591-
if (ref !== undefined) {
592-
referenceable.set(item, ref);
593-
result.push(ESCAPE, ref - currentPos);
594-
return;
588+
case "string": {
589+
if (item.length > 1) {
590+
// short strings are shorter when not emitting a reference (this saves 1 byte per empty string)
591+
// check if we can emit a reference
592+
const ref = referenceable.get(item);
593+
if (ref !== undefined) {
594+
const offset = ref - currentPos;
595+
if (
596+
// One far ref → 5 bytes (`null` + offset), one near ref -> 2/3 bytes
597+
offset >= -32768 ||
598+
// long enough that the ref wins, keep it
599+
item.length >= 4 ||
600+
// multibyte → inline would be even bigger, keep it
601+
Buffer.byteLength(item) !== item.length
602+
) {
603+
// A back-reference is `null` + the offset.
604+
result.push(ESCAPE, offset);
605+
return;
606+
}
607+
}
608+
addReferenceable(item);
595609
}
596-
item = alreadyUsedBuffer;
597-
}
598-
addReferenceable(item);
599610

600-
result.push(/** @type {Buffer} */ (item));
601-
} else if (item === ESCAPE) {
602-
result.push(ESCAPE, ESCAPE_ESCAPE_VALUE);
603-
} else if (
604-
typeof item === "object"
605-
// We don't have to check for null as ESCAPE is null and this has been checked before
606-
) {
607-
// check if we can emit a reference
608-
const ref = referenceable.get(item);
609-
if (ref !== undefined) {
610-
result.push(ESCAPE, ref - currentPos);
611+
if (item.length > 102400 && context.logger) {
612+
context.logger.warn(
613+
`Serializing big strings (${Math.round(
614+
item.length / 1024
615+
)}kiB) impacts deserialization performance (consider using Buffer instead and decode when needed)`
616+
);
617+
}
618+
619+
result.push(item);
611620
return;
612621
}
622+
case "undefined":
623+
result.push(ESCAPE, ESCAPE_UNDEFINED);
624+
return;
625+
case "object": {
626+
// ESCAPE is null, so this covers null too
627+
if (item === ESCAPE) {
628+
result.push(ESCAPE, ESCAPE_ESCAPE_VALUE);
629+
return;
630+
}
613631

614-
if (cycleStack.has(item)) {
615-
throw new Error(
616-
"This is a circular references. To serialize circular references use 'setCircularReference' somewhere in the circle during serialize and deserialize."
617-
);
618-
}
632+
if (Buffer.isBuffer(item)) {
633+
// check if we can emit a reference
634+
const ref = referenceable.get(item);
635+
if (ref !== undefined) {
636+
result.push(ESCAPE, ref - currentPos);
637+
return;
638+
}
639+
const alreadyUsedBuffer = dedupeBuffer(item);
640+
if (alreadyUsedBuffer !== item) {
641+
const ref = referenceable.get(alreadyUsedBuffer);
642+
if (ref !== undefined) {
643+
referenceable.set(item, ref);
644+
result.push(ESCAPE, ref - currentPos);
645+
return;
646+
}
647+
item = alreadyUsedBuffer;
648+
}
649+
addReferenceable(item);
619650

620-
const { request, name, serializer } = ObjectMiddleware.getSerializerFor(
621-
/** @type {Constructor} */
622-
(item)
623-
);
624-
const key = `${request}/${name}`;
625-
const lastIndex = objectTypeLookup.get(key);
651+
result.push(/** @type {Buffer} */ (item));
652+
return;
653+
}
626654

627-
if (lastIndex === undefined) {
628-
objectTypeLookup.set(key, currentPosTypeLookup++);
655+
// check if we can emit a reference
656+
const ref = referenceable.get(item);
657+
if (ref !== undefined) {
658+
result.push(ESCAPE, ref - currentPos);
659+
return;
660+
}
629661

630-
result.push(ESCAPE, request, name);
631-
} else {
632-
result.push(ESCAPE, currentPosTypeLookup - lastIndex);
633-
}
662+
if (cycleStack.has(item)) {
663+
throw new Error(
664+
"This is a circular references. To serialize circular references use 'setCircularReference' somewhere in the circle during serialize and deserialize."
665+
);
666+
}
634667

635-
cycleStack.add(item);
668+
// Keyed by the registration itself: one per class, so it stands in for
669+
// `${request}/${name}` without building that string per object.
670+
const config = ObjectMiddleware.getSerializerFor(
671+
/** @type {Constructor} */
672+
(item)
673+
);
674+
const { request, name, serializer } = config;
675+
const lastIndex = objectTypeLookup.get(config);
636676

637-
try {
638-
serializer.serialize(item, ctx);
639-
} finally {
640-
cycleStack.delete(item);
641-
}
677+
if (lastIndex === undefined) {
678+
objectTypeLookup.set(config, currentPosTypeLookup++);
642679

643-
result.push(ESCAPE, ESCAPE_END_OBJECT);
680+
result.push(ESCAPE, request, name);
681+
} else {
682+
result.push(ESCAPE, currentPosTypeLookup - lastIndex);
683+
}
644684

645-
addReferenceable(item);
646-
} else if (typeof item === "string") {
647-
if (item.length > 1) {
648-
// short strings are shorter when not emitting a reference (this saves 1 byte per empty string)
649-
// check if we can emit a reference
650-
const ref = referenceable.get(item);
651-
if (ref !== undefined) {
652-
const offset = ref - currentPos;
653-
if (
654-
// One far ref → 5 bytes (`null` + offset), one near ref -> 2/3 bytes
655-
offset >= -32768 ||
656-
// long enough that the ref wins, keep it
657-
item.length >= 4 ||
658-
// multibyte → inline would be even bigger, keep it
659-
Buffer.byteLength(item) !== item.length
660-
) {
661-
// A back-reference is `null` + the offset.
662-
result.push(ESCAPE, offset);
663-
return;
664-
}
685+
cycleStack.add(item);
686+
687+
try {
688+
serializer.serialize(item, ctx);
689+
} finally {
690+
cycleStack.delete(item);
665691
}
666-
addReferenceable(item);
667-
}
668692

669-
if (item.length > 102400 && context.logger) {
670-
context.logger.warn(
671-
`Serializing big strings (${Math.round(
672-
item.length / 1024
673-
)}kiB) impacts deserialization performance (consider using Buffer instead and decode when needed)`
674-
);
675-
}
693+
result.push(ESCAPE, ESCAPE_END_OBJECT);
676694

677-
result.push(item);
678-
} else if (typeof item === "function") {
679-
if (!SerializerMiddleware.isLazy(item)) {
680-
throw new Error(`Unexpected function ${item}`);
695+
addReferenceable(item);
696+
return;
681697
}
698+
case "function": {
699+
if (!SerializerMiddleware.isLazy(item)) {
700+
throw new Error(`Unexpected function ${item}`);
701+
}
682702

683-
/** @type {SerializedType | undefined} */
684-
const serializedData =
685-
SerializerMiddleware.getLazySerializedValue(item);
703+
/** @type {SerializedType | undefined} */
704+
const serializedData =
705+
SerializerMiddleware.getLazySerializedValue(item);
686706

687-
if (serializedData !== undefined) {
688-
if (typeof serializedData === "function") {
689-
result.push(serializedData);
690-
} else {
707+
if (serializedData !== undefined) {
708+
if (typeof serializedData === "function") {
709+
result.push(serializedData);
710+
} else {
711+
throw new Error("Not implemented");
712+
}
713+
} else if (SerializerMiddleware.isLazy(item, this)) {
691714
throw new Error("Not implemented");
715+
} else {
716+
const data =
717+
/** @type {() => PrimitiveSerializableType[] | Promise<PrimitiveSerializableType[]>} */
718+
(
719+
SerializerMiddleware.serializeLazy(item, (data) =>
720+
this.serialize([data], context)
721+
)
722+
);
723+
SerializerMiddleware.setLazySerializedValue(item, data);
724+
result.push(data);
692725
}
693-
} else if (SerializerMiddleware.isLazy(item, this)) {
694-
throw new Error("Not implemented");
695-
} else {
696-
const data =
697-
/** @type {() => PrimitiveSerializableType[] | Promise<PrimitiveSerializableType[]>} */
698-
(
699-
SerializerMiddleware.serializeLazy(item, (data) =>
700-
this.serialize([data], context)
701-
)
702-
);
703-
SerializerMiddleware.setLazySerializedValue(item, data);
704-
result.push(data);
726+
return;
705727
}
706-
} else if (item === undefined) {
707-
result.push(ESCAPE, ESCAPE_UNDEFINED);
708-
} else {
709-
result.push(item);
728+
default:
729+
result.push(item);
710730
}
711731
};
712732

0 commit comments

Comments
 (0)