Skip to content

Commit 95988b3

Browse files
feat(copy): add an ignore hook and never read from output.path (#22216)
* fix(copy): never read a file from inside output.path A pattern whose base contains `output.path` copied the compilation's own output back into itself, one directory deeper on every build: the watch never settled and the output grew without bound. Exclude the output directory from the walk, and do not watch a base that sits inside it. * fix(copy): exclude a symlink resolving into output.path too The lexical check saw only the link's own path, so a followed directory symlink reached the output directory under a name it never matched. Compare the resolved path where the walk already calls `realpath`. * test(copy): cover a pattern whose own base is such a symlink The alias case reaches the walk's check but not `walkBase`'s, which is the one a `from` naming the link directly takes. * feat(copy): add an ignore hook to CopyPlugin Never reading from `output.path` is the right default, but it left a pattern that deliberately read from there copying nothing and warning about it, with no way to say otherwise. Give plugins the say instead, the way CleanPlugin's `keep` hook does for what it deletes. `CopyPlugin.getCompilationHooks(compilation).ignore` is consulted for every path before `output.path` is: true ignores it, false copies it even from inside the output directory, and nothing keeps webpack's answer. A directory is answered like a file, because one that stays ignored is never walked — so opting a subtree back in means answering the directory too. The pattern's own `globOptions.ignore` still applies, so a plugin cannot override what the user configured. * fix(copy): exclude a file symlink resolving into output.path too The symlink guard only reached directories. A followed link to a FILE is reported by `stat` as a plain file and returned before anything canonicalizes it, so the lexical check saw the link's own name and copied the build's own output under it. A pattern naming such a link directly took the same path. Reproduced before fixing: with output.path at static/build and a link static/link.txt pointing into it, the copy emitted link.txt carrying the stale output, and `from: static/link.txt` emitted named.txt/link.txt. Resolve a followed file the same way a followed directory already is, in the walk and for a directly named one. Reverting lib/CopyPlugin.js fails both new assertions. * fix(copy): skip the dependency for a base resolving into output.path `walkBase` returns nothing for a base that resolves into the output directory, but the dependency guard after it still read the base as written. A symlink naming the output directory therefore copied nothing and was watched anyway, so a write webpack made under output.path invalidated the build through the alias. Resolve the base once, before both, and let the guard read that. Reverting lib/CopyPlugin.js fails the new assertion, which found the alias in `contextDependencies`. * fix(copy): copy a named symlink as a link when not following `stat` follows the link, so a `from` naming one was classified as what it points at: a `followSymlinks: false` read the target's bytes instead of copying the link, which is what the same link reached through the walk gets. * chore(changeset): note the named-symlink copy fix * refactor(copy): make the ignore hook the one answer for a path The hook sat inside the `output.path` check, so it overruled only that: a `false` still lost to `globOptions.ignore`, and a followed symlink asked it about the target's real path rather than the name the pattern reached it under. It now answers first and alone, the way `CleanPlugin`'s `keep` does. * chore(changeset): fold the copy changesets into one * feat(copy): leave output.path to globOptions.ignore and the ignore hook Reading from output.path can be intended, so it is no longer excluded by default; a configuration opts out with globOptions.ignore or the hook. * test(copy): cover lstat fallbacks and an ignored link when not following * chore(changeset): keep the copy entry within 80 characters
1 parent 592c065 commit 95988b3

9 files changed

Lines changed: 452 additions & 48 deletions

File tree

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"webpack": minor
3+
---
4+
5+
Add an `ignore` hook to `output.copy` and keep unfollowed `from` symlinks.

‎lib/output/CopyPlugin.js‎

Lines changed: 123 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -6,13 +6,15 @@
66
"use strict";
77

88
const path = require("path");
9+
const { SyncBailHook } = require("tapable");
910
const { RawSource } = require("webpack-sources");
1011
const Compilation = require("../Compilation");
1112
const EmptyCopyPatternWarning = require("../errors/EmptyCopyPatternWarning");
1213
const WebpackError = require("../errors/WebpackError");
1314
const { PERMISSIONS_MASK, join } = require("../fs/fs");
1415
const asyncLib = require("../util/async");
1516
const createHash = require("../util/createHash");
17+
const createHooksRegistry = require("../util/createHooksRegistry");
1618
const {
1719
escapeGlobPattern,
1820
extractGlobBaseDir,
@@ -23,6 +25,19 @@ const {
2325
} = require("../util/globUtils");
2426
const { ABSOLUTE_PATH_REGEXP } = require("../util/identifier");
2527

28+
const createCompilationHooks = () => ({
29+
/**
30+
* Answers whether a path is copied, ahead of `globOptions.ignore`: true never copies it, false copies it whatever `globOptions.ignore` says, and nothing leaves the decision to webpack. A directory has to be answered as well as the files below it, because one that is ignored is never walked.
31+
* @type {SyncBailHook<[string], boolean | void>}
32+
* @since 5.112.0
33+
*/
34+
ignore: new SyncBailHook(["path"])
35+
});
36+
37+
/**
38+
* @typedef {ReturnType<typeof createCompilationHooks>} CopyPluginCompilationHooks
39+
*/
40+
2641
/** @import { CopyGlobOptions, CopyObjectPattern, CopyOptions, CopyPattern } from "../../declarations/WebpackOptions" */
2742
/** @import Compiler from "../Compiler" */
2843
/** @import { AssetInfo, AssetSymlink, AssetTimestamps, PathData } from "../Compilation" */
@@ -183,6 +198,31 @@ const stat = (fs, filePath) =>
183198
});
184199
});
185200

201+
/**
202+
* Answers whether the path is a symbolic link itself rather than what one points at.
203+
* @param {InputFileSystem} fs input file system
204+
* @param {string} filePath path
205+
* @returns {Promise<boolean>} true when it is a symbolic link
206+
*/
207+
const isSymbolicLink = (fs, filePath) =>
208+
new Promise((resolve, reject) => {
209+
// a file system without it reports what the link points at, which is the
210+
// behavior this answer then leaves in place
211+
if (!fs.lstat) {
212+
resolve(false);
213+
return;
214+
}
215+
fs.lstat(filePath, (err, stats) => {
216+
if (err) {
217+
if (err.code === "ENOENT" || err.code === "ENOTDIR") {
218+
return resolve(false);
219+
}
220+
return reject(err);
221+
}
222+
resolve(/** @type {IStats} */ (stats).isSymbolicLink());
223+
});
224+
});
225+
186226
/**
187227
* @param {InputFileSystem} fs input file system
188228
* @param {string} filePath path
@@ -260,11 +300,12 @@ const walkDirectory = async (options, directory, visited, maxDepth) => {
260300
DIRECTORY_CONCURRENCY,
261301
async (entry) => {
262302
const filePath = join(fs, directory, entry.name);
303+
const normalizedPath = normalizePathSeparatorsForPath(filePath);
263304
const isSymlink = entry.isSymbolicLink();
264305
// the link itself is what is copied, so neither its target nor what is
265306
// below it is read
266307
if (isSymlink && !followSymlinks) {
267-
if (ignore.file(normalizePathSeparatorsForPath(filePath))) return [];
308+
if (ignore.file(normalizedPath)) return [];
268309
links.add(filePath);
269310
return [filePath];
270311
}
@@ -273,11 +314,13 @@ const walkDirectory = async (options, directory, visited, maxDepth) => {
273314
const stats = isSymlink ? await stat(fs, filePath) : entry;
274315
// the target is gone, or is neither a file nor a directory
275316
if (stats === undefined) return [];
276-
if (stats.isFile()) return [filePath];
317+
if (stats.isFile()) {
318+
return ignore.file(normalizedPath) ? [] : [filePath];
319+
}
277320
if (!stats.isDirectory() || maxDepth === 0) return [];
278321
// an ignored directory is not listed at all, which is what makes an
279322
// `ignore` naming one cheaper than dropping its files afterwards
280-
if (ignore.directory(normalizePathSeparatorsForPath(filePath))) return [];
323+
if (ignore.directory(normalizedPath)) return [];
281324
if (isSymlink) {
282325
const realPath = await realpath(fs, filePath);
283326
if (visited.has(realPath)) return [];
@@ -301,16 +344,13 @@ const walkDirectory = async (options, directory, visited, maxDepth) => {
301344
* @returns {Promise<string[]>} absolute paths of the files below it, in a stable order
302345
*/
303346
const walkBase = async (options, base, maxDepth) => {
304-
// the base is a directory like any below it, so an `ignore` naming it skips
305-
// what the whole pattern would have copied
306-
if (options.ignore.directory(normalizePathSeparatorsForPath(base))) return [];
307-
return walkDirectory(
308-
options,
309-
base,
310-
// the base counts as walked, so a symlink pointing back at it ends there
311-
new Set(options.followSymlinks ? [await realpath(options.fs, base)] : []),
312-
maxDepth
313-
);
347+
/** @type {Set<string>} */
348+
const visited = new Set();
349+
// the base counts as walked, so a symlink pointing back at it ends there
350+
if (options.followSymlinks) {
351+
visited.add(await realpath(options.fs, base));
352+
}
353+
return walkDirectory(options, base, visited, maxDepth);
314354
};
315355

316356
/**
@@ -321,6 +361,25 @@ const notIgnored = () => false;
321361
/** @type {IgnoreMatcher} */
322362
const NOTHING_IGNORED = { file: notIgnored, directory: notIgnored };
323363

364+
/**
365+
* Puts the `ignore` hook ahead of `globOptions.ignore`, so a plugin can overrule it either way.
366+
* @param {CopyPluginCompilationHooks} hooks the compilation's copy hooks
367+
* @param {IgnoreMatcher} ignore what `globOptions.ignore` does not copy
368+
* @returns {IgnoreMatcher} the one answer for a path
369+
*/
370+
const createIgnoreMatcher = (hooks, ignore) => {
371+
/**
372+
* @param {(normalizedPath: string) => boolean} byGlobOptions what `globOptions.ignore` says
373+
* @returns {(normalizedPath: string) => boolean} what the copy does
374+
*/
375+
const answer = (byGlobOptions) => (normalizedPath) => {
376+
const decision = hooks.ignore.call(normalizedPath);
377+
if (decision !== undefined) return decision;
378+
return byGlobOptions(normalizedPath);
379+
};
380+
return { file: answer(ignore.file), directory: answer(ignore.directory) };
381+
};
382+
324383
// the tail of a glob naming everything inside a directory, which names the
325384
// directory itself once it is gone
326385
const IGNORED_DIRECTORY_CONTENTS_REGEXP = /\/\*\*(?:\/\*)?$/;
@@ -474,10 +533,9 @@ class CopyPlugin {
474533
caseSensitive: globOptions.caseSensitive,
475534
requireLiteralLeadingDot: globOptions.dot === false
476535
};
477-
const ignore = this._createIgnoreMatcher(
478-
context,
479-
globOptions.ignore,
480-
matchOptions
536+
const ignore = createIgnoreMatcher(
537+
CopyPlugin.getCompilationHooks(compilation),
538+
this._createIgnoreMatcher(context, globOptions.ignore, matchOptions)
481539
);
482540
/** @type {Set<string>} */
483541
const links = new Set();
@@ -584,15 +642,22 @@ class CopyPlugin {
584642
const absoluteFrom = path.isAbsolute(from)
585643
? path.normalize(from)
586644
: path.resolve(context, from);
645+
// `stat` follows the link, so a `from` naming one is classified as what it
646+
// points at — a directory to walk into, or a file whose bytes to read —
647+
// where a `followSymlinks: false` copies the link itself instead
648+
const namesLink =
649+
!followSymlinks && (await isSymbolicLink(fs, absoluteFrom));
587650
const stats = await stat(fs, absoluteFrom);
588651
/** @type {TypeOfFrom} */
589-
const typeOfFrom = !stats
590-
? "glob"
591-
: stats.isDirectory()
592-
? "dir"
593-
: stats.isFile()
594-
? "file"
595-
: "glob";
652+
const typeOfFrom = namesLink
653+
? "file"
654+
: !stats
655+
? "glob"
656+
: stats.isDirectory()
657+
? "dir"
658+
: stats.isFile()
659+
? "file"
660+
: "glob";
596661

597662
/** @type {string} */
598663
let glob;
@@ -634,39 +699,47 @@ class CopyPlugin {
634699
// the glob is normalized already, and the base once here, so the match
635700
// does not redo either for every entry
636701
const normalizedBase = normalizePathSeparatorsForPath(base);
637-
const entries =
702+
// asked once for the whole pattern, so the walk and the watch agree on it
703+
const ignored =
638704
typeOfFrom === "file"
639705
? ignore.file(normalizePathSeparatorsForPath(absoluteFrom))
640-
? []
641-
: [absoluteFrom]
706+
: ignore.directory(normalizedBase);
707+
const entries = ignored
708+
? []
709+
: typeOfFrom === "file"
710+
? [absoluteFrom]
642711
: (
643712
await walkBase(
644713
{ fs, followSymlinks, ignore, links },
645714
base,
646715
maxDepth
647716
)
648-
).filter((filePath) => {
649-
const normalizedPath = normalizePathSeparatorsForPath(filePath);
650-
return (
651-
globMatchNormalizedWithExplicitDot(
652-
glob,
653-
normalizedPath,
654-
normalizedBase,
655-
matchOptions
656-
) && !ignore.file(normalizedPath)
657-
);
658-
});
717+
).filter((filePath) =>
718+
globMatchNormalizedWithExplicitDot(
719+
glob,
720+
normalizePathSeparatorsForPath(filePath),
721+
normalizedBase,
722+
matchOptions
723+
)
724+
);
659725

660-
await this._addPatternDependency(
661-
fs,
662-
compilation,
663-
typeOfFrom,
664-
absoluteFrom,
665-
base,
666-
// only a glob can name a base which is not there — a directory `from`
667-
// was stat'd above, and a file one watches itself
668-
entries.length === 0 && typeOfFrom === "glob"
669-
);
726+
// what the link says is what is copied, the same as one the walk reaches
727+
if (namesLink && entries.length > 0) links.add(absoluteFrom);
728+
729+
// a base which copies nothing is not watched either, or the compilation
730+
// would rebuild on what it wrote there itself
731+
if (!ignored) {
732+
await this._addPatternDependency(
733+
fs,
734+
compilation,
735+
typeOfFrom,
736+
absoluteFrom,
737+
base,
738+
// only a glob can name a base which is not there — a directory `from`
739+
// was stat'd above, and a file one watches itself
740+
entries.length === 0 && typeOfFrom === "glob"
741+
);
742+
}
670743

671744
return { from, typeOfFrom, glob, base, entries };
672745
}
@@ -1051,4 +1124,6 @@ class CopyPlugin {
10511124
}
10521125
}
10531126

1127+
CopyPlugin.getCompilationHooks = createHooksRegistry(createCompilationHooks);
1128+
10541129
module.exports = CopyPlugin;
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
a
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
deep
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
secret
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
const fs = require("fs");
2+
const path = require("path");
3+
4+
const read = (name) => fs.readFileSync(path.resolve(__dirname, name), "utf-8");
5+
6+
const exists = (name) => fs.existsSync(path.resolve(__dirname, name));
7+
8+
it("should not copy a path the ignore hook claims", () => {
9+
expect(exists("secret.txt")).toBe(false);
10+
});
11+
12+
it("should copy what the hook leaves to webpack", () => {
13+
expect(read("a.txt")).toBe("a");
14+
expect(read("nested/deep.txt")).toBe("deep");
15+
});
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
"use strict";
2+
3+
const webpack = require("../../../../");
4+
5+
/** @type {import("../../../../").Configuration} */
6+
module.exports = {
7+
output: {
8+
copy: ["files"]
9+
},
10+
plugins: [
11+
(compiler) => {
12+
compiler.hooks.thisCompilation.tap("Test", (compilation) => {
13+
webpack.CopyPlugin.getCompilationHooks(compilation).ignore.tap(
14+
"Test",
15+
(copiedPath) => {
16+
if (copiedPath.endsWith("/secret.txt")) return true;
17+
}
18+
);
19+
});
20+
}
21+
]
22+
};

0 commit comments

Comments
 (0)