I resolved all the CI problems I had here except for one that's happening on Windows and this one is actually something that I cannot really fix on our side. I suspect it's a bug in boost::filesystem. I reported it in boostorg/filesystem#201.

The issue is basically that some paths that have redundant .. segments (e.g. /a/../a/b/) make weakly_canonical() throw on Windows. This was not a problem with --base-path because it requires the directory to actually exist. --allow-paths does not and I don't think it can - if it did, the targets of remappings would have to always exist on disk.

One workaround I can think of (other than waiting for this to be fixed in boost) would be to just require these paths not to have such redundant segments. Possibly only on Windows. The user can always avoid it by using a normalized path anyway.

This now passes all tests. I disabled tests that trigger the bug in boost (but only on Windows and on boost <= 1.76). The problem is still there until boost 1.77 gets released and we start using it for release binaries. And even then anyone who builds with older boost will run into it.

I think that's better than waiting until 1.77 is released and increasing the required version. Increasing it might cause problems for people who build on distros that do not have cutting-edge libraries and the bug is Windows-only anyway. To trigger it someone has to specifically use --allow-paths with a path to a directory that does not exist and is unnormalized in a very specific way - either enters and exits a directory (e.g. a/../a/b/) or goes beyond FS root and ends with .. (e.g. /../a/..). It also does not affect --base-path even though we normalize it before checking if the dir exists - that's because the bug is triggered only when we use weakly_canonical() to get symlinks resolved.

#11686 approved, which now unblocks this PR.

Commits squashed.

@cameel It's a huge PR, took a while, sorry. But I think that's all "right" :)

Thanks!

Well, it's big but it's either that or splitting it into multiple PRs that depend on each other and in the past I've seen that long chains of dependent PRs get pretty confusing. Still, not sure if this turned out better. I'll probably go back to splitting them next time because that way the parts are smaller and easier to review.