Depends on #11408.
Related to #9353.
Part of a set of issues that replaces #11138.

Abstract

Add --include-paths option which specifies a list of directories from which files can be loaded using relative paths (in addition to --base-path). This is meant to be similar to $PATH variable in the shell or $PYTHONPATH in Python.

The new feature makes it possible to reliably use relative paths in imports without relying on remappings. This is the first step towards disallowing absolute import paths, though for now it will still be possible to use an empty path as --base-path (in fact it remains the default) to keep using them.

Motivation

Currently the compiler provides the --base-path option, which is a way to specify how import paths should be resolved by the default file loader. By default base path is empty, which means that relative paths are relative to the current working directory while absolute paths remain absolute. This makes imports change their meaning depending on the directory from which the compiler is invoked. Specifying a custom base path, on the other hand, results in both relative and absolute paths being interpreted as relative to the base directory. This prevents the file loader from accessing files outside of that directory and thus breaks the common usage pattern where libraries are installed using a package manager and can reside in different locations depending on how they were installed (e.g. local vs global node_nodules/).

From what I have seen, frameworks generally ignore --base-path and have their own mechanisms for dealing with importing libraries: they either don't rely on the default file loader at all (Truffle, Hardhat) or use import remappings to redirect relative paths to the right directories (Brownie, dapp.tools). The goal of this proposal is to extend the base path mechanism to be actually usable in direct CLI usage and optionally also for those frameworks that do rely on the default file loader. It might may also make it easier to use LSP with projects that rely on frameworks that do path resolution on their own (#7763 (comment)).

Specification

1. Add a command-line option called --include-path.
   • The value is a single directory path. The option can be used multiple times to specify multiple directories.
   • The path must not be empty and must point at an existing directory.
   • The path is made canonical according to the same rules as --base-path (see Stripping --base-path from CLI paths + CLI path normalization (path spec v3) #11408).
   • The option is allowed in combination with --standard-json, just like --base-path currently is.
2. When the default file loader receives a source unit name to load, it first tries to prepend base path to it and see if a file exists under that path. Then it does the same with all directories passed to --include-path in the order they were specified.
   • The loader returns the content of the first matching file.
   • If there is more than one matching file, the compiler prints a warning.
   • NOTE: This is different from how some frameworks handle this. For example Truffle's path resolver stops at the first match so it's possible for a file in one library to "shadow" a file in another.
3. Paths from --include-paths are automatically added to --allowed-paths.
4. Include paths are stripped from CLI file paths the same way --base-path is (this part depends on Stripping --base-path from CLI paths + CLI path normalization (path spec v3) #11408).
5. Implement the same logic in solcjs.

As a result base path simply becomes the first entry on a list of directories that are searched for import files. The only difference between it and other include paths is that it can be empty (and that difference will be removed by #11410).

NOTE: The paths specified in sources.urls in Standard JSON are also processed by the default file loader so they will also be affected by the new option.

Backwards Compatibility

The change is fully backwards-compatible. Without using --include-paths we get the current behavior.