Motivation for this change

Edit: PR description slightly outdated, please see the commit message for the current one (which has feedback addressed).

This has the intention to address various issues with the existing
builtins.filterSource and surrounding library functions:

• The library functions existing so far are not straightforward to use correctly and are hard to remember. Consequently only few people use them.

• Most existing functions are about excluding unwanted files, making various hardcodes (such as .o and .so in cleanSourceFilter) that typically still forget some other unwanted files. The added approach is more about explicitly listing wanted files,
  thus being more reliable.

• This very frequently results in non-reproducible builds when working from project working trees (as opposed to nixpkgs pkgs builds, which always operate from clean checkouts), with "build", "gen" or similar directories being accidentally included as sources.
  It is also a common sources of the dreaded
  warning: dumping very large path (> 256 MiB); this may run out of memory
  message (and as indicated, bad for performance).

• The most common antipattern regarding the above is src = ./..
  To prevent it, we need to provide functionality that's straightforward to understand and convenient.

• builtins.filterSource f ./. can result in build impurities depending on the basename of the parent directory (see Basename of current directory goes into derivation for filterSource on ./. nix#1305).
  A solution to that is builtins.path. The added function makes its use very convenient.

• The existing functions make it difficult to debug what files are included and why.
  The added function provides an argument debugEnableTracing, that makes debugging easy.
  Output looks like:

  trace: myproject: include regular   /home/user/myproject/Setup.hs
  trace: myproject: skip    regular   /home/user/myproject/myproject.nix
  trace: myproject: skip    directory /home/user/myproject/dist
  trace: myproject: include directory /home/user/myproject/images
  trace: myproject: include regular   /home/user/myproject/images/image.svg
  trace: myproject: include directory /home/user/myproject/src
  trace: myproject: include directory /home/user/myproject/src/MyDir
  trace: myproject: include regular   /home/user/myproject/src/MyDir/File1.hs
  trace: myproject: include regular   /home/user/myproject/src/MyDir/File2.hs
  

After spending tens of hours of tracking down irreproducible builds resulting from bad src = assignments in various projects, I hope that this function will put an end to it.

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Assured whether relevant documentation is up to date
• Fits CONTRIBUTING.md.
• Manual testing of the added functions
  • Examples:
    • lib.explicitFilterSource "lib" ./lib { srcDirs = [ ./lib ]; pathComponentExcludes = ["tests"]; }

TODO

• Get general feedback on the idea.
• Update nixpkgs manual, adding a section on using nix packages in project working trees, explaining how build files are often accidentally included, and recommending explicitFilterSource as a solution.
• Update nix manual, mentioning in the sections about builtins.filterSource and builtins.path's filter that nixpkgs has convenience functions to handle common situations.