Packages should be compiled independently and then linked together. This provides several advantages in the long term:

• Packages can be compiled in parallel so compile time can be shorter.
• Unchanged packages can be cached, further cutting down on compile time.
• We can optionally avoid LTO to make incremental (debug) builds much faster.

I would strive for the following architecture:

• Change the compiler package to compile just a single package (and not dependencies).
• Add a linker package that links several packages together and performs whole-program optimizations on them (LTO).
• Add a driver package (or repurpose the loader package?) that orchestrates all this work.
• Add some subcommands (go tool compile, go tool link) that call the compiler/linker directly. This can be useful for debugging.

I have taken a look at the Go build cache and I think I'd implement a similar system. Every package has a hash of the input (build tags, file list, hashes of files, output hashes of packages it depends on) and produces an output hash, which is a hash over all the artifacts that it builds (mostly the serialized exported types).

My thinking is that a cached package would be an uncompressed zip file stored in the cache directory, with the following contents as files:

• a list of packages it depends upon and their hashes (plain text)
• a list of files included in the build and their mtimes / hashes (plain text)
• a serialized form of the public API (some binary format, like this one)
• LLVM bytecode for this file (including C/C++ files, they are all linked together in one module / compile unit)

This is most likely too big to do at once, so I'd do it in multiple steps. But I'm creating this issue to give a high-level overview of what I would want to achieve at some point.