nix-local-build has to attempt to recompile EVERY component, even if files from only one component has changed. There seems to be some non-zero latency associated with thinking about a file, so when nix-local-build ends up hitting every component, this combined with GHC's recompilation avoidance latency makes the latency pretty bad.