Summary

Currently, Spack only builds one package at a time. This PR allows N packages to build concurrently through refactoring parts of installer.py and build_environment.py to allow more multiprocessing to occur. This feature improves the build process's parallelism, speeding up installation times for multi-package builds.

Changes to Codebase

In the installer, the main while loop, while self.peek_ready_task() or active_tasks: continues to execute as long as there are items ready to be installed or active_tasks. The inner while loop for this condition, while len(active_tasks) < self.max_active_tasks:, ensures that there is space available for starting another process for a task. max_active_tasks is assigned the value of -p/--concurrent-packages. If the inner while loop's condition is met, the lowest priority task gets popped off of the queue and a child process is started.

After breaking out of the inner-while loop, all of the active tasks are polled, checking that the child process has received information from the read pipe: ready = [task for task in active_tasks if task.poll()]. Once this occurs, the installation of the task is completed and the task gets removed from the active_tasks list, opening up space for another task to begin running.

• installer.py: BuildTask's execute() method has been split up into three new methods: start(), poll(), and complete().

• build_environment.py: The addition of class BuildProcess, which manages and monitors the state of a child process for a task used for building/installing a given package.

Usage

This PR adds the command line argument -p/--nprocs to spack install <package>, which allows users to specify the maximum number of packages that can be built concurrently when installing a given package. If -p/--nprocs is not specified, the default value is 4, meaning up to four packages could be built concurrently.

To-Do

• Performance testing to decide what value to assign as the default for -p/--concurrent-packages.
• Create and run unit-tests to exercise different build scenarios with new implementations.
• Refactor code that handles binary installation, currently commented out.
• Make -p/--nprocs a config option.