Allow using -j to control the parallelism of concretization#37608
Conversation
fixes spack#29464 This PR allows to use ``` $ spack concretize -j X ``` to set a cap on the parallelism of concretization from the command line
spackbot-app
bot
added
commands
core
alecbcs
left a comment
alecbcs
left a comment
There was a problem hiding this comment.
Awesome! Great to see this getting added!
| max_processes = min(len(arguments), 16) # Number of specs # Cap on 16 cores | ||
| max_processes = min( | ||
| len(arguments), # Number of specs | ||
| spack.config.get("config:build_jobs"), # Cap on build jobs |
There was a problem hiding this comment.
Are you sure we should still be taking the min of the two arguments here? If a user declares -j 16 but only specifies a single spec they might be surprised to see clingo only using a single thread no? Or is this not controlling threads to clingo?
There was a problem hiding this comment.
clingo is currently single threaded. We checked in the early days and using parallelism at that level was detrimental to performance. Parallelism is obtained by spawning multiple processes, each of which is concretizing an independent spec.
There was a problem hiding this comment.
Reference for using a single thread:
spack/lib/spack/spack/solver/asp.py
Line 83 in 384f5f9
There was a problem hiding this comment.
Corollary to this is if we switch to when_possible in our CI environments, we're back to basically one thread. IMO this is fine, as it'll still (hopefully) be faster b/c it's a single solve. But the win with parallelism here is b/c we're concretizing so many things separately.
There was a problem hiding this comment.
clingois currently single threaded. We checked in the early days and using parallelism at that level was detrimental to performance. Parallelism is obtained by spawning multiple processes, each of which is concretizing an independent spec.
Gotcha! Thanks for the explanation @alalazo! I was definitely confused on the meaning of processes here.
|
Bit too late, but this should use |
4 participants
fixes #29464
This PR allows to use
$ spack concretize -j Xto set a cap on the parallelism of concretization from the command line