depfile: reduce dag with --use-buildcache flag.#33315
Conversation
|
@spackbot fix style |
|
Let me see if I can fix that for you! |
|
I was able to run spack style --fix==> Running style checks on spack
selected: isort, mypy, black, flake8
==> Modified files
lib/spack/spack/cmd/env.py
==> Running isort checks
isort checks were clean
==> Running mypy checks
Success: no issues found in 556 source files
mypy checks were clean
==> Running black checks
reformatted lib/spack/spack/cmd/env.py
All done! ✨ 🍰 ✨
1 file reformatted.
black checks were clean
==> Running flake8 checks
flake8 checks were clean
==> spack style checks were clean
I've updated the branch with isort fixes. |
|
@trws do you have opinions here? So, But maybe
|
haampie
changed the title
haampie
changed the title
haampie
changed the title
|
@alalazo mentioned: If we do that, we can immediately add the right But actually it's kinda similar to the amazing |
| pass | ||
|
|
||
|
|
||
| def traverse_breadth_first(visitor, specs=[]): |
There was a problem hiding this comment.
Spec.traverse(order='pre') should match this in all the desired ways and might avoid some corner cases like:
1____
| \ \
2 3 4
| | |
| 5 6
|_/__/
|
7
i.e., we want something more-sophisticated than breadth first or we may end up visiting 7 before [5, 6] (if your implementation avoids this, we should probably call it something like "topological" rather than "breadth-first").
There was a problem hiding this comment.
Hm, the point is I have multiple root specs, and need to handle the roots differently from the dependencies. Given that the second root may appear as a dependency of the first root, I don't want depth-first search. Breadth-first search initialized with all root specs guarantees that roots are found at depth 0 and that all connected nodes are visited once. I don't see the need for topological sort, for make targets all I need is the adjacency list.
See the bit
def neighbors(self, node):
# At depth zero we return build / test deps too
if node.depth == 0:
return node.spec.dependencies(deptype="all")
# Otherwise we just return link/run type deps
return node.spec.dependencies(deptype=("link", "run"))
haampie
force-pushed
the
depfile/shallow-install
branch
from
d91d94b to
c92bde9
Compare
haampie
changed the title
--use-buildcache flag.
haampie
force-pushed
the
depfile/shallow-install
branch
2 times, most recently
from
94f3c45 to
5e031ba
Compare
|
This is ready for review, again. Includes tests now. |
When installing some/all specs from a buildcache, build edges are pruned from those specs. This can result in a much smaller effective DAG. Until now, `spack env depfile` would always generate a full DAG. Ths PR adds the `spack env depfile --use-buildcache` flag that was introduced for `spack install` before. This way, not only can we drop build edges, but also we can automatically set the right buildcache related flags on the specific specs that are gonna get installed. This way we get parallel installs of binary deps without redundancy, which is useful for Gitlab CI.
haampie
force-pushed
the
depfile/shallow-install
branch
from
5e031ba to
0760894
Compare
|
@scheibelp I've moved the breadth-first traversal bits to #33406 so that this PR doesn't introduce its own "custom" traversal. |
3 participants
When installing some/all specs from a buildcache, build edges are pruned
from those specs. This can result in a much smaller effective DAG. Until
now,
spack env depfilewould always generate a full DAG.Ths PR adds the
spack env depfile --use-buildcacheflag that wasintroduced for
spack installbefore. This way, not only can we dropbuild edges, but also we can automatically set the right buildcache
related flags on the specific specs that are gonna get installed.
On top of that, this PR introduces
spack env depfile [spec 1] [spec 2]...that reduces the dag to particular specs of interest in the environment.
This way we get parallel installs of binary deps of a particular spec
that's built from sources, without redundancy, which is useful for
Gitlab CI.
Example use: