Consolidate concretization output for environments by tgamblin · Pull Request #44489 · spack/spack

tgamblin · 2024-06-02T05:30:28Z

When Spack concretizes environments, it prints every (newly concretized) root spec individually with all of its dependencies. For most reasonably sized environments, this is too much output. This is true for three commands:

spack concretize when concretizing an environment with newly added specs
spack install when installing an environment with newly added specs
spack spec with no arguments in an environment

The output dates back to before we had unified environments or nicer spec traversal routines, and we can improve it.

This PR makes environment concretization output analogous to what we do for regular specs. Just like spack spec for a single spec, we show all root specs with no indentation, so you can easily see the specs you explicitly requested. Dependencies are shown:

With indentation according to their depth in a breadth-first traversal starting at the roots;
Only once if they appear on paths from multiple roots

So, the default is now consistent with spack spec for one spec--it's --cover=nodes. i.e., if there are 100 specs in your environment, you'll get 100 lines of output. Here's an example:

You can see that the roots, python, tcl, and zlib-ng are at the top level of the
output. Specs like gnuconfig, a shared dependency of tcl and zlib-ng, are only
shown for the first thing that requires them (tcl here).

If you want to see more details, you can do that with spack spec using the arguments you're already familiar with. For example, if you wanted to see dependency types and all dependencies, you could use spack spec -l --cover=edges. Or you could add deptypes and namespaces with, e.g. spack spec -ltN.

With no arguments in an environment, spack spec concretizes (if necessary) and shows the concretized environment. If you run spack concretize first, inspecting the environment repeatedly with spack spec will be fast, as everything is already in the spack.lock file.

factor most logic of Spec.tree() out of Spec class into spack.spec.tree(), which can take multiple specs as roots.
make Spec.tree() call spack.spec.tree()
spack.environment.display_specs() now uses spack.spec.tree()
Update spack concretize
Update spack install
Update spack spec to call spack.spec.tree() for environments.
Continue to output specs individually for spack spec when using --yaml or --json

haampie · 2024-06-02T08:20:20Z

I like this 👍 much more compact. The thing that's lost is associating abstract with concrete spec, I wonder if people rely on that.

tgamblin · 2024-06-02T14:37:26Z

I feel like in the spack find view for an environment, we show that, and if you're running concretize, you've likely either just looked at a spack.yaml, spack find output or something where you would know what abstract specs you're looking for.

For non-environment spack spec invocations, I've actually been meaning to remove the input spec part, as I don't think it adds anything -- it's already on the command line just before the spack spec invocation.

tgamblin · 2024-06-02T14:38:13Z

This reminds me that another place we should be concretizing together and displaying together is when people provide multiple specs on the spack spec CLI.

tgamblin · 2024-06-02T14:52:18Z

Ok looking at that I think it's a separate PR -- there is some additional consolidation that needs to happen to merge all the ways we currently invoke concretize(). I think I will leave this as-is and fix up spack spec in a later PR.

alecbcs

Looks good to me. Unifying the ways in which we print spec trees will be great, but I agree that can be done in a follow up PR without blocking this work

tgamblin · 2024-06-03T16:50:25Z

@kennyweiss FYI

When Spack concretizes environments, it prints every (newly concretized) root spec individually with all of its dependencies. For most reasonably sized environments, this is too much output. This is true for three commands: * `spack concretize` when concretizing an environment with newly added specs * `spack install` when installing an environment with newly added specs * `spack spec` with no arguments in an environment The output dates back to before we had unified environments or nicer spec traversal routines, and we can improve it. This PR makes environment concretization output analogous to what we do for regular specs. Just like `spack spec` for a single spec, we show all root specs with no indentation, so you can easily see the specs you explicitly requested. Dependencies are shown: 1. With indentation according to their depth in a breadth-first traversal starting at the roots; 2. Only once if they appear on paths from multiple roots So, the default is now consistent with `spack spec` for one spec--it's `--cover=nodes`. i.e., if there are 100 specs in your environment, you'll get 100 lines of output. If you want to see more details, you can do that with `spack spec` using the arguments you're already familiar with. For example, if you wanted to see dependency types and *all* dependencies, you could use `spack spec -l --cover=edges`. Or you could add deptypes and namespaces with, e.g. `spack spec -ltN`. With no arguments in an environment, `spack spec` concretizes (if necessary) and shows the concretized environment. If you run `spack concretize` *first*, inspecting the environment repeatedly with `spack spec` will be fast, as everything is already in the `spack.lock` file. - [x] factor most logic of `Spec.tree()` out of `Spec` class into `spack.spec.tree()`, which can take multiple specs as roots. - [x] make `Spec.tree()` call `spack.spec.tree()` - [x] `spack.environment.display_specs()` now uses `spack.spec.tree()` - [x] Update `spack concretize` - [x] Update `spack install` - [x] Update `spack spec` to call `spack.spec.tree()` for environments. - [x] Continue to output specs individually for `spack spec` when using `--yaml` or `--json`

scheibelp

Overall this seems good. There appears to be a couple spots with minor inconsistencies/redundancies, but not something that should hold up the PR IMO.

scheibelp · 2024-06-03T17:04:41Z

lib/spack/spack/cmd/solve.py

+                elif required_format == "json":
+                    sys.stdout.write(spec.to_json(hash=ht.dag_hash))
+        else:
+            sys.stdout.write(spack.spec.tree(result.specs, color=sys.stdout.isatty(), **kwargs))


(minor) It seems like this could be consolidated into a function that replaces this and lines 108-129 of cmd/spec.py.

lib/spack/spack/environment/environment.py

lib/spack/spack/spec.py

scheibelp · 2024-06-03T20:28:05Z

There is some discussion on refactoring at

When Spack concretizes environments, it prints every (newly concretized) root spec individually with all of its dependencies. For most reasonably sized environments, this is too much output. This is true for three commands: * `spack concretize` when concretizing an environment with newly added specs * `spack install` when installing an environment with newly added specs * `spack spec` with no arguments in an environment The output dates back to before we had unified environments or nicer spec traversal routines, and we can improve it. This PR makes environment concretization output analogous to what we do for regular specs. Just like `spack spec` for a single spec, we show all root specs with no indentation, so you can easily see the specs you explicitly requested. Dependencies are shown: 1. With indentation according to their depth in a breadth-first traversal starting at the roots; 2. Only once if they appear on paths from multiple roots So, the default is now consistent with `spack spec` for one spec--it's `--cover=nodes`. i.e., if there are 100 specs in your environment, you'll get 100 lines of output. If you want to see more details, you can do that with `spack spec` using the arguments you're already familiar with. For example, if you wanted to see dependency types and *all* dependencies, you could use `spack spec -l --cover=edges`. Or you could add deptypes and namespaces with, e.g. `spack spec -ltN`. With no arguments in an environment, `spack spec` concretizes (if necessary) and shows the concretized environment. If you run `spack concretize` *first*, inspecting the environment repeatedly with `spack spec` will be fast, as everything is already in the `spack.lock` file. - [x] factor most logic of `Spec.tree()` out of `Spec` class into `spack.spec.tree()`, which can take multiple specs as roots. - [x] make `Spec.tree()` call `spack.spec.tree()` - [x] `spack.environment.display_specs()` now uses `spack.spec.tree()` - [x] Update `spack concretize` - [x] Update `spack install` - [x] Update `spack spec` to call `spack.spec.tree()` for environments. - [x] Continue to output specs individually for `spack spec` when using `--yaml` or `--json`

`spack spec` output has looked like this for a while: ```console > spack spec /v5fn6xo /wd2p2v7 Input spec -------------------------------- - /v5fn6xo Concretized -------------------------------- [+] [email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=darwin-sequoia-m1 [+] ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=darwin-sequoia-m1 ... Input spec -------------------------------- - /wd2p2v7 Concretized -------------------------------- [+] [email protected]%[email protected] build_system=python_pip arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 ``` But the input spec is right there on the CLI, and it doesn't add anything to the output. Also, since #44843, specs concretized in the CLI line can be unified, so it makes sense to display them as we did in #44489 -- as one multi-root tree instead of as multiple single-root trees. With this PR, concretize output now looks like this: ```console > spack spec /v5fn6xo /wd2p2v7 [+] [email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=darwin-sequoia-m1 [+] ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~guile build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+cpanm+opcode+open+shared+threads build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~debug~pic+shared build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools libs=shared,static arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools patches=bbf97f1 arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+compat+new_strategies+opt+pic+shared build_system=autotools arch=darwin-sequoia-m1 [+] ^gnuconfig@2022-09-17%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] [email protected]%[email protected] build_system=python_pip arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 [-] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 ``` With no input spec displayed -- just the concretization output shown as one consolidated tree and multiple roots. - [x] remove "Input Spec" section and "Concretized" header from `spack spec` output - [x] print concretized specs as one BFS tree instead of multiple Signed-off-by: Todd Gamblin <[email protected]>

`spack spec` output has looked like this for a while: ```console > spack spec /v5fn6xo /wd2p2v7 Input spec -------------------------------- - /v5fn6xo Concretized -------------------------------- [+] [email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=darwin-sequoia-m1 [+] ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=darwin-sequoia-m1 ... Input spec -------------------------------- - /wd2p2v7 Concretized -------------------------------- [+] [email protected]%[email protected] build_system=python_pip arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 ``` But the input spec is right there on the CLI, and it doesn't add anything to the output. Also, since #44843, specs concretized in the CLI line can be unified, so it makes sense to display them as we did in #44489 -- as one multi-root tree instead of as multiple single-root trees. With this PR, concretize output now looks like this: ```console > spack spec /v5fn6xo /wd2p2v7 [+] [email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=darwin-sequoia-m1 [+] ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~guile build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+cpanm+opcode+open+shared+threads build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~debug~pic+shared build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools libs=shared,static arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools patches=bbf97f1 arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+compat+new_strategies+opt+pic+shared build_system=autotools arch=darwin-sequoia-m1 [+] ^gnuconfig@2022-09-17%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] [email protected]%[email protected] build_system=python_pip arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 [-] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 ... ``` With no input spec displayed -- just the concretization output shown as one consolidated tree and multiple roots. - [x] remove "Input Spec" section and "Concretized" header from `spack spec` output - [x] print concretized specs as one BFS tree instead of multiple Signed-off-by: Todd Gamblin <[email protected]>

`spack spec` output has looked like this for a while: ```console > spack spec /v5fn6xo /wd2p2v7 Input spec -------------------------------- - /v5fn6xo Concretized -------------------------------- [+] [email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=darwin-sequoia-m1 [+] ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=darwin-sequoia-m1 ... Input spec -------------------------------- - /wd2p2v7 Concretized -------------------------------- [+] [email protected]%[email protected] build_system=python_pip arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 ``` But the input spec is right there on the CLI, and it doesn't add anything to the output. Also, since #44843, specs concretized in the CLI line can be unified, so it makes sense to display them as we did in #44489 -- as one multi-root tree instead of as multiple single-root trees. With this PR, concretize output now looks like this: ```console > spack spec /v5fn6xo /wd2p2v7 [+] [email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=darwin-sequoia-m1 [+] ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~guile build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+cpanm+opcode+open+shared+threads build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~debug~pic+shared build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools libs=shared,static arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools patches=bbf97f1 arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+compat+new_strategies+opt+pic+shared build_system=autotools arch=darwin-sequoia-m1 [+] ^gnuconfig@2022-09-17%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] [email protected]%[email protected] build_system=python_pip arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 [-] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 ... ``` With no input spec displayed -- just the concretization output shown as one consolidated tree and multiple roots. - [x] remove "Input Spec" section and "Concretized" header from `spack spec` output - [x] print concretized specs as one BFS tree instead of multiple --------- Signed-off-by: Todd Gamblin <[email protected]> Co-authored-by: Harmen Stoppels <[email protected]>

`spack spec` output has looked like this for a while: ```console > spack spec /v5fn6xo /wd2p2v7 Input spec -------------------------------- - /v5fn6xo Concretized -------------------------------- [+] [email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=darwin-sequoia-m1 [+] ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=darwin-sequoia-m1 ... Input spec -------------------------------- - /wd2p2v7 Concretized -------------------------------- [+] [email protected]%[email protected] build_system=python_pip arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 ``` But the input spec is right there on the CLI, and it doesn't add anything to the output. Also, since spack#44843, specs concretized in the CLI line can be unified, so it makes sense to display them as we did in spack#44489 -- as one multi-root tree instead of as multiple single-root trees. With this PR, concretize output now looks like this: ```console > spack spec /v5fn6xo /wd2p2v7 [+] [email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=darwin-sequoia-m1 [+] ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~guile build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+cpanm+opcode+open+shared+threads build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~debug~pic+shared build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools libs=shared,static arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools patches=bbf97f1 arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+compat+new_strategies+opt+pic+shared build_system=autotools arch=darwin-sequoia-m1 [+] ^gnuconfig@2022-09-17%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] [email protected]%[email protected] build_system=python_pip arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 [-] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 ... ``` With no input spec displayed -- just the concretization output shown as one consolidated tree and multiple roots. - [x] remove "Input Spec" section and "Concretized" header from `spack spec` output - [x] print concretized specs as one BFS tree instead of multiple --------- Signed-off-by: Todd Gamblin <[email protected]> Co-authored-by: Harmen Stoppels <[email protected]>

spackbot-app bot added commands core PR affects Spack core functionality environments labels Jun 2, 2024

tgamblin requested review from alalazo, becker33 and haampie and removed request for alalazo, becker33 and haampie June 2, 2024 05:37

tgamblin mentioned this pull request Jun 2, 2024

Truncate spack concretize output in environments #44386

Closed

tgamblin requested a review from scheibelp June 2, 2024 14:32

haampie previously approved these changes Jun 3, 2024

View reviewed changes

alalazo assigned haampie Jun 3, 2024

alecbcs previously approved these changes Jun 3, 2024

View reviewed changes

tgamblin assigned scheibelp Jun 3, 2024

tgamblin dismissed stale reviews from alecbcs and haampie via d9d7d9d June 3, 2024 16:53

tgamblin force-pushed the better-env-concretize-view branch from f14fc4f to d9d7d9d Compare June 3, 2024 16:53

alecbcs approved these changes Jun 3, 2024

View reviewed changes

scheibelp approved these changes Jun 3, 2024

View reviewed changes

scheibelp merged commit 85f1344 into develop Jun 3, 2024

scheibelp deleted the better-env-concretize-view branch June 3, 2024 20:29

This was referenced Jun 13, 2024

[WIP] Spec printing: fold dependencies into parents #44706

Open

spack find -c: search all concretized-but-not-installed specs #44713

Merged

greenc-FNAL mentioned this pull request Jul 30, 2024

Update from upstream 2024-07-22 and re-apply scopes FNALssi/spack#408

Merged

tgamblin mentioned this pull request Nov 13, 2024

spack spec: simplify and unify output #47574

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Consolidate concretization output for environments#44489

Consolidate concretization output for environments#44489
scheibelp merged 1 commit intodevelopfrom
better-env-concretize-view

tgamblin commented Jun 2, 2024 •

edited

Loading

Uh oh!

haampie commented Jun 2, 2024

Uh oh!

tgamblin commented Jun 2, 2024

Uh oh!

tgamblin commented Jun 2, 2024

Uh oh!

tgamblin commented Jun 2, 2024

Uh oh!

alecbcs left a comment

Uh oh!

tgamblin commented Jun 3, 2024

Uh oh!

scheibelp left a comment

Uh oh!

scheibelp Jun 3, 2024

Uh oh!

Uh oh!

Uh oh!

scheibelp commented Jun 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

tgamblin commented Jun 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haampie commented Jun 2, 2024

Uh oh!

tgamblin commented Jun 2, 2024

Uh oh!

tgamblin commented Jun 2, 2024

Uh oh!

tgamblin commented Jun 2, 2024

Uh oh!

alecbcs left a comment

Choose a reason for hiding this comment

Uh oh!

tgamblin commented Jun 3, 2024

Uh oh!

scheibelp left a comment

Choose a reason for hiding this comment

Uh oh!

scheibelp Jun 3, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

scheibelp commented Jun 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tgamblin commented Jun 2, 2024 •

edited

Loading