parse_specs: unify specs based on concretizer:unify by becker33 · Pull Request #44843 · spack/spack

becker33 · 2024-06-24T18:10:56Z

Currently, the concretizer:unify: config option only affects environments.

With this PR, it now affects any group of specs given to a command using the parse_specs(*, concretize=True) interface.

implementation in parse_specs
tests
ensure all commands that accept multiple specs and concretize use parse_specs interface

tgamblin

One minor request and one for a future PR. In the future, concretize_separately, concretize_together, and concretize_together_when_possible should be one method in spack.concretize. This PR gets us closer to that but not quite there, but I think that is fine b/c it's a localized change for parse_specs, and the bigger refactor would likely touch many more parts of the code.

lib/spack/spack/cmd/__init__.py

Co-authored-by: Todd Gamblin <[email protected]>

…ment

alalazo · 2024-11-02T09:06:20Z

lib/spack/spack/concretize.py

    return [s.copy() for s in result.specs]


+def concretize_together(*spec_list, **kwargs):


Just wanted to note that in this file we added **kwargs and no type-hinting to new functions.

`spack spec` output has looked like this for a while: ```console > spack spec /v5fn6xo /wd2p2v7 Input spec -------------------------------- - /v5fn6xo Concretized -------------------------------- [+] [email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=darwin-sequoia-m1 [+] ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=darwin-sequoia-m1 ... Input spec -------------------------------- - /wd2p2v7 Concretized -------------------------------- [+] [email protected]%[email protected] build_system=python_pip arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 ``` But the input spec is right there on the CLI, and it doesn't add anything to the output. Also, since #44843, specs concretized in the CLI line can be unified, so it makes sense to display them as we did in #44489 -- as one multi-root tree instead of as multiple single-root trees. With this PR, concretize output now looks like this: ```console > spack spec /v5fn6xo /wd2p2v7 [+] [email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=darwin-sequoia-m1 [+] ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~guile build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+cpanm+opcode+open+shared+threads build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~debug~pic+shared build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools libs=shared,static arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools patches=bbf97f1 arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+compat+new_strategies+opt+pic+shared build_system=autotools arch=darwin-sequoia-m1 [+] ^gnuconfig@2022-09-17%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] [email protected]%[email protected] build_system=python_pip arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 [-] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 ``` With no input spec displayed -- just the concretization output shown as one consolidated tree and multiple roots. - [x] remove "Input Spec" section and "Concretized" header from `spack spec` output - [x] print concretized specs as one BFS tree instead of multiple Signed-off-by: Todd Gamblin <[email protected]>

`spack spec` output has looked like this for a while: ```console > spack spec /v5fn6xo /wd2p2v7 Input spec -------------------------------- - /v5fn6xo Concretized -------------------------------- [+] [email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=darwin-sequoia-m1 [+] ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=darwin-sequoia-m1 ... Input spec -------------------------------- - /wd2p2v7 Concretized -------------------------------- [+] [email protected]%[email protected] build_system=python_pip arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 ``` But the input spec is right there on the CLI, and it doesn't add anything to the output. Also, since #44843, specs concretized in the CLI line can be unified, so it makes sense to display them as we did in #44489 -- as one multi-root tree instead of as multiple single-root trees. With this PR, concretize output now looks like this: ```console > spack spec /v5fn6xo /wd2p2v7 [+] [email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=darwin-sequoia-m1 [+] ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~guile build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+cpanm+opcode+open+shared+threads build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~debug~pic+shared build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools libs=shared,static arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools patches=bbf97f1 arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+compat+new_strategies+opt+pic+shared build_system=autotools arch=darwin-sequoia-m1 [+] ^gnuconfig@2022-09-17%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] [email protected]%[email protected] build_system=python_pip arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 [-] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 ... ``` With no input spec displayed -- just the concretization output shown as one consolidated tree and multiple roots. - [x] remove "Input Spec" section and "Concretized" header from `spack spec` output - [x] print concretized specs as one BFS tree instead of multiple Signed-off-by: Todd Gamblin <[email protected]>

`spack spec` output has looked like this for a while: ```console > spack spec /v5fn6xo /wd2p2v7 Input spec -------------------------------- - /v5fn6xo Concretized -------------------------------- [+] [email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=darwin-sequoia-m1 [+] ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=darwin-sequoia-m1 ... Input spec -------------------------------- - /wd2p2v7 Concretized -------------------------------- [+] [email protected]%[email protected] build_system=python_pip arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 ``` But the input spec is right there on the CLI, and it doesn't add anything to the output. Also, since #44843, specs concretized in the CLI line can be unified, so it makes sense to display them as we did in #44489 -- as one multi-root tree instead of as multiple single-root trees. With this PR, concretize output now looks like this: ```console > spack spec /v5fn6xo /wd2p2v7 [+] [email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=darwin-sequoia-m1 [+] ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~guile build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+cpanm+opcode+open+shared+threads build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~debug~pic+shared build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools libs=shared,static arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools patches=bbf97f1 arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+compat+new_strategies+opt+pic+shared build_system=autotools arch=darwin-sequoia-m1 [+] ^gnuconfig@2022-09-17%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] [email protected]%[email protected] build_system=python_pip arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 [-] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 ... ``` With no input spec displayed -- just the concretization output shown as one consolidated tree and multiple roots. - [x] remove "Input Spec" section and "Concretized" header from `spack spec` output - [x] print concretized specs as one BFS tree instead of multiple --------- Signed-off-by: Todd Gamblin <[email protected]> Co-authored-by: Harmen Stoppels <[email protected]>

alalazo · 2024-11-18T10:13:13Z

lib/spack/spack/environment/environment.py

-        msg = "Starting concretization"
-        if sys.platform not in ("darwin", "win32") and num_procs > 1:
-            msg += f" pool with {num_procs} processes"
-        tty.msg(msg)


Was it intentional to remove this print ☝️ ? It was fairly useful in debugging pipeliines.

It was lost in #44843

It was lost in spack#44843

It was lost in #44843

`spack spec` output has looked like this for a while: ```console > spack spec /v5fn6xo /wd2p2v7 Input spec -------------------------------- - /v5fn6xo Concretized -------------------------------- [+] [email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=darwin-sequoia-m1 [+] ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=darwin-sequoia-m1 ... Input spec -------------------------------- - /wd2p2v7 Concretized -------------------------------- [+] [email protected]%[email protected] build_system=python_pip arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 ``` But the input spec is right there on the CLI, and it doesn't add anything to the output. Also, since spack#44843, specs concretized in the CLI line can be unified, so it makes sense to display them as we did in spack#44489 -- as one multi-root tree instead of as multiple single-root trees. With this PR, concretize output now looks like this: ```console > spack spec /v5fn6xo /wd2p2v7 [+] [email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=darwin-sequoia-m1 [+] ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~guile build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+cpanm+opcode+open+shared+threads build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~debug~pic+shared build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools libs=shared,static arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools patches=bbf97f1 arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+compat+new_strategies+opt+pic+shared build_system=autotools arch=darwin-sequoia-m1 [+] ^gnuconfig@2022-09-17%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] [email protected]%[email protected] build_system=python_pip arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 [-] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 ... ``` With no input spec displayed -- just the concretization output shown as one consolidated tree and multiple roots. - [x] remove "Input Spec" section and "Concretized" header from `spack spec` output - [x] print concretized specs as one BFS tree instead of multiple --------- Signed-off-by: Todd Gamblin <[email protected]> Co-authored-by: Harmen Stoppels <[email protected]>

It was lost in spack#44843

`spack spec` output has looked like this for a while: ```console > spack spec /v5fn6xo /wd2p2v7 Input spec -------------------------------- - /v5fn6xo Concretized -------------------------------- [+] [email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=darwin-sequoia-m1 [+] ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=darwin-sequoia-m1 ... Input spec -------------------------------- - /wd2p2v7 Concretized -------------------------------- [+] [email protected]%[email protected] build_system=python_pip arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 ``` But the input spec is right there on the CLI, and it doesn't add anything to the output. Also, since spack#44843, specs concretized in the CLI line can be unified, so it makes sense to display them as we did in spack#44489 -- as one multi-root tree instead of as multiple single-root trees. With this PR, concretize output now looks like this: ```console > spack spec /v5fn6xo /wd2p2v7 [+] [email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=darwin-sequoia-m1 [+] ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~guile build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+cpanm+opcode+open+shared+threads build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~debug~pic+shared build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools libs=shared,static arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools patches=bbf97f1 arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=autotools arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected]+compat+new_strategies+opt+pic+shared build_system=autotools arch=darwin-sequoia-m1 [+] ^gnuconfig@2022-09-17%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] [email protected]%[email protected] build_system=python_pip arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 [+] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 [-] ^[email protected]%[email protected] build_system=generic arch=darwin-sequoia-m1 ... ``` With no input spec displayed -- just the concretization output shown as one consolidated tree and multiple roots. - [x] remove "Input Spec" section and "Concretized" header from `spack spec` output - [x] print concretized specs as one BFS tree instead of multiple --------- Signed-off-by: Todd Gamblin <[email protected]> Co-authored-by: Harmen Stoppels <[email protected]>

It was lost in spack#44843

spackbot-app bot added commands core PR affects Spack core functionality tests General test capability(ies) labels Oct 7, 2024

becker33 added this to the v0.23 milestone Oct 28, 2024

becker33 force-pushed the features/unify-for-non-environment-concretization branch from 318fa2d to 45af646 Compare October 31, 2024 00:05

spackbot-app bot added the environments label Oct 31, 2024

tgamblin requested changes Nov 1, 2024

View reviewed changes

lib/spack/spack/cmd/__init__.py Outdated Show resolved Hide resolved

becker33 and others added 8 commits November 1, 2024 15:36

parse_specs: unify specs based on concretizer:unify

069708f

bugfix non-concretizing parse_specs

99939ab

update test for new mock_package names

31672fd

commands: get matching specs from env while respecting unification

315e29f

style

3a31f59

remove vestigial import

00efb7b

Update lib/spack/spack/cmd/__init__.py

794b9dd

Co-authored-by: Todd Gamblin <[email protected]>

include entire environment context when concretizing matching environ…

842b5bf

…ment

becker33 force-pushed the features/unify-for-non-environment-concretization branch from c025314 to 842b5bf Compare November 1, 2024 22:38

becker33 added 2 commits November 1, 2024 15:45

missing rebase resolution

682841b

variable naming error

1fd7d5c

tgamblin enabled auto-merge (squash) November 1, 2024 23:37

tgamblin approved these changes Nov 1, 2024

View reviewed changes

tgamblin merged commit e42a4a8 into develop Nov 1, 2024

tgamblin deleted the features/unify-for-non-environment-concretization branch November 1, 2024 23:49

alalazo reviewed Nov 2, 2024

View reviewed changes

becker33 mentioned this pull request Nov 12, 2024

parse_specs: special case for concretizing lookups quickly #47556

Merged

tgamblin mentioned this pull request Nov 13, 2024

spack spec: simplify and unify output #47574

Merged

2 tasks

alalazo reviewed Nov 18, 2024

View reviewed changes

alalazo added a commit that referenced this pull request Nov 18, 2024

Restore message when concretizing in parallel

50be415

It was lost in #44843

alalazo added a commit that referenced this pull request Nov 18, 2024

Restore message when concretizing in parallel

2c649eb

It was lost in #44843

alalazo added a commit to alalazo/spack that referenced this pull request Nov 18, 2024

Restore message when concretizing in parallel

67cf181

It was lost in spack#44843

alalazo mentioned this pull request Nov 18, 2024

Restore message when concretizing in parallel #47663

Merged

alalazo added a commit that referenced this pull request Nov 18, 2024

Restore message when concretizing in parallel

fb89def

It was lost in #44843

tldahlgren pushed a commit that referenced this pull request Nov 19, 2024

Restore message when concretizing in parallel (#47663)

f182032

It was lost in #44843

alalazo added a commit that referenced this pull request Nov 19, 2024

Restore message when concretizing in parallel

3121612

It was lost in #44843

This was referenced Nov 25, 2024

maintenance/merge from upstream 2024 11 25 FNALssi/spack#467

Merged

maintenance/v0.23.0 fermi/fix duplication style FNALssi/spack#468

Closed

fryeguy52 pushed a commit to fryeguy52/spack that referenced this pull request Dec 17, 2024

Restore message when concretizing in parallel (spack#47663)

21997de

It was lost in spack#44843

kshea21 pushed a commit to kshea21/spack that referenced this pull request Dec 26, 2024

Restore message when concretizing in parallel (spack#47663)

a09d376

It was lost in spack#44843

teaguesterling pushed a commit to teaguesterling/spack that referenced this pull request Feb 5, 2025

Restore message when concretizing in parallel (spack#47663)

c7325bf

It was lost in spack#44843

nkoukpaizan pushed a commit to nkoukpaizan/spack that referenced this pull request Apr 4, 2025

Restore message when concretizing in parallel (spack#47663)

418d3a1

It was lost in spack#44843

vjranagit pushed a commit to vjranagit/spack that referenced this pull request Jan 18, 2026

Restore message when concretizing in parallel

e5d7942

It was lost in spack#44843

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

parse_specs: unify specs based on concretizer:unify#44843

parse_specs: unify specs based on concretizer:unify#44843
tgamblin merged 10 commits intodevelopfrom
features/unify-for-non-environment-concretization

becker33 commented Jun 24, 2024 •

edited by tgamblin

Loading

Uh oh!

tgamblin left a comment

Uh oh!

Uh oh!

alalazo Nov 2, 2024

Uh oh!

alalazo Nov 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		return [s.copy() for s in result.specs]


		def concretize_together(spec_list, *kwargs):

Conversation

becker33 commented Jun 24, 2024 • edited by tgamblin Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tgamblin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alalazo Nov 2, 2024

Choose a reason for hiding this comment

Uh oh!

alalazo Nov 18, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

becker33 commented Jun 24, 2024 •

edited by tgamblin

Loading