Include concrete environments with `include_concrete` by RikkiButler20 · Pull Request #33768 · spack/spack

RikkiButler20 · 2022-11-08T21:57:33Z

Resolves #33313

Add the ability to include any number of (potentially nested) concrete environments, e.g.:

   spack:
     specs: []
     concretizer:
         unify: true
     include_concrete:
     - /path/to/environment1
     - /path/to/environment2

or, from the CLI:

   $ spack env create myenv
   $ spack -e myenv add python
   $ spack -e myenv concretize
   $ spack env create --include-concrete myenv included_env

The contents of included concrete environments' spack.lock files are
included in the environment's lock file at creation time. Any changes
to included concrete environments are only reflected after the environment
is re-concretized from the re-concretized included environments.

RikkiButler20 · 2022-11-09T21:46:26Z

@spackbot fix style

spackbot-app · 2022-11-09T21:46:28Z

Let me see if I can fix that for you!

spackbot-app · 2022-11-09T21:48:21Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: isort, mypy, black, flake8
==> Modified files
  lib/spack/spack/cmd/env.py
==> Running isort checks
  isort checks were clean
==> Running mypy checks
Success: no issues found in 569 source files
  mypy checks were clean
==> Running black checks
reformatted lib/spack/spack/cmd/env.py
All done! ✨ 🍰 ✨
1 file reformatted.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> spack style checks were clean

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

RikkiButler20 · 2022-11-15T20:00:42Z

@spackbot fix style

spackbot-app · 2022-11-15T20:00:44Z

Let me see if I can fix that for you!

spackbot-app · 2022-11-15T20:03:16Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: isort, mypy, black, flake8
==> Modified files
  lib/spack/spack/cmd/env.py
  lib/spack/spack/environment/environment.py
==> Running isort checks
  isort checks were clean
==> Running mypy checks
lib/spack/llnl/util/lang.py:1076: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
lib/spack/llnl/util/lock.py:84: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
lib/spack/spack/bootstrap.py:89: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
Success: no issues found in 569 source files
  mypy checks were clean
==> Running black checks
reformatted lib/spack/spack/cmd/env.py
All done! ✨ 🍰 ✨
1 file reformatted, 1 file left unchanged.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> spack style checks were clean

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

RikkiButler20 · 2022-11-16T20:27:56Z

@spackbot fix style

spackbot-app · 2022-11-16T20:27:58Z

Let me see if I can fix that for you!

spackbot-app · 2022-11-16T20:29:45Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: isort, mypy, black, flake8
==> Modified files
  lib/spack/spack/cmd/env.py
  lib/spack/spack/environment/environment.py
==> Running isort checks
  isort checks were clean
==> Running mypy checks
lib/spack/llnl/util/lang.py:1076: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
lib/spack/llnl/util/lock.py:84: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
lib/spack/spack/bootstrap.py:89: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
Success: no issues found in 569 source files
  mypy checks were clean
==> Running black checks
reformatted lib/spack/spack/cmd/env.py
All done! ✨ 🍰 ✨
1 file reformatted, 1 file left unchanged.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> spack style checks were clean

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

RikkiButler20 · 2022-12-13T18:50:05Z

@spackbot fix style

spackbot-app · 2022-12-13T18:50:07Z

Let me see if I can fix that for you!

spackbot-app · 2022-12-13T18:51:04Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: isort, mypy, black, flake8
==> Modified files
  lib/spack/spack/cmd/env.py
  lib/spack/spack/environment/environment.py
  lib/spack/spack/schema/env.py
==> Running isort checks
  isort checks were clean
==> Running mypy checks
lib/spack/llnl/util/lang.py:1076: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
lib/spack/llnl/util/lock.py:84: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
lib/spack/spack/bootstrap.py:89: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
Success: no issues found in 569 source files
  mypy checks were clean
==> Running black checks
reformatted lib/spack/spack/cmd/env.py
reformatted lib/spack/spack/environment/environment.py
All done! ✨ 🍰 ✨
2 files reformatted, 1 file left unchanged.
  black checks were clean
==> Running flake8 checks
lib/spack/spack/environment/environment.py:934: [F841] local variable 'new_dict_roots' is assigned to but never used
lib/spack/spack/environment/environment.py:935: [F841] local variable 'lockfile_meta' is assigned to but never used
lib/spack/spack/environment/environment.py:945: [F841] local variable 'old_env' is assigned to but never used
lib/spack/spack/environment/environment.py:945: [F821] undefined name 'Envinronment'
  flake8 found errors

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

RikkiButler20 · 2022-12-15T17:38:17Z

@spackbot fix style

spackbot-app · 2022-12-15T17:38:19Z

Let me see if I can fix that for you!

spackbot-app · 2022-12-15T17:39:16Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: isort, mypy, black, flake8
==> Modified files
  lib/spack/spack/cmd/env.py
  lib/spack/spack/environment/environment.py
  lib/spack/spack/schema/env.py
==> Running isort checks
  isort checks were clean
==> Running mypy checks
lib/spack/llnl/util/lang.py:1064: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
lib/spack/spack/util/timer.py:68: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
lib/spack/spack/build_environment.py:291: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
lib/spack/spack/bootstrap/_common.py:52: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
Success: no issues found in 571 source files
  mypy checks were clean
==> Running black checks
reformatted lib/spack/spack/cmd/env.py
reformatted lib/spack/spack/environment/environment.py
All done! ✨ 🍰 ✨
2 files reformatted, 1 file left unchanged.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> spack style checks were clean

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

RikkiButler20 · 2023-01-13T18:40:59Z

@spackbot fix style

spackbot-app · 2023-01-13T18:41:01Z

Let me see if I can fix that for you!

spackbot-app · 2023-01-13T18:42:15Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: isort, black, flake8, mypy
==> Modified files
  lib/spack/spack/cmd/env.py
  lib/spack/spack/environment/environment.py
  lib/spack/spack/schema/env.py
==> Running isort checks
  isort checks were clean
==> Running black checks
All done! ✨ 🍰 ✨
3 files left unchanged.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> Running mypy checks
Success: no issues found in 571 source files
  mypy checks were clean
==> spack style checks were clean

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I wasn't able to make any further changes, but please see the message above for remaining issues you can fix locally!

spackbot-app · 2024-03-25T21:05:00Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: isort, black, flake8, mypy
==> Modified files
  lib/spack/spack/cmd/env.py
  lib/spack/spack/cmd/find.py
  lib/spack/spack/environment/__init__.py
  lib/spack/spack/environment/environment.py
  lib/spack/spack/schema/env.py
  lib/spack/spack/test/cmd/env.py
  lib/spack/spack/test/cmd/find.py
==> Running isort checks
  isort checks were clean
==> Running black checks
reformatted lib/spack/spack/environment/environment.py
All done! ✨ 🍰 ✨
1 file reformatted, 6 files left unchanged.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> Running mypy checks
Success: no issues found in 619 source files
  mypy checks were clean
==> spack style checks were clean

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

RikkiButler20 · 2024-03-29T22:20:48Z

@spackbot fix style

spackbot-app · 2024-03-29T22:20:50Z

Let me see if I can fix that for you!

spackbot-app · 2024-03-29T22:22:14Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: isort, black, flake8, mypy
==> Modified files
  lib/spack/spack/cmd/env.py
  lib/spack/spack/cmd/find.py
  lib/spack/spack/environment/__init__.py
  lib/spack/spack/environment/environment.py
  lib/spack/spack/schema/env.py
  lib/spack/spack/test/cmd/env.py
  lib/spack/spack/test/cmd/find.py
==> Running isort checks
  isort checks were clean
==> Running black checks
reformatted lib/spack/spack/cmd/env.py
All done! ✨ 🍰 ✨
1 file reformatted, 6 files left unchanged.
  black checks were clean
==> Running flake8 checks
lib/spack/spack/test/cmd/env.py:1819: [F841] local variable 'lockfile_as_dict' is assigned to but never used
  flake8 found errors
==> Running mypy checks
Success: no issues found in 619 source files
  mypy checks were clean

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

tldahlgren · 2024-04-23T00:40:55Z

Interesting that it has been 6 hours and prechecks are still in Waiting for status to be reported with only readthedocs completed. Could it be conflicts related?

tldahlgren · 2024-04-23T00:41:22Z

@spackbot help

spackbot-app · 2024-04-23T00:41:23Z

You can interact with me in many ways!

@spackbot hello: say hello and get a friendly response back!
@spackbot help or @spackbot commands: see this message
@spackbot run pipeline or @spackbot re-run pipeline: to request a new run of the GitLab CI pipeline
@spackbot rebuild everything: to run a pipeline rebuilding all specs from source.
@spackbot fix style if you have write and would like me to run spack style --fix for you.
@spackbot maintainers or @spackbot request review: to look for and assign reviewers for the pull request.

I'll also help to label your pull request and assign reviewers!
If you need help or see there might be an issue with me, open an issue here

tldahlgren · 2024-04-23T00:42:12Z

TBD. Will closing and re-opening force pre-checks to complete despite conflicts?

lib/spack/spack/test/cmd/env.py

lib/spack/spack/environment/environment.py

tldahlgren · 2024-04-30T21:35:58Z

I think this still needs a test for the following:

$ spack env create test1
$ spack -e test1 add foo
$ spack -e test1 concretize
$ spack env create --include-concrete test1 test2
$ spack -e test2 add bar
$ spack -e test2 concretize
$ spack env create --include-concrete test2 test3
$ spack -e test3 concretize
# should include foo and bar

There is a unit test -- test_concretize_nested_included_concrete -- that goes further to not only demonstrate the expected effects when two environments include the same environment concretized before and after a spec change (respectively) but to also exercise a case where those two environments are included in a fourth before and after reconcretization.

Goals of the test are to demonstrate:

each environment including a concrete environment has the concretized specs at creation time;
changes to the included environment are not reflected in the including environment until both are re-concretized; and
numbers 1 and 2 apply for nested included environments.

tldahlgren · 2024-04-30T21:58:08Z

What is the synchronization policy between this environment and the included one (if e.g. the included environment changes independently)?

IMO test_concrete_nested_included_concrete illustrates the sequence as I understand it.

Creation of an environment that includes a concrete environment will have its concrete specs at creation time. The only way the environment will reflect changes in an updated and re-concretized included environment is if the environment itself is re-concretized. The same principle applies for nested included environments.

For example, if a includes b and b includes c, a can only reflect changes to c if b is re-concretized to reflect those changes before a is re-concretized.

At least that's how the code is currently implemented.

tldahlgren · 2024-05-06T17:03:14Z

@alalazo @becker33 Is it okay to resolve the remaining conversations so we can merge this PR?

tgamblin · 2024-05-06T17:06:43Z

planning to after I deconflict it

tldahlgren · 2024-05-06T17:08:47Z

planning to after I deconflict it

Okay. I was just starting to do that but won't if you've already started.

Add the ability to include any number of (potentially nested) concrete environments. The contents of included concrete environments' spack.lock files are included in the environment's lock file at creation time. Any changes to included concrete environments are only reflected after the environment is re-concretized from the re-concretized included environments. Co-authored-by: Kayla Butler <<[email protected]> Co-authored-by: Tamara Dahlgren <[email protected] m> Co-authored-by: Todd Gamblin <[email protected]>

tldahlgren · 2024-05-07T00:48:39Z

The CodeCov numbers currently reported in the checks do not look right at all.

Following Details -> View this Pull Request on CodeCov I see +.01% change (head 83.06%, patch 90.70%).

lib/spack/spack/environment/environment.py

alalazo · 2024-05-07T13:23:38Z

Is it okay to resolve the remaining conversations so we can merge this PR?

If we are in a hurry to merge a PR, in my opinion we should leave open discussions open and force merge. In that way point of discussions are easier to spot later. Otherwise the "Unresolved conversations" will just push us to close stuff only to the detriment of our workflows.

RikkiButler20 added environments views labels Nov 8, 2022

RikkiButler20 self-assigned this Nov 8, 2022

spackbot-app bot added commands core PR affects Spack core functionality labels Nov 8, 2022

RikkiButler20 marked this pull request as draft November 9, 2022 21:46

spackbot-app bot added architecture binary-packages build-environment build-systems compilers conflicts labels Jan 24, 2023

tldahlgren reviewed Apr 29, 2024

View reviewed changes

lib/spack/spack/test/cmd/env.py Outdated Show resolved Hide resolved

tldahlgren reviewed Apr 29, 2024

View reviewed changes

lib/spack/spack/environment/environment.py Outdated Show resolved Hide resolved

tldahlgren reviewed Apr 29, 2024

View reviewed changes

lib/spack/spack/environment/environment.py Outdated Show resolved Hide resolved

heatherkellyucl mentioned this pull request Apr 30, 2024

Building parts of Spack environments on different nodes: option 2 (no environment chaining) UCL-ARC/hpc-spack#56

Closed

tgamblin previously approved these changes May 6, 2024

View reviewed changes

tgamblin approved these changes May 7, 2024

View reviewed changes

alalazo requested changes May 7, 2024

View reviewed changes

lib/spack/spack/environment/environment.py Show resolved Hide resolved

tldahlgren mentioned this pull request May 7, 2024

Remove dead environment creation code #44065

Merged

alalazo mentioned this pull request May 7, 2024

Backports v0.22.0 #44057

Closed

7 tasks

haampie mentioned this pull request May 8, 2024

Backports v0.22.0 #44070

Merged

9 tasks

This was referenced May 14, 2024

Merge from fnal/v0.22.0-fermi FNALssi/spack#352

Merged

Merge from upstream/develop FNALssi/spack#353

Merged

becker33 mentioned this pull request Jun 5, 2024

Initial work on include environment. #24200

Closed

tgamblin mentioned this pull request Oct 10, 2024

Support conditional and optional includes #46792

Closed

9 tasks

Conversation

RikkiButler20 commented Nov 8, 2022 • edited by tgamblin Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RikkiButler20 commented Nov 9, 2022

Uh oh!

spackbot-app bot commented Nov 9, 2022

Uh oh!

spackbot-app bot commented Nov 9, 2022

Uh oh!

RikkiButler20 commented Nov 15, 2022

Uh oh!

spackbot-app bot commented Nov 15, 2022

Uh oh!

spackbot-app bot commented Nov 15, 2022

Uh oh!

RikkiButler20 commented Nov 16, 2022

Uh oh!

spackbot-app bot commented Nov 16, 2022

Uh oh!

spackbot-app bot commented Nov 16, 2022

Uh oh!

RikkiButler20 commented Dec 13, 2022

Uh oh!

spackbot-app bot commented Dec 13, 2022

Uh oh!

spackbot-app bot commented Dec 13, 2022

Uh oh!

RikkiButler20 commented Dec 15, 2022

Uh oh!

spackbot-app bot commented Dec 15, 2022

Uh oh!

spackbot-app bot commented Dec 15, 2022

Uh oh!

RikkiButler20 commented Jan 13, 2023

Uh oh!

spackbot-app bot commented Jan 13, 2023

Uh oh!

spackbot-app bot commented Jan 13, 2023

Uh oh!

spackbot-app bot commented Mar 25, 2024

Uh oh!

RikkiButler20 commented Mar 29, 2024

Uh oh!

spackbot-app bot commented Mar 29, 2024

Uh oh!

spackbot-app bot commented Mar 29, 2024

Uh oh!

tldahlgren commented Apr 23, 2024

Uh oh!

tldahlgren commented Apr 23, 2024

Uh oh!

spackbot-app bot commented Apr 23, 2024

Uh oh!

tldahlgren commented Apr 23, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tldahlgren commented Apr 30, 2024

Uh oh!

tldahlgren commented Apr 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tldahlgren commented May 6, 2024

Uh oh!

tgamblin commented May 6, 2024

Uh oh!

tldahlgren commented May 6, 2024

Uh oh!

tldahlgren commented May 7, 2024

Uh oh!

Uh oh!

alalazo commented May 7, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

RikkiButler20 commented Nov 8, 2022 •

edited by tgamblin

Loading

tldahlgren commented Apr 30, 2024 •

edited

Loading