Improve and unify developer shell interfaces by psakievich · Pull Request #47492 · spack/spack

psakievich · 2024-11-08T05:02:36Z

Updated Scope:

This PR adds functionality to spack dev-build and spack build-env to increase compatibility and add a longstanding external feature build-env-dive to spack core.

build-env-dive [spec] is a shell function that

Launches a subshell with the build-environment for the spec active
Navigates to the build directory of the spec
Changes the prompt to indicate to the user they are in the build-env subshell

The features added are:

spack build-env
- add --dive for diving into the build-env
- change the prompt as part of the dive process (bourne shells and csh only for now)
- add spack build-env --status for people using shells that don't have a changed prompt (aka fish, powershell)
- add --cd for navigation with subshell commands, (reproduces spack cd as a precursor to the dive/cmd)
spack dev-build
- allow it to work with spack environments (requires specs to be marked as develop)
- remove constraint for concrete spec
- add --prompt and unify code for the --drop-in with spack build-env --dive
aliases
- create a spack dev-dive alias to replace build-env-dive completely

Example developer workflow

$ spack env activate --temp
$ spack add zlib-ng
$ spack stage -p $SPACK_ENV zlib-ng
$ spack develop zlib-ng
$ spack install -u autoreconf zlib-ng
# fails
$ spack dev-dive zlib-ng
# prompt changed, let's check the location
zlib-ng-build-env $ pwd
/private/var/folders/ln/1_3kxbwd35s_ylsjlm3zmqmc00307v/T/spack-ne8_m488/zlib-ng
# confirm build-env in case we forget where we are in subshell hierarchy
zlib-ng-build-env $ spack build-env --status
==> In build env zlib-ng-wrsaadvkbg7rjj7kfjw5rhdrrfdswcmm
# fix code
zlib-ng-build-env $ make -j6
# builds  now
zlib-ng-build-env $ exit
$ spack build-env --status
==> build environment not detected
$ spack install

Original message:

There is really no reason why spack develop and dev-build shouldn't play along with each other. dev-build just assumes it can't work in an environment. This PR makes it so dev-build will look for matching develop specs in a concrete environment.

We could optionally make it do a spack add && spack develop if they don't exist but I think that is too far reaching.

We have constantly been re-writing a build-env-dive shell function that

loads the build environment for a develop spec in a subshell
spack cd -b to the build directory

Item 1 is already in place with spack dev-build so rather than write a whole new funciton, I think we should make them compatible.

That said I think the dev-build interface is a little clunky for how build-env-dive has been used and I think it needs to be improved, but it might just be doable through aliases.

Things I want to add still:

prompt to indicate you are in the build environment when using --build-dive
navigation option, dive then go to build or src directory

lib/spack/spack/cmd/dev_build.py

haampie · 2024-11-08T07:20:51Z

spack dev-build --help and docs should be updated.

As for a shell with the build environment, what's wrong with spack build-env? That it's lacking --drop-in and doesn't change the working dir? Adding that flag sounds useful and trivial.

I've never used spack dev-build --drop-in, looks very weird to me that it happens past .install() and won't happen on install failure. Do you agree?

To me it would make more sense if:

spack build-env --drop-in would do what it says without install
spack install/dev-build --drop-in would drop in on install failure

lib/spack/spack/cmd/dev_build.py

psakievich · 2024-11-08T13:02:33Z

spack dev-build --help and docs should be updated.

As for a shell with the build environment, what's wrong with spack build-env? That it's lacking --drop-in and doesn't change the working dir? Adding that flag sounds useful and trivial.

I've never used spack dev-build --drop-in, looks very weird to me that it happens past .install() and won't happen on install failure. Do you agree?

To me it would make more sense if:

spack build-env --drop-in would do what it says without install

spack install/dev-build --drop-in would drop in on install failure

Yes I agree. Adding a --drop-in, I would prefer --dive, for build-env makes a lot of sense. The drop-in still happens even if the install fails for this command. The PackageInstaller.install() can fail but the command continues to the drop-in section. Overall I find this pretty unintuitive.

I started working on this yesterday because some SNL staff mentioned they use this, but it's not great, and they haven't used spack develop. 😭 I'm hoping this will also be a bridge to convert people over to spack develop.

I would propose the following for dev-build:

Stage 1. Make it compatible with environments and spack develop (this PR).
Stage 2. Advertise to the user base and determine how much use dev-build actually gets
Stage 3. Deprecate and then major refactor or remove after a release cycle.

Overall I think the dev-build command may have a place, but when I'm developing I don't want to run a command with a ton of options since development CLI commands tend to be repetitive and frequent.

I don't like that it is essentially wrapping spack install for non-environments and I find the current interface clunky. However for quick and dirty develop it seems like it could be useful if it is refactored and updated.

psakievich · 2024-11-08T17:02:13Z

I think I might give up on the prompt for this PR. I think adding that plus a --drop-in/--dive option to build-env might be suitable for a follow on PR. I think I'll put the navigation option in either with or after that. Really I'd like to give more thought to cleaning up the relationship between these two. I spent a couple hours today trying to get the prompt stuff to work, but I don't think it is necessary for this PR.

I personally would classify this as a bug fix since there was no real reason for this command to not work with environments in the past.

@haampie I've updated the help section. This command does not appear in the docs 😬 . Not sure I want to add it right now given I consider it a candiadate to be reworked or deleted.

I tried consolidating but I am not convinced it will work correctly

lib/spack/spack/test/cmd/build_env.py

johnwparent · 2024-12-16T23:00:08Z

Left a review comment, still need to take a closer pass for Windows concerns, but this should at least unblock the CI.

Co-authored-by: John W. Parent <[email protected]>

lib/spack/spack/test/cmd/build_env.py

johnwparent

Partial review, still need to look closer at the dev_build and location commands.

Trying this on Windows results in an error, tracing that down now.

lib/spack/spack/test/cmd/build_env.py

johnwparent · 2025-03-06T19:44:00Z

lib/spack/spack/util/shell_detection.py

+                    pass
+            except subprocess.CalledProcessError:
+                pass
+            raise SpackError("Unknown shell type being used on Windows")


Do you want me to add support for CMD?

Hmm what do you think is best? I could just punt on windows for this PR and you could do a follow on? Otherwise likely yes? I never use windows so I don't really know what is best here.

I can also just commit directly to this PR if you're cool with that, shouldn't take long

Yeah that is fine if the changes are small. If I run into too many more windows issues then I think we should break it into two PR's since I can't really test windows behavior outside of CI at the moment.

Agreed, that's reasonable.

johnwparent · 2025-03-06T19:46:18Z

lib/spack/spack/util/shell_detection.py

+        else:
+            try:
+                output = subprocess.check_output(
+                    'powershell -Command "echo $PSVersionTable"', universal_newlines=True


This will always pass (fine if we don't want to support CMD), as the powershell executable is always in the path unless a user removed it explicitly.

I think something like:

pproc_name = psutil.Process(os.getppid()).name() is_power_shell = bool(re.fullmatch('pwsh|pwsh.exe|powershell|powershell.exe', pproc_name))

With a better regex would be best.

johnwparent · 2025-03-06T19:47:17Z

lib/spack/spack/prompt.py

+    mods = EnvironmentModifications()
+
+    if shell == "fish" or shell == "pwsh" or shell == "bat":
+        # requires a function and can't be set with os.environ


bat can be set from env variables

Co-authored-by: John W. Parent <[email protected]>

Signed-off-by: psakiev <[email protected]>

Signed-off-by: Phil Sakievich <[email protected]>

Signed-off-by: psakiev <[email protected]>

Make dev-build compatible with spack develop

259a1d9

psakievich added the snl-core-team Issue for SNL Spack developers label Nov 8, 2024

spackbot-app bot added commands core PR affects Spack core functionality labels Nov 8, 2024

haampie reviewed Nov 8, 2024

View reviewed changes

lib/spack/spack/cmd/dev_build.py Outdated Show resolved Hide resolved

tgamblin reviewed Nov 8, 2024

View reviewed changes

lib/spack/spack/cmd/dev_build.py Outdated Show resolved Hide resolved

psakievich added 2 commits November 8, 2024 06:32

Fix some things from review

33ae096

Update help

43604f6

psakievich marked this pull request as ready for review November 8, 2024 17:03

Add prompt capability to dev_build drop-in

9b3c200

spackbot-app bot added build-environment environments labels Nov 11, 2024

psakievich added 2 commits November 11, 2024 16:46

Merge remote-tracking branch 'origin' into psakiev/dev-build

8d5e71f

WIP unify

2acf90f

psakievich requested a review from RikkiButler20 November 12, 2024 17:27

psakievich added 2 commits November 12, 2024 13:16

Add build-env --dive

4216a06

Merge remote-tracking branch 'origin' into psakiev/dev-build

973f59a

spackbot-app bot added the utilities label Nov 12, 2024

psakievich marked this pull request as draft November 12, 2024 20:28

Add completion

4d49a65

spackbot-app bot added defaults shell-support labels Nov 12, 2024

psakievich added 4 commits November 12, 2024 16:09

Fixes

ab5c2d5

Compatibility

707a8da

Add navigation to dive feature

8eb4354

Style

d6b937d

psakievich changed the title ~~Make dev-build compatible with spack develop~~ Improve and unify spack development shell interfaces Nov 13, 2024

psakievich added 2 commits December 10, 2024 22:40

Revert prompt changes for environment activation

1d536ce

I tried consolidating but I am not convinced it will work correctly

Update imports

3f88278

psakievich force-pushed the psakiev/dev-build branch from ae257a4 to 3f88278 Compare December 11, 2024 05:55

psakievich requested a review from johnwparent December 16, 2024 19:12

johnwparent reviewed Dec 16, 2024

View reviewed changes

lib/spack/spack/test/cmd/build_env.py Outdated Show resolved Hide resolved

Update build_env.py

b68b581

Co-authored-by: John W. Parent <[email protected]>

psakievich commented Dec 17, 2024

View reviewed changes

lib/spack/spack/test/cmd/build_env.py Outdated Show resolved Hide resolved

Update lib/spack/spack/test/cmd/build_env.py

2cae2d0

johnwparent mentioned this pull request Feb 10, 2025

Windows: add env name to prompts #48819

Merged

johnwparent reviewed Mar 6, 2025

View reviewed changes

lib/spack/spack/test/cmd/build_env.py Outdated Show resolved Hide resolved

johnwparent reviewed Mar 6, 2025

View reviewed changes

psakievich and others added 6 commits June 2, 2025 08:46

Update lib/spack/spack/test/cmd/build_env.py

12620c8

Co-authored-by: John W. Parent <[email protected]>

Merge branch 'develop' into psakiev/dev-build

8510652

Style

5689022

Fix unit test

7ca5162

Signed-off-by: psakiev <[email protected]>

Delete stale doc

36a039f

Signed-off-by: psakiev <[email protected]>

Style

bbcec77

Signed-off-by: psakiev <[email protected]>

psakievich added this to the v1.1.0 milestone Jul 25, 2025

psakievich added 3 commits July 25, 2025 11:43

License

fcca6d9

Signed-off-by: psakiev <[email protected]>

Merge branch 'develop' into psakiev/dev-build

a1368b4

Signed-off-by: Phil Sakievich <[email protected]>

Delete prompt mods for now

f56c825

Signed-off-by: psakiev <[email protected]>

psakievich force-pushed the psakiev/dev-build branch from ca72e17 to f56c825 Compare August 12, 2025 22:45

Fix failing test

c3f164b

Signed-off-by: psakiev <[email protected]>

psakievich marked this pull request as draft August 13, 2025 11:44

psakievich added 2 commits August 13, 2025 14:14

Fix more tests

2482f64

Signed-off-by: psakiev <[email protected]>

Refactor more tests

f8b9dfb

Signed-off-by: psakiev <[email protected]>

psakievich mentioned this pull request Aug 26, 2025

spack develop: fast automatic reconcretization #51140

Merged

2 tasks

tgamblin modified the milestones: v1.1.0, v1.2.0 Nov 3, 2025

Conversation

psakievich commented Nov 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Updated Scope:

Example developer workflow

Original message:

Uh oh!

Uh oh!

haampie commented Nov 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

psakievich commented Nov 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

psakievich commented Nov 8, 2024

Uh oh!

Uh oh!

johnwparent commented Dec 16, 2024

Uh oh!

Uh oh!

Uh oh!

johnwparent left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

johnwparent Mar 6, 2025

Choose a reason for hiding this comment

Uh oh!

psakievich Mar 6, 2025

Choose a reason for hiding this comment

Uh oh!

johnwparent Mar 6, 2025

Choose a reason for hiding this comment

Uh oh!

psakievich Mar 6, 2025

Choose a reason for hiding this comment

Uh oh!

johnwparent Mar 6, 2025

Choose a reason for hiding this comment

Uh oh!

johnwparent Mar 6, 2025

Choose a reason for hiding this comment

Uh oh!

johnwparent Mar 6, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

psakievich commented Nov 8, 2024 •

edited

Loading

haampie commented Nov 8, 2024 •

edited

Loading

psakievich commented Nov 8, 2024 •

edited

Loading