cmd: Add new mark command by michaelkuhn · Pull Request #16662 · spack/spack

michaelkuhn · 2020-05-15T14:04:30Z

This adds a new mark command that can be used to mark packages as either explicitly or implicitly installed. Apart from fixing the package database after installing a dependency manually, it can be used to implement upgrade workflows as outlined in #13385.

The following commands demonstrate how the mark and gc commands can be used to only keep the current version of a package installed:

$ spack install pkgA
$ spack install pkgB
$ git pull # Imagine new versions for pkgA and/or pkgB are introduced
$ spack mark -i -a
$ spack install pkgA
$ spack install pkgB
$ spack gc

If there is no new version for a package, install will simply mark it as explicitly installed and gc will not remove it.

Fixes #9626

michaelkuhn · 2020-05-15T14:10:22Z

cc @SteVwonder (better late than never 😆)

michaelkuhn · 2020-05-25T10:30:19Z

@alalazo Could you please review this? Thanks!

alalazo

In general I agree with the functionality added by this PR. I wonder if we should try to group this and other commands under a common umbrella. For instance:

$ spack store mark
$ spack store reindex
$ spack store find
...

instead of adding a new one. This is because:

$ ls -1 lib/spack/spack/cmd/ | wc -l
74

In case this discussion can be held in a separate issue.

lib/spack/spack/cmd/mark.py

michaelkuhn · 2020-05-26T14:19:57Z

I don't have a strong opinion regarding the amount of commands. Git also has a lot of commands and I think useful help output also solves the problem. Especially for common operations such as spack find, I would rather keep them short. Having to type spack store find every time could get annoying quickly. But I agree, this is probably best discussed in a separate issue.

alalazo · 2020-05-26T14:28:20Z

Especially for common operations such as spack find, I would rather keep them short.

We must have a spack find alias for spack store find in case. I agree discussing this in another issue. I don't have objections, but would like somebody else to chime in and comment on the opportunity to add a new command with such a narrow scope.

michaelkuhn · 2020-06-16T08:53:09Z

@tgamblin @scheibelp @adamjstewart @becker33 ping

I use Spack for dependency management for projects outside of Spack. The mark command would allow an upgrade workflow as described in the initial post. Currently, whenever users run into problems with dependencies, I more or less have to tell them "delete and then rebuild everything".

lib/spack/spack/installer.py

michaelkuhn · 2020-06-28T20:55:19Z

@alalazo @becker33 ping

I just did a rebase to fix some conflicts and pushed the updated version.

becker33

I think this is useful, and an appropriately named command. I agree that the proliferation of commands is a concern, but I don't really see an obvious place to combine this command, and I think having poorly factored commands with suboptions that are unintuitive is a worse sin than too many commands. I think most users are able to find the (relatively few) commands they need, and documentation and the community support are able to point power users towards the more advanced commands they desire.

I think this PR probably requires documentation. I also have one note for code cleanup.

lib/spack/spack/cmd/mark.py

michaelkuhn · 2020-06-29T11:15:44Z

I have just pushed an updated version that also includes documentation and some tests.

SteVwonder

Thanks @michaelkuhn! This is a very useful feature. I just had a question about some of the terminology used in the documentation (see below):

SteVwonder · 2020-06-30T20:31:25Z

lib/spack/docs/basic_usage.rst

+on different versions of ``libsigsegv``. ``spack gc`` will not remove any of
+the packages since both versions of ``m4`` have been installed explicitly.
+


Why are both versions of m4 considered being explicitly installed? Were they installed with spack install m4 ^[email protected] and spack install m4 ^[email protected]? If so, that makes sense to me.

If spack install m4 results in [email protected] getting marked as explicit, I'm confused. Are all packages marked as explict until someone comes by and manually marks them as implicit?

I could be wrong, but I think other package managers like apt use explicit to mean the package was explicitly mentioned by the user in a command line invocation, and implicit means the package was pulled in as a dependency of an explicit package.

If the terms used here differ, it might be worth precisely defining what explicit and implicit mean in this documentation.

Why are both versions of m4 considered being explicitly installed? Were they installed with spack install m4 ^[email protected] and spack install m4 ^[email protected]? If so, that makes sense to me.

Exactly. I have now added some more lines to the example that should hopefully make this more clear.

If spack install m4 results in [email protected] getting marked as explicit, I'm confused. Are all packages marked as explict until someone comes by and manually marks them as implicit?

No, only the two m4 packages will be marked as explicitly installed. I have also added a few sentences explaining this.

Thanks for your feedback! Let me know whether you think my changes are enough to clear things up.

Awesome! Thanks for the clarifications. LGTM!

lib/spack/spack/database.py

becker33

@michaelkuhn sorry I haven't gotten back to this sooner. I have one more request on this PR.

becker33 · 2020-09-02T00:56:28Z

lib/spack/spack/cmd/mark.py

+            print()
+            spack.cmd.display_specs(matching, **display_args)
+            print()


This is part of an error message, and should be printed to stderr

I have added a new commit that should take care of this. It also fixes uninstall's error output.

michaelkuhn · 2020-10-02T08:06:22Z

@becker33 ping

michaelkuhn · 2020-11-10T10:28:48Z

@becker33 I have rebased the PR to current develop and it now applies again. Anything else I should work on to get this merged? Thanks!

becker33

It needs a rebase again, but it looks good. Sorry I dropped the ball on reviewing this.

This adds a new `mark` command that can be used to mark packages as either explicitly or implicitly installed. Apart from fixing the package database after installing a dependency manually, it can be used to implement upgrade workflows as outlined in spack#13385. The following commands demonstrate how the `mark` and `gc` commands can be used to only keep the current version of a package installed: ```console $ spack install pkgA $ spack install pkgB $ git pull # Imagine new versions for pkgA and/or pkgB are introduced $ spack mark -i -a $ spack install pkgA $ spack install pkgB $ spack gc ``` If there is no new version for a package, `install` will simply mark it as explicitly installed and `gc` will not remove it. Fixes spack#9626

`output` can be used to redirect `display_specs`'s output to stderr.

becker33 · 2020-11-18T10:14:27Z

@michaelkuhn I took care of the rebase to get this into v0.16.0

* releases/v0.16: (6003 commits) update CHANGELOG.md for v0.16.0 bump version number to 0.16.0 clingo: add `master` branch version (spack#19958) cmd: add `spack mark` command (spack#16662) spack test (spack#15702) Added -level_zero -rocm -opencl flags and sha256 for TAU v2.30. (spack#19962) Improve warning message for deprecated attributes in "packages.yaml" Documentation: spack load/environments prefix inspections (spack#19961) spack load/environments: allow customization of prefix inspections (spack#18260) spack containerize: allow users to customize the base image (spack#15028) concretizer: modified weights for providers and matching for externals concretizer: maximize the number of default values used for a single variant concretizer: don't require a provider for virtual deps if spec is external concretizer: spec_clauses() shouldn't emit node_compiler_hard for rule bodies. concretizer: don't generate rules for empty version lists concretizer: add a rule to avoid cycles in the graph of dependencies External packages have a consistent hash across different concretizers Don't fail if MV variants have a tuple as default value Fixup for target preferences Added unit tests to for regressions on open concretizer bugs ... # Conflicts: # .gitignore # lib/spack/spack/binary_distribution.py # lib/spack/spack/cmd/setup.py # lib/spack/spack/modules/common.py # var/spack/repos/builtin/packages/med/package.py # var/spack/repos/builtin/packages/mofem-cephas/package.py # var/spack/repos/builtin/packages/mofem-fracture-module/package.py # var/spack/repos/builtin/packages/mofem-users-modules/package.py # var/spack/repos/builtin/packages/mpich/package.py # var/spack/repos/builtin/packages/petsc/package.py # var/spack/repos/builtin/packages/python/package.py

michaelkuhn force-pushed the cmd-mark branch from e2c0981 to 35f336e Compare May 16, 2020 18:15

adamjstewart added the new-command label May 16, 2020

alalazo reviewed May 26, 2020

View reviewed changes

lib/spack/spack/cmd/mark.py Show resolved Hide resolved

michaelkuhn force-pushed the cmd-mark branch from 35f336e to 9c74d87 Compare May 26, 2020 14:15

alalazo requested review from adamjstewart, becker33, scheibelp and tgamblin May 26, 2020 14:29

becker33 reviewed Jun 16, 2020

View reviewed changes

lib/spack/spack/installer.py Outdated Show resolved Hide resolved

michaelkuhn force-pushed the cmd-mark branch 3 times, most recently from b765672 to 0d140c2 Compare June 18, 2020 19:58

michaelkuhn force-pushed the cmd-mark branch from 0d140c2 to f5668ab Compare June 28, 2020 20:54

becker33 requested changes Jun 29, 2020

View reviewed changes

lib/spack/spack/cmd/mark.py Outdated Show resolved Hide resolved

michaelkuhn force-pushed the cmd-mark branch from 5f9caf1 to f2f12bb Compare June 29, 2020 11:15

michaelkuhn force-pushed the cmd-mark branch 2 times, most recently from d162c27 to 0d04bdf Compare June 30, 2020 19:06

SteVwonder reviewed Jun 30, 2020

View reviewed changes

michaelkuhn force-pushed the cmd-mark branch from 0d04bdf to 2447d7c Compare June 30, 2020 20:57

becker33 reviewed Jul 9, 2020

View reviewed changes

lib/spack/spack/database.py Outdated Show resolved Hide resolved

michaelkuhn force-pushed the cmd-mark branch 2 times, most recently from 9aa923f to 53deb4c Compare August 20, 2020 08:23

becker33 requested changes Sep 2, 2020

View reviewed changes

michaelkuhn force-pushed the cmd-mark branch from 828988d to 88a8576 Compare November 10, 2020 09:53

becker33 approved these changes Nov 17, 2020

View reviewed changes

michaelkuhn and others added 2 commits November 18, 2020 02:12

cmd: Add output argument for display_specs

dd3c0f1

`output` can be used to redirect `display_specs`'s output to stderr.

becker33 force-pushed the cmd-mark branch from 88a8576 to dd3c0f1 Compare November 18, 2020 10:12

Merge branch 'develop' into cmd-mark

6ea9177

tgamblin merged commit 20367e4 into spack:develop Nov 18, 2020

		on different versions of ``libsigsegv``. ``spack gc`` will not remove any of
		the packages since both versions of ``m4`` have been installed explicitly.

Conversation

michaelkuhn commented May 15, 2020

Uh oh!

michaelkuhn commented May 15, 2020

Uh oh!

michaelkuhn commented May 25, 2020

Uh oh!

alalazo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

michaelkuhn commented May 26, 2020

Uh oh!

alalazo commented May 26, 2020

Uh oh!

michaelkuhn commented Jun 16, 2020

Uh oh!

Uh oh!

michaelkuhn commented Jun 28, 2020

Uh oh!

becker33 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

michaelkuhn commented Jun 29, 2020

Uh oh!

SteVwonder left a comment

Choose a reason for hiding this comment

Uh oh!

SteVwonder Jun 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michaelkuhn Jun 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SteVwonder Jun 30, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

becker33 left a comment

Choose a reason for hiding this comment

Uh oh!

becker33 Sep 2, 2020

Choose a reason for hiding this comment

Uh oh!

michaelkuhn Sep 3, 2020

Choose a reason for hiding this comment

Uh oh!

michaelkuhn commented Oct 2, 2020

Uh oh!

michaelkuhn commented Nov 10, 2020

Uh oh!

becker33 left a comment

Choose a reason for hiding this comment

Uh oh!

becker33 commented Nov 18, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

SteVwonder Jun 30, 2020 •

edited

Loading

michaelkuhn Jun 30, 2020 •

edited

Loading