BundlePackage by citibeth · Pull Request #3133 · spack/spack

citibeth · 2017-02-12T02:40:01Z

TODO:

Unit tests
Docs

This provides an implementation for "clean" Bundle packages --- i.e. packages that don't install anything, but whose purpose is to depend on other packages. See example below.

Addresses #1926 #2049

from spack import *

class ProjNcurses(BundlePackage):
    """Sample Bundle"""

    version('1.0')

    depends_on('ncurses', type='run')

@tgamblin @becker33
Notes from past conversation:

I would really like to have a MetaPackage class for this, which you could just extend, and Spack would know it's not real.

I don't like the name MetaPackage, it sounds too much like "meta class," which is not a class. I think BundlePackage is less confusing.

Are you doing this because spack doesn't have something like bundler or brew bundle? Would you like something like that? It's something I've been thinking a lot about implementing recently, as our code teams need it for building their dependencies. The Gemfile and Gemfile.lock concepts map nicely onto Spack's abstract and concrete packages (though in both cases Spack can be more specific than gem). This would be a larger feature than what you're describing, but I think it's something spack needs to adequately encapsulate build environments.

Or, is this more like X11, where it's a bunch of packages that people will probably depend on together as part of a larger system?

I would say some of both. This also addresses the issues in #3131. Although I've used Spack configuration scopes and packages.yaml to good effect, I'm coming to believe that users aren't warming up to this way of working. I realized that we could get the same thing with packages, thereby foisting fewer abstractions on our users. (There are some subtle differences in semantics of putting version, variant, etc. directives in packages.yaml vs. a Bundle Package. That discussion should happen in Spack Environments WG).

How would I re-do #3049 with this PR in place... suppose you have A->B, and you want A and B's +debug flag to be configured together. You can make a BundlePackage we will call proj-A, something like this:

class ProjA(BundlePackage):
    variant('debug', default=False)
    depends_on('A')
    depends_on('A+debug', when='+debug')
    depends_on('B')
    depends_on('B+debug', when='+debug')

This approach is quite flexible. It can set multiple forwarded variants, create its own project-level variants to control constellations of variants in the DAG, choose which packages get the +debug (or whatever), etc. The examples I work with are significantly more complex than this one; I will try to see how they turn out using this approach to project management.

If we decide to adopt this approach (part of a larger discussion on Spack Environments), then we will be better able to ask users to remove variant forwarding, site-specific stuff, etc. from their main packages -- and put it in project-level packages like the one above. The hope is that users will be more willing to do this than the current state of affairs, where there seems to be resistance to putting stuff in packages.yaml.

citibeth · 2017-02-12T02:40:40Z

@hartzell @davydden @BarrySmith

citibeth · 2017-02-12T03:31:32Z

Can someone please help decipher the codecov errors? I think it's saying it wants tests for bundle.py, which is reasonable. Is it asking for anything else?

(I would want to add such tests after some positive feedback on this PR).

davydden · 2017-02-12T06:34:35Z

I like the idea and I would certainly use this.
But I think the name MetaPackage fits better because it's not really a package with url, tarbal, description, installation phase etc, but an auxiliary class to make it easier to have consistent toolchain installed with a single command. I would use it around the following lines:

class MyWorkSuite(MetaPackage):
    depends_on('dealii+doc') # use Boost and MPI, among others
    depends_on('valgrind')  # use Boost and MPI
    depends_on('[email protected]')
    depends_on('numdiff')
    depends_on('the-silver-searcher')
    # add whatever i may be using in conjunction with the above in personal C++ projects

I think by definition such packages should never be submitted upstream to Spack. Thereby this means that they either need to be in user's fork -- this would require extra Git skills and may not be appropriate for all users; or such packages should be placed into custom Repositories to be added in repos.yaml. I think the latter is easier to maintain. One could also keep this repo in ./spack so such packages won't be deleted if a user decided to wipe out Spack and re-install everything.

davydden · 2017-02-12T06:38:58Z

p.s. However BundlePackage is more descriptive in what this package does, whereas MetaPackage implies that it's not really a package, but does not really say anything about how it is to be used. So I am fine with both names.

davydden · 2017-02-12T06:40:23Z

lib/spack/spack/package.py

        self._fetcher = None
        self.url = getattr(self.__class__, 'url', None)

-        # Fix up self.url if this package fetches with a URLFetchStrategy.


i suppose this is the only change that is not directly related to the PR.

davydden · 2017-02-12T06:41:51Z

@citibeth if others also like the idea, would you mind adding a bit of documentation somewhere?

citibeth · 2017-02-12T17:16:46Z

I think by definition such packages should never be submitted upstream to Spack. Thereby this means that they either need to be in user's fork -- this would require extra Git skills and may not be appropriate for all users; or such packages should be placed into custom Repositories to be added in repos.yaml. I think the latter is easier to maintain. One could also keep this repo in ./spack so such packages won't be deleted if a user decided to wipe out Spack and re-install everything.

I understand the reasoning; but we need to be mindful of user convenience as well. Consider, for example, the case in #3049. Would @bvanessen be inconvenienced if (for example) project-level bundle packages for LBANN were not allowed into core Spack? That would certainly add more steps for LBANN users. It's easy to think of cases where we add (say) 10 packages, and then one top-level package controlling them all.

citibeth · 2017-02-12T17:17:37Z

I'd be happy to put project-level packages in a separate namespace that is built into Spack and turned on by default. That is essentially what the proj- prefix does.

davydden · 2017-02-12T17:21:05Z

I understand the reasoning; but we need to be mindful of user convenience as well

fair point. I am fine with proj- prefix for those.

tgamblin · 2017-02-14T17:55:29Z

@citibeth: thanks for the awesome feature. I will do a more thorough review later (currently at the OSLS in Tahoe), but I definitely want to get this into Spack.

RE: some of the points above:

I actually like BundlePackage and agree with @citibeth's point about Meta being overloaded. I think BundlePackage is a little more specific to this use case.
I think there is a place for BundlePackages in the builtin repo. We have commonly known suites of packages like x11 and the xsdk that @BarrySmith is working on, and I think it's reasonable to put some of those in the main repo.
I agree with @davydden that we probably shouldn't put every user's set of concretization settings in the main repo, but I think this package could serve as an easy infrastructure base for bundle files like other package managers have. These could live in project directories outside of Spack, and could potentially be implemented by instantiating a subclass of BundlePackage and installing it. So we'll probably need this infrastructure anyway. I think we can hold off on the specification of such things for now.
Slightly more future-looking points w.r.t. redoing LBANN #3049: I'm not sure we necessarily need BundlePackages. I think that when we have a smarter concretizer, it'll be possible to put preferences into packages instead of requirements. Currently, the main distinction between packages.yaml and package-level constraints is that the former are soft constraints, and the latter are required. Making package-level constraints soft could allow a developer to have some "preferred" defaults in their package but also allow higher-level dependencies to override those settings. That doesn't make me completely happy because it doesn't completely solve our package consistency concerns, but we can think about policy w.r.t. package configuration consistency once we have the capability to do this.

tgamblin · 2017-02-14T17:56:21Z

@citibeth:

If we're using this to install "suites" of software, do you think it makes sense for BundlePackage to have special install/remove semantics? e.g., if I do this from a clean install:

spack install xsdk
spack uninstall xsdk

Should I be left with a clean slate? Seems like uninstalling a bundle package like X11 should try to uninstall all the dependencies.

Then again, maybe we should rely on explicit/non-explicit packages to do this, and auto-remove unneeded implicit dependencies on uninstall. That would handle regular and bundle packages the same way. I might feel better about such a policy if we had garbage collection -- where we don't really remove things until some time after uninstall runs, in case someone needed the eagerly deleted packages. Thoughts?

luigi-calori · 2017-02-14T18:45:53Z

I like the idea to have a package that define a quite detailed installation recipe.
This would help in the goal of a precise definition of the deployment of some bundle of packages.
I' ll try to test it

citibeth · 2017-02-14T18:59:32Z

Slightly more future-looking points w.r.t. redoing #3049...

I like that idea. But this still begs the question of where these constraints should be put. I still think we should have a policy against forwarding variants in the "plain vanilla" packages. Otherwise, it will be too random, with no good way to manage the process.

Then again, maybe we should rely on explicit/non-explicit packages...

Yes. Although this is a topic I think needs more thought in the Environments WG. Yes I have lots of thoughts for another topic...

davydden · 2017-10-11T16:00:10Z

@tgamblin @scheibelp @becker33 ping.

1. Add source_path option to specify where the downloaded source is found. (Defaults to '.', previously hardcoded to '.') 2. Add missing `-j` option to match `spack install`

# Conflicts: # lib/spack/spack/__init__.py # lib/spack/spack/package.py

Basic BundlePackage works.

62dd705

citibeth added the feature A feature is missing in Spack label Feb 12, 2017

Elizabeth Fischer added 2 commits February 11, 2017 21:54

Added missing source file.

0764cb9

flake8

a0536bc

davydden approved these changes Feb 12, 2017

View reviewed changes

This was referenced Feb 13, 2017

Package-Level Module Autoloading #3134

Closed

Specify additional configuration directories using command arguments #3142

Closed

tgamblin mentioned this pull request Mar 2, 2017

spack install paraview fails with cmake #3296

Closed

citibeth mentioned this pull request May 4, 2017

Adding FASTMath uber package and missing FASTMath packages #4120

Merged

citibeth mentioned this pull request Aug 11, 2017

Bundle Packages #1926

Closed

junghans mentioned this pull request Oct 11, 2017

Add a bundle package for the ECP proxy applications suite #5689

Merged

Merge branch 'develop' into efischer/170211-BundlePackage

1dcac6a

JavierCVilla referenced this pull request in JavierCVilla/fcc-spack Nov 14, 2017

Add dummy package to install a software stack

e507a6c

Elizabeth Fischer added 3 commits April 17, 2018 13:03

Minor upgrades to spack diy:

4323245

1. Add source_path option to specify where the downloaded source is found. (Defaults to '.', previously hardcoded to '.') 2. Add missing `-j` option to match `spack install`

Fix bug/oversight in formatting error message.

192007f

Merge branch 'develop' into efischer/170211-BundlePackage

0c4edb4

# Conflicts: # lib/spack/spack/__init__.py # lib/spack/spack/package.py

citibeth force-pushed the efischer/170211-BundlePackage branch from ff6a789 to 0c4edb4 Compare April 23, 2018 02:04

Undo change that is in spack#5963

861c691

citibeth closed this Apr 23, 2018

citibeth reopened this Apr 23, 2018

citibeth closed this Apr 23, 2018

citibeth mentioned this pull request Apr 23, 2018

BundlePackage #7862

Closed

Conversation

citibeth commented Feb 12, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

citibeth commented Feb 12, 2017

Uh oh!

citibeth commented Feb 12, 2017

Uh oh!

davydden commented Feb 12, 2017

Uh oh!

davydden commented Feb 12, 2017

Uh oh!

davydden Feb 12, 2017

Choose a reason for hiding this comment

Uh oh!

davydden commented Feb 12, 2017

Uh oh!

citibeth commented Feb 12, 2017

Uh oh!

citibeth commented Feb 12, 2017

Uh oh!

davydden commented Feb 12, 2017

Uh oh!

tgamblin commented Feb 14, 2017

Uh oh!

tgamblin commented Feb 14, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

luigi-calori commented Feb 14, 2017

Uh oh!

citibeth commented Feb 14, 2017

Uh oh!

davydden commented Oct 11, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

citibeth commented Feb 12, 2017 •

edited

Loading

tgamblin commented Feb 14, 2017 •

edited

Loading