Skip to content

concretizer: reuse existing packages#25310

Merged
tgamblin merged 30 commits intodevelopfrom
reuse-existing-packages
Nov 5, 2021
Merged

concretizer: reuse existing packages#25310
tgamblin merged 30 commits intodevelopfrom
reuse-existing-packages

Conversation

@tgamblin
Copy link
Copy Markdown
Member

@tgamblin tgamblin commented Aug 8, 2021 •
edited
Loading

Draft PR because it still needs some work, but it's in a state that @adamjstewart could start playing with.

Fixes #311.
Fixes #22613.
Fixes #20429.

This PR makes the concretizer aware of already-installed packages, and it tries to optimize the solve by reusing them instead of building when it can.

--reuse option

Currently, this adds a --reuse option to spack solve. You can try it out with spack solve -Il --reuse <spec>. I think the interface is TBD; we probably want to think about what to call this vs. the old solve, and about which one should be default.

Minimizing builds

The way this works is:

  1. Figure out which installed packages are possible dependencies of the things to be installed.
  2. Add information about installed hashes that are possible dependencies to the ASP program.
    It looks kind of like this openssl example, but for everything that is installed: 
    installed_hash("openssl","lwatuuysmwkhuahrncywvn77icdhs6mn").
imposed_constraint("lwatuuysmwkhuahrncywvn77icdhs6mn","node","openssl").
imposed_constraint("lwatuuysmwkhuahrncywvn77icdhs6mn","version","openssl","1.1.1g").
imposed_constraint("lwatuuysmwkhuahrncywvn77icdhs6mn","node_platform","openssl","darwin").
imposed_constraint("lwatuuysmwkhuahrncywvn77icdhs6mn","node_os","openssl","catalina").
imposed_constraint("lwatuuysmwkhuahrncywvn77icdhs6mn","node_target","openssl","x86_64").
imposed_constraint("lwatuuysmwkhuahrncywvn77icdhs6mn","node_compiler","openssl","apple-clang").
imposed_constraint("lwatuuysmwkhuahrncywvn77icdhs6mn","node_compiler_version","openssl","apple-clang","12.0.0").
imposed_constraint("lwatuuysmwkhuahrncywvn77icdhs6mn","hash","openssl","lwatuuysmwkhuahrncywvn77icdhs6mn").
imposed_constraint("lwatuuysmwkhuahrncywvn77icdhs6mn","depends_on","openssl","zlib","build").
imposed_constraint("lwatuuysmwkhuahrncywvn77icdhs6mn","depends_on","openssl","zlib","link").
imposed_constraint("lwatuuysmwkhuahrncywvn77icdhs6mn","hash","zlib","x2anksgssxsxa7pcnhzg5k3dhgacglze").
    You can see from the example that if we use this hash for openssl, it also forces a bunch of constraints and a dependency on its particular zlib installation on the solve. i.e., it's concrete -- we have to use it the way it was built and we cannot twiddle things like variant values on things we already installed.
  3. Try to minimize the number of packages that are built. That looks like of like: 
    % the solver is free to choose at most one installed hash for each package in the solution
{ hash(Package, Hash) : installed_hash(Package, Hash) } 1 :- node(Package).

% if a hash is selected, we impose all of its constraints
impose(Hash) :- hash(Package, Hash).

% if we haven't selected a hash for a package, and if it is not external, we build it
build(Package) :- not hash(Package, _), not external(Package), node(Package).
#minimize { 1@18,Package : build(Package) }.

OS handling

The solver was still handling OS defaults through hard constraints (this is how we did it before we knew how to do it the right way -- with optimization), so this reworks OS handling in the solver. I believe this is the last part of the spec that needed converting -- targets and compilers were already changed in past PRs.

We now optimize for OS matches in the Spec, but we allow you to depend on a package for an older OS, with some limits. Specifically, I noticed when testing this that I wanted to reuse still-working Catalina installations on Big Sur. This adds the ability to specify, e.g.: os_compatible("catalina", "bigsur") so that the solver knows that it can safely reuse certain packages. I think we can also leverage this for, e.g., reusing CentOS binaries on RHEL.

Relaxing existing constraints

Many of the existing constraints in the solver are meant for things we plan to build. When reusing concrete specs, We have to take the installed specs as they are. We can't do things like disallow unknown variant names (the variants may not exist) or compiler version OS support (we only know about the compilers on the current system, not the ones the concrete specs were built with). So, these types of constraints have been restricted to only apply to specs we intend to build. If we reuse something concrete, we still have to consider things like dependency version constraints (e.g. if you want to reuse openssl you should still check if the version that's installed satisfies your depends_on("openssl@:1.0") constraint).

Example

Here's a comparison of reusing vs. not reusing with llvm:

With reuse:

`spack solve -Il --reuse llvm` output 
(spackle):spack> spack solve -Il --reuse llvm
==> Best of 7 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                 Value
  1         number of packages to build (vs. reuse)       5
  2         deprecated versions used                      2
  3         version weight                                0
  4         number of non-default variants (roots)        1
  5         multi-valued variants                        -1
  6         preferred providers for roots                 0
  7         number of non-default variants (non-roots)    5
  8         preferred providers (non-roots)              10
  9         compiler mismatches                           0
  10        os mismatches                                 8
  11        version badness                              51
  12        count of non-root multi-valued variants      -5
  13        non-preferred compilers                       0
  14        target mismatches                             0
  15        non-preferred targets                       253

 -   ikfi254  [email protected]%[email protected]~all_targets+clang~code_signing+compiler-rt~cuda~flang~gold+internal_unwind~ipo+libcxx+lld~lldb~llvm_dylib~mlir~omp_debug~omp_tsan+polly~python~shared_libs~split_dwarf build_type=Release cuda_arch=none arch=darwin-bigsur-x86_64
 -   xtkcrrd      ^[email protected]%[email protected]~doc+ncurses+openssl+ownlibs~qt build_type=Release arch=darwin-bigsur-x86_64
[+]  eli4aax          ^[email protected]%[email protected]~symlinks+termlib arch=darwin-catalina-x86_64
[+]  lwatuuy          ^[email protected]%[email protected]+systemcerts arch=darwin-catalina-x86_64
[+]  x2anksg              ^[email protected]%[email protected]+optimize+pic+shared arch=darwin-catalina-x86_64
 -   oc2m54j      ^[email protected]%[email protected]~cairo~cuda~gl~libudev+libxml2~netloc~nvml~pci+shared arch=darwin-bigsur-x86_64
[+]  dqzlvh6          ^[email protected]%[email protected]~python arch=darwin-catalina-x86_64
[+]  xrjlxcv              ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]  gfa5h76              ^[email protected]%[email protected]~pic arch=darwin-catalina-x86_64
 -   ozoef2w          ^[email protected]%[email protected] arch=darwin-bigsur-x86_64
 -   tzk3evu      ^[email protected]%[email protected] arch=darwin-bigsur-x86_64
[+]  jjgcc6o          ^[email protected]%[email protected]+cpanm+shared+threads arch=darwin-catalina-x86_64
[+]  df3hxmr              ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]  4d3hjqd                  ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]  24lf5pr      ^[email protected]%[email protected]+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4~uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87,210df3f28cde02a8135b58cc4168e70ab91dbf9097359d05938f1e2843875e57 arch=darwin-catalina-x86_64
[+]  6ntaixw          ^[email protected]%[email protected]+shared arch=darwin-catalina-x86_64
[+]  2jdnmfp          ^[email protected]%[email protected]~libbsd arch=darwin-catalina-x86_64
[+]  g5lf2co          ^[email protected]%[email protected]+bzip2+curses+git~libunistring+libxml2+tar+xz arch=darwin-catalina-x86_64
[+]  dc675hk              ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]  5ye4c7r          ^[email protected]%[email protected] patches=26f26c6f29a7ce9bf370ad3ab2610f99365b4bdd7b82e7c31df41a3370d685c0 arch=darwin-catalina-x86_64
[+]  nwrcpot          ^[email protected]%[email protected]+column_metadata+fts~functions~rtree arch=darwin-catalina-x86_64
[+]  de6nj46      ^[email protected]%[email protected]+python arch=darwin-catalina-x86_64
[+]  mjlsl7o          ^[email protected]%[email protected] arch=darwin-catalina-x86_64

Without reuse:

`spack solve -Il llvm` output 
(spackle):spack> spack solve -Il llvm
==> Best of 7 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                 Value
  1         number of packages to build (vs. reuse)      16
  2         deprecated versions used                      0
  3         version weight                                9
  4         number of non-default variants (roots)        1
  5         multi-valued variants                        -1
  6         preferred providers for roots                 0
  7         number of non-default variants (non-roots)   10
  8         preferred providers (non-roots)              20
  9         compiler mismatches                           0
  10        os mismatches                                 0
  11        version badness                               9
  12        count of non-root multi-valued variants       0
  13        non-preferred compilers                       0
  14        target mismatches                             0
  15        non-preferred targets                         0

 -   2p6rpiv  [email protected]%[email protected]~all_targets+clang~code_signing+compiler-rt~cuda~flang~gold+internal_unwind~ipo+libcxx+lld~lldb~llvm_dylib~mlir~omp_debug~omp_tsan+polly~python~shared_libs~split_dwarf build_type=Release cuda_arch=none patches=3fa900e31763cc7ff604430b86faa4eb650048c1796ebd39db2dcf84f6168a8b arch=darwin-bigsur-skylake
 -   rs5nwwn      ^[email protected]%[email protected]~doc+ncurses~openssl+ownlibs~qt build_type=Release arch=darwin-bigsur-skylake
 -   xrklbl5          ^[email protected]%[email protected]~symlinks+termlib abi=none arch=darwin-bigsur-skylake
 -   ztt3pgs              ^[email protected]%[email protected] arch=darwin-bigsur-skylake
 -   hjw2gx7      ^[email protected]%[email protected]~cairo~cuda~gl~libudev~libxml2~netloc~nvml~pci+shared arch=darwin-bigsur-skylake
 -   zipu7bi      ^[email protected]%[email protected] arch=darwin-bigsur-skylake
 -   raet62c          ^[email protected]%[email protected]+cpanm+shared+threads arch=darwin-bigsur-skylake
 -   akjl2di              ^[email protected]%[email protected]+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=darwin-bigsur-skylake
 -   l7gigul              ^[email protected]%[email protected]~debug~pic+shared arch=darwin-bigsur-skylake
 -   vazus7p                  ^[email protected]%[email protected] arch=darwin-bigsur-skylake
 -   vtlbzpn                      ^[email protected]%[email protected] arch=darwin-bigsur-skylake
 -   xkdly6s              ^[email protected]%[email protected] arch=darwin-bigsur-skylake
 -   gmhw2jt                  ^[email protected]%[email protected] arch=darwin-bigsur-skylake
[+]  xflsoui              ^[email protected]%[email protected]+optimize+pic+shared arch=darwin-bigsur-skylake
 -   inxzdsq      ^[email protected]%[email protected]+bz2~ctypes+dbm~debug~libxml2~lzma~nis~optimizations+pic~pyexpat+pythoncmd+readline+shared~sqlite3~ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87 arch=darwin-bigsur-skylake
 -   m6ef5no          ^[email protected]%[email protected] arch=darwin-bigsur-skylake
 -   otyemkg          ^[email protected]%[email protected]+bzip2+curses+git~libunistring~libxml2~tar~xz arch=darwin-bigsur-skylake

You can see we reuse a lot more with the reuse option on -- whereas without it Spack wants to build clean versions of everything but zlib. You can see this in the optimization criteria as well -- the difference between using --reuse and not is 16 built packages. In the case without --reuse, we're just not telling the solver about the installed packages so it has no hashes to swap in for different builds.

Optimization criteria

There are still some tricky things in here -- specifically around the priority of these optimizations. You can see in the solve calls above that we're optimizing by minimizing the number of packages built. That has some weird consequences for things that aren't installed yet, like:

(spackle):spack> spack solve -Il --reuse cmake
[...]
 -   fw3sb6l  [email protected]%[email protected]~doc+ncurses~openssl+ownlibs~qt build_type=Release arch=darwin-bigsur-skylake
[+]  ywtu57g      ^[email protected]%[email protected]~symlinks+termlib abi=none arch=darwin-catalina-skylake

Here, the concretizer decided to disable openssl on a new cmake install, to avoid installing more things. So it did exactly what we told it to, but not what we likely want. We probably need to add some nuance to the optimization criterion. In particular, sometimes you care more about default variants, and sometimes you'd rather it reuse what's already installed.

Currently, I am thinking we want to only minimize builds if something that is installed could be reused. If there's some package we already have to build, we should just respect its defaults and build dependencies as we normally would. I haven't worked out how best to model that yet or if it'll end up fitting peoples' intuition, so please provide suggestions.

On a similar note, you can see that we also used a couple of deprecated versions of things because they were already installed. Seems like maybe we should put the reuse optimization below the "don't use deprecated versions" optimization so that this won't happen. That would mean that you'd need to be explicit on the command line about using deprecated versions to get them into the solve. I suspect that might surprise some people so I'm not sure whether we shouldn't just warn instead.

Tasks

  • Add information about installed dependencies to the solve
  • Rework OS semantics in the solver to allow for certain OS's to be compatible
  • Make certain integrity constraints only apply to specs we intend to build and not to reused specs
  • Resolve issues with semantics and ordering of optimization criteria (i.e. do the right thing when we're not reusing)
  • Figure out whether this should be the default and what the option should be called (e.g., this is --reuse -- should the old way be --deterministic? --upgrade? --latest?)
  • add options to spack install
  • enable reuse from binary caches and upstreams too (so that if you register them, you automatically reuse prebuilt things)
  • add --reuse to docs

That last task will make build caches and upstreams significantly easier to use the way we intended them, and it'll mean that you can both pull from develop and easily reuse binaries from older versions of Spack. This will reduce the perception of "hashes changing" that people have with the old, more deterministic concretizer.

@tgamblin tgamblin requested a review from becker33 August 8, 2021 20:49
@tgamblin tgamblin force-pushed the reuse-existing-packages branch from 8d40878 to fab5fae Compare August 8, 2021 21:35
@adamjstewart
Copy link
Copy Markdown
Member

adamjstewart commented Aug 8, 2021 •
edited
Loading

Thanks @tgamblin, you have no idea how happy this makes me! Gave this a quick spin, here are my thoughts.

CLI

I definitely think this should be configurable from the command-line. Can we move the option into common_arguments since multiple commands need to share it? Specifically, can you add the flag to concretize and install so I can play around with an environment and really give this PR a good test?

P.S. When are we merging spack solve into spack spec? I still want to be able to use spack spec --concretizer=clingo or spack spec --concretizer=original so I can see how things differ, which isn't possible with spack spec or spack solve if the default concretizer is set to clingo already. Would also love to merge spack concretize into spack spec too. But I can open an issue for that at some point, don't want to derail this PR.

Configuration

Since most Spack users will want to install things many times, I think this flag should be configurable in config.yaml. At SC a few years ago (has it really been that long already?) we talked about how users of the new concretizer could configure it to prefer certain behaviors, like:

  • minimize reinstalls
  • maximize variants enabled
  • minimize variants enabled
  • minimize dependencies
  • etc.

Is this still something you think we should do? I would actually love to make the entire optimization criteria API public so that I can configure the order in my config.yaml.

On this thread, the behavior I would personally like doesn't exactly fall under --reuse or --deterministic. What I would personally like to use would be something in between. Let's say I have an environment with a list of root specs. These are likely packages that I care about, and would like to be able to test with the latest version whenever they get updated. However, non-root specs (or dependencies of specs I ask for on the command-line) are something I don't care about. So I would like to --reuse but only for dependencies of root specs.

Default

Since we're planning on switching the default concretizer in 0.17, and since concretization will change as a result anyway, I think we have a rare opportunity to change default concretization behavior without worrying too much about backwards compatibility. I think --reuse being on by default is the correct way to go, and works very similarly to literally every other PM in existence.

Bugs

This PR seems to drastically change concretization behavior. That's expected, since --reuse will tell the concretizer to prefer existing installations even if they aren't exactly what were expected. What isn't expected is that this PR drastically changes concretization behavior when --reuse isn't being used! For example:

develop, no reuse 
$ spack solve -I python
==> Best of 10 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                 Value
  1         deprecated versions used                      0
  2         version weight                                0
  3         number of non-default variants (roots)        0
  4         multi-valued variants                         0
  5         preferred providers for roots                 0
  6         number of non-default variants (non-roots)    0
  7         preferred providers (non-roots)              20
  8         compiler mismatches                           0
  9         version badness                              -1
  10        count of non-root multi-valued variants      -2
  11        non-preferred compilers                       0
  12        target mismatches                             0
  13        non-preferred targets                      -600

[+]  [email protected]%[email protected]+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87 arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]~debug~pic+shared arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]~libbsd arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected]~symlinks+termlib abi=none arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]+bzip2+curses+git~libunistring+libxml2+tar+xz arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected]~python arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected]~pic libs=shared,static arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected]+optimize+pic+shared arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected] patches=26f26c6f29a7ce9bf370ad3ab2610f99365b4bdd7b82e7c31df41a3370d685c0 arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]~docs+systemcerts arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected]+cpanm+shared+threads arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected]+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]+column_metadata+fts~functions~rtree arch=darwin-catalina-x86_64
this branch, no reuse 
$ spack solve -I python
==> Best of 8 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                 Value
  1         number of packages to build (vs. reuse)       4
  2         deprecated versions used                      0
  3         version weight                                0
  4         number of non-default variants (roots)       10
  5         multi-valued variants                         0
  6         preferred providers for roots                 0
  7         number of non-default variants (non-roots)    4
  8         preferred providers (non-roots)              20
  9         compiler mismatches                           0
  10        OS mismatches                                 0
  11        non-preferred OSes                            0
  12        version badness                              -1
  13        count of non-root multi-valued variants       0
  14        non-preferred compilers                       0
  15        target mismatches                             0
  16        non-preferred targets                      -150

 -   [email protected]%[email protected]~bz2~ctypes~dbm~debug~libxml2~lzma~nis~optimizations+pic~pyexpat+pythoncmd~readline+shared~sqlite3~ssl~tix~tkinter~ucs4+uuid~zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87 arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected] arch=darwin-catalina-x86_64
 -       ^[email protected]%[email protected]~bzip2~curses+git~libunistring~libxml2~tar~xz arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected] arch=darwin-catalina-x86_64
this branch, reuse 
$ spack solve -I --reuse python
==> Best of 3 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                 Value
  1         number of packages to build (vs. reuse)       0
  2         deprecated versions used                      0
  3         version weight                                0
  4         number of non-default variants (roots)        1
  5         multi-valued variants                        -1
  6         preferred providers for roots                 0
  7         number of non-default variants (non-roots)    1
  8         preferred providers (non-roots)               0
  9         compiler mismatches                           0
  10        OS mismatches                                 0
  11        non-preferred OSes                            0
  12        version badness                              -1
  13        count of non-root multi-valued variants      -3
  14        non-preferred compilers                       0
  15        target mismatches                             0
  16        non-preferred targets                      -480

[+]  [email protected]%[email protected]+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87 arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]~debug~pic+shared arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]~libbsd arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected]~symlinks+termlib abi=none arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]+bzip2+curses+git~libunistring+libxml2+tar+xz arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected]~python arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected]~pic libs=shared,static arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected]+optimize+pic+shared arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected] patches=26f26c6f29a7ce9bf370ad3ab2610f99365b4bdd7b82e7c31df41a3370d685c0 arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]~docs+systemcerts arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]+column_metadata+fts~functions~rtree arch=darwin-catalina-x86_64

It seems that even when --reuse isn't being used, the concretizer is still trying to minimize the number of packages that are built. I think that's a useful feature to have (see configuration section above), but seems to be orthogonal to the feature we're trying to implement here.

@tgamblin
Copy link
Copy Markdown
Member Author

tgamblin commented Aug 9, 2021 •
edited
Loading

It seems that even when --reuse isn't being used, the concretizer is still trying to minimize the number of packages that are built.

Yep this was true. It would not have any installations to consider, but it would minimize the number of built packages -- and everything was built so this boiled down to minimizing nodes as the top priority. Pushed a fix for just this problem.

I would still like to make the reuse a bit smarter -- basically I would like it to aggressively reuse when possible, without minimizing builds quite so aggressively. I am not sure I really know how to structure the optimization criteria to get that result, but I'm thinking about it.

This was referenced Aug 9, 2021
Closed
Merged
@tldahlgren
Copy link
Copy Markdown
Contributor

tldahlgren commented Aug 10, 2021

Will this resolve multiple dependency spec issues such as the one reported for font-util at https://gitlab.spack.io/spack/spack/-/jobs/619732?

...
==> [2021-08-09-20:06:17.817550] Error: [email protected]%[email protected] fonts=encodings,font-adobe-100dpi,font-adobe-75dpi,font-adobe-utopia-100dpi,font-adobe-utopia-75dpi,font-adobe-utopia-type1,font-alias,font-arabic-misc,font-bh-100dpi,font-bh-75dpi,font-bh-lucidatypewriter-100dpi,font-bh-lucidatypewriter-75dpi,font-bh-type1,font-bitstream-100dpi,font-bitstream-75dpi,font-bitstream-speedo,font-bitstream-type1,font-cronyx-cyrillic,font-cursor-misc,font-daewoo-misc,font-dec-misc,font-ibm-type1,font-isas-misc,font-jis-misc,font-micro-misc,font-misc-cyrillic,font-misc-ethiopic,font-misc-meltho,font-misc-misc,font-mutt-misc,font-schumacher-misc,font-screen-cyrillic,font-sun-misc,font-winitzki-cyrillic,font-xfree86-type1 arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected]+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected]~debug~pic+shared arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected]+sigsegv arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected]~symlinks+termlib abi=none arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected]+cpanm+shared+threads arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected] arch=linux-ubuntu18.04-x86_64 ^[email protected]%[email protected]+optimize+pic+shared arch=linux-ubuntu18.04-x86_64 matches multiple specs in the environment /builds/spack/spack/jobs_scratch_dir/concrete_environment: 
Dependency spec
  skmlfuy  [email protected]%[email protected] fonts=encodings,font-adobe-100dpi,font-adobe-75dpi,font-adobe-utopia-100dpi,font-adobe-utopia-75dpi,font-adobe-utopia-type1,font-alias,font-arabic-misc,font-bh-100dpi,font-bh-75dpi,font-bh-lucidatypewriter-100dpi,font-bh-lucidatypewriter-75dpi,font-bh-type1,font-bitstream-100dpi,font-bitstream-75dpi,font-bitstream-speedo,font-bitstream-type1,font-cronyx-cyrillic,font-cursor-misc,font-daewoo-misc,font-dec-misc,font-ibm-type1,font-isas-misc,font-jis-misc,font-micro-misc,font-misc-cyrillic,font-misc-ethiopic,font-misc-meltho,font-misc-misc,font-mutt-misc,font-schumacher-misc,font-screen-cyrillic,font-sun-misc,font-winitzki-cyrillic,font-xfree86-type1 arch=linux-ubuntu18.04-x86_64
Dependency spec
  skmlfuy  [email protected]%[email protected] fonts=encodings,font-adobe-100dpi,font-adobe-75dpi,font-adobe-utopia-100dpi,font-adobe-utopia-75dpi,font-adobe-utopia-type1,font-alias,font-arabic-misc,font-bh-100dpi,font-bh-75dpi,font-bh-lucidatypewriter-100dpi,font-bh-lucidatypewriter-75dpi,font-bh-type1,font-bitstream-100dpi,font-bitstream-75dpi,font-bitstream-speedo,font-bitstream-type1,font-cronyx-cyrillic,font-cursor-misc,font-daewoo-misc,font-dec-misc,font-ibm-type1,font-isas-misc,font-jis-misc,font-micro-misc,font-misc-cyrillic,font-misc-ethiopic,font-misc-meltho,font-misc-misc,font-mutt-misc,font-schumacher-misc,font-screen-cyrillic,font-sun-misc,font-winitzki-cyrillic,font-xfree86-type1 arch=linux-ubuntu18.04-x86_64
...

@alalazo
Copy link
Copy Markdown
Member

alalazo commented Aug 20, 2021

@tldahlgren @tgamblin Fyi, I moved all the pipelines to clingo in #25502 and I encounter several errors similar to #25310 (comment), see e.g. https://gitlab.spack.io/spack/spack/-/jobs/659647

@alalazo alalazo mentioned this pull request Sep 11, 2021
Closed
4 tasks
@becker33
Copy link
Copy Markdown
Member

becker33 commented Sep 29, 2021

I haven't looked into whether it's feasible yet, but I think what we want for the optimization criteria is a heuristic for how unlikely we are to care about something -- for example, we could score each node by the minimal depth to traverse from the root to the node, and then put the optimization "cost" of building that node at magic_weight * depth and minimize on that criterion.

We may also want to prioritize default variants for root packages above minimizing builds, but default variants for non-root packages below minimizing builds. That seems to track more with what we think we're doing when we install a package "unconstrained" (I want the default for the package, and then as efficient as possible to build deps).

I'll start looking into how feasible this is in case people like it.

@becker33
Copy link
Copy Markdown
Member

becker33 commented Sep 29, 2021

I also agree with @adamjstewart that we want to make these options as configurable as possible. I don't think we want to expose the API for this because it's probably not sufficiently user friends to make a good user interface, but we should basically create a config option for every decision that we feel is non-obvious what people will want, because that probably means different people will want different things.

@becker33
Copy link
Copy Markdown
Member

becker33 commented Sep 29, 2021

Another thought -- if we want to minimize the extent to which turning off default variants improves the score without preventing the concretizer from toggling variants to get better reuse, we could combine the variant issue into the criterion. The "score" for the reuse optimization criterion could be something like sum(depth * weight1) * sum(weight2 * non_default_variants / depth) or something equally insane that gets the behavior that users will find intuitive. The variant default criterion could then be handled independently at a different priority.

In general I think some of our optimization criteria may be too complicated to be modeled at discrete priorities, and we may need to write more complicated fitness functions to explore the trade-offs in the space. Those fitness functions could well be named and selected by config options, to get back to the point Adam was making.

@becker33
Copy link
Copy Markdown
Member

becker33 commented Sep 29, 2021

Thinking about this more, I think the default behavior probably does not require a complicated priority-mixing fitness function, because I think my earlier idea about alternating the priority root_variants > reuse > non_root_variants is probably close enough to a good default. But we may need complicated functions for some of the other concretization options we may consider.

becker33
becker33 requested changes Sep 29, 2021
View reviewed changes
Copy link
Copy Markdown
Member

@becker33 becker33 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partial review

lib/spack/spack/solver/asp.py Outdated

# create set of OS's to consider
possible = set([
buildable = set([
Copy link
Copy Markdown
Member

@becker33 becker33 Sep 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should include everything from platform.operating_systems.values(). That's really the list of operating systems we could target from this build node.

Copy link
Copy Markdown
Member

@becker33 becker33 Nov 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this was addressed

Copy link
Copy Markdown
Member Author

@tgamblin tgamblin Nov 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This worked with platform.operating_sys.keys()

lib/spack/spack/solver/asp.py
return (
os == platform.default_os, # prefer default
os not in buildable, # then prefer buildables
os, # then sort by name
Copy link
Copy Markdown
Member

@becker33 becker33 Sep 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need a much more formal model of what a compatible os is before we can try to mix operating systems in a meaningful way -- I think we currently get away with it because we put the OS optimization at such a high priority that the algorithm never chooses something else, but if we really want to allow centos7 binaries to be selected for rhel7 nodes we need a module for operating system compatibility which we can call into to compare any two operating systems and determine whether binaries for the first can run on the second. Then we could populate our possible_oses list with something like all the operating systems that we know could run on any of the operating systems in the buildable list.

@tgamblin tgamblin force-pushed the reuse-existing-packages branch from ac9dfd8 to 6a40b97 Compare October 22, 2021 09:55
@spackbot-app spackbot-app bot added the tests General test capability(ies) label Oct 22, 2021
@tgamblin
Copy link
Copy Markdown
Member Author

tgamblin commented Oct 22, 2021 •
edited
Loading

@adamjstewart @alalazo I still need to clean this up a bit, rework how spack solve displays opt criteria, and address @becker33's comments but can you see if the version here fixes the issue that minimizing nodes was drastically affecting the way we concretize built specs?

I'll write up a better description of how this works later, but short story is that now the opt criteria are structured like this:

  1. criteria for specs to be built (all minimize -- i.e. "best" value is zero)
  2. number of built (minimize)
  3. criteria for reused specs (exactly the same criteria as 1 above, but for the already installed stuff)

The idea is that if we build nothing, all the criteria in (1) are zero. If we do build something, then the top priority is making the build specs look like we've come to expect (i.e., criteria in (1)), but we still prioritize reusing to satisfy their dependencies. This will still reuse as much as possible, as reusing will always be "as good" as the best possible build configuration (because all the criteria are minimizing and their best value is zero). No build configuration will be inherently better than reusing because reusing will not worsen any criteria in (1).

It's a bit hard to think about at first but once you get the idea that the reused specs can't make the criteria in (1) worse, and (2) dominates the optimization process until we determine that something has to be built, it starts to make sense.

@tgamblin
Copy link
Copy Markdown
Member Author

tgamblin commented Oct 22, 2021

@haampie can you give this a try as well?

@haampie
Copy link
Copy Markdown
Member

haampie commented Oct 22, 2021

Hey Todd, I'll have a look on Monday!

@adamjstewart
Copy link
Copy Markdown
Member

adamjstewart commented Oct 23, 2021

@tgamblin with the latest commit, all calls to spack spec/solve result in the following warning:

/Users/Adam/spack/lib/spack/spack/solver/concretize.lp:840:7-40: info: atom does not occur in any rule head:
  provider_weight(Provider,Weight)

Also, I'm not seeing any dependency reuse anymore. spack spec -I results in the same thing on develop as it does on your branch, even though I have all of these dependencies installed somewhere:

$ spack solve -I py-torch
/Users/Adam/spack/lib/spack/spack/solver/concretize.lp:840:7-40: info: atom does not occur in any rule head:
  provider_weight(Provider,Weight)

==> Best of 12 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                         Value
  1         (build) deprecated versions used                      0
  2         (build) version weight                                0
  3         (build) number of non-default variants (roots)        0
  4         (build) preferred providers for roots                 0
  5         (build) preferred providers for roots                 0
  6         (build) number of non-default variants (non-roots)    0
  7         (build) preferred providers (non-roots)               0
  8         (build) compiler mismatches                           0
  9         (build) OS mismatches                                 0
  10        (build) non-preferred OSes                            0
  11        (build) version badness                               5
  12        (build) non-preferred compilers                       0
  13        (build) target mismatches                             0
  14        (build) non-preferred targets                     -1230
  15        number of packages to build (vs. reuse)               0
  16        deprecated versions used                              0
  17        version weight                                        0
  18        number of non-default variants (roots)                0
  19        preferred providers for roots                         0
  20        preferred providers for roots                         0
  21        number of non-default variants (non-roots)            0
  22        preferred providers (non-roots)                       0
  23        compiler mismatches                                   0
  24        OS mismatches                                         0
  25        version badness                                       0
  26        non-preferred compilers                               0
  27        target mismatches                                     0
  28        non-preferred targets                                 0

 -   [email protected]%[email protected]+caffe2~cuda~cudnn~distributed~fbgemm~gloo+kineto~magma+metal+mkldnn~mpi~nccl+nnpack~numa+numpy+onnx_ml+openmp+qnnpack~rocm~tensorpipe~test~valgrind+xnnpack cuda_arch=none patches=e37afffe45cf7594c22050109942370e49983ad772d12ebccf508377dc9dcfc9 arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]~doc+ncurses+openssl+ownlibs~qt build_type=Release arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected]~symlinks+termlib abi=none arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected]~docs certs=system arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected]+cpanm+shared+threads arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected]+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected]~debug~pic+shared arch=darwin-catalina-x86_64
[+]                      ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]                          ^[email protected]%[email protected] libs=shared,static arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]                      ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected]+optimize+pic+shared arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]~ipo build_type=RelWithDebInfo arch=darwin-catalina-x86_64
[+]      ^fxdiv@2020-04-17%[email protected]~ipo build_type=RelWithDebInfo arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected]+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87,4c2457325f2b608b1b6a2c63087df8c26e07db3e3d493caf36a56f0ecf6fb768,f2fd060afc4b4618fe8104c4c5d771f36dc55b1db5a4623785a4ea707ec72fb4 arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected]~libbsd arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected]+bzip2+curses+git~libunistring+libxml2+tar+xz arch=darwin-catalina-x86_64
[+]                      ^[email protected]%[email protected]~python arch=darwin-catalina-x86_64
[+]                          ^[email protected]%[email protected]~pic libs=shared,static arch=darwin-catalina-x86_64
[+]                      ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected] patches=26f26c6f29a7ce9bf370ad3ab2610f99365b4bdd7b82e7c31df41a3370d685c0 arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected]+column_metadata+fts~functions~rtree arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]~ilp64+shared threads=none arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]~ipo~multicompat build_type=RelWithDebInfo arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]+shared build_type=Release arch=darwin-catalina-x86_64
[+]      ^psimd@2020-05-17%[email protected]~ipo build_type=RelWithDebInfo arch=darwin-catalina-x86_64
[+]      ^pthreadpool@2021-04-13%[email protected]~ipo build_type=RelWithDebInfo arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected] arch=darwin-catalina-x86_64
 -           ^[email protected]%[email protected] arch=darwin-catalina-x86_64
 -       ^[email protected]%[email protected]+blas+lapack patches=873745d7b547857fcfec9cae90b09c133b42a4f0c23b6c2d84cf37e2dd816604 arch=darwin-catalina-x86_64
 -           ^[email protected]%[email protected] arch=darwin-catalina-x86_64
 -       ^[email protected]%[email protected]~cpp arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]~ipo build_type=RelWithDebInfo arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]+libyaml arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected] arch=darwin-catalina-x86_64
 -       ^[email protected]%[email protected]~notebook~telegram arch=darwin-catalina-x86_64
 -           ^[email protected]%[email protected]+toml arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected] arch=darwin-catalina-x86_64

@tgamblin
Copy link
Copy Markdown
Member Author

tgamblin commented Oct 23, 2021 •
edited
Loading

@adamjstewart can you test with spack solve --reuse? Looks like you didn’t give the option.

the provider weight thing needs to be fixed with a #defined but it’s minor.

@branee

This comment has been minimized.

Sign in to view
@adamjstewart
Copy link
Copy Markdown
Member

adamjstewart commented Oct 24, 2021

@tgamblin even with --reuse I'm seeing almost no reuse:

$ spack solve -I --reuse py-torch
/Users/Adam/spack/lib/spack/spack/solver/concretize.lp:840:7-40: info: atom does not occur in any rule head:
  provider_weight(Provider,Weight)

==> Best of 11 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                         Value
  1         (build) deprecated versions used                      0
  2         (build) version weight                                0
  3         (build) number of non-default variants (roots)        0
  4         (build) preferred providers for roots                 0
  5         (build) preferred providers for roots                 0
  6         (build) number of non-default variants (non-roots)    0
  7         (build) preferred providers (non-roots)               0
  8         (build) compiler mismatches                           0
  9         (build) OS mismatches                                 0
  10        (build) non-preferred OSes                            0
  11        (build) version badness                               0
  12        (build) non-preferred compilers                       0
  13        (build) target mismatches                             0
  14        (build) non-preferred targets                      -630
  15        number of packages to build (vs. reuse)              21
  16        deprecated versions used                              0
  17        version weight                                        0
  18        number of non-default variants (roots)                0
  19        preferred providers for roots                         0
  20        preferred providers for roots                         0
  21        number of non-default variants (non-roots)            0
  22        preferred providers (non-roots)                       0
  23        compiler mismatches                                   0
  24        OS mismatches                                         0
  25        version badness                                       0
  26        non-preferred compilers                               5
  27        target mismatches                                     0
  28        non-preferred targets                                 0

 -   [email protected]%[email protected]+caffe2~cuda~cudnn~distributed~fbgemm~gloo+kineto~magma+metal+mkldnn~mpi~nccl+nnpack~numa+numpy+onnx_ml+openmp+qnnpack~rocm~tensorpipe~test~valgrind+xnnpack cuda_arch=none patches=e37afffe45cf7594c22050109942370e49983ad772d12ebccf508377dc9dcfc9 arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]~doc+ncurses+openssl+ownlibs~qt build_type=Release arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected]~symlinks+termlib abi=none arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected]~docs certs=system arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected]+optimize+pic+shared arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]~ipo build_type=RelWithDebInfo arch=darwin-catalina-x86_64
[+]      ^fxdiv@2020-04-17%[email protected]~ipo build_type=RelWithDebInfo arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected]+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87,4c2457325f2b608b1b6a2c63087df8c26e07db3e3d493caf36a56f0ecf6fb768,f2fd060afc4b4618fe8104c4c5d771f36dc55b1db5a4623785a4ea707ec72fb4 arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected]~debug~pic+shared arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected]~libbsd arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]                      ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected]+bzip2+curses+git~libunistring+libxml2+tar+xz arch=darwin-catalina-x86_64
[+]                      ^[email protected]%[email protected] libs=shared,static arch=darwin-catalina-x86_64
[+]                      ^[email protected]%[email protected]~python arch=darwin-catalina-x86_64
[+]                          ^[email protected]%[email protected]~pic libs=shared,static arch=darwin-catalina-x86_64
[+]                      ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected] patches=26f26c6f29a7ce9bf370ad3ab2610f99365b4bdd7b82e7c31df41a3370d685c0 arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected]+column_metadata+fts~functions~rtree arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]~ilp64+shared threads=none arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]~ipo~multicompat build_type=RelWithDebInfo arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]+shared build_type=Release arch=darwin-catalina-x86_64
[+]      ^psimd@2020-05-17%[email protected]~ipo build_type=RelWithDebInfo arch=darwin-catalina-x86_64
[+]      ^pthreadpool@2021-04-13%[email protected]~ipo build_type=RelWithDebInfo arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected] arch=darwin-catalina-x86_64
 -           ^[email protected]%[email protected] arch=darwin-catalina-x86_64
 -       ^[email protected]%[email protected]+blas+lapack patches=873745d7b547857fcfec9cae90b09c133b42a4f0c23b6c2d84cf37e2dd816604 arch=darwin-catalina-x86_64
 -           ^[email protected]%[email protected] arch=darwin-catalina-x86_64
 -       ^[email protected]%[email protected]~cpp arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]~ipo build_type=RelWithDebInfo arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]+libyaml arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected] arch=darwin-catalina-x86_64
 -       ^[email protected]%[email protected]~notebook~telegram arch=darwin-catalina-x86_64
 -           ^[email protected]%[email protected]+toml arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected] arch=darwin-catalina-x86_64

$ spack solve -I --reuse gdal
/Users/Adam/spack/lib/spack/spack/solver/concretize.lp:840:7-40: info: atom does not occur in any rule head:
  provider_weight(Provider,Weight)

==> Best of 8 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                         Value
  1         (build) deprecated versions used                      0
  2         (build) version weight                                0
  3         (build) number of non-default variants (roots)        0
  4         (build) preferred providers for roots                 0
  5         (build) preferred providers for roots                 0
  6         (build) number of non-default variants (non-roots)    0
  7         (build) preferred providers (non-roots)               0
  8         (build) compiler mismatches                           0
  9         (build) OS mismatches                                 0
  10        (build) non-preferred OSes                            0
  11        (build) version badness                               0
  12        (build) non-preferred compilers                       0
  13        (build) target mismatches                             0
  14        (build) non-preferred targets                     -1140
  15        number of packages to build (vs. reuse)              38
  16        deprecated versions used                              0
  17        version weight                                        0
  18        number of non-default variants (roots)                0
  19        preferred providers for roots                         0
  20        preferred providers for roots                         0
  21        number of non-default variants (non-roots)            0
  22        preferred providers (non-roots)                       0
  23        compiler mismatches                                   0
  24        OS mismatches                                         0
  25        version badness                                       0
  26        non-preferred compilers                               2
  27        target mismatches                                     0
  28        non-preferred targets                                 0

 -   [email protected]%[email protected]~armadillo~cfitsio~crypto~cryptopp~curl~expat~geos~gif~grib~hdf4~hdf5~jasper~java+jpeg~kea~libiconv~libkml+liblzma+libtool+libz~mdb~netcdf~odbc~opencl~openjpeg~pcre~perl~pg~png~poppler+proj~python~qhull~sosi~sqlite3~xerces~xml2~zstd arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]~guile+nls arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected]+bzip2+curses+git~libunistring+libxml2+tar+xz arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected]~debug~pic+shared arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]                      ^[email protected]%[email protected] libs=shared,static arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected]~python arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected]~pic libs=shared,static arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected]+optimize+pic+shared arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected]~symlinks+termlib abi=none arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected] patches=12f6edb0c6b270b8c8dba2ce17998c580db01182d871ee32b7b6e4129bd1d23a,1732115f651cff98989cb0215d8f64da5e0f7911ebf0c13b064920f088f2ffe1 arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected]+cpanm+shared+threads arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected]+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=darwin-catalina-x86_64
[+]                  ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]                      ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]      ^[email protected]%[email protected]~ipo build_type=RelWithDebInfo arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected]~doc+ncurses+openssl+ownlibs~qt build_type=Release arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected]~docs certs=system arch=darwin-catalina-x86_64
 -       ^[email protected]%[email protected]+jpeg+proj+zlib arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]          ^[email protected]%[email protected] arch=darwin-catalina-x86_64
 -           ^[email protected]%[email protected]+curl+tiff arch=darwin-catalina-x86_64
 -               ^[email protected]%[email protected]~gssapi~ldap~libidn2~librtmp~libssh~libssh2~nghttp2 tls=gnutls,mbedtls,nss,secure_transport arch=darwin-catalina-x86_64
 -                   ^[email protected]%[email protected]~guile+zlib arch=darwin-catalina-x86_64
 -                       ^[email protected]%[email protected] arch=darwin-catalina-x86_64
 -                           ^[email protected]%[email protected] arch=darwin-catalina-x86_64
 -                       ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]                          ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]                              ^[email protected]%[email protected] patches=35c449281546376449766f92d49fc121ca50e330e60fefcfc9be2af3253082c2,7793209b33013dc0f81208718c68440c5aae80e7a1c4b8d336e382525af791a7,a49dd5bac3b62daa0ff688ab4d508d71dbd2f4f8d7e2a02321926346161bf3ee arch=darwin-catalina-x86_64
[+]                                  ^[email protected]%[email protected]+sigsegv patches=9dc5fbd0d5cb1037ab1e6d0ecc74a30df218d0a94bdd5a02759a97f62daca573,bfdffa7c2eb01021d5849b36972c069693654ad826c1a20b53534009a4ec7a89 arch=darwin-catalina-x86_64
[+]                                      ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]                              ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]                              ^[email protected]%[email protected] arch=darwin-catalina-x86_64
 -                   ^[email protected]%[email protected]~pic build_type=Release libs=static arch=darwin-catalina-x86_64
 -                   ^[email protected]%[email protected] arch=darwin-catalina-x86_64
 -                       ^[email protected]%[email protected] arch=darwin-catalina-x86_64
[+]                      ^[email protected]%[email protected]+column_metadata+fts~functions~rtree arch=darwin-catalina-x86_64
[+]              ^[email protected]%[email protected]~gmock~ipo+pthreads+shared build_type=RelWithDebInfo arch=darwin-catalina-x86_64

Both of these packages are already installed (several times unfortunately) so it should definitely be possible to build them without installing any dependencies.

@tgamblin tgamblin force-pushed the reuse-existing-packages branch from 6a40b97 to 3580b0d Compare October 24, 2021 02:19
@tgamblin
Copy link
Copy Markdown
Member Author

tgamblin commented Oct 24, 2021

@adamjstewart can you attach the tarball from spack debug create-db-tarball?

tgamblin and others added 20 commits November 4, 2021 22:10
@tgamblin
spec: ensure_valid_variants() should not validate concrete specs
72e7752 
Variants in concrete specs are "always" correct -- or at least we assume
them to be b/c they were concretized before. Their variants need not match
the current version of the package.
@tgamblin
concretizer: rework spack solve output to handle reuse better
b1ad90a
@alalazo @tgamblin
Fix a type in "variant_not_default" rule
9cd8357
@alalazo @tgamblin
ASP-based solve: if an OS is set, respect the value
956ce6c
@alalazo @tgamblin
Fix a unit test to match the new OS semantics
301dfb5 
CNL, debian6 and Suse are not compatible
@alalazo @tgamblin
Trim dependencies on externals
23180ae
@alalazo @tgamblin
Update command completion
f95f221
@tgamblin
concretizer: get rid of last maximize directive in concretize.lp
4c71c4e 
- [x] Get rid of forgotten maximize directive.
- [x] Simplify variant handling
- [x] Fix bug in treatment of defaults on externals (don't count
      non-default variants on externals against them)
@alalazo @tgamblin
spack spec: add --reuse argument
bd1ec68
@alalazo @tgamblin
spack concretize: add --reuse argument
253b609
@alalazo @tgamblin
spack install: add --reuse argument
d695961
@alalazo @tgamblin
Add buildcache to reusable specs
302d578
@alalazo @tgamblin
Add a missing definition in the logic program
45c73b9
@tgamblin
spack diff: more flexible tests, restore transitive diff with spec_cl…
2d1882c 
…auses

In switching to hash facts for concrete specs, we lost the transitive facts
from dependencies. This was fine for solves, because they were implied by
the imposed constraints from every hash. However, for `spack diff`, we want
to see what the hashes mean, so we need another mode for `spec_clauses()` to
show that.

This adds a `expand_hashes` argument to `spec_clauses()` that allows us to
output *both* the hashes and their implications on dependencies. We use
this mode in `spack diff`.
@tgamblin
concretizer: exempt already-installed specs from compiler and variant…
d3eb1ca 
… rules

Concrete specs that are already installed or that come from a buildcache
may have compilers and variant settings that we do not recognize, but that
shouldn't prevent reuse (at least not until we have a more detailed compiler
model).

- [x] make sure compiler and variant consistency rules only apply to
      built specs
- [x] don't validate concrete specs on input, either -- they're concrete
      and we shouldn't apply today's rules to yesterday's build
@tgamblin
bugfix: handle hashes that only exist in input specs
64a956b 
In our tests, we use concrete specs generated from mock packages,
which *only* occur as inputs to the solver. This fixes two problems:

1. We weren't previously adding facts to encode the necessary
   `depends_on()` relationships, and specs were unsatisfiable on
   reachability.

2. Our hash lookup for reconstructing the DAG does not
   consider that a hash may have come from the inputs.
@alalazo @tgamblin
Fix logic program for multi-valued variants
7afa0f3 
Reformulate variant rules so that we minimize both

1. The number of non-default values being used
2. The number of default values not-being used

This is crucial for MV variants where we may have
more than one default value
@tgamblin
concretizer: add error messages and simplify asp.py
1fd8b3f
@becker33 @tgamblin
error message for reusing multiple hashes for package
0b21415
@tgamblin
docs for experimental --reuse argument to spack install
2978c0f 
Add docs for `--reuse`, along with a warning that it will likely be
removed and refactored.
@tgamblin tgamblin force-pushed the reuse-existing-packages branch from a38e368 to 2978c0f Compare November 5, 2021 05:48
@spackbot-app spackbot-app bot added the documentation Improvements or additions to documentation label Nov 5, 2021
alalazo
alalazo approved these changes Nov 5, 2021
View reviewed changes
Copy link
Copy Markdown
Member

@alalazo alalazo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the last commits and gave this PR another fast test-drive. LGTM.

@tgamblin tgamblin merged commit 8e76244 into develop Nov 5, 2021
@tgamblin tgamblin deleted the reuse-existing-packages branch November 5, 2021 07:15
@tgamblin tgamblin mentioned this pull request Nov 6, 2021
Merged
@psakievich
Copy link
Copy Markdown
Contributor

psakievich commented Nov 8, 2021

Congrats all for getting this in. I followed the dev conversation from a far. I'm excited to try this out. 🥳

@tgamblin tgamblin mentioned this pull request Dec 24, 2021
Merged
7 tasks
@alalazo alalazo mentioned this pull request Mar 20, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@becker33 becker33 becker33 left review comments

@alalazo alalazo alalazo approved these changes

@adamjstewart adamjstewart Awaiting requested review from adamjstewart

Assignees

@tgamblin tgamblin

@alalazo alalazo

@becker33 becker33

Labels

binary-packages commands concretization dependencies documentation Improvements or additions to documentation environments tests General test capability(ies)

Projects

None yet

Milestone

No milestone

8 participants

@tgamblin @adamjstewart @tldahlgren @alalazo @becker33 @haampie @branee @psakievich