Skip to content

ASP-based solver: allow configuring target selection#29835

Merged
becker33 merged 6 commits intospack:developfrom
alalazo:features/target_selection
Apr 26, 2022
Merged

ASP-based solver: allow configuring target selection#29835
becker33 merged 6 commits intospack:developfrom
alalazo:features/target_selection

Conversation

@alalazo
Copy link
Copy Markdown
Member

@alalazo alalazo commented Apr 1, 2022

fixes #29672

This commit adds a new concretizer:targets configuration section, and two options under it.

  1. concretizer:targets:granularity allows switching from considering only generic targets to consider all possible microarchitectures.
  2. concretizer:targets:host_compatible instead controls whether we can concretize for microarchitectures that are incompatible with the current host.

Modifications:

  • Add new handles to configure the solver
  • Add documentation
  • Add unit tests

@spackbot-app spackbot-app bot added the defaults label Apr 1, 2022
@alalazo
Copy link
Copy Markdown
Member Author

alalazo commented Apr 1, 2022

@bvanessen This is still a draft, but I think it might be one way to solve your issue with Spack reusing binaries that are incompatible with the host. Can you try this PR with:

concretizer:
  targets:
    host_compatible: true

?

I drafted the PR after having observed an increase in the time it takes to concretize a spec if we add additional rules as in #29694. Since Spack seems sensitive to the number of targets that are used as candidates, here I simply avoid emitting facts for certain targets (i.e. I reduce the problem space) rather than having additional rules in the logic program.

This is the result on my host:

  • Spack: 0.18.0.dev0 (0190890)
  • Python: 3.8.10
  • Platform: linux-ubuntu20.04-icelake
  • Concretizer: clingo

when I concretize hdf5 using all the microarchitectures in archspec which are in the x86_64 family:

$ spack solve --timers hdf5
Time:
    setup:         4.1003
    load:          0.0191
    ground:        1.3516
    solve:         1.7280
Total: 7.2267
[ ... ]

and when I only allow for generic x86_64 levels:

$ spack solve --timers hdf5
Time:
    setup:         3.9343
    load:          0.0191
    ground:        1.0835
    solve:         1.4872
Total: 6.5506
[ ... ]

@alalazo
Copy link
Copy Markdown
Member Author

alalazo commented Apr 1, 2022

Here's an example concretization when using:

concretizer:
  targets:
    granularity: generic
$ spack solve --timers trilinos
Time:
    setup:         5.0338
    load:          0.0192
    ground:        1.5389
    solve:         2.0918
Total: 8.7160

==> Best of 3 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                            Installed  ToBuild
  1         number of packages to build (vs. reuse)                      -        0
  2         deprecated versions used                                     0        0
  3         version weight                                               0        0
  4         number of non-default variants (roots)                       0        0
  5         preferred providers for roots                                0        0
  6         default values of variants not being used (roots)            0        3
  7         number of non-default variants (non-roots)                   0        0
  8         preferred providers (non-roots)                              0        0
  9         compiler mismatches                                          0        0
  10        OS mismatches                                                0        0
  11        non-preferred OS's                                           0        0
  12        version badness                                              0        4
  13        default values of variants not being used (non-roots)        0        1
  14        non-preferred compilers                                      0        0
  15        target mismatches                                            0        0
  16        non-preferred targets                                        0        0

[email protected]%[email protected]~adios2+amesos+amesos2+anasazi+aztec~basker+belos~boost~chaco~complex~cuda~cuda_rdc~debug~dtk+epetra+epetraext~epetraextbtf~epetraextexperimental~epetraextgraphreorderings~exodus+explicit_template_instantiation~float+fortran~gtest~hdf5~hypre+ifpack+ifpack2~intrepid~intrepid2~ipo~isorropia+kokkos~mesquite~minitensor+ml+mpi+muelu~mumps~nox~openmp~panzer~phalanx~piro~python~rocm~rocm_rdc~rol~rythmos+sacado~scorec~shards+shared~shylu~stk~stokhos~stratimikos~strumpack~suite-sparse~superlu~superlu-dist~teko~tempus~thyra+tpetra~trilinoscouplings~wrapper~x11~zoltan~zoltan2 build_type=RelWithDebInfo cxxstd=14 gotype=long_long arch=linux-ubuntu20.04-x86_64_v4
    ^[email protected]%[email protected]~doc+ncurses+openssl+ownlibs~qt build_type=Release arch=linux-ubuntu20.04-x86_64_v4
        ^[email protected]%[email protected]~symlinks+termlib abi=none arch=linux-ubuntu20.04-x86_64_v4
            ^[email protected]%[email protected] arch=linux-ubuntu20.04-x86_64_v4
        ^[email protected]%[email protected]~docs~shared certs=system arch=linux-ubuntu20.04-x86_64_v4
            ^[email protected]%[email protected]+cpanm+shared+threads arch=linux-ubuntu20.04-x86_64_v4
                ^[email protected]%[email protected]+cxx~docs+stl patches=b231fcc arch=linux-ubuntu20.04-x86_64_v4
                ^[email protected]%[email protected]~debug~pic+shared arch=linux-ubuntu20.04-x86_64_v4
                    ^[email protected]%[email protected] arch=linux-ubuntu20.04-x86_64_v4
                        ^[email protected]%[email protected] libs=shared,static arch=linux-ubuntu20.04-x86_64_v4
                ^[email protected]%[email protected] arch=linux-ubuntu20.04-x86_64_v4
                    ^[email protected]%[email protected] arch=linux-ubuntu20.04-x86_64_v4
                ^[email protected]%[email protected]+optimize+pic+shared arch=linux-ubuntu20.04-x86_64_v4
    ^[email protected]%[email protected]~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared arch=linux-ubuntu20.04-x86_64_v4
        ^[email protected]%[email protected] arch=linux-ubuntu20.04-x86_64_v4
            ^[email protected]%[email protected] arch=linux-ubuntu20.04-x86_64_v4
                ^[email protected]%[email protected]+sigsegv patches=9dc5fbd,bfdffa7 arch=linux-ubuntu20.04-x86_64_v4
                    ^[email protected]%[email protected] arch=linux-ubuntu20.04-x86_64_v4
            ^[email protected]%[email protected] arch=linux-ubuntu20.04-x86_64_v4
        ^[email protected]%[email protected]~python arch=linux-ubuntu20.04-x86_64_v4
            ^[email protected]%[email protected]~pic libs=shared,static arch=linux-ubuntu20.04-x86_64_v4
    ^[email protected]%[email protected]~bignuma~consistent_fpcsr~ilp64+locking+pic+shared symbol_suffix=none threads=none arch=linux-ubuntu20.04-x86_64_v4
    ^[email protected]%[email protected]~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~pmi~pmix+romio~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-ubuntu20.04-x86_64_v4
        ^[email protected]%[email protected]+openssl arch=linux-ubuntu20.04-x86_64_v4
        ^[email protected]%[email protected] patches=4e1d78c,62fc8a8,ff37630 arch=linux-ubuntu20.04-x86_64_v4
            ^[email protected]%[email protected] patches=35c4492,7793209,a49dd5b arch=linux-ubuntu20.04-x86_64_v4
            ^[email protected]%[email protected] arch=linux-ubuntu20.04-x86_64_v4
        ^[email protected]%[email protected] arch=linux-ubuntu20.04-x86_64_v4
            ^[email protected]%[email protected] arch=linux-ubuntu20.04-x86_64_v4

Copy link
Copy Markdown
Member

@becker33 becker33 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One comment, otherwise LGTM

if target not in compatible_targets:
compatible_targets.append(target)
if target not in candidate_targets:
candidate_targets.append(target)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to add the ancestors of the target here as well. If I'm on an aarch64 system, and try to concretize hdf5 target=broadwell, then I should have haswell and x86_64 available as allbacks for dependency nodes that can't build with broadwell.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See ec1bb5f I think we'll need to revisit anyhow when adding proper support for cross-compilation.

@spackbot-app spackbot-app bot added documentation Improvements or additions to documentation tests General test capability(ies) labels Apr 4, 2022
@alalazo alalazo marked this pull request as ready for review April 4, 2022 13:58
@alalazo alalazo force-pushed the features/target_selection branch from 16c4f88 to 4488a05 Compare April 5, 2022 09:15
@tgamblin tgamblin self-assigned this Apr 12, 2022
Copy link
Copy Markdown
Member

@becker33 becker33 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alalazo any chance you can get the test coverage on this up a bit?

alalazo added 5 commits April 25, 2022 15:37
This commit adds a new "concretizer:targets" configuration
section, and two options under it.

 - "concretizer:targets:granularity" allows switching from
considering only generic targets to consider all possible
microarchitectures.

 - "concretizer:targets:host_compatible" instead controls
whether we can concretize for microarchitectures that
are incompatible with the current host.
@alalazo alalazo force-pushed the features/target_selection branch from 4488a05 to 2fbdc7b Compare April 25, 2022 14:29
@becker33 becker33 enabled auto-merge (squash) April 26, 2022 00:19
@becker33 becker33 merged commit b9d6a51 into spack:develop Apr 26, 2022
@alalazo alalazo deleted the features/target_selection branch April 26, 2022 06:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

defaults documentation Improvements or additions to documentation tests General test capability(ies)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

--reuse concretizes to a new architecture than the current target architecture

3 participants