Skip to content

externals: Model as concrete specs and add dependency definitions#51118

Merged
tgamblin merged 42 commits intospack:developfrom
alalazo:solver/external-as-concrete
Oct 19, 2025
Merged

externals: Model as concrete specs and add dependency definitions#51118
tgamblin merged 42 commits intospack:developfrom
alalazo:solver/external-as-concrete

Conversation

@alalazo
Copy link
Copy Markdown
Member

@alalazo alalazo commented Aug 6, 2025 •
edited
Loading

refers to #49697
closes #36179
closes #38638
closes #41682

fixes #26034
fixes #28827
fixes #6420 see #6420 (comment)

In this PR externals are treated as concrete specs so there is a 1:1 mapping between an entry in packages.yaml and any installed external spec, for a fixed repository.

Their YAML specification has been extended, following #49697, to allow modeling dependencies of external specs. This might be quite useful to better capture e.g. ROCm installations that are already installed on a given system, or in similar cases.

To be backward compatible with external specs specifying a compiler, for instance mpich %gcc@9, Spack will match the compiler specification to an existing external. It will fail when the specification is ambiguous, or leads to no match.

@alalazo alalazo mentioned this pull request Aug 6, 2025
Merged
2 tasks
@alalazo alalazo added feature A feature is missing in Spack concretization external-packages labels Aug 6, 2025
@alalazo alalazo force-pushed the solver/external-as-concrete branch from aed154c to aeb5538 Compare August 6, 2025 06:55
@alalazo alalazo mentioned this pull request Aug 6, 2025
Merged
@alalazo alalazo force-pushed the solver/external-as-concrete branch from aeb5538 to 3042e18 Compare August 6, 2025 13:57
@alalazo alalazo mentioned this pull request Aug 7, 2025
Merged
@alalazo alalazo force-pushed the solver/external-as-concrete branch 3 times, most recently from 4ff2359 to 3d0a7e7 Compare August 8, 2025 17:18
@scheibelp
Copy link
Copy Markdown
Member

scheibelp commented Aug 8, 2025

Concerns:

1. controlling compiler mixing in matrices

For a matrix environment that combines [x, y] with [%gcc, %clang], right now you can generally prevent mixing with reuse: false and unify: false (I think we want a better way, but right now this is the best way I know of to minimize it). If the Clang/GCC entries are external, I'm curious if the changes in this PR will de-emphasize using pre-existing compiler definitions in a matrix environment (in other words, if externals are becoming concrete specs, is reuse the mechanism by which they are considered?).

2. 1:many behavior was useful sometimes

1:many behavior was useful for externals in the past because it gave the user some discretion to effectively mark many details of an external as unimportant: anything that a dependent requires could be "enforced"; if x->z+v1 and y->z~v1, then a z external could omit v1 from its spec and pretend to be both (and if x/y both run, the user wouldn't mind).

  • It was particularly useful for build dependencies. This is less of an issue if the concretizer doesn't force compiler matching onto build-only dependencies, but if that falls apart anywhere it will be felt acutely here (I think Treat externals as concrete specs #49697 (comment) is a potential example of this)
  • For libraries, Spack may get more-insistent on details that were not specified, or has the concretizer been changed in this PR to relax about concrete-and-external specs?

@alalazo
Copy link
Copy Markdown
Member Author

alalazo commented Aug 10, 2025

If the Clang/GCC entries are external, I'm curious if the changes in this PR will de-emphasize using pre-existing compiler definitions in a matrix environment (in other words, if externals are becoming concrete specs, is reuse the mechanism by which they are considered?).

This shouldn't be an issue, since the PR is designed so that concretizer:reuse keeps the same semantics as before. This means that:

  1. Externals are the only concrete specs considered when using reuse:false
  2. The only way to exclude externals from concretization is to use reuse:from and have a type:external with filters appropriately set, as shown here https://spack.readthedocs.io/en/latest/build_settings.html#reuse-already-installed-packages
  1. 1:many behavior was useful sometimes

I'd start from the claim that 1:many behavior is objectively wrong, since most specs cannot be both +foo and ~foo at the same time for most of the variants.

On top of that, this PR adds the ability to model dependencies for external specs. With the previous model the many in the "1:many" behavior grows in a combinatorial way, which makes things even worse.

Finally, after this PR, people can inspect their externals and know them up to their hash - so potentially externals can be also referenced by hash.

(I think Treat externals as concrete specs #49697 (comment) is a potential example of this)

I don't get your example of #49697 (comment). If anything it seems a claim that the PR goes in the correct direction. The use case as I understand it:

  1. Have an external llvm in packages.yaml
  2. Install julia, which requires e.g. version_suffix=jl +link_llvm_dylib
  3. The concretizer right now is happy to add those to your external, since those variants are rarely specified
  4. The spec concretizes, but fails at runtime with "The resulting errors are missing files in the llvm directory while building julia"

For libraries, Spack may get more-insistent on details that were not specified, or has the concretizer been changed in this PR to relax about concrete-and-external specs?

I don't get the question, sorry. I wonder if that would be answered by #49697, in particular by the "Complete missing information on an external node" subsection.

@scheibelp
Copy link
Copy Markdown
Member

scheibelp commented Aug 11, 2025

Externals are the only concrete specs considered when using reuse:false

Thanks, that sounds good.

For libraries, Spack may get more-insistent on details that were not specified, or has the concretizer been changed in this PR to relax about concrete-and-external specs?

I don't get the question, sorry. I wonder if that would be answered by #49697, in particular by the "Complete missing information on an external node" subsection.

The proposals for unspecified variants were

  1. inferring nothing (so e.g. a spec wouldn't satisfy +shared or ~shared)
  2. inferring default values

[2] is most likely to be useful (I predict that externals autocompleted with [1] would be frustrating to use). [2] still might be pessimistic: I think there was some convenience to a user effectively being able to say "this thing I installed is good, use it"; I'm a bit worried if they have to fight Spack about it; other alternatives that might help are

  1. determine_variants could be called on it to flesh them out
  2. there could be an option that the variants are all set once the first time it participates as a dependent in a concretization

@alalazo alalazo force-pushed the solver/external-as-concrete branch 2 times, most recently from b67e5db to e2277c8 Compare August 12, 2025 12:54
@alalazo alalazo marked this pull request as ready for review August 12, 2025 16:45
This was referenced Aug 13, 2025
Closed
Open
@haampie haampie requested a review from Copilot August 13, 2025 07:42
@alalazo alalazo review requested due to automatic review settings August 13, 2025 07:42
@haampie haampie requested a review from Copilot August 13, 2025 07:46

This comment was marked as outdated.

Sign in to view

@alalazo alalazo force-pushed the solver/external-as-concrete branch 3 times, most recently from c470033 to d21839e Compare August 13, 2025 11:24
@alalazo alalazo mentioned this pull request Aug 13, 2025
Closed
1 task
@tgamblin tgamblin self-requested a review August 14, 2025 16:42
@psakievich psakievich self-requested a review August 16, 2025 13:04
@tgamblin tgamblin self-assigned this Aug 18, 2025
@tgamblin tgamblin force-pushed the solver/external-as-concrete branch from 8c0a8f9 to a70504a Compare October 19, 2025 06:35
tgamblin
tgamblin reviewed Oct 19, 2025
View reviewed changes
tgamblin
tgamblin approved these changes Oct 19, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@tgamblin tgamblin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming the coverage is there, I think this is great! Thanks for the iteration and for addressing most of the points.

I think the error messages and upgrade paths need work before the release but we have time to improve those. See comment above but don't let I hold up merging.

@tgamblin tgamblin merged commit 0c83856 into spack:develop Oct 19, 2025
33 checks passed
@tgamblin
Copy link
Copy Markdown
Member

tgamblin commented Oct 19, 2025

Merged! TODO in a follow-on:

@tgamblin tgamblin changed the title solver: treat externals as concrete specs with dependencies externals: Model as concrete specs and add dependency definitions Oct 19, 2025
@alalazo alalazo deleted the solver/external-as-concrete branch October 19, 2025 21:02
bcumming pushed a commit to eth-cscs/alps-cluster-config that referenced this pull request Oct 22, 2025
@albestro
Update external packages spec (#66)
df6a5ea 
With spack/spack#51118 external specs are
treated in a slightly different way.

In particular. An external package will be mapped 1:1 with a specific
spec, which implies that an external entry will not anymore act as a
jolly in terms of unspecified variants, but it will just represent the
spec with the set of variants as they result from the configuration file
(explicitly or implicitly). Moreover, it will be possible (required?) to
specify also dependencies for the external packages.

This PR aims at anticipating potential problems that might arise with
this change.

- `xpmem` by default has `+kernel-module`, but this would require also
specifying the `gcc` dependency. since no one seems to care about this
variant, the proposed solution is to explicitly disable this variant
`~kernel-module`
@alalazo alalazo mentioned this pull request Oct 23, 2025
Open
3 tasks
@psakievich psakievich mentioned this pull request Nov 12, 2025
Merged
3 tasks
harshula added a commit to ACCESS-NRI/spack-config that referenced this pull request Nov 23, 2025
@harshula
common-api-v2/gadi/packages.yaml: externals are treated as concrete s…
3cb4b04 
…pecs

* As of spack/spack#51118 we need to accommodate:
  "To be backward compatible with external specs specifying a compiler, for
   instance mpich %gcc@9, Spack will match the compiler specification to an
   existing external. It will fail when the specification is ambiguous, or
   if it does not match any other externals."
harshula added a commit to ACCESS-NRI/spack-config that referenced this pull request Nov 24, 2025
@harshula
common-api-v2/gadi/packages.yaml: externals are treated as concrete s…
6088a9b 
…pecs

* As of spack/spack#51118 we need to accommodate:
  "To be backward compatible with external specs specifying a compiler, for
   instance mpich %gcc@9, Spack will match the compiler specification to an
   existing external. It will fail when the specification is ambiguous, or
   if it does not match any other externals."
harshula added a commit to ACCESS-NRI/spack-config that referenced this pull request Nov 25, 2025
@harshula
common-api-v2/gadi/packages.yaml: externals are treated as concrete s…
2cca835 
…pecs

* As of spack/spack#51118 we need to accommodate:
  "To be backward compatible with external specs specifying a compiler, for
   instance mpich %gcc@9, Spack will match the compiler specification to an
   existing external. It will fail when the specification is ambiguous, or
   if it does not match any other externals."
harshula added a commit to ACCESS-NRI/spack-config that referenced this pull request Nov 25, 2025
@harshula
common-api-v2/gadi/packages.yaml: externals are treated as concrete s…
23e3511 
…pecs

* As of spack/spack#51118 we need to accommodate:
  "To be backward compatible with external specs specifying a compiler, for
   instance mpich %gcc@9, Spack will match the compiler specification to an
   existing external. It will fail when the specification is ambiguous, or
   if it does not match any other externals."
harshula added a commit to ACCESS-NRI/spack-config that referenced this pull request Nov 25, 2025
@harshula
common-api-v2/gadi/packages.yaml: externals are treated as concrete s…
12b5300 
…pecs

* As of spack/spack#51118 we need to accommodate:
  "To be backward compatible with external specs specifying a compiler, for
   instance mpich %gcc@9, Spack will match the compiler specification to an
   existing external. It will fail when the specification is ambiguous, or
   if it does not match any other externals."
harshula added a commit to ACCESS-NRI/spack-config that referenced this pull request Nov 25, 2025
@harshula
common-api-v2/gadi/packages.yaml: externals are treated as concrete s…
6027e1e 
…pecs

* As of spack/spack#51118 we need to accommodate:
  "To be backward compatible with external specs specifying a compiler, for
   instance mpich %gcc@9, Spack will match the compiler specification to an
   existing external. It will fail when the specification is ambiguous, or
   if it does not match any other externals."
harshula added a commit to ACCESS-NRI/spack-config that referenced this pull request Nov 28, 2025
@harshula
common-api-v2/gadi/packages.yaml: externals are treated as concrete s…
9db16e1 
…pecs

* As of spack/spack#51118 we need to accommodate:
  "To be backward compatible with external specs specifying a compiler, for
   instance mpich %gcc@9, Spack will match the compiler specification to an
   existing external. It will fail when the specification is ambiguous, or
   if it does not match any other externals."
harshula added a commit to ACCESS-NRI/spack-config that referenced this pull request Nov 28, 2025
@harshula
common-api-v2/gadi/packages.yaml: externals are treated as concrete s…
976cae7 
…pecs

* As of spack/spack#51118 we need to accommodate:
  "To be backward compatible with external specs specifying a compiler, for
   instance mpich %gcc@9, Spack will match the compiler specification to an
   existing external. It will fail when the specification is ambiguous, or
   if it does not match any other externals."
@becker33 becker33 mentioned this pull request Dec 8, 2025
Closed
@alalazo alalazo mentioned this pull request Dec 15, 2025
Merged
@sethrj sethrj mentioned this pull request Dec 29, 2025
Closed
harshula added a commit to ACCESS-NRI/spack-config that referenced this pull request Jan 28, 2026
@harshula
common-api-v2/gadi/packages.yaml: externals are treated as concrete s…
e070b4e 
…pecs

* As of spack/spack#51118 we need to accommodate:
  "To be backward compatible with external specs specifying a compiler, for
   instance mpich %gcc@9, Spack will match the compiler specification to an
   existing external. It will fail when the specification is ambiguous, or
   if it does not match any other externals."
harshula added a commit to ACCESS-NRI/spack-config that referenced this pull request Feb 6, 2026
@harshula
common-api-v2/gadi/packages.yaml: externals are treated as concrete s…
e2d7669 
…pecs

* As of spack/spack#51118 we need to accommodate:
  "To be backward compatible with external specs specifying a compiler, for
   instance mpich %gcc@9, Spack will match the compiler specification to an
   existing external. It will fail when the specification is ambiguous, or
   if it does not match any other externals."
@harshula harshula mentioned this pull request Feb 6, 2026
Merged
harshula added a commit to ACCESS-NRI/spack-config that referenced this pull request Feb 6, 2026
@harshula
common-api-v2/gadi/packages.yaml: externals are treated as concrete s…
567b698 
…pecs (#99)

* As of spack/spack#51118 we need to accommodate:
  "To be backward compatible with external specs specifying a compiler, for
   instance mpich %gcc@9, Spack will match the compiler specification to an
   existing external. It will fail when the specification is ambiguous, or
   if it does not match any other externals."
* We can longer specify only the compiler name. If we specify the compiler 
  name, then we must include a compiler version.
harshula added a commit to ACCESS-NRI/spack-config that referenced this pull request Feb 6, 2026
@harshula
common-api-v2/gadi/packages.yaml: externals treated as concrete specs (…
ea5ee55 
…#99)

* As of spack/spack#51118 we need to accommodate:
  "To be backward compatible with external specs specifying a compiler, for
   instance mpich %gcc@9, Spack will match the compiler specification to an
   existing external. It will fail when the specification is ambiguous, or
   if it does not match any other externals."
* We can no longer specify only the compiler name. If we specify the compiler
  name, then we must include a compiler version.
harshula added a commit to ACCESS-NRI/spack-config that referenced this pull request Feb 6, 2026
@harshula
common-api-v2/gadi/packages.yaml: externals treated as concrete specs (…
6c3df35 
…#99)

* As of spack/spack#51118 we need to accommodate:
  "To be backward compatible with external specs specifying a compiler, for
   instance mpich %gcc@9, Spack will match the compiler specification to an
   existing external. It will fail when the specification is ambiguous, or
   if it does not match any other externals."
* We can no longer specify only the compiler name. If we specify the
  compiler name, then we must include a compiler version.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment