Summary

I have had multiple issues over the years where I would have liked for Spack to support external dependencies of external packages.

I always could workaround, but as toolchain dependency chains grow (cuda, hip, llvm, tbb, etc.), reliably defining external toolchains will become harder.

Rationale

Here is a recap of my experience with external MPI toolchains on production HPC clusters, where an "external toolchain" support would have helped a lot:

• Coherence of dependencies with multiple multi-package external toolchains

  Some HPC clusters provide multiple MPI toolchains, where the compatibility is guaranteed only within a toolchain.

  But if I configure all toolchains simultaneously, Spack would choose the last version of individual external packages, independently of their toolchain compatibility.

  For example, if I configure 4 external packages from 2 toolchains, openmpi@2 ^hwloc@1 and openmpi@4 ^hwloc@2, and I concretize a package like pkg ^openmpi@2 ^hwloc, then Spack will choose the latest hwloc@4 which is incompatible with openmpi@2.

• External toolchain, but allow buildable

  When more flexibility is needed, It'd be nice to be able to configure an external toolchain but leave it as buildable. But currently, Spack can inadvertently mix external and spack-installed packages breaking the toolchain coherence/compatibility.

• External (module) load order

  For example, if I have a package with both hwloc and openmpi as dependencies, Spack currently does not ensure it will load external hwloc before loading external openmpi.

  This can be critical for infrastructures that rely on specific module load orders to enable toolchains. The unload conflict code can especially be an issue when using the wrong order (unloading hwloc can unload a previously loaded openmpi).

There were a couple of Spack versions when it kind of worked (0.15?, 0.17? maybe), but each time it was quickly disabled: I can recall it would cause issues around checks like spec["pkg"] and "pkg" in spec when pkg was actually hidden behind an external. I think I can recall re-use and upstreams issues.

For some time now, I have been using a couple of quick-and-dirty non-friendly workarounds to get reliable software stacks, but it'd be nice if vanilla Spack could support this in some way or another. (for example, separate each toolchain configuration, patch spack, etc.)

I've seen other issues reported around this subject over the years, most recently #38263 about an MPI behind an external intel-oneapi-mkl, or #29298 about hip toolchains.

Description

I think Spack could allow the declaration of dependencies in external specs:

packages:
  zlib:
    externals:
    - spec: [email protected]
      prefix: /usr
  zstd:
    externals:
    - spec: [email protected] +programs compression=zlib ^[email protected]
      prefix: /usr

The new part is ^[email protected].

As of spack 0.20, you get an error Attempted to use external for 'zstd' which does not satisfy any configured external spec, but I think this should be allowed.

And checks like "zlib" in zstd_spec could become valid here. i.e., Looking for declared dependencies of external packages could work.

Additional information

No response

General information

• I have searched the issues of this repo and believe this is not a duplicate