Skip to content

Add an audit to prevent virtual packages with variants specified#41747

Merged
haampie merged 12 commits intospack:developfrom
alalazo:audits/variants-in-depends-on
Dec 19, 2023
Merged

Add an audit to prevent virtual packages with variants specified#41747
haampie merged 12 commits intospack:developfrom
alalazo:audits/variants-in-depends-on

Conversation

@alalazo
Copy link
Copy Markdown
Member

@alalazo alalazo commented Dec 18, 2023

Currently, a virtual spec is composed of just a name and a version. When a virtual spec contains other components, such as variants, Spack won't emit warnings or errors but will silently drop them - which is unexpected by users.

This PR adds an audit to catch where we use virtual specs with variants, and fixes the directive with the tools we currently have. Some changes, like the one for fftw-api are definitely repetitive and should be improved later.

@spackbot-app spackbot-app bot added the core PR affects Spack core functionality label Dec 18, 2023
@alalazo
Copy link
Copy Markdown
Member Author

alalazo commented Dec 18, 2023

Failures on develop:

Screenshot from 2023-12-18 21-17-24

The package recipe is much more complicated than it needs to be,
and tried to work around virtuals - which is effectively working
against Spack software model.

This needs to be addressed in a later PR
@alalazo
Copy link
Copy Markdown
Member Author

alalazo commented Dec 18, 2023

@spackbot maintainers

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Dec 18, 2023

Hi @alalazo! I noticed that the following package(s) don't yet have maintainers:

  • elk
  • hpcc
  • qscintilla

Are you interested in adopting any of these package(s)? If so, simply add the following to the package class:

    maintainers("alalazo")

If not, could you contact the developers of this package and see if they are interested? You can quickly see who has worked on a package with spack blame:

$ spack blame elk

Thank you for your help! Please don't add maintainers without their consent.

You don't have to be a Spack expert or package developer in order to be a "maintainer," it just gives us a list of users willing to review PRs or debug issues relating to this package. A package can have multiple maintainers; just add a list of GitHub handles of anyone who wants to volunteer.

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Dec 18, 2023

@patrickb314 can you review this PR?

This PR modifies the following package(s), for which you are listed as a maintainer:

  • arrayfire
  • beatnik
  • berkeleygw
  • clblast
  • hpctoolkit
  • ngspice
  • octopus
  • py-torchgeo

Copy link
Copy Markdown
Contributor

@patrickb314 patrickb314 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. Thanks for the fix.

@tgamblin
Copy link
Copy Markdown
Member

tgamblin commented Dec 18, 2023

@alalazo I think this is fine for now but we should also make variants, etc. actually work on virtuals. There are design questions there -- like whether they should be passed directly through (e.g., mpi+fortran or mpi+cuda could be quite useful if we just passed the variants through) or should the variants of a virtual be well known and defined on the virtual, e.g., you'd have to explicitly expose and map them on the providers:

# map virtual variants to package variants
# they don't have to be the same but it probably makes sense most of the time.
provides("+cuda", when="+cuda")
provides("fortran", when="+fortran")

I think it's probably the latter. We should merge this for now, but also take note of the removed lines in these packages and try to express what they're saying in the DSL in the long term.

@patrickb314
Copy link
Copy Markdown
Contributor

Automatically passing them through would be great. I agree that the virtual variant should probably explicitly map how it passes through. Is there a list of "standard variant names" somewhere?

@tgamblin
Copy link
Copy Markdown
Member

tgamblin commented Dec 18, 2023

@patrickb314: no, but this could be one way to formalize them, at least for virtuals. We have talked about having classes for virtual packages, which could act like contracts and allow us to document what the legal variants, versions, etc. are. We don't have that yet so even this is kind of by convention. We could make a page listing all the variants with well known meanings.

Copy link
Copy Markdown
Member

@mwkrentel mwkrentel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I approve for hpctoolkit.

@alalazo
Copy link
Copy Markdown
Member Author

alalazo commented Dec 19, 2023

We should merge this for now, but also take note of the removed lines in these packages and try to express what they're saying in the DSL in the long term

I agree, and indeed tried to put the same concept briefly in the description. I think we should specify better what virtuals are, and ideally:

  1. Define which concrete versions are available for the virtual
  2. Define which variants are associated with the virtual

Each implementation can then provide a specific version range and set of variants. Also, each implementation might also be able to provide all the values of a virtual variant with the same installation (e.g. if it installs two different libraries to be used under different circumstances).

We should discuss this, but I suspect we'll end up formalizing having classes for virtual packages too.

@haampie haampie merged commit 0eca79e into spack:develop Dec 19, 2023
@alalazo alalazo deleted the audits/variants-in-depends-on branch December 19, 2023 17:10
fangohr added a commit to fangohr/spack-ci-octopus that referenced this pull request Dec 19, 2023
(in the meantime, this merge request spack/spack#41747
might affect the result of the compilation for the development branch of spack.)
RikkiButler20 pushed a commit to RikkiButler20/spack that referenced this pull request Jan 31, 2024
…ck#41747)

Currently, a virtual spec is composed of just a name and a version. When a virtual spec contains other components, such as variants, Spack won't emit warnings or errors but will silently drop them - which is unexpected by users.
iamashwin99 added a commit to fangohr/octopus-in-spack that referenced this pull request Mar 17, 2024
Includes changes from :
- spack/spack#41747
- spack/spack#41003
- spack/spack#41919
- spack/spack#40685
- Add hash for octopus 14 and Update BerkeleyGW dependency version in Octopus package #101 ( done at octopus: Support new version octopus@14  spack/spack#43160)
push changes in Disable gdlib #90 (done at octopus: disable gdlib by default spack/spack#43161)
- include new maintainer spack/spack#43163
kwryankrattiger pushed a commit to kwryankrattiger/spack that referenced this pull request Jul 9, 2024
…ck#41747)

Currently, a virtual spec is composed of just a name and a version. When a virtual spec contains other components, such as variants, Spack won't emit warnings or errors but will silently drop them - which is unexpected by users.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core PR affects Spack core functionality dependencies python update-package

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants