Skip to content

Improve version, version range, and version list syntax and behavior#36273

Merged
tgamblin merged 83 commits intospack:developfrom
haampie:fix/concrete-version-satisfies
May 6, 2023
Merged

Improve version, version range, and version list syntax and behavior#36273
tgamblin merged 83 commits intospack:developfrom
haampie:fix/concrete-version-satisfies

Conversation

@haampie
Copy link
Copy Markdown
Member

@haampie haampie commented Mar 21, 2023

Version types, parsing and printing

  • The version classes have changed: VersionBase is removed, there is now a
    ConcreteVersion base class. StandardVersion and GitVersion both inherit
    from this.

  • The public api (Version, VersionRange, ver) has changed a bit:

    1. Version produces either StandardVersion or GitVersion instances.
    2. VersionRange produces a ClosedOpenRange, but this shouldn't affect the user.
    3. ver produces any of VersionList, ClosedOpenRange, StandardVersion
      or GitVersion.
  • No unexpected type promotion, so that the following is no longer an identity:
    Version(x) != VersionRange(x, x).

  • VersionList.concrete now returns a version if it contains only a single element
    subtyping ConcreteVersion (i.e. StandardVersion(...) or GitVersion(...))

  • In version lists, the parser turns @x into VersionRange(x, x) instead
    of Version(x).

  • The above also means that ver("x") produces a range, whereas
    ver("=x") produces a StandardVersion. The = is part of VersionList
    syntax.

  • VersionList.__str__ now outputs =x.y.z for specific version entries,
    and x.y.z as a short-hand for ranges x.y.z:x.y.z.

  • Spec.format no longer aliases {version} to {versions}, but pulls the
    concrete version out of the list and prints that -- except when the list is
    is not concrete, then is falls back to {versions} to avoid a pedantic error.
    For projections of concrete specs, {version} should be used to render
    1.2.3 instead of =1.2.3 (which you would get with {versions}).
    The default Spec format string used in Spec.__str__ now uses
    {versions} so that str(Spec(string)) == string holds.

Changes to GitVersion

  • GitVersion is a small wrapper around StandardVersion which enriches it
    with a git ref. It no longer inherits from it.

  • GitVersion always needs to be able to look up an associated Spack version
    if it was not assigned (yet). It throws a VersionLookupError whenever ref_version
    is accessed but it has no means to look up the ref; in the past Spack would
    not error and use the commit sha as a literal version, which was incorrect.

  • GitVersion is never equal to StandardVersion, nor is satisfied by it. This
    is such that we don't lose transitivity. This fixes the following bug on develop
    where git_version_a == standard_version == git_version_b does not imply
    git_version_a == git_version_b. It also ensures equality always implies equal
    hash, which is also currently broken on develop; inclusion tests of a set of
    versions + git versions would behave differently from inclusion tests of a
    list of the same objects.

  • The above means ver("ref=1.2.3) != ver("=1.2.3") could break packages that branch
    on specific versions, but that was brittle already, since the same happens with
    externals: [email protected] suffixes wouldn't be exactly equal either. Instead,
    those checks should be x.satisfies("@1.2.3") which works both for git versions and
    custom version suffixes.

  • GitVersion from commit will now print as <hash>=<version> once the
    git ref is resolved to a spack version. This is for reliability -- version is frozen
    when added to the database and queried later. It also improves performance
    since there is no need to clone all repos of all git versions after spack clean -m
    is run and something queries the database, triggering version comparison, such
    as potentially reuse concretization.

  • The "empty VerstionStrComponent trick" for GitVerison is dropped since it wasn't
    representable as a version string (by design). Instead, it's replaced by git,
    so you get 1.2.3.git.4 (which reads 4 commits after a tag 1.2.3). This means
    that there's an edge case for version schemes 1.1.1, 1.1.1a, since the
    generated git version 1.1.1.git.1 (1 commit after 1.1.1) compares larger
    than 1.1.1a, since a < git are compared as strings. This is currently a
    wont-fix edge case, but if really required, could be fixed by special casing
    the git string.

  • Saved, concrete specs (database, lock file, ...) that only had a git sha as their
    version, but have no means to look the effective Spack version anymore, will
    now see their version mapped to hash=develop. Previously these specs
    would always have their sha literally interpreted as a version string (even when
    it could be looked up). This only applies to databases, lock files and spec.json
    files created before Spack 0.20; after this PR, we always have a Spack version
    associated to the relevant GitVersion).

  • Fixes a bug where previously to_dict / from_dict (de)serialization would not
    reattach the repo to the GitVersion, causing the git hash to be used as a literal
    (bogus) version instead of the resolved version. This was in particularly breaking
    version comparison in the build process on macOS/Windows.

Installing or matching specific versions

  • In the past, spack install [email protected] would install pkg@=3.2 if it was a
    known specific version defined in the package, even when newer patch releases
    3.2.1, 3.2.2, ... were available. This behavior was only there because
    there was no syntax to distinguish between 3.2 and 3.2.1. Since there is
    syntax for this now through pkg@=3.2, the old exact matching behavior is
    removed. This means that spack install [email protected] constrains the pkg version
    to the range 3.2, and spack install pkg@=3.2 constrains it to the specific
    version 3.2.

  • Also in directives such as depends_on("[email protected]") and their when
    conditions conflicts("...", when="@2.3") ranges are ranges, and specific
    version matches require @=2.3..

  • No matching version: in the case [email protected] matches nothing, concretization
    errors. However, if you run spack install pkg@=3.2 and this version
    doesn't exist, Spack will define it; this allows you to install non-registered
    versions.

  • For consistency, you can now do %gcc@10 and let it match a configured
    10.x.y compiler. It errors when there is no matching compiler.
    In the past it was interpreted like a specific gcc@=10 version, which
    would get bootstrapped.

  • When compiler bootstrapping is enabled, %gcc@=10.2.0 can be used to
    bootstrap a specific compiler version.

Other changes

  • Externals, compilers, and develop spec definitions are backwards compatible.
    They are typically defined as [email protected] even though they should be
    saying pkg@=3.2.1. Spack now transforms pkg@3 into pkg@=3 in those cases.

  • Finally, fix strictness of version(...) directive/declaration. It just does a simple
    type check, and now requires strings/integers. Floats are not allowed because
    they are ambiguous str(3.10) == "3.1".

Closes #36183

@spackbot-app spackbot-app bot added core PR affects Spack core functionality versions directives fetching tests General test capability(ies) compilers labels Mar 21, 2023
@haampie
Copy link
Copy Markdown
Member Author

haampie commented Mar 22, 2023

PKG-DIRECTIVES: 3 issues found
1. py-apache-beam: dependency on [email protected]:0.3.1 cannot be satisfied by known versions of py-dill
    happening in /home/runner/work/spack/spack/var/spack/repos/builtin/packages/py-apache-beam/package.py
    known versions of py-dill are 0.3.6, 0.3.5.1, 0.3.4, 0.3.1, 0.2.9, 0.2.7, 0.2.6, 0.2.5, 0.2.4, 0.2.3, 0.2.2, 0.2.1, 0.2
2. py-cutadapt: dependency on [email protected]:0.3 cannot be satisfied by known versions of py-dnaio
    happening in /home/runner/work/spack/spack/var/spack/repos/builtin/packages/py-cutadapt/package.py
    known versions of py-dnaio are 0.9.1, 0.4.2, 0.3
3. py-sphinx-tabs: dependency on [email protected]:0.16 cannot be satisfied by known versions of py-docutils
    happening in /home/runner/work/spack/spack/var/spack/repos/builtin/packages/py-sphinx-tabs/package.py
    known versions of py-docutils are 0.19, 0.18.1, 0.18, 0.17.1, 0.17, 0.16, 0.15.2, 0.14, 0.13.1, 0.12

looks like correct audit results to me, so develop was broken

@haampie
Copy link
Copy Markdown
Member Author

haampie commented Mar 29, 2023

Looking at spack solve --timers cp2k+sirius setup time goes from 8.5s to 7.5s, so that's something.

@haampie haampie force-pushed the fix/concrete-version-satisfies branch from d5a0d01 to ffb87bb Compare March 30, 2023 12:47
@spackbot-app spackbot-app bot added documentation Improvements or additions to documentation python labels Mar 30, 2023
@haampie haampie force-pushed the fix/concrete-version-satisfies branch 2 times, most recently from 4a7a945 to 9f0c66a Compare March 30, 2023 12:52
@haampie haampie marked this pull request as ready for review March 30, 2023 13:43
@tgamblin tgamblin merged commit fa7719a into spack:develop May 6, 2023
@haampie haampie deleted the fix/concrete-version-satisfies branch May 8, 2023 09:47
@snehring snehring mentioned this pull request May 10, 2023
3 tasks
greenc-FNAL added a commit to FNALssi/spack that referenced this pull request May 11, 2023
greenc-FNAL added a commit to FNALssi/spack that referenced this pull request May 11, 2023
greenc-FNAL added a commit to FNALssi/spack that referenced this pull request May 11, 2023
greenc-FNAL added a commit to FNALssi/spack that referenced this pull request May 12, 2023
greenc-FNAL added a commit to FNALssi/spack that referenced this pull request May 17, 2023
greenc-FNAL added a commit to FNALssi/spack that referenced this pull request May 17, 2023
greenc-FNAL added a commit to FNALssi/spack that referenced this pull request May 18, 2023
greenc-FNAL added a commit to FNALssi/spack that referenced this pull request May 18, 2023
brenthuisman added a commit to arbor-sim/arbor that referenced this pull request Jul 10, 2023
- Upstream Spack changes forces any version that will be built to be
explicitly listed in `package.py`. See
spack/spack#36273
- This version is only provided if a local `VERSION` is found (implying
presence of Arbor source). If not, building that version would fail.
- Temporarily disabled recently added Python tests in the workflow file,
because `dev-build`s don't install the package, and thus the tests and
examples won't run. I'm looking into a solution, but don't want to hold
merging this PR up.
@brenthuisman brenthuisman mentioned this pull request Jul 19, 2023
4 tasks
greenc-FNAL added a commit to FNALssi/spack that referenced this pull request Aug 22, 2023
greenc-FNAL added a commit to greenc-FNAL/spack that referenced this pull request Aug 22, 2023
greenc-FNAL added a commit to FNALssi/spack that referenced this pull request Aug 23, 2023
greenc-FNAL added a commit to FNALssi/spack that referenced this pull request Aug 23, 2023
marcmengel pushed a commit to FNALssi/spack that referenced this pull request Aug 23, 2023
* Test directive enhancement

* [geant4] Accommodate spack#36273

* Fix style
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GitVersion.intersects errors sometimes when no =reference is given

5 participants