Skip to content

go,gcc: Support external Go compilers for Go bootstrap#27769

Merged
trws merged 1 commit intospack:developfrom
bernhardkaindl:bk/go/bootstrap-external
Oct 18, 2022
Merged

go,gcc: Support external Go compilers for Go bootstrap#27769
trws merged 1 commit intospack:developfrom
bernhardkaindl:bk/go/bootstrap-external

Conversation

@bernhardkaindl
Copy link
Copy Markdown
Contributor

@bernhardkaindl bernhardkaindl commented Dec 3, 2021

This PR implements/fixes a TODO for using an external go compiler to bootstrap Go.

It enables using spack external find go-bootstrap or spack external find gcc (with gccgo, gcc's go variant is detected)

Note: To bootstrap go@17 using gccgo, gccgo needs a recent fix which this PR adds also.

Being able to use an external Go compiler for bootstrapping go speeds up bootstrapping Go on ARM64 a lot:

It avoids having to build gccgo(and its dependencies), while of course still falling back to building gcc langunages=go when needed.

So far, the Go bootstrap was hard-coded in var/spack/repos/builtin/packages/go/package.py:
On aarch64, we bootstraped Go using gccgo, because Go 1.4 (which is the last Go to boostrap from C) does not support ARM64:

-    # TODO: Make non-c self-hosting compilers feasible without backflips
-    # should be a dep on external go compiler
-    if platform.machine() == 'aarch64':
-        depends_on('gcc languages=go', type='build')
-    else:
-        depends_on('go-bootstrap', type='build')
+
+    # Use an external go compiler from packages.yaml/`spack external find go-bootstrap`,
+    # but fallback to build [email protected] or to gcc with languages=go (for aarch64):
+    depends_on('golang-bootstrap', type='build')

Now, we can use golang-bootstrap, which defaults to go-bootstrap and falls back to gcc (with languages=go) when it is not available when to target=aarch64 and go-bootstrap is not used as external package.

gcc itself can be added as an external package with gccgo installed, or be built from source by spack:

etc/spack/defaults/packages.yaml:
       golang: [gcc]
+      golang-bootstrap: [go-bootstrap, gcc]

Summary:

  • On aarch64, we can now avoid having to build gccgo(gcc languages=go) from source as bootstrap go compiler when using either spack external find go-bootstrap or spack external find gcc (with gccgo-10 or newer installed)
  • On other architectures, it avoids the download and build of Go 1.4 using a C compiler, which is also a (smaller) win.

@spack spack deleted a comment from spackbot-app bot Dec 3, 2021
@bernhardkaindl bernhardkaindl force-pushed the bk/go/bootstrap-external branch from c498f73 to 1b82bb8 Compare December 3, 2021 08:07
@spackbot-app spackbot-app bot added the patch label Dec 3, 2021
@bernhardkaindl bernhardkaindl force-pushed the bk/go/bootstrap-external branch from 1b82bb8 to f4c5b6b Compare December 3, 2021 08:17
@bernhardkaindl bernhardkaindl changed the title go,gcc: Support external go compilers for Go bootstrap go,gcc: Support external Go compilers for Go bootstrap Dec 3, 2021
Copy link
Copy Markdown
Member

@michaelkuhn michaelkuhn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't actually tried this yet, just two small comments up front.

Copy link
Copy Markdown
Member

@scheibelp scheibelp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good - I'm wondering what combos you have tried in terms of:

  • arm vs. not-arm
  • using gcc vs. go-bootstrap as the golang-bootstrap provider
  • using external go-bootstrap vs. not

I'm also wondering if having go-bootstrap and golang-bootstrap will be confusing, but I think it is fairly.

Also, have you run spack external find on a system with go already installed? Originally I thought there might be an issue with using the same executable names in executables for two packages (in this case go and go-bootstrap) but I'm pretty sure I was wrong about that (my guess based on another look at the code is that it will add an external entry for each, both with the same prefix).

@scheibelp scheibelp self-assigned this Dec 17, 2021
@bernhardkaindl
Copy link
Copy Markdown
Contributor Author

bernhardkaindl commented Dec 17, 2021

This looks good - I'm wondering what combos you have tried in terms of:

* arm vs. not-arm

* using gcc vs. go-bootstrap as the golang-bootstrap provider

* using external go-bootstrap vs. not

I'm also wondering if having go-bootstrap and golang-bootstrap will be confusing, but I think it is fairly.

Also, have you run spack external find on a system with go already installed? Originally I thought there might be an issue with using the same executable names in executables for two packages (in this case go and go-bootstrap) but I'm pretty sure I was wrong about that (my guess based on another look at the code is that it will add an external entry for each, both with the same prefix).

I confirm: I tested each of these points.

I tested bootstrapping [email protected] with using external go-bootstrap and with gcc, on aarch64 and x86_64.

There is also no problem on running e.g. even both spack external find go and spack external find go-bootstrap when e.g. go-1.16.2 is installed on the system and then compiling a different go version using spack install [email protected]

And it also is still possible to use the external go to to build some go package directly with the external go: bin/spack install gh ^[email protected]

So yes, all these cases work.

@bernhardkaindl
Copy link
Copy Markdown
Contributor Author

@michaelkuhn I adressed your comments. Would you review/approve?

trws
trws previously approved these changes Jan 13, 2022
@bernhardkaindl bernhardkaindl requested review from michaelkuhn and removed request for alalazo, michaelkuhn and tgamblin January 16, 2022 16:17
alecbcs
alecbcs previously approved these changes Oct 15, 2022
@alecbcs
Copy link
Copy Markdown
Member

alecbcs commented Oct 15, 2022

@michaelkuhn were there any lasting blockers to this PR? I've realized there isn't a way to install Go on an Arm Mac without this support for externally bootstrapping Spack.

@trws
Copy link
Copy Markdown
Contributor

trws commented Oct 15, 2022

Other than new conflicts I don't recall there being an issue here, I recall approving actually.

@bernhardkaindl bernhardkaindl dismissed stale reviews from alecbcs and trws via a965579 October 16, 2022 01:47
@bernhardkaindl bernhardkaindl force-pushed the bk/go/bootstrap-external branch from f7c58a4 to a965579 Compare October 16, 2022 01:47
@spackbot-app spackbot-app bot added the core PR affects Spack core functionality label Oct 16, 2022
@bernhardkaindl
Copy link
Copy Markdown
Contributor Author

@alecbcs @trws I resolved the conflicts with current develop and merged the commits into one commit, please review/approve.

@michaelkuhn @michaelkuhn We also need your reviews to go into approved state to merge this since you added comments.

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Oct 16, 2022

I had a problem triggering the pipeline.

@alecbcs
Copy link
Copy Markdown
Member

alecbcs commented Oct 16, 2022

Just tested this on an M1 MacBook and I'm actually seeing platform.machine() report the system architecture as arm64 instead of aarch64 and so it's falling through to go-bootstrap and failing. @trws is there a technical difference between aarch64 and arm64 or should we be reporting the two as equivalent in Spack? (And standardizing on one naming convention vs another?)

@bernhardkaindl bernhardkaindl force-pushed the bk/go/bootstrap-external branch 4 times, most recently from a990106 to 39cb050 Compare October 16, 2022 22:23
@bernhardkaindl
Copy link
Copy Markdown
Contributor Author

bernhardkaindl commented Oct 16, 2022

@reviewers: The diffs behind the "Compare" buttons are a bit spammed - sorry, commits themselfes are fixed now.

Just tested this on an M1 MacBook and I'm actually seeing platform.machine() report the system architecture as arm64 instead of aarch64 and so it's falling through to go-bootstrap and failing. @trws is there a technical difference between aarch64 and arm64 or should we be reporting the two as equivalent in Spack? (And standardizing on one naming convention vs another?)

@alecbcs Yes, the use of platform.machine() is completely wrong. It returns the machine of the Python platform, and because it's from platform, anybody (Microsoft, Apple) can return whatever they want, and do not align this with what Linux returns. Microsoft guys return amd64 and I've seen may reports that the x86_64 python installations running thru Rosetta2 return x86_64. I found two search results (one is sqlalchemy/sqlalchemy#7714) where indeed arm64 was mentioned, so it would be plausible that a python build for MacOS-M1 could return arm64 while Linux python interpreter on the same machine would return aarch64.

This answer shows the extent of the chaos:
https://stackoverflow.com/questions/71548156/how-to-get-architecture-of-running-python-interpreter-on-macos-with-apple-silico

Spack has and uses self.spec.target.family to get the target architecture of the current spec and I presume it is also used for matches of target= in when clauses, so this is what spack should use exclusively everywhere.

It was totally unsuitable to use platform.machine() in #30350, but unfortunately the use of platform.machine() is endemic in spack, so the proliferation if it appears to happen just by example. I also didn't know about this problem until I looked it up now.

If possible, we should add a stage to spack style which checks if platform.machine() is used should be rejected, and reject it to slowly get rid of this trap.

Even import platform could be forbidden because spack has e.g. spec.platform == "linux" to check the OS of the spec.

@trws
Copy link
Copy Markdown
Contributor

trws commented Oct 16, 2022 via email

@bernhardkaindl
Copy link
Copy Markdown
Contributor Author

I'm not sure, that sounds like an archspec artifact if anything, for this purpose I would think they're the same but I'm not sure that's always true.

@trws: 👍
Interesting, I see that archspec is used in spack in a few places. This idea of mine of course cannot work for depends_on():

if self.spec.target.family == "aarch64":
  depends_on("go-external-or-gccgo-bootstrap", type="build")
else:
  ...

Of course self is only present in methods, and this would need the support != operator in the constructor of Spec:

depends_on("go-bootstrap", when="target!=aarch64:", type="build")
depends_on("go-external-or-gccgo-bootstrap", when="target=aarch64:", type="build")

So I'm a but stuck at the moment.

Maybe just use if platform.machine() in ["arm64", "aarch64"]: until we have something nicer.

@bernhardkaindl bernhardkaindl force-pushed the bk/go/bootstrap-external branch from 39cb050 to 2524f93 Compare October 16, 2022 23:56
For ARM64, fallback to gccgo. ([email protected] can't support ARM64)
@bernhardkaindl bernhardkaindl force-pushed the bk/go/bootstrap-external branch from 2524f93 to eac875b Compare October 17, 2022 00:00
@bernhardkaindl
Copy link
Copy Markdown
Contributor Author

JFYI: eac875b only merged the commits to have no broken commit.

@trws trws merged commit 1ae32ff into spack:develop Oct 18, 2022
luke-dt added a commit to dantaslab/spack that referenced this pull request Oct 19, 2022
* hypre 2.26.0 (spack#33299)

* Github Discussions can be used for Q&A (spack#33364)

* Support spackbot rebuilding all specs from source (spack#32596)

Support spackbot rebuilding all specs from source when asked (with "rebuild everything")

- Allow overriding --prune-dag cli opt with env var
- Use job variable to optionally prevent rebuild jobs early exit behavior
- ci rebuild: Use new install argument to insist deps are always installed from binary, but
package is only installed from source.
- gitlab: fix bug w/ untouched pruning
- ci rebuild: install from hash rather than json file
- When doing a "rebuild everything" pipeline, make sure that each install job only consumes
binary dependencies from the mirror being populated by the current pipeline.  This avoids
using, e.g. binaries from develop, when rebuilding everything on a PR.
- When running a pipeline to rebuild everything, do not die because we generated a hash on
the broken specs list.  Instead only warn in that case.
- bugfix: Replace broken no-args tty.die() with sys.exit(1)

* python: add 3.10.7, 3.9.14, 3.8.14, 3.7.14 (spack#32623)

* Add checksum for py-secretstorage 3.3.3 (spack#33366)

* Add checksum for py-ipykernel 6.15.2 (spack#33360)

* GnuPG: add v2.3.8 and update stack (spack#33368)

* installer.py: traverse_dependencies has local deptype (spack#33367)

Currently `traverse_dependencies` fixes deptypes to traverse once and
for all in the recursion, but this is incorrect, since deptypes depend
on the node (e.g. if it's a dependency and cache-only, don't follow
build type edges, even if the parent is build from sources and needs
build deps.)

* py-horovod: add v0.26 (spack#33311)

* py-horovod: add v0.26

* py-petastorm: add v0.12.0

* database: don't warn adding missing build deps (spack#33361)

When installing an individual spec `spack --only=package --cache-only /xyz`
from a buildcache, Spack currently issues tons of warnings about
missing build deps (and their deps) in the database.

This PR disables these warnings, since it's fine to have a spec without
its build deps in the db (they are just "missing").

* Classic Intel compilers do not support gcc-toolchain (spack#33281)

* Classic Intel compilers do not support gcc-toolchain

This fix removes `--gcc-toolchain=` from the ~.fcg` files for the classic Intel
compilers. AFAIK this option is only supported for Clang based compilers.

This lead to an issue when installing cmake. Reproducer:
```
spack install [email protected]%[email protected]~doc+ncurses+ownlibs~qt
build_type=Release arch=linux-amzn2-skylake_avx512
```

Tagging maintainer @rscohn2

* Add `-gcc-name` for icc

.. and `-gxx-name` for icpc.

AFAIK this is used for modern C++ support, so we can ignore `ifort`.

Co-authored-by: Stephen Sachs <[email protected]>

* py-xopen: version bump to 1.6.0 (spack#33231)

* version bump to 1.6.0

* added py-isal, updated URL

* Initial contribution of LibPressio ecosystem (spack#32630)

* Add libpressio and dependencies; some of these packages are
  maintained as forks of the original repositories and in those
  cases the docstring mentions this.
* Add optional dependency in adios2 on libpressio
* cub package: set CUB_DIR environment variable for dependent
  installations
* Clear R_HOME/R_ENVIRON before Spack installation (avoid sources
  outside of Spack from affecting the installation in Spack)
* Rename dlib to dorian3d-dlib and update dependents; add new dlib
  implementation. Pending an official policy on how to handle
  packages with short names, reviewer unilaterally decided that
  the rename was acceptable given that the new Spack dlib package
  is referenced more widely (by orders of magnitude) than the
  original

Co-authored-by: Samuel Li <[email protected]>

* New packages: libbigwig, methyldackel (spack#33273)

* libbigwig: adding new package libbigwig
* methyldackel: adding new package methyldackel
* libbigwig: tighten up curl variant

* grid: reference `fftw-api` instead of `fftw` (spack#33374)

This makes it possible to compile with, e.g., `cray-fftw`, not just `fftw`.

* Bugfix HIP and aluminum rocm build (spack#33344)

* Fixed two bugs in the HIP package recipe.  The first is that the
HIP_PATH was being set to the actual spec, and not the spec prefix.

The second bug is that HIP is expected to be in /opt/rocm-x.y.z/hip
but it's libraries can exist at both /opt/rocm-x.y.z/hip/lib and
/opt/rocm-x.y.z/lib.  This means that the external detection logic may
find it in either and it turns out that some modules only expose one
of those two locations.  Logic is added to ensure that the internal
HIP_PATH and associated ROCM_PATH are correctly set in both cases.

* Added support for Aluminum to use the libfabric plugin with either
RCCL or NCCL.

* lz4: add 1.9.4 (spack#33334)

* e4s ci stack: add trilinos +rocm (spack#31601)

* py-fiona: add v1.8.22 (spack#33372)

* intel-oneapi-compilers: fix Python 2.7 compliance (spack#33383)

* build(deps): bump docker/setup-buildx-action from 2.1.0 to 2.2.0 (spack#33384)

Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 2.1.0 to 2.2.0.
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](docker/setup-buildx-action@95cb08c...c74574e)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* mothur: add v1.48.0 and variants (spack#33326)

* vsearch: add v2.22.1 (spack#33327)

* papi: fix for Intel OneAPI compiler (spack#33225)

Without this patch one hits this error trying to compiler papi with Intel OneAPI:

icx: error: Note that use of '-g' without any optimization-level option will turn off most compiler optimizations similar to use of '-O0' [-Werror,-Wdebug-disables-optimization]

Signed-off-by: Howard Pritchard <[email protected]>

Signed-off-by: Howard Pritchard <[email protected]>

* go,gcc: Support external go compilers for Go bootstrap (spack#27769)

For ARM64, fallback to gccgo. ([email protected] can't support ARM64)

* Reusable --use-buildcache with better validation (spack#33388)

Co-authored-by: Massimiliano Culpo <[email protected]>

* Docs: Spack info option updates (spack#33376)

* intel-oneapi-compilers: do not pass -Wno-unused-command-line-argument to icc + refactor (spack#33389)

* spack checksum: warn if version is deprecated (spack#32438)

Co-authored-by: Massimiliano Culpo <[email protected]>

* Update the binary index before attempting direct fetches (spack#32137)

"spack install" will not update the binary index if given a concrete
spec, which causes it to fall back to direct fetches when a simple
index update would have helped. For S3 buckets in particular, this
significantly and needlessly slows down the install process.

This commit alters the logic so that the binary index is updated
whenever a by-hash lookup fails. The lookup is attempted again with
the updated index before falling back to direct fetches. To avoid
updating too frequently (potentially once for each spec being
installed), BinaryCacheIndex.update now includes a "cooldown"
option, and when this option is enabled it will not update more
than once in a cooldown window (set in config.yaml).

Co-authored-by: Tamara Dahlgren <[email protected]>

* Relocate "run" type dependencies too (spack#33191)

When downloading from binary cache not only replace RPATHs to dependencies, but
also text references to dependencies.

Example:
`[email protected]` contains a text reference to the executable of its dependency
`perl`:

```
$ grep perl-5 /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/autoconf-2.69-q3lo/bin/autoreconf
eval 'case $# in 0) exec /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/perl-5.34.1-yphg/bin/perl -S "$0";; *) exec /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/perl-5.34.1-yphg/bin/perl -S "$0" "$@";; esac'
```

These references need to be replace or any package using `autoreconf` will fail
as it cannot find the installed `perl`.

Co-authored-by: Stephen Sachs <[email protected]>

* Add a command to bootstrap Spack right now (spack#33407)

* axom: python only reliably available when +python, +devtools (spack#33414)

* pilercr: new package (spack#33251)

* new package
* fixed style
* actually building now

* raja +rocm: use hipcc as CMAKE_CXX_COMPILER (spack#33375)

Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Howard Pritchard <[email protected]>
Co-authored-by: Rui Peng Li <[email protected]>
Co-authored-by: Harmen Stoppels <[email protected]>
Co-authored-by: Scott Wittenburg <[email protected]>
Co-authored-by: Carlos Bederián <[email protected]>
Co-authored-by: iarspider <[email protected]>
Co-authored-by: Massimiliano Culpo <[email protected]>
Co-authored-by: Adam J. Stewart <[email protected]>
Co-authored-by: Stephen Sachs <[email protected]>
Co-authored-by: Stephen Sachs <[email protected]>
Co-authored-by: Robert Underwood <[email protected]>
Co-authored-by: Samuel Li <[email protected]>
Co-authored-by: snehring <[email protected]>
Co-authored-by: Mosè Giordano <[email protected]>
Co-authored-by: Brian Van Essen <[email protected]>
Co-authored-by: Michael Kuhn <[email protected]>
Co-authored-by: eugeneswalker <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Howard Pritchard <[email protected]>
Co-authored-by: Bernhard Kaindl <[email protected]>
Co-authored-by: Tamara Dahlgren <[email protected]>
Co-authored-by: Robert Cohn <[email protected]>
Co-authored-by: Jonathon Anderson <[email protected]>
luke-dt added a commit to dantaslab/spack that referenced this pull request Oct 19, 2022
* hypre 2.26.0 (spack#33299)

* Github Discussions can be used for Q&A (spack#33364)

* Support spackbot rebuilding all specs from source (spack#32596)

Support spackbot rebuilding all specs from source when asked (with "rebuild everything")

- Allow overriding --prune-dag cli opt with env var
- Use job variable to optionally prevent rebuild jobs early exit behavior
- ci rebuild: Use new install argument to insist deps are always installed from binary, but
package is only installed from source.
- gitlab: fix bug w/ untouched pruning
- ci rebuild: install from hash rather than json file
- When doing a "rebuild everything" pipeline, make sure that each install job only consumes
binary dependencies from the mirror being populated by the current pipeline.  This avoids
using, e.g. binaries from develop, when rebuilding everything on a PR.
- When running a pipeline to rebuild everything, do not die because we generated a hash on
the broken specs list.  Instead only warn in that case.
- bugfix: Replace broken no-args tty.die() with sys.exit(1)

* python: add 3.10.7, 3.9.14, 3.8.14, 3.7.14 (spack#32623)

* Add checksum for py-secretstorage 3.3.3 (spack#33366)

* Add checksum for py-ipykernel 6.15.2 (spack#33360)

* GnuPG: add v2.3.8 and update stack (spack#33368)

* installer.py: traverse_dependencies has local deptype (spack#33367)

Currently `traverse_dependencies` fixes deptypes to traverse once and
for all in the recursion, but this is incorrect, since deptypes depend
on the node (e.g. if it's a dependency and cache-only, don't follow
build type edges, even if the parent is build from sources and needs
build deps.)

* py-horovod: add v0.26 (spack#33311)

* py-horovod: add v0.26

* py-petastorm: add v0.12.0

* database: don't warn adding missing build deps (spack#33361)

When installing an individual spec `spack --only=package --cache-only /xyz`
from a buildcache, Spack currently issues tons of warnings about
missing build deps (and their deps) in the database.

This PR disables these warnings, since it's fine to have a spec without
its build deps in the db (they are just "missing").

* Classic Intel compilers do not support gcc-toolchain (spack#33281)

* Classic Intel compilers do not support gcc-toolchain

This fix removes `--gcc-toolchain=` from the ~.fcg` files for the classic Intel
compilers. AFAIK this option is only supported for Clang based compilers.

This lead to an issue when installing cmake. Reproducer:
```
spack install [email protected]%[email protected]~doc+ncurses+ownlibs~qt
build_type=Release arch=linux-amzn2-skylake_avx512
```

Tagging maintainer @rscohn2

* Add `-gcc-name` for icc

.. and `-gxx-name` for icpc.

AFAIK this is used for modern C++ support, so we can ignore `ifort`.

Co-authored-by: Stephen Sachs <[email protected]>

* py-xopen: version bump to 1.6.0 (spack#33231)

* version bump to 1.6.0

* added py-isal, updated URL

* Initial contribution of LibPressio ecosystem (spack#32630)

* Add libpressio and dependencies; some of these packages are
  maintained as forks of the original repositories and in those
  cases the docstring mentions this.
* Add optional dependency in adios2 on libpressio
* cub package: set CUB_DIR environment variable for dependent
  installations
* Clear R_HOME/R_ENVIRON before Spack installation (avoid sources
  outside of Spack from affecting the installation in Spack)
* Rename dlib to dorian3d-dlib and update dependents; add new dlib
  implementation. Pending an official policy on how to handle
  packages with short names, reviewer unilaterally decided that
  the rename was acceptable given that the new Spack dlib package
  is referenced more widely (by orders of magnitude) than the
  original

Co-authored-by: Samuel Li <[email protected]>

* New packages: libbigwig, methyldackel (spack#33273)

* libbigwig: adding new package libbigwig
* methyldackel: adding new package methyldackel
* libbigwig: tighten up curl variant

* grid: reference `fftw-api` instead of `fftw` (spack#33374)

This makes it possible to compile with, e.g., `cray-fftw`, not just `fftw`.

* Bugfix HIP and aluminum rocm build (spack#33344)

* Fixed two bugs in the HIP package recipe.  The first is that the
HIP_PATH was being set to the actual spec, and not the spec prefix.

The second bug is that HIP is expected to be in /opt/rocm-x.y.z/hip
but it's libraries can exist at both /opt/rocm-x.y.z/hip/lib and
/opt/rocm-x.y.z/lib.  This means that the external detection logic may
find it in either and it turns out that some modules only expose one
of those two locations.  Logic is added to ensure that the internal
HIP_PATH and associated ROCM_PATH are correctly set in both cases.

* Added support for Aluminum to use the libfabric plugin with either
RCCL or NCCL.

* lz4: add 1.9.4 (spack#33334)

* e4s ci stack: add trilinos +rocm (spack#31601)

* py-fiona: add v1.8.22 (spack#33372)

* intel-oneapi-compilers: fix Python 2.7 compliance (spack#33383)

* build(deps): bump docker/setup-buildx-action from 2.1.0 to 2.2.0 (spack#33384)

Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 2.1.0 to 2.2.0.
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](docker/setup-buildx-action@95cb08c...c74574e)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* mothur: add v1.48.0 and variants (spack#33326)

* vsearch: add v2.22.1 (spack#33327)

* papi: fix for Intel OneAPI compiler (spack#33225)

Without this patch one hits this error trying to compiler papi with Intel OneAPI:

icx: error: Note that use of '-g' without any optimization-level option will turn off most compiler optimizations similar to use of '-O0' [-Werror,-Wdebug-disables-optimization]

Signed-off-by: Howard Pritchard <[email protected]>

Signed-off-by: Howard Pritchard <[email protected]>

* go,gcc: Support external go compilers for Go bootstrap (spack#27769)

For ARM64, fallback to gccgo. ([email protected] can't support ARM64)

* Reusable --use-buildcache with better validation (spack#33388)

Co-authored-by: Massimiliano Culpo <[email protected]>

* Docs: Spack info option updates (spack#33376)

* intel-oneapi-compilers: do not pass -Wno-unused-command-line-argument to icc + refactor (spack#33389)

* spack checksum: warn if version is deprecated (spack#32438)

Co-authored-by: Massimiliano Culpo <[email protected]>

* Update the binary index before attempting direct fetches (spack#32137)

"spack install" will not update the binary index if given a concrete
spec, which causes it to fall back to direct fetches when a simple
index update would have helped. For S3 buckets in particular, this
significantly and needlessly slows down the install process.

This commit alters the logic so that the binary index is updated
whenever a by-hash lookup fails. The lookup is attempted again with
the updated index before falling back to direct fetches. To avoid
updating too frequently (potentially once for each spec being
installed), BinaryCacheIndex.update now includes a "cooldown"
option, and when this option is enabled it will not update more
than once in a cooldown window (set in config.yaml).

Co-authored-by: Tamara Dahlgren <[email protected]>

* Relocate "run" type dependencies too (spack#33191)

When downloading from binary cache not only replace RPATHs to dependencies, but
also text references to dependencies.

Example:
`[email protected]` contains a text reference to the executable of its dependency
`perl`:

```
$ grep perl-5 /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/autoconf-2.69-q3lo/bin/autoreconf
eval 'case $# in 0) exec /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/perl-5.34.1-yphg/bin/perl -S "$0";; *) exec /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/perl-5.34.1-yphg/bin/perl -S "$0" "$@";; esac'
```

These references need to be replace or any package using `autoreconf` will fail
as it cannot find the installed `perl`.

Co-authored-by: Stephen Sachs <[email protected]>

* Add a command to bootstrap Spack right now (spack#33407)

* axom: python only reliably available when +python, +devtools (spack#33414)

* pilercr: new package (spack#33251)

* new package
* fixed style
* actually building now

* raja +rocm: use hipcc as CMAKE_CXX_COMPILER (spack#33375)

Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Howard Pritchard <[email protected]>
Co-authored-by: Rui Peng Li <[email protected]>
Co-authored-by: Harmen Stoppels <[email protected]>
Co-authored-by: Scott Wittenburg <[email protected]>
Co-authored-by: Carlos Bederián <[email protected]>
Co-authored-by: iarspider <[email protected]>
Co-authored-by: Massimiliano Culpo <[email protected]>
Co-authored-by: Adam J. Stewart <[email protected]>
Co-authored-by: Stephen Sachs <[email protected]>
Co-authored-by: Stephen Sachs <[email protected]>
Co-authored-by: Robert Underwood <[email protected]>
Co-authored-by: Samuel Li <[email protected]>
Co-authored-by: snehring <[email protected]>
Co-authored-by: Mosè Giordano <[email protected]>
Co-authored-by: Brian Van Essen <[email protected]>
Co-authored-by: Michael Kuhn <[email protected]>
Co-authored-by: eugeneswalker <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Howard Pritchard <[email protected]>
Co-authored-by: Bernhard Kaindl <[email protected]>
Co-authored-by: Tamara Dahlgren <[email protected]>
Co-authored-by: Robert Cohn <[email protected]>
Co-authored-by: Jonathon Anderson <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants