go,gcc: Support external Go compilers for Go bootstrap#27769
go,gcc: Support external Go compilers for Go bootstrap#27769trws merged 1 commit intospack:developfrom
Conversation
c498f73 to
1b82bb8
Compare
1b82bb8 to
f4c5b6b
Compare
michaelkuhn
left a comment
There was a problem hiding this comment.
I haven't actually tried this yet, just two small comments up front.
scheibelp
left a comment
There was a problem hiding this comment.
This looks good - I'm wondering what combos you have tried in terms of:
- arm vs. not-arm
- using gcc vs. go-bootstrap as the golang-bootstrap provider
- using external go-bootstrap vs. not
I'm also wondering if having go-bootstrap and golang-bootstrap will be confusing, but I think it is fairly.
Also, have you run spack external find on a system with go already installed? Originally I thought there might be an issue with using the same executable names in executables for two packages (in this case go and go-bootstrap) but I'm pretty sure I was wrong about that (my guess based on another look at the code is that it will add an external entry for each, both with the same prefix).
I confirm: I tested each of these points. I tested bootstrapping There is also no problem on running e.g. even both And it also is still possible to use the external So yes, all these cases work. |
|
@michaelkuhn I adressed your comments. Would you review/approve? |
|
@michaelkuhn were there any lasting blockers to this PR? I've realized there isn't a way to install Go on an Arm Mac without this support for externally bootstrapping Spack. |
|
Other than new conflicts I don't recall there being an issue here, I recall approving actually. |
f7c58a4 to
a965579
Compare
|
@alecbcs @trws I resolved the conflicts with current develop and merged the commits into one commit, please review/approve. @michaelkuhn @michaelkuhn We also need your reviews to go into approved state to merge this since you added comments. |
|
I had a problem triggering the pipeline. |
|
Just tested this on an M1 MacBook and I'm actually seeing |
a990106 to
39cb050
Compare
|
@reviewers: The diffs behind the "Compare" buttons are a bit spammed - sorry, commits themselfes are fixed now.
@alecbcs Yes, the use of This answer shows the extent of the chaos: Spack has and uses It was totally unsuitable to use platform.machine() in #30350, but unfortunately the use of platform.machine() is endemic in spack, so the proliferation if it appears to happen just by example. I also didn't know about this problem until I looked it up now. If possible, we should add a stage to Even |
|
I'm not sure, that sounds like an archspec artifact if anything, for this purpose I would think they're the same but I'm not sure that's always true.
|
@trws: 👍 if self.spec.target.family == "aarch64":
depends_on("go-external-or-gccgo-bootstrap", type="build")
else:
...Of course depends_on("go-bootstrap", when="target!=aarch64:", type="build")
depends_on("go-external-or-gccgo-bootstrap", when="target=aarch64:", type="build")So I'm a but stuck at the moment. Maybe just use |
39cb050 to
2524f93
Compare
For ARM64, fallback to gccgo. ([email protected] can't support ARM64)
2524f93 to
eac875b
Compare
|
JFYI: eac875b only merged the commits to have no broken commit. |
* hypre 2.26.0 (spack#33299) * Github Discussions can be used for Q&A (spack#33364) * Support spackbot rebuilding all specs from source (spack#32596) Support spackbot rebuilding all specs from source when asked (with "rebuild everything") - Allow overriding --prune-dag cli opt with env var - Use job variable to optionally prevent rebuild jobs early exit behavior - ci rebuild: Use new install argument to insist deps are always installed from binary, but package is only installed from source. - gitlab: fix bug w/ untouched pruning - ci rebuild: install from hash rather than json file - When doing a "rebuild everything" pipeline, make sure that each install job only consumes binary dependencies from the mirror being populated by the current pipeline. This avoids using, e.g. binaries from develop, when rebuilding everything on a PR. - When running a pipeline to rebuild everything, do not die because we generated a hash on the broken specs list. Instead only warn in that case. - bugfix: Replace broken no-args tty.die() with sys.exit(1) * python: add 3.10.7, 3.9.14, 3.8.14, 3.7.14 (spack#32623) * Add checksum for py-secretstorage 3.3.3 (spack#33366) * Add checksum for py-ipykernel 6.15.2 (spack#33360) * GnuPG: add v2.3.8 and update stack (spack#33368) * installer.py: traverse_dependencies has local deptype (spack#33367) Currently `traverse_dependencies` fixes deptypes to traverse once and for all in the recursion, but this is incorrect, since deptypes depend on the node (e.g. if it's a dependency and cache-only, don't follow build type edges, even if the parent is build from sources and needs build deps.) * py-horovod: add v0.26 (spack#33311) * py-horovod: add v0.26 * py-petastorm: add v0.12.0 * database: don't warn adding missing build deps (spack#33361) When installing an individual spec `spack --only=package --cache-only /xyz` from a buildcache, Spack currently issues tons of warnings about missing build deps (and their deps) in the database. This PR disables these warnings, since it's fine to have a spec without its build deps in the db (they are just "missing"). * Classic Intel compilers do not support gcc-toolchain (spack#33281) * Classic Intel compilers do not support gcc-toolchain This fix removes `--gcc-toolchain=` from the ~.fcg` files for the classic Intel compilers. AFAIK this option is only supported for Clang based compilers. This lead to an issue when installing cmake. Reproducer: ``` spack install [email protected]%[email protected]~doc+ncurses+ownlibs~qt build_type=Release arch=linux-amzn2-skylake_avx512 ``` Tagging maintainer @rscohn2 * Add `-gcc-name` for icc .. and `-gxx-name` for icpc. AFAIK this is used for modern C++ support, so we can ignore `ifort`. Co-authored-by: Stephen Sachs <[email protected]> * py-xopen: version bump to 1.6.0 (spack#33231) * version bump to 1.6.0 * added py-isal, updated URL * Initial contribution of LibPressio ecosystem (spack#32630) * Add libpressio and dependencies; some of these packages are maintained as forks of the original repositories and in those cases the docstring mentions this. * Add optional dependency in adios2 on libpressio * cub package: set CUB_DIR environment variable for dependent installations * Clear R_HOME/R_ENVIRON before Spack installation (avoid sources outside of Spack from affecting the installation in Spack) * Rename dlib to dorian3d-dlib and update dependents; add new dlib implementation. Pending an official policy on how to handle packages with short names, reviewer unilaterally decided that the rename was acceptable given that the new Spack dlib package is referenced more widely (by orders of magnitude) than the original Co-authored-by: Samuel Li <[email protected]> * New packages: libbigwig, methyldackel (spack#33273) * libbigwig: adding new package libbigwig * methyldackel: adding new package methyldackel * libbigwig: tighten up curl variant * grid: reference `fftw-api` instead of `fftw` (spack#33374) This makes it possible to compile with, e.g., `cray-fftw`, not just `fftw`. * Bugfix HIP and aluminum rocm build (spack#33344) * Fixed two bugs in the HIP package recipe. The first is that the HIP_PATH was being set to the actual spec, and not the spec prefix. The second bug is that HIP is expected to be in /opt/rocm-x.y.z/hip but it's libraries can exist at both /opt/rocm-x.y.z/hip/lib and /opt/rocm-x.y.z/lib. This means that the external detection logic may find it in either and it turns out that some modules only expose one of those two locations. Logic is added to ensure that the internal HIP_PATH and associated ROCM_PATH are correctly set in both cases. * Added support for Aluminum to use the libfabric plugin with either RCCL or NCCL. * lz4: add 1.9.4 (spack#33334) * e4s ci stack: add trilinos +rocm (spack#31601) * py-fiona: add v1.8.22 (spack#33372) * intel-oneapi-compilers: fix Python 2.7 compliance (spack#33383) * build(deps): bump docker/setup-buildx-action from 2.1.0 to 2.2.0 (spack#33384) Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 2.1.0 to 2.2.0. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](docker/setup-buildx-action@95cb08c...c74574e) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * mothur: add v1.48.0 and variants (spack#33326) * vsearch: add v2.22.1 (spack#33327) * papi: fix for Intel OneAPI compiler (spack#33225) Without this patch one hits this error trying to compiler papi with Intel OneAPI: icx: error: Note that use of '-g' without any optimization-level option will turn off most compiler optimizations similar to use of '-O0' [-Werror,-Wdebug-disables-optimization] Signed-off-by: Howard Pritchard <[email protected]> Signed-off-by: Howard Pritchard <[email protected]> * go,gcc: Support external go compilers for Go bootstrap (spack#27769) For ARM64, fallback to gccgo. ([email protected] can't support ARM64) * Reusable --use-buildcache with better validation (spack#33388) Co-authored-by: Massimiliano Culpo <[email protected]> * Docs: Spack info option updates (spack#33376) * intel-oneapi-compilers: do not pass -Wno-unused-command-line-argument to icc + refactor (spack#33389) * spack checksum: warn if version is deprecated (spack#32438) Co-authored-by: Massimiliano Culpo <[email protected]> * Update the binary index before attempting direct fetches (spack#32137) "spack install" will not update the binary index if given a concrete spec, which causes it to fall back to direct fetches when a simple index update would have helped. For S3 buckets in particular, this significantly and needlessly slows down the install process. This commit alters the logic so that the binary index is updated whenever a by-hash lookup fails. The lookup is attempted again with the updated index before falling back to direct fetches. To avoid updating too frequently (potentially once for each spec being installed), BinaryCacheIndex.update now includes a "cooldown" option, and when this option is enabled it will not update more than once in a cooldown window (set in config.yaml). Co-authored-by: Tamara Dahlgren <[email protected]> * Relocate "run" type dependencies too (spack#33191) When downloading from binary cache not only replace RPATHs to dependencies, but also text references to dependencies. Example: `[email protected]` contains a text reference to the executable of its dependency `perl`: ``` $ grep perl-5 /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/autoconf-2.69-q3lo/bin/autoreconf eval 'case $# in 0) exec /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/perl-5.34.1-yphg/bin/perl -S "$0";; *) exec /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/perl-5.34.1-yphg/bin/perl -S "$0" "$@";; esac' ``` These references need to be replace or any package using `autoreconf` will fail as it cannot find the installed `perl`. Co-authored-by: Stephen Sachs <[email protected]> * Add a command to bootstrap Spack right now (spack#33407) * axom: python only reliably available when +python, +devtools (spack#33414) * pilercr: new package (spack#33251) * new package * fixed style * actually building now * raja +rocm: use hipcc as CMAKE_CXX_COMPILER (spack#33375) Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Howard Pritchard <[email protected]> Co-authored-by: Rui Peng Li <[email protected]> Co-authored-by: Harmen Stoppels <[email protected]> Co-authored-by: Scott Wittenburg <[email protected]> Co-authored-by: Carlos Bederián <[email protected]> Co-authored-by: iarspider <[email protected]> Co-authored-by: Massimiliano Culpo <[email protected]> Co-authored-by: Adam J. Stewart <[email protected]> Co-authored-by: Stephen Sachs <[email protected]> Co-authored-by: Stephen Sachs <[email protected]> Co-authored-by: Robert Underwood <[email protected]> Co-authored-by: Samuel Li <[email protected]> Co-authored-by: snehring <[email protected]> Co-authored-by: Mosè Giordano <[email protected]> Co-authored-by: Brian Van Essen <[email protected]> Co-authored-by: Michael Kuhn <[email protected]> Co-authored-by: eugeneswalker <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Howard Pritchard <[email protected]> Co-authored-by: Bernhard Kaindl <[email protected]> Co-authored-by: Tamara Dahlgren <[email protected]> Co-authored-by: Robert Cohn <[email protected]> Co-authored-by: Jonathon Anderson <[email protected]>
* hypre 2.26.0 (spack#33299) * Github Discussions can be used for Q&A (spack#33364) * Support spackbot rebuilding all specs from source (spack#32596) Support spackbot rebuilding all specs from source when asked (with "rebuild everything") - Allow overriding --prune-dag cli opt with env var - Use job variable to optionally prevent rebuild jobs early exit behavior - ci rebuild: Use new install argument to insist deps are always installed from binary, but package is only installed from source. - gitlab: fix bug w/ untouched pruning - ci rebuild: install from hash rather than json file - When doing a "rebuild everything" pipeline, make sure that each install job only consumes binary dependencies from the mirror being populated by the current pipeline. This avoids using, e.g. binaries from develop, when rebuilding everything on a PR. - When running a pipeline to rebuild everything, do not die because we generated a hash on the broken specs list. Instead only warn in that case. - bugfix: Replace broken no-args tty.die() with sys.exit(1) * python: add 3.10.7, 3.9.14, 3.8.14, 3.7.14 (spack#32623) * Add checksum for py-secretstorage 3.3.3 (spack#33366) * Add checksum for py-ipykernel 6.15.2 (spack#33360) * GnuPG: add v2.3.8 and update stack (spack#33368) * installer.py: traverse_dependencies has local deptype (spack#33367) Currently `traverse_dependencies` fixes deptypes to traverse once and for all in the recursion, but this is incorrect, since deptypes depend on the node (e.g. if it's a dependency and cache-only, don't follow build type edges, even if the parent is build from sources and needs build deps.) * py-horovod: add v0.26 (spack#33311) * py-horovod: add v0.26 * py-petastorm: add v0.12.0 * database: don't warn adding missing build deps (spack#33361) When installing an individual spec `spack --only=package --cache-only /xyz` from a buildcache, Spack currently issues tons of warnings about missing build deps (and their deps) in the database. This PR disables these warnings, since it's fine to have a spec without its build deps in the db (they are just "missing"). * Classic Intel compilers do not support gcc-toolchain (spack#33281) * Classic Intel compilers do not support gcc-toolchain This fix removes `--gcc-toolchain=` from the ~.fcg` files for the classic Intel compilers. AFAIK this option is only supported for Clang based compilers. This lead to an issue when installing cmake. Reproducer: ``` spack install [email protected]%[email protected]~doc+ncurses+ownlibs~qt build_type=Release arch=linux-amzn2-skylake_avx512 ``` Tagging maintainer @rscohn2 * Add `-gcc-name` for icc .. and `-gxx-name` for icpc. AFAIK this is used for modern C++ support, so we can ignore `ifort`. Co-authored-by: Stephen Sachs <[email protected]> * py-xopen: version bump to 1.6.0 (spack#33231) * version bump to 1.6.0 * added py-isal, updated URL * Initial contribution of LibPressio ecosystem (spack#32630) * Add libpressio and dependencies; some of these packages are maintained as forks of the original repositories and in those cases the docstring mentions this. * Add optional dependency in adios2 on libpressio * cub package: set CUB_DIR environment variable for dependent installations * Clear R_HOME/R_ENVIRON before Spack installation (avoid sources outside of Spack from affecting the installation in Spack) * Rename dlib to dorian3d-dlib and update dependents; add new dlib implementation. Pending an official policy on how to handle packages with short names, reviewer unilaterally decided that the rename was acceptable given that the new Spack dlib package is referenced more widely (by orders of magnitude) than the original Co-authored-by: Samuel Li <[email protected]> * New packages: libbigwig, methyldackel (spack#33273) * libbigwig: adding new package libbigwig * methyldackel: adding new package methyldackel * libbigwig: tighten up curl variant * grid: reference `fftw-api` instead of `fftw` (spack#33374) This makes it possible to compile with, e.g., `cray-fftw`, not just `fftw`. * Bugfix HIP and aluminum rocm build (spack#33344) * Fixed two bugs in the HIP package recipe. The first is that the HIP_PATH was being set to the actual spec, and not the spec prefix. The second bug is that HIP is expected to be in /opt/rocm-x.y.z/hip but it's libraries can exist at both /opt/rocm-x.y.z/hip/lib and /opt/rocm-x.y.z/lib. This means that the external detection logic may find it in either and it turns out that some modules only expose one of those two locations. Logic is added to ensure that the internal HIP_PATH and associated ROCM_PATH are correctly set in both cases. * Added support for Aluminum to use the libfabric plugin with either RCCL or NCCL. * lz4: add 1.9.4 (spack#33334) * e4s ci stack: add trilinos +rocm (spack#31601) * py-fiona: add v1.8.22 (spack#33372) * intel-oneapi-compilers: fix Python 2.7 compliance (spack#33383) * build(deps): bump docker/setup-buildx-action from 2.1.0 to 2.2.0 (spack#33384) Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 2.1.0 to 2.2.0. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](docker/setup-buildx-action@95cb08c...c74574e) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * mothur: add v1.48.0 and variants (spack#33326) * vsearch: add v2.22.1 (spack#33327) * papi: fix for Intel OneAPI compiler (spack#33225) Without this patch one hits this error trying to compiler papi with Intel OneAPI: icx: error: Note that use of '-g' without any optimization-level option will turn off most compiler optimizations similar to use of '-O0' [-Werror,-Wdebug-disables-optimization] Signed-off-by: Howard Pritchard <[email protected]> Signed-off-by: Howard Pritchard <[email protected]> * go,gcc: Support external go compilers for Go bootstrap (spack#27769) For ARM64, fallback to gccgo. ([email protected] can't support ARM64) * Reusable --use-buildcache with better validation (spack#33388) Co-authored-by: Massimiliano Culpo <[email protected]> * Docs: Spack info option updates (spack#33376) * intel-oneapi-compilers: do not pass -Wno-unused-command-line-argument to icc + refactor (spack#33389) * spack checksum: warn if version is deprecated (spack#32438) Co-authored-by: Massimiliano Culpo <[email protected]> * Update the binary index before attempting direct fetches (spack#32137) "spack install" will not update the binary index if given a concrete spec, which causes it to fall back to direct fetches when a simple index update would have helped. For S3 buckets in particular, this significantly and needlessly slows down the install process. This commit alters the logic so that the binary index is updated whenever a by-hash lookup fails. The lookup is attempted again with the updated index before falling back to direct fetches. To avoid updating too frequently (potentially once for each spec being installed), BinaryCacheIndex.update now includes a "cooldown" option, and when this option is enabled it will not update more than once in a cooldown window (set in config.yaml). Co-authored-by: Tamara Dahlgren <[email protected]> * Relocate "run" type dependencies too (spack#33191) When downloading from binary cache not only replace RPATHs to dependencies, but also text references to dependencies. Example: `[email protected]` contains a text reference to the executable of its dependency `perl`: ``` $ grep perl-5 /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/autoconf-2.69-q3lo/bin/autoreconf eval 'case $# in 0) exec /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/perl-5.34.1-yphg/bin/perl -S "$0";; *) exec /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/perl-5.34.1-yphg/bin/perl -S "$0" "$@";; esac' ``` These references need to be replace or any package using `autoreconf` will fail as it cannot find the installed `perl`. Co-authored-by: Stephen Sachs <[email protected]> * Add a command to bootstrap Spack right now (spack#33407) * axom: python only reliably available when +python, +devtools (spack#33414) * pilercr: new package (spack#33251) * new package * fixed style * actually building now * raja +rocm: use hipcc as CMAKE_CXX_COMPILER (spack#33375) Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Howard Pritchard <[email protected]> Co-authored-by: Rui Peng Li <[email protected]> Co-authored-by: Harmen Stoppels <[email protected]> Co-authored-by: Scott Wittenburg <[email protected]> Co-authored-by: Carlos Bederián <[email protected]> Co-authored-by: iarspider <[email protected]> Co-authored-by: Massimiliano Culpo <[email protected]> Co-authored-by: Adam J. Stewart <[email protected]> Co-authored-by: Stephen Sachs <[email protected]> Co-authored-by: Stephen Sachs <[email protected]> Co-authored-by: Robert Underwood <[email protected]> Co-authored-by: Samuel Li <[email protected]> Co-authored-by: snehring <[email protected]> Co-authored-by: Mosè Giordano <[email protected]> Co-authored-by: Brian Van Essen <[email protected]> Co-authored-by: Michael Kuhn <[email protected]> Co-authored-by: eugeneswalker <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Howard Pritchard <[email protected]> Co-authored-by: Bernhard Kaindl <[email protected]> Co-authored-by: Tamara Dahlgren <[email protected]> Co-authored-by: Robert Cohn <[email protected]> Co-authored-by: Jonathon Anderson <[email protected]>
This PR implements/fixes a TODO for using an external go compiler to bootstrap Go.
It enables using
spack external find go-bootstraporspack external find gcc(with gccgo, gcc's go variant is detected)Being able to use an external Go compiler for bootstrapping
gospeeds up bootstrapping Go on ARM64 a lot:So far, the Go bootstrap was hard-coded in
var/spack/repos/builtin/packages/go/package.py:On aarch64, we bootstraped Go using
gccgo, because Go 1.4 (which is the last Go to boostrap from C) does not support ARM64:Now, we can use
golang-bootstrap, which defaults togo-bootstrapand falls back to gcc (with languages=go) when it is not available when totarget=aarch64andgo-bootstrapis not used as external package.gcc itself can be added as an external package with gccgo installed, or be built from source by spack:
Summary:
aarch64, we can now avoid having to build gccgo(gcc languages=go) from source as bootstrap go compiler when using eitherspack external find go-bootstraporspack external find gcc(with gccgo-10 or newer installed)