Skip to content

Two bugs that prevent creating and using bootstrap mirrors with spack develop as of 2024/12/27 (and v0.23) #48163

Closed
#48252
Closed
Two bugs that prevent creating and using bootstrap mirrors with spack develop as of 2024/12/27 (and v0.23)#48163
#48252
Assignees
alalazoclimbfuji
Labels
bugSomething isn't workingimpact-medium
@climbfuji

Description

@climbfuji
climbfuji
opened on Dec 17, 2024

Steps to reproduce

On a system with Internet access (in order to create the bootstrap mirror):

> spack debug report
* **Spack:** 0.24.0.dev0 (60b4882d4e74a77c9ef4fb5f5b41c337c82045a8)
* **Python:** 3.6.15
* **Platform:** linux-sles15-zen2

> spack bootstrap mirror --binary-packages ./bootstrap-mirror
==> Adding "clingo-bootstrap@spack+python %gcc platform=linux target=x86_64" and dependencies to the mirror at /p/work2/heinzell/spack-dev-20241217/bootstrap-mirror/bootstrap_cache
==> Adding "[email protected]: %gcc platform=linux target=x86_64" and dependencies to the mirror at /p/work2/heinzell/spack-dev-20241217/bootstrap-mirror/bootstrap_cache
==> Adding "[email protected]: %gcc platform=linux target=x86_64" and dependencies to the mirror at /p/work2/heinzell/spack-dev-20241217/bootstrap-mirror/bootstrap_cache
==> Adding "file %gcc platform=linux target=x86_64" and dependencies to the mirror at /p/work2/heinzell/spack-dev-20241217/bootstrap-mirror/bootstrap_cache
==> Adding "gnuconfig" and dependencies to the mirror at /p/work2/heinzell/spack-dev-20241217/bootstrap-mirror/bootstrap_cache
==> Adding binary packages from "https://github.com/spack/spack-bootstrap-mirrors/releases/download/v0.4/bootstrap-buildcache.tar.gz" to the mirror at /p/work2/heinzell/spack-dev-20241217/bootstrap-mirror/bootstrap_cache
==> Error: [Errno 2] No such file or directory: '/p/work2/heinzell/spack-dev-20241217/share/spack/bootstrap/github-actions-v0.4/clingo.json'

> grep -Rie v0\.4 lib
lib/spack/docs/getting_started.rst:   ==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.4/build_cache/linux-centos7-x86_64-gcc-10.2.1-clingo-bootstrap-spack-ba5ijauisd3uuixtmactc36vps7yfsrl.spec.json
lib/spack/docs/getting_started.rst:   ==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.4/build_cache/linux-centos7-x86_64/gcc-10.2.1/clingo-bootstrap-spack/linux-centos7-x86_64-gcc-10.2.1-clingo-bootstrap-spack-ba5ijauisd3uuixtmactc36vps7yfsrl.spack
lib/spack/docs/getting_started.rst:   ==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.4/build_cache/linux-centos7-x86_64-gcc-10.2.1-patchelf-0.16.1-p72zyan5wrzuabtmzq7isa5mzyh6ahdp.spec.json
lib/spack/docs/getting_started.rst:   ==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.4/build_cache/linux-centos7-x86_64/gcc-10.2.1/patchelf-0.16.1/linux-centos7-x86_64-gcc-10.2.1-patchelf-0.16.1-p72zyan5wrzuabtmzq7isa5mzyh6ahdp.spack
lib/spack/docs/getting_started.rst:   $ spack bootstrap disable github-actions-v0.4
lib/spack/docs/getting_started.rst:   ==> "github-actions-v0.4" is now disabled and will not be used for bootstrapping
Binary file lib/spack/spack/cmd/__pycache__/bootstrap.cpython-36.pyc matches
lib/spack/spack/cmd/bootstrap.py:BINARY_TARBALL = "https://github.com/spack/spack-bootstrap-mirrors/releases/download/v0.4/bootstrap-buildcache.tar.gz"
lib/spack/spack/cmd/bootstrap.py:CLINGO_JSON = "$spack/share/spack/bootstrap/github-actions-v0.4/clingo.json"
lib/spack/spack/cmd/bootstrap.py:GNUPG_JSON = "$spack/share/spack/bootstrap/github-actions-v0.4/gnupg.json"
lib/spack/spack/cmd/bootstrap.py:PATCHELF_JSON = "$spack/share/spack/bootstrap/github-actions-v0.4/patchelf.json"
lib/spack/spack/test/data/unparse/py-torch.txt:        tag="v0.4.1",
lib/spack/spack/test/data/unparse/py-torch.txt:    version("0.4.0", tag="v0.4.0", submodules=True, deprecated=True)

> sed -i 's/v0.4/v0.5/g' lib/spack/spack/cmd/bootstrap.py

> spack bootstrap mirror --binary-packages ./bootstrap-mirror
==> Adding "clingo-bootstrap@spack+python %gcc platform=linux target=x86_64" and dependencies to the mirror at /p/work2/heinzell/spack-dev-20241217/bootstrap-mirror/bootstrap_cache
==> Adding "[email protected]: %gcc platform=linux target=x86_64" and dependencies to the mirror at /p/work2/heinzell/spack-dev-20241217/bootstrap-mirror/bootstrap_cache
==> Adding "[email protected]: %gcc platform=linux target=x86_64" and dependencies to the mirror at /p/work2/heinzell/spack-dev-20241217/bootstrap-mirror/bootstrap_cache
==> Adding "file %gcc platform=linux target=x86_64" and dependencies to the mirror at /p/work2/heinzell/spack-dev-20241217/bootstrap-mirror/bootstrap_cache
==> Adding "gnuconfig" and dependencies to the mirror at /p/work2/heinzell/spack-dev-20241217/bootstrap-mirror/bootstrap_cache
==> Adding binary packages from "https://github.com/spack/spack-bootstrap-mirrors/releases/download/v0.5/bootstrap-buildcache.tar.gz" to the mirror at /p/work2/heinzell/spack-dev-20241217/bootstrap-mirror/bootstrap_cache

To register the mirror on the platform where it's supposed to be used, move "./bootstrap-mirror" to its final location and run the following command(s):

  % spack bootstrap add --trust local-sources <final-path>/metadata/sources
  % spack bootstrap add --trust local-binaries <final-path>/metadata/binaries

Then, I am copying this version of spack (develop as of 2024/12/17) to the air-gapped system to /p/work1/heinzell/spack-stack/spack-dev-20241217, and the bootstrap mirror to /p/work1/heinzell/spack-stack/bootstrap-mirror. The air-gapped system doesn't have a $HOME/.spack directory yet, this is the first time to run spack there.

On the air-gapped system:

> spack debug report
* **Spack:** 0.24.0.dev0 (60b4882d4e74a77c9ef4fb5f5b41c337c82045a8)
* **Python:** 3.6.15
* **Platform:** linux-sles15-zen4

> spack bootstrap add --trust local-sources /p/work1/heinzell/spack-stack/bootstrap-mirror/metadata/sources
==> New bootstrapping source "local-sources" added in the "user" configuration scope
==> "local-sources" is now enabled for bootstrapping
> spack bootstrap add --trust local-binaries /p/work1/heinzell/spack-stack/bootstrap-mirror/metadata/binaries
==> New bootstrapping source "local-binaries" added in the "user" configuration scope
==> "local-binaries" is now enabled for bootstrapping

> spack bootstrap status
Spack v0.24.0.dev0 - [email protected]

[FAIL] Core Functionalities
  [B] MISSING "clingo": required to concretize specs

[FAIL] Binary packages
  [B] MISSING "patchelf": required to relocate binaries

Spack will take care of bootstrapping any missing dependency marked as [B]. Dependencies marked as [-] are instead required to be found on the system.

> spack bootstrap list
Name: local-binaries ENABLED

  Type: buildcache

  Info:
    url: ../../bootstrap_cache
    homepage: https://github.com/spack/spack-bootstrap-mirrors
    releases: https://github.com/spack/spack-bootstrap-mirrors/releases
    tarball: https://github.com/spack/spack-bootstrap-mirrors/releases/download/v0.5/bootstrap-buildcache.tar.gz

  Description:
    Buildcache copied from a public tarball available on Github.The sha256 checksum of binaries is checked before installation.

Name: local-sources ENABLED

  Type: install

  Info:
    url: ../../bootstrap_cache

  Description:
    Mirror with software needed to bootstrap Spack

Name: github-actions-v0.6 ENABLED

  Type: buildcache

  Info:
    url: oci://ghcr.io/spack/bootstrap-buildcache-v1
    homepage: https://github.com/spack/spack-bootstrap-mirrors
    releases: https://github.com/spack/spack-bootstrap-mirrors/releases

  Description:
    Buildcache generated from a public workflow using GitHub Actions hosted on GitHub Packages.
    The sha256 checksum of binaries is checked before installation.

Name: github-actions-v0.5 ENABLED

  Type: buildcache

  Info:
    url: https://mirror.spack.io/bootstrap/github-actions/v0.5
    homepage: https://github.com/spack/spack-bootstrap-mirrors
    releases: https://github.com/spack/spack-bootstrap-mirrors/releases

  Description:
    Buildcache generated from a public workflow using Github Actions.
    The sha256 checksum of binaries is checked before installation.

Name: spack-install ENABLED

  Type: install

  Info:
    url: https://mirror.spack.io

  Description:
    Specs built from sources downloaded from the Spack public mirror.
        
> spack bootstrap root
/p/home/heinzell/.spack/bootstrap

> spack bootstrap now 2>&1 | tee log.bootstrap.001
[+] /usr (external bison-3.0.4-a4mdd4sch7tspipgpqurmb5m2kojzu5t)
[+] /usr (external cmake-3.20.4-ostdc67uvigqj6qbnlvxdtqf6c22wi2h)
[+] /usr (external glibc-2.31-ypeurjk3ao7mgyz4p763vp66j37wdo77)
[+] /usr (external python-3.6-24wx6hce2s5mafhwumyxr7qsvryimydx)
==> Installing gcc-runtime-13.3.0-awzllsnkbz4vgndlhpxqcn44lhngpukq [5/9]
==> No binary for gcc-runtime-13.3.0-awzllsnkbz4vgndlhpxqcn44lhngpukq found: installing from source
==> No patches needed for gcc-runtime
==> gcc-runtime: Executing phase: 'install'
==> gcc-runtime: Successfully installed gcc-runtime-13.3.0-awzllsnkbz4vgndlhpxqcn44lhngpukq
  Stage: 0.00s.  Install: 0.19s.  Post-install: 0.09s.  Total: 0.32s
[+] /p/home/heinzell/.spack/bootstrap/store/linux-sles15-x86_64/gcc-13.3.0/gcc-runtime-13.3.0-awzllsnkbz4vgndlhpxqcn44lhngpukq
==> Installing python-venv-1.0-6c5iktcwg2zeei4aqh3iprgmvdtzdk5d [6/9]
==> No binary for python-venv-1.0-6c5iktcwg2zeei4aqh3iprgmvdtzdk5d found: installing from source
==> No patches needed for python-venv
==> python-venv: Executing phase: 'install'
==> python-venv: Successfully installed python-venv-1.0-6c5iktcwg2zeei4aqh3iprgmvdtzdk5d
  Stage: 0.00s.  Install: 0.07s.  Post-install: 0.06s.  Total: 0.19s
[+] /p/home/heinzell/.spack/bootstrap/store/linux-sles15-x86_64/gcc-13.3.0/python-venv-1.0-6c5iktcwg2zeei4aqh3iprgmvdtzdk5d
==> Installing gmake-4.4.1-25hu4ufpfdcmke5dfr63ytcb7cvvufvw [7/9]
==> No binary for gmake-4.4.1-25hu4ufpfdcmke5dfr63ytcb7cvvufvw found: installing from source
==> Error: FetchError: All fetchers failed for spack-stage-gmake-4.4.1-25hu4ufpfdcmke5dfr63ytcb7cvvufvw
        file:///p/work1/heinzell/bootstrap_cache/_source-cache/archive/dd/dd16fb1d67bfab79a72f5e8390735c49e3e8e70b4945a15ab1f81ddb78658fb3.tar.gz: URLError: <urlopen error [Errno 2] No such file or directory: '/p/work1/heinzell/bootstrap_cache/_source-cache/archive/dd/dd16fb1d67bfab79a72f5e8390735c49e3e8e70b4945a15ab1f81ddb78658fb3.tar.gz'>
    https://ftpmirror.gnu.org/make/make-4.4.1.tar.gz: URLError: <urlopen error [Errno 101] Network is unreachable>
    https://ftpmirror.gnu.org/make/make-4.4.1.tar.gz: URLError: <urlopen error [Errno 101] Network is unreachable>
    https://ftpmirror.gnu.org/make/make-4.4.1.tar.gz: URLError: <urlopen error [Errno 101] Network is unreachable>
==> Warning: Skipping build of clingo-bootstrap-spack-zdraelqnmaiebssvxme3ls6bxnkx4fuw since gmake-4.4.1-25hu4ufpfdcmke5dfr63ytcb7cvvufvw failed
==> Warning: Skipping build of re2c-3.0-n6fkyitoujjxfoyoanwoxxtbglws6al5 since gmake-4.4.1-25hu4ufpfdcmke5dfr63ytcb7cvvufvw failed

> cat ~/.spack/bootstrap.yaml
bootstrap:
  sources:
  - name: local-binaries
    metadata: /p/work1/heinzell/spack-stack/bootstrap-mirror/metadata/binaries
  - name: local-sources
    metadata: /p/work1/heinzell/spack-stack/bootstrap-mirror/metadata/sources
  trusted:
    local-sources: true
    local-binaries: true

> ls /p/work1/heinzell/spack-stack/bootstrap-mirror/bootstrap_cache/_source-cache/archive/dd/dd16fb1d67bfab79a72f5e8390735c49e3e8e70b4945a15ab1f81ddb78658fb3.tar.gz
/p/work1/heinzell/spack-stack/bootstrap-mirror/bootstrap_cache/_source-cache/archive/dd/dd16fb1d67bfab79a72f5e8390735c49e3e8e70b4945a15ab1f81ddb78658fb3.tar.gz

> pwd
/p/work1/heinzell/spack-stack/spack-dev-20241217

> cd ../..
> pwd
/p/work1/heinzell/
> ln -sf spack-stack/bootstrap-mirror/bootstrap_cache .
heinzell@cole01:/p/work1/heinzell> ls -l
total 8
lrwxrwxrwx  1 heinzell 0379GYH2   44 Dec 17 15:32 bootstrap_cache -> spack-stack/bootstrap-mirror/bootstrap_cache
> cd -
> pwd
/p/work1/heinzell/spack-stack/spack-dev-20241217

> spack bootstrap now 2>&1 | tee log.bootstrap.002
==> Fetching file:///p/work1/heinzell/bootstrap_cache/build_cache/linux-centos7-x86_64-gcc-10.2.1-patchelf-0.18.0-afv7arjarb7nzmlh7c5slkfxykybuqce.spec.json
==> Fetching file:///p/work1/heinzell/bootstrap_cache/build_cache/linux-centos7-x86_64/gcc-10.2.1/patchelf-0.18.0/linux-centos7-x86_64-gcc-10.2.1-patchelf-0.18.0-afv7arjarb7nzmlh7c5slkfxykybuqce.spack
==> Installing "patchelf@=0.18.0%gcc@=10.2.1 ldflags='-static-libstdc++ -static-libgcc'  build_system=autotools arch=linux-centos7-x86_64" from a buildcache
==> Fetching file:///p/work1/heinzell/bootstrap_cache/build_cache/linux-centos7-x86_64-gcc-10.2.1-clingo-bootstrap-spack-t4yf34cuvquqp5xd66zybmcfyhwbdlsf.spec.json
==> Fetching file:///p/work1/heinzell/bootstrap_cache/build_cache/linux-centos7-x86_64/gcc-10.2.1/clingo-bootstrap-spack/linux-centos7-x86_64-gcc-10.2.1-clingo-bootstrap-spack-t4yf34cuvquqp5xd66zybmcfyhwbdlsf.spack
==> Installing "clingo-bootstrap@=spack%gcc@=10.2.1~docs+ipo+optimized+python+static_libstdcpp build_system=cmake build_type=Release generator=make patches=bebb819,ec99431 arch=linux-centos7-x86_64" from a buildcache

> spack bootstrap status
Spack v0.24.0.dev0 - [email protected]

[PASS] Core Functionalities

[PASS] Binary packages

Error message

The error is described in detail in the above "Steps to reproduce" box. The problem is likely the relative url entry in the spack bootstrap list:

> spack bootstrap list
Name: local-binaries ENABLED

  Type: buildcache

  Info:
    url: ../../bootstrap_cache

Information on your system

See above. I reported a similar error for v0.23 in #47827. This was flagged as "impact-low", which I don't understand. Not being able to bootstrap spack on an air-gapped system is a show stopper, and the problem isn't only in v0.23, but also in develop as of 2024/12/17.

General information

  • I have run spack debug report and reported the version of Spack/Python/Platform
  • I have searched the issues of this repo and believe this is not a duplicate
  • I have run the failing commands in debug mode and reported the output

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingimpact-medium

Type

No type

Projects

Triaged bugs

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions