Add macOS ML CI stacks by adamjstewart · Pull Request #36586 · spack/spack

adamjstewart · 2023-03-31T18:20:52Z

See #31551

share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml

eugeneswalker · 2023-04-05T21:45:21Z

Curl failed in mps pipeline:

(specs) curl/e75gmhu 7.88.1 [email protected] darwin-ventura-aarch64 Machine Learning MPS

But succeeded in cpu pipeline:

(specs) curl/e75gmhu 7.88.1 [email protected] darwin-ventura-aarch64 Machine Learning

From the failed build:

...
configure: error: one or more libs available at link-time are not available run-time. Libs used at link-time: -lidn2  -lz 
==> Error: ProcessError: Command exited with status 1:
    '/var/folders/6n/pw9tmf6n08dfgg5td4_w9wh00000gx/T/gitlab-runner/spack-stage/spack-stage-curl-7.88.1-e75gmhuqyanoxw7lvl6t4vt7hqq7u6i4/spack-src/configure' '--prefix=/Users/gitlab-runner/builds/yezPTxFu/0/spack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spac/morepadding/darwin-ventura-aarch64/apple-clang-14.0.0/curl-7.88.1-e75gmhuqyanoxw7lvl6t4vt7hqq7u6i4' '--with-zlib=/Users/gitlab-runner/builds/yezPTxFu/0/spack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spac/darwin-ventura-aarch64/apple-clang-14.0.0/zlib-1.2.13-2iaqlis6z5ogtxwuupsg2o7mli5kxycf' '--without-brotli' '--without-libgsasl' '--without-libpsl' '--without-zstd' '--enable-shared' '--enable-static' '--without-gssapi' '--without-gnutls' '--without-mbedtls' '--without-nss' '--without-openssl' '--with-secure-transport' '--without-sspi' '--with-libidn2=/Users/gitlab-runner/builds/yezPTxFu/0/spack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spac/darwin-ventura-aarch64/apple-clang-14.0.0/libidn2-2.3.4-77ws22fcd5jtyi77aue2sa24iu7tqqr3' '--without-librtmp' '--without-nghttp2' '--without-libssh2' '--without-libssh' '--disable-ldap'
1 error found in build log:
     262    checking for size_t... yes
     263    checking for sys/types.h... (cached) yes
     264    checking for sys/time.h... (cached) yes
     265    checking for sys/socket.h... (cached) yes
     266    checking for struct timeval... yes
     267    checking run-time libs availability... failed
  >> 268    configure: error: one or more libs available at link-time are not a
            vailable run-time. Libs used at link-time: -lidn2  -lz

Notice the hash is the same, and both of those jobs ran on the same runner uo-lambda. I would expect if the same package is being built on the same runner, the result would be the same...

eugeneswalker · 2023-04-10T20:10:53Z

@spackbot run pipeline

spackbot-app · 2023-04-10T20:11:03Z

I've started that pipeline for you!

eugeneswalker · 2023-04-13T17:31:50Z

The failures here look like mostly, if not entirely, relocation issues:

TEXINFO

checking for perl... /Users/gitlab-runner/builds/yezPTxFu/0/spack/spack/opt/spack/[padded-to-256-chars]/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/bin/perl
checking Perl version and modules... /var/folders/6n/pw9tmf6n08dfgg5td4_w9wh00000gx/T/gitlab-runner/spack-stage/spack-stage-texinfo-7.0-iiakzqw4vf2w6bxmy5bngzomklkdsxgr/spack-src/configure: line 5034: 80028 Abort trap: 6           $PERL -e "use 5.008_001; use Encode; use Data::Dumper" > /dev/null 2>&1
no
configure: error: perl >= 5.8.1 with Encode and Data::Dumper required by Texinfo.

PERL-DATA-DUMPER

dyld[78828]: Library not loaded: /Users/gitlab-runner/builds/yezPTxFu/0/spack/spack/opt/spack/[padded-to-256-chars]/morepadding/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib/5.36.0/darwin-thread-multi-2level/CORE/libperl.dylib
  Referenced from: <0A51BC43-CDA1-369A-B7D5-E1AD4480232C> /Users/gitlab-runner/builds/yezPTxFu/0/spack/spack/opt/spack/[padded-to-256-chars]/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/bin/perl
  Reason: tried: '/Users/gitlab-runner/builds/yezPTxFu/0/spack/spack/opt/spack/[padded-to-256-chars]/morepadding/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib/5.36.0/darwin-thread-multi-2level/CORE/libperl.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/gitlab-runner/builds/yezPTxFu/0/spack/spack/opt/spack/[padded-to-292-chars]/morepadding/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib/5.36.0/darwin-thread-multi-2level/CORE/libperl.dylib' (no such file), '/Users/gitlab-runner/builds/yezPTxFu/0/spack/spack/opt/spack/[padded-to-256-chars]/morepadding/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib/5.36.0/darwin-thread-multi-2level/CORE/libperl.dylib' (no such file), '/usr/local/lib/libperl.dylib' (no such file), '/usr/lib/libperl.dylib' (no such file, not in dyld cache)

AUTOCONF

     26    dyld[78133]: Library not loaded: /Users/gitlab-runner/builds/yezPTxF
           u/0/spack/spack/opt/spack/__spack_path_placeholder__/__spack_path_pl
           aceholder__/__spack_path_placeholder__/__spack_path_placeholder__/__
           spack_path_placeholder__/__spack_path_placeholder__/__spack_path_pla
           ceholder__/__spac/morepadding/darwin-ventura-aarch64/apple-clang-14.
           0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib/5.36.0/darwin-t
           hread-multi-2level/CORE/libperl.dylib
     27      Referenced from: <0A51BC43-CDA1-369A-B7D5-E1AD4480232C> /Users/git
           lab-runner/builds/yezPTxFu/0/spack/spack/opt/spack/__spack_path_plac
           eholder__/__spack_path_placeholder__/__spack_path_placeholder__/__sp
           ack_path_placeholder__/__spack_path_placeholder__/__spack_path_place
           holder__/__spack_path_placeholder__/__spac/darwin-ventura-aarch64/ap
           ple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/bin/pe
           rl
     28      Reason: tried: '/Users/gitlab-runner/builds/yezPTxFu/0/spack/spack
           /opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__s
           pack_path_placeholder__/__spack_path_placeholder__/__spack_path_plac
           eholder__/__spack_path_placeholder__/__spack_path_placeholder__/__sp
           ac/morepadding/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0
           -i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib/5.36.0/darwin-thread-multi-2le
           vel/CORE/libperl.dylib' (no such file), '/System/Volumes/Preboot/Cry
           ptexes/OS/Users/gitlab-runner/builds/yezPTxFu/0/spack/spack/opt/spac
           k/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path
           _placeholder__/__spack_path_placeholder__/__spack_path_placeholder__
           /__spack_path_placeholder__/__spack_path_placeholder__/__spac/morepa
           dding/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47d
           ed7p5o2fkwimbhsw6rxxfip3/lib/5.36.0/darwin-thread-multi-2level/CORE/
           libperl.dylib' (no such file), '/Users/gitlab-runner/builds/yezPTxFu
           /0/spack/spack/opt/spack/__spack_path_placeholder__/__spack_path_pla
           ceholder__/__spack_path_placeholder__/__spack_path_placeholder__/__s
           pack_path_placeholder__/__spack_path_placeholder__/__spack_path_plac
           eholder__/__spac/morepadding/darwin-ventura-aarch64/apple-clang-14.0
           .0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib/5.36.0/darwin-th
           read-multi-2level/CORE/libperl.dylib' (no such file), '/usr/local/li
           b/libperl.dylib' (no such file), '/usr/lib/libperl.dylib' (no such f
           ile, not in dyld cache)
     29    /var/folders/6n/pw9tmf6n08dfgg5td4_w9wh00000gx/T/gitlab-runner/spack
           -stage/spack-stage-autoconf-2.69-v5qspithe3yxjlb7i66qinusk5n5r5kv/sp
           ack-src/configure: line 2667: 78133 Abort trap: 6           $PERL -e
            'require 5.006;'

All of these packages I am able to build, using the same spack.yaml, same git commit, building locally on the same runner machine where they failed in the pipeline, if I (1) use no binaries from cache and (2) use no padding.

adamjstewart · 2023-04-13T17:53:03Z

^ @haampie

eugeneswalker · 2023-04-13T17:55:00Z

Here is the problem:

$> git clone https://github.com/adamjstewart/spack.git
$> (cd spack && git checkout ci/ml-darwin)
$> . spack/share/spack/setup-env.sh

$> cp spack/share/spack/gitlab/cloud_pipelines/stacks/ml-darwin-aarch64-mps/spack.yaml .
$> spack  -e . concretize -f |tee concretize.log

$> spack -e . install --cache-only --no-check-signature --only dependencies --include-build-deps /v5qsp
OK

$> spack -e . install --no-cache --only package /v5qsp
==> Installing autoconf-2.69-v5qspithe3yxjlb7i66qinusk5n5r5kv
==> Using cached archive: /Users/lpeyrala/spack/var/spack/cache/_source-cache/archive/95/954bd69b391edc12d6a4a51a2dd1476543da5c6bbf05a95b59dc0dd6fd4c2969.tar.gz
==> Using cached archive: /Users/lpeyrala/spack/var/spack/cache/_source-cache/archive/77/7793209b33013dc0f81208718c68440c5aae80e7a1c4b8d336e382525af791a7
==> Using cached archive: /Users/lpeyrala/spack/var/spack/cache/_source-cache/archive/35/35c449281546376449766f92d49fc121ca50e330e60fefcfc9be2af3253082c2
==> Using cached archive: /Users/lpeyrala/spack/var/spack/cache/_source-cache/archive/a4/a49dd5bac3b62daa0ff688ab4d508d71dbd2f4f8d7e2a02321926346161bf3ee
==> Applied patch http://mirrors.mit.edu/gentoo-portage/sys-devel/autoconf/files/autoconf-2.69-fix-libtool-test.patch
==> Applied patch http://mirrors.mit.edu/gentoo-portage/sys-devel/autoconf/files/autoconf-2.69-perl-5.26.patch
==> Applied patch http://mirrors.mit.edu/gentoo-portage/sys-devel/autoconf/files/autoconf-2.69-perl-5.26-2.patch
==> Ran patch() for autoconf
==> autoconf: Executing phase: 'autoreconf'
==> autoconf: Executing phase: 'configure'
==> Error: ProcessError: Command exited with status 134:
    '/var/folders/_2/r23rjwn13yjfwll2z58635l00000gs/T/lpeyrala/spack-stage/spack-stage-autoconf-2.69-v5qspithe3yxjlb7i66qinusk5n5r5kv/spack-src/configure' '--prefix=/Users/lpeyrala/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/darwin-ventura-aarch64/apple-clang-14.0.0/autoconf-2.69-v5qspithe3yxjlb7i66qinusk5n5r5kv'

1 error found in build log:
     24    checking how m4 supports trace files... --debugfile
     25    checking for perl... /Users/lpeyrala/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__sp
           ack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.
           0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/bin/perl
     26    dyld[69044]: Library not loaded: /Users/lpeyrala/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeh
           older__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/morepadding/darwin-ventura-aarch64/apple
           -clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib/5.36.0/darwin-thread-multi-2level/CORE/libperl.dylib
     27      Referenced from: <0A51BC43-CDA1-369A-B7D5-E1AD4480232C> /Users/lpeyrala/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placehol
           der__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/darwin-ventura-
           aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/bin/perl
     28      Reason: tried: '/Users/lpeyrala/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack
           _path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/morepadding/darwin-ventura-aarch64/apple-clang-14.0.0/p
           erl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib/5.36.0/darwin-thread-multi-2level/CORE/libperl.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/lpe
           yrala/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_
           path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/morepadding/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fk
           wimbhsw6rxxfip3/lib/5.36.0/darwin-thread-multi-2level/CORE/libperl.dylib' (no such file), '/Users/lpeyrala/spack/opt/spack/__spack_path_placeholder__/__spack_path_pl
           aceholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_place
           holder__/__spack_/morepadding/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib/5.36.0/darwin-thread-multi-2level/CORE/libpe
           rl.dylib' (no such file), '/usr/local/lib/libperl.dylib' (no such file), '/usr/lib/libperl.dylib' (no such file, not in dyld cache)
     29    /var/folders/_2/r23rjwn13yjfwll2z58635l00000gs/T/lpeyrala/spack-stage/spack-stage-autoconf-2.69-v5qspithe3yxjlb7i66qinusk5n5r5kv/spack-src/configure: line 2667: 6904
           4 Abort trap: 6           $PERL -e 'require 5.006;'
  >> 30    configure: error: Perl 5.006 or better is required

The problem is, the relocated perl is non-functional!

$> $(spack location -i perl)/bin/perl
dyld[69080]: Library not loaded: /Users/lpeyrala/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/morepadding/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib/5.36.0/darwin-thread-multi-2level/CORE/libperl.dylib
  Referenced from: <0A51BC43-CDA1-369A-B7D5-E1AD4480232C> /Users/lpeyrala/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/bin/perl
  Reason: tried: '/Users/lpeyrala/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/morepadding/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib/5.36.0/darwin-thread-multi-2level/CORE/libperl.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/lpeyrala/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/morepadding/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib/5.36.0/darwin-thread-multi-2level/CORE/libperl.dylib' (no such file), '/Users/lpeyrala/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/morepadding/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib/5.36.0/darwin-thread-multi-2level/CORE/libperl.dylib' (no such file), '/usr/local/lib/libperl.dylib' (no such file), '/usr/lib/libperl.dylib' (no such file, not in dyld cache)
zsh: abort      $(spack location -i perl)/bin/perl

I don't yet know why... looking into it...

Looking at the debug logs showing relocation of perl:

$> spack -d -e . install --cache-only --no-check-signature /i3wjv # PERL
...
install_name_tool \
 -change \
   /Users/gitlab-runner/builds/4NXyRFit/0/spack/spack/opt/spack/[padded-to-256-chars]/morepadding/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib/5.36.0/darwin-thread-multi-2level/CORE/libperl.dylib \
   /Users/lpeyrala/spack/opt/spack/[padded-to-256-chars]/morepadding/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib/5.36.0/darwin-thread-multi-2level/CORE/libperl.dylib \
 -rpath  \
   /Users/gitlab-runner/builds/4NXyRFit/0/spack/spack/opt/spack/[padded-to-256-chars]/morepadding/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib \
   /Users/lpeyrala/spack/opt/spack/[padded-to-256-chars]/morepadding/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib \
 -rpath  \
   /Users/gitlab-runner/builds/4NXyRFit/0/spack/spack/opt/spack/[padded-to-256-chars]/morepadding/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib64  \
   /Users/lpeyrala/spack/opt/spack/[padded-to-256-chars]/morepadding/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib64 \
 -rpath  \
   /Users/gitlab-runner/builds/4NXyRFit/0/spack/spack/opt/spack/[padded-to-256-chars]/darwin-ventura-aarch64/apple-clang-14.0.0/zlib-1.2.13-2iaqlis6z5ogtxwuupsg2o7mli5kxycf/lib \
   /Users/lpeyrala/spack/opt/spack/[padded-to-256-chars]/darwin-ventura-aarch64/apple-clang-14.0.0/zlib-1.2.13-2iaqlis6z5ogtxwuupsg2o7mli5kxycf/lib \
  -rpath \
    /Users/gitlab-runner/builds/4NXyRFit/0/spack/spack/opt/spack/[padded-to-256-chars]/darwin-ventura-aarch64/apple-clang-14.0.0/gdbm-1.23-ok7zncf655l6gvnoovvss2redrnuvhnj/lib \
    /Users/lpeyrala/spack/opt/spack/[padded-to-256-chars]/darwin-ventura-aarch64/apple-clang-14.0.0/gdbm-1.23-ok7zncf655l6gvnoovvss2redrnuvhnj/lib \
  -rpath \
    /Users/gitlab-runner/builds/4NXyRFit/0/spack/spack/opt/spack/[padded-to-256-chars]/darwin-ventura-aarch64/apple-clang-14.0.0/readline-8.2-2pultgbxeglbzvlndod3ubcox3xxraag/lib \
    /Users/lpeyrala/spack/opt/spack/[padded-to-256-chars]/darwin-ventura-aarch64/apple-clang-14.0.0/readline-8.2-2pultgbxeglbzvlndod3ubcox3xxraag/lib \
  -rpath \
    /Users/gitlab-runner/builds/4NXyRFit/0/spack/spack/opt/spack/[padded-to-256-chars]/darwin-ventura-aarch64/apple-clang-14.0.0/ncurses-6.4-qr3fb54bkephwivkukhu4bkti2vn3vhc/lib \
    /Users/lpeyrala/spack/opt/spack/[padded-to-256-chars]/darwin-ventura-aarch64/apple-clang-14.0.0/ncurses-6.4-qr3fb54bkephwivkukhu4bkti2vn3vhc/lib \
  -rpath \
    /Users/gitlab-runner/builds/4NXyRFit/0/spack/spack/opt/spack/[padded-to-256-chars]/darwin-ventura-aarch64/apple-clang-14.0.0/bzip2-1.0.8-dcbfv4cnrpbqbrg4t6z2skv4jag2smeg/lib \
    /Users/lpeyrala/spack/opt/spack/[padded-to-256-chars]/darwin-ventura-aarch64/apple-clang-14.0.0/bzip2-1.0.8-dcbfv4cnrpbqbrg4t6z2skv4jag2smeg/lib \
  -rpath \
    /Users/gitlab-runner/builds/4NXyRFit/0/spack/spack/opt/spack/[padded-to-256-chars]/darwin-ventura-aarch64/apple-clang-14.0.0/berkeley-db-18.1.40-oty6bbsmdlkmv3k3b6wlllvsedijsl42/lib \
    /Users/lpeyrala/spack/opt/spack/[padded-to-256-chars]/darwin-ventura-aarch64/apple-clang-14.0.0/berkeley-db-18.1.40-oty6bbsmdlkmv3k3b6wlllvsedijsl42/lib \
  /Users/lpeyrala/spack/opt/spack/[padded-to-256-chars]/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/bin/perl

Focusing on the first part of install_name_tool, you can see:

install_name_tool \
 -change \
   /Users/gitlab-runner/builds/4NXyRFit/0/spack/spack/opt/spack/[padded-to-256-chars]/morepadding/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib/5.36.0/darwin-thread-multi-2level/CORE/libperl.dylib \
   /Users/lpeyrala/spack/opt/spack/[padded-to-256-chars]/morepadding/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib/5.36.0/darwin-thread-multi-2level/CORE/libperl.dylib \

This doesn't look right, because the second path here does not match where we are installing perl, as there is no morepadding in the target prefix:

$> spack location -i perl
/Users/lpeyrala/spack/opt/spack/[padded-to-256-chars]/darwin-ventura-aarch64/apple-clang-14.0.0/perl-5.36.0-i3wjv47ded7p5o2fkwimbhsw6rxxfip3/lib/5.36.0/darwin-thread-multi-2level/CORE/libperl.dylib
...

eugeneswalker · 2023-04-13T22:51:06Z

Looking here:

spack/lib/spack/spack/relocate.py

Lines 232 to 235 in a9db562

    
           for dep in deps: 
        
               for old_prefix, new_prefix in prefix_to_prefix.items(): 
        
                   if dep.startswith(old_prefix): 
        
                       paths_to_paths[dep] = re.sub(re.escape(old_prefix), new_prefix, dep)

It first sets the relocation mapping correctly:

path_to_paths[.../libperl.dylib] = /Users/lpeyrala/spack/opt/spack/[padded-to-256-chars]/darwin-ventura-aarch64/.../libperl.dylib  # CORRECT

But then immediately overwrites the prior, correct relocation path, with an incorrect one:

path_to_paths[.../libperl.dylib] = /Users/lpeyrala/spack/opt/spack/[padded-to-256-chars]/morepadding/darwin-ventura-aarch64/.../libperl.dylib # WRONG

haampie · 2023-04-18T07:03:49Z

Thanks for digging into that @eugeneswalker. Not sure why the logic is custom for macho files.

adamjstewart · 2023-04-18T15:45:27Z

macho sounds way better than Mach-O 💪

eugeneswalker · 2023-04-30T15:20:24Z

@spackbot run pipeline

spackbot-app · 2023-04-30T15:20:27Z

I'm sorry, gitlab does not have your latest revision yet, I can't run that pipeline for you right now.

One likely possibility is that your PR pipeline has been temporarily deferred, in which case, it is awaiting a develop pipeline, and will be run when that finishes.

Please check the gitlab commit status message to see if more information is available.

Details

Unexpected response from gitlab: {'message': '404 Commit Not Found'}

eugeneswalker · 2023-05-01T13:36:51Z

@spackbot run pipeline

spackbot-app · 2023-05-01T13:37:04Z

I've started that pipeline for you!

eugeneswalker · 2023-05-01T15:14:01Z

@spackbot run pipeline

spackbot-app · 2023-05-01T15:14:12Z

I've started that pipeline for you!

eugeneswalker · 2023-05-01T23:09:00Z

@spackbot run pipeline

spackbot-app · 2023-05-01T23:09:06Z

I've started that pipeline for you!

eugeneswalker · 2023-05-02T20:02:40Z

@spackbot run pipeline

spackbot-app · 2023-05-02T20:02:47Z

I've started that pipeline for you!

eugeneswalker · 2023-05-02T20:36:49Z

A number of current CI errors are due to the fact I forgot to install boto3 on the new Mac runners. Will fix.

eugeneswalker · 2023-05-03T13:56:55Z

@spackbot run pipeline

spackbot-app · 2023-05-03T13:57:02Z

I've started that pipeline for you!

…or now

eugeneswalker · 2023-05-21T19:30:48Z

All CI finally passing here. I have one last tweak to make before we merge though, so let us defer merge until then. Should be ready very soon.

adamjstewart

Great job!

share/spack/gitlab/cloud_pipelines/configs/darwin/aarch64/compilers.yaml

share/spack/gitlab/cloud_pipelines/stacks/ml-darwin-aarch64-mps/spack.yaml

adamjstewart · 2023-05-22T18:35:15Z

@haampie we're likely going to merge this soon, but if you can investigate the remaining code signing/relocation issues in a follow-up PR, that would be much appreciated. Let me know if you want us to open an issue to help track this.

share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml

share/spack/gitlab/cloud_pipelines/configs/darwin/aarch64/compilers.yaml

eugeneswalker · 2023-05-25T03:49:02Z

@spackbot run pipeline because one of the aws-pcluster-skylake jobs cgal is failing due to missing dependency binary that is expected to exist:

==> Installing gmp-6.2.1-r3arlbl2twfz33i2ru5vcv26mc5uuoza
==> Removing failure mark on gmp-6.2.1-r3arlbl2twfz33i2ru5vcv26mc5uuoza
==> Error: No binary for gmp-6.2.1-r3arlbl2twfz33i2ru5vcv26mc5uuoza found when cache-only specified
==> Error: Failed to install gmp due to SystemExit: 1
==> Error: gmp-6.2.1-r3arlbl2twfz33i2ru5vcv26mc5uuoza: Package was not installed
==> Error: Installation request failed.  Refer to reported errors for failing package(s).

spackbot-app · 2023-05-25T03:49:10Z

I've started that pipeline for you!

spackbot-app bot added core PR affects Spack core functionality gitlab Issues related to gitlab integration labels Mar 31, 2023

adamjstewart commented Mar 31, 2023

View reviewed changes

share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml Outdated Show resolved Hide resolved

eugeneswalker self-requested a review March 31, 2023 18:31

adamjstewart mentioned this pull request Mar 31, 2023

CI and public binaries for ML libraries #31551

Open

6 tasks

eugeneswalker force-pushed the ci/ml-darwin branch from da27d02 to 8291fab Compare April 11, 2023 00:26

adamjstewart mentioned this pull request Apr 17, 2023

macos: set new ad-hoc signature with codesign after binary string replacement #35585

Merged

eugeneswalker force-pushed the ci/ml-darwin branch from 0dad414 to f2caede Compare April 26, 2023 22:26

eugeneswalker added 2 commits May 15, 2023 13:53

nproc || true

4c914d2

comment out specs failing due to bazel from cache codesign issue

c68e42e

eugeneswalker force-pushed the ci/ml-darwin branch from e66794e to c68e42e Compare May 15, 2023 20:54

eugeneswalker previously approved these changes May 16, 2023

View reviewed changes

eugeneswalker enabled auto-merge (squash) May 16, 2023 00:58

update to use new simplified ci workflow

de650df

eugeneswalker dismissed their stale review via de650df May 19, 2023 20:56

eugeneswalker and others added 6 commits May 19, 2023 16:09

Merge branch 'develop' into ci/ml-darwin

af048eb

remove .pr-*

6c92eba

remove expired gitlab-ci section

baca975

use .build

5081b47

cannot use tags in trigger job

3c53e3e

remove specs failing due to bazel codesign issue; limit # of builds f…

27b9888

…or now

eugeneswalker added the don't-merge-yet label May 20, 2023

adamjstewart commented May 22, 2023

View reviewed changes

share/spack/gitlab/cloud_pipelines/configs/darwin/aarch64/compilers.yaml Outdated Show resolved Hide resolved

share/spack/gitlab/cloud_pipelines/stacks/ml-darwin-aarch64-mps/spack.yaml Show resolved Hide resolved

add newline before eof

40a1ca3

adamjstewart commented May 22, 2023

View reviewed changes

share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml Show resolved Hide resolved

eugeneswalker and others added 3 commits May 22, 2023 20:49

Merge branch 'develop' into ci/ml-darwin

10d726d

remove old mac stack

53a8710

remove quantum-espresso

78fb237

eugeneswalker previously approved these changes May 24, 2023

View reviewed changes

add newline before EOF

733b00b

eugeneswalker dismissed their stale review via 733b00b May 24, 2023 20:01

eugeneswalker approved these changes May 24, 2023

View reviewed changes

share/spack/gitlab/cloud_pipelines/configs/darwin/aarch64/compilers.yaml Outdated Show resolved Hide resolved

eugeneswalker merged commit 2f8cea2 into spack:develop May 25, 2023

adamjstewart deleted the ci/ml-darwin branch May 25, 2023 05:16

Conversation

adamjstewart commented Mar 31, 2023

Uh oh!

Uh oh!

eugeneswalker commented Apr 5, 2023

Uh oh!

eugeneswalker commented Apr 10, 2023

Uh oh!

spackbot-app bot commented Apr 10, 2023

Uh oh!

eugeneswalker commented Apr 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adamjstewart commented Apr 13, 2023

Uh oh!

eugeneswalker commented Apr 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eugeneswalker commented Apr 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haampie commented Apr 18, 2023

Uh oh!

adamjstewart commented Apr 18, 2023

Uh oh!

eugeneswalker commented Apr 30, 2023

Uh oh!

spackbot-app bot commented Apr 30, 2023

Uh oh!

eugeneswalker commented May 1, 2023

Uh oh!

spackbot-app bot commented May 1, 2023

Uh oh!

eugeneswalker commented May 1, 2023

Uh oh!

spackbot-app bot commented May 1, 2023

Uh oh!

eugeneswalker commented May 1, 2023

Uh oh!

spackbot-app bot commented May 1, 2023

Uh oh!

eugeneswalker commented May 2, 2023

Uh oh!

spackbot-app bot commented May 2, 2023

Uh oh!

eugeneswalker commented May 2, 2023

Uh oh!

eugeneswalker commented May 3, 2023

Uh oh!

spackbot-app bot commented May 3, 2023

Uh oh!

eugeneswalker commented May 21, 2023

Uh oh!

adamjstewart left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

adamjstewart commented May 22, 2023

Uh oh!

Uh oh!

Uh oh!

eugeneswalker commented May 25, 2023

Uh oh!

spackbot-app bot commented May 25, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

eugeneswalker commented Apr 13, 2023 •

edited

Loading

eugeneswalker commented Apr 13, 2023 •

edited

Loading

eugeneswalker commented Apr 13, 2023 •

edited

Loading