Ci reactivate darwin pipelines by eugeneswalker · Pull Request #48453 · spack/spack

eugeneswalker · 2025-01-07T22:21:44Z

Re-activating the darwin CI pipelines

eugeneswalker · 2025-01-10T20:56:20Z

@spackbot run pipeline

spackbot-app · 2025-01-10T20:56:27Z

I've started that pipeline for you!

eugeneswalker · 2025-01-11T13:36:07Z

@spackbot run pipelines - tweaked local runner environment

spackbot-app · 2025-01-11T13:36:13Z

I've started that pipeline for you!

eugeneswalker · 2025-01-13T18:38:17Z

@spackbot run pipelines -- due to pipeline failures in generate jobs that seem unrelated to this PR specifically?

spackbot-app · 2025-01-13T18:38:24Z

I've started that pipeline for you!

eugeneswalker · 2025-01-21T19:37:13Z

There is an issue with the stage dir and permissions such that files are being created which cannot be deleted!

[email protected] /au2dystiuwtwgkx52wmjkqbkiwq4wsdw Spack CI

-r--r--r--  1 gitlab-runner-0  staff  609 Jan 21 10:40 tmp/stage/spack-stage-fzf-0.57.0-xa5f3xwhin2xgkvbpkhck74xzdbauvsz/pkg/mod/github.com/mattn/[email protected]/isatty_bsd.go

One of the first actions the gitlab-runner takes on processing a new job is to delete files from the old job, and that is failing because of the permission shown above:

...
warning: failed to remove tmp/stage/spack-stage-fzf-0.57.0-xa5f3xwhin2xgkvbpkhck74xzdbauvsz/pkg/mod/golang.org/x/[email protected]/term_unix_other.go: Permission denied
warning: failed to remove tmp/stage/spack-stage-fzf-0.57.0-xa5f3xwhin2xgkvbpkhck74xzdbauvsz/pkg/mod/golang.org/x/[email protected]/go.mod: Permission denied
warning: failed to remove tmp/stage/spack-stage-fzf-0.57.0-xa5f3xwhin2xgkvbpkhck74xzdbauvsz/pkg/mod/golang.org/x/[email protected]/terminal_test.go: Permission denied
...

We do want to keep the stage dir like below, but we need the permissions to be such that the files can be cleaned-up/deleted!

20ac391

config:
  build_stage:
  - $spack/tmp/stage

eugeneswalker · 2025-01-24T22:55:52Z

There is an issue with the stage dir and permissions such that files are being created which cannot be deleted!

* [[email protected] /au2dystiuwtwgkx52wmjkqbkiwq4wsdw Spack CI](https://gitlab.spack.io/spack/spack/-/jobs/14744491)

-r--r--r--  1 gitlab-runner-0  staff  609 Jan 21 10:40 tmp/stage/spack-stage-fzf-0.57.0-xa5f3xwhin2xgkvbpkhck74xzdbauvsz/pkg/mod/github.com/mattn/[email protected]/isatty_bsd.go

The issue here appeared to be due to golang mod files being created with read only permissions, by design. We can simply invoke spack clean at the end of darwin jobs to ensure the staged files get removed, rather than leaving them for the gitlab-runner to clean up.

eugeneswalker · 2025-01-26T04:53:34Z

@spackbot run pipeline

spackbot-app · 2025-01-26T04:53:40Z

I've started that pipeline for you!

eugeneswalker · 2025-01-26T23:03:10Z

@spackbot run pipeline

spackbot-app · 2025-01-26T23:03:16Z

I've started that pipeline for you!

eugeneswalker · 2025-01-30T15:19:23Z

There are a few build errors blocking this:

From: bootstrap-aarch64-darwin-build

clingo-bootstrap@spack /x2a3vkokdagmv3c3de5hvduocmqshbru Spack CI

...
error: /Library/Developer/CommandLineTools/usr/bin/install_name_tool: for: /Users/gitlab-runner-0/builds/t1_TDJH3y/0/spack/spack/opt/spack/[padded-to-256-chars]/darwin-sequoia-aarch64/apple-clang-16.0.0/clingo-bootstrap-spack-x2a3vkokdagmv3c3de5hvduocmqshbru/lib/libclingo.4.0.dylib (for architecture arm64) option "-add_rpath /Users/gitlab-runner-0/builds/t1_TDJH3y/0/spack/spack/opt/spack/[padded-to-256-chars]/darwin-sequoia-aarch64/apple-clang-16.0.0/clingo-bootstrap-spack-x2a3vkokdagmv3c3de5hvduocmqshbru/lib" would duplicate path, file already has LC_RPATH for: /Users/gitlab-runner-0/builds/t1_TDJH3y/0/spack/spack/opt/spack/[padded-to-256-chars]/darwin-sequoia-aarch64/apple-clang-16.0.0/clingo-bootstrap-spack-x2a3vkokdagmv3c3de5hvduocmqshbru/lib
...
  The C compiler
    "/Users/gitlab-runner-0/builds/t1_TDJH3y/0/spack/spack/lib/spack/env/clang/clang"
  is not able to compile a simple test program.
  It fails with the following output:
    Change Dir: '/Users/gitlab-runner-0/builds/t1_TDJH3y/0/spack/spack/tmp/stage/spack-stage-clingo-bootstrap-spack-x2a3vkokdagmv3c3de5hvduocmqshbru/spack-build-x2a3vko/CMakeFiles/CMakeScratch/TryCompile-9cNMK2'
...
    error: Error in reading profile /Users/gitlab-runner-0/builds/t1_TDJH3y/0/spack/spack/tmp/stage/spack-stage-clingo-bootstrap-spack-x2a3vkokdagmv3c3de5hvduocmqshbru/spack-src/reports/default.profdata: No such file or directory
    gmake[1]: *** [CMakeFiles/cmTC_260ce.dir/build.make:82: CMakeFiles/cmTC_260ce.dir/testCCompiler.c.o] Error 1
    gmake[1]: Leaving directory '/Users/gitlab-runner-0/builds/t1_TDJH3y/0/spack/spack/tmp/stage/spack-stage-clingo-bootstrap-spack-x2a3vkokdagmv3c3de5hvduocmqshbru/spack-build-x2a3vko/CMakeFiles/CMakeScratch/TryCompile-9cNMK2'
    gmake: *** [Makefile:134: cmTC_260ce/fast] Error 2
...

[email protected] /2pvlbhhqutmkbmrjo7sgeliijfprmzgm Spack CI

...
error: /Library/Developer/CommandLineTools/usr/bin/install_name_tool: for: /Users/gitlab-runner-0/builds/t1_TDJH3y/0/spack/spack/opt/spack/[padded-to-256-chars]/darwin-sequoia-aarch64/apple-clang-16.0.0/clingo-bootstrap-5.7.1-2pvlbhhqutmkbmrjo7sgeliijfprmzgm/lib/libclingo.4.0.dylib (for architecture arm64) option "-add_rpath /Users/gitlab-runner-0/builds/t1_TDJH3y/0/spack/spack/opt/spack/[padded-to-256-chars]/darwin-sequoia-aarch64/apple-clang-16.0.0/clingo-bootstrap-5.7.1-2pvlbhhqutmkbmrjo7sgeliijfprmzgm/lib" would duplicate path, file already has LC_RPATH for: /Users/gitlab-runner-0/builds/t1_TDJH3y/0/spack/spack/opt/spack/[padded-to-256-chars]/darwin-sequoia-aarch64/apple-clang-16.0.0/clingo-bootstrap-5.7.1-2pvlbhhqutmkbmrjo7sgeliijfprmzgm/lib
...
  The C compiler
    "/Users/gitlab-runner-0/builds/t1_TDJH3y/0/spack/spack/lib/spack/env/clang/clang"
  is not able to compile a simple test program.
  It fails with the following output:
    Change Dir: '/Users/gitlab-runner-0/builds/t1_TDJH3y/0/spack/spack/tmp/stage/spack-stage-clingo-bootstrap-5.7.1-2pvlbhhqutmkbmrjo7sgeliijfprmzgm/spack-build-2pvlbhh/CMakeFiles/CMakeScratch/TryCompile-Zh6pVg'
...
    error: Error in reading profile /Users/gitlab-runner-0/builds/t1_TDJH3y/0/spack/spack/tmp/stage/spack-stage-clingo-bootstrap-5.7.1-2pvlbhhqutmkbmrjo7sgeliijfprmzgm/spack-src/reports/default.profdata: No such file or directory
    gmake[1]: *** [CMakeFiles/cmTC_480d8.dir/build.make:82: CMakeFiles/cmTC_480d8.dir/testCCompiler.c.o] Error 1
    gmake[1]: Leaving directory '/Users/gitlab-runner-0/builds/t1_TDJH3y/0/spack/spack/tmp/stage/spack-stage-clingo-bootstrap-5.7.1-2pvlbhhqutmkbmrjo7sgeliijfprmzgm/spack-build-2pvlbhh/CMakeFiles/CMakeScratch/TryCompile-Zh6pVg'
    gmake: *** [Makefile:134: cmTC_480d8/fast] Error 2

From: developer-tools-darwin-build

[email protected] /3fuavwi5f3v7wweq6fvsyasg6kglrxuh Developer Tools Darwin

...
ld: duplicate LC_RPATH '/usr/local/spack-tools/spack/opt/spack/darwin-sonoma-aarch64/apple-clang-16.0.0/gcc-14.2.0-dxv7yb3pasmje7wnkq4eizyvyvsdl2d3/lib' in '/Users/gitlab-runner-0/builds/t1_a84b7x/0/spack/spack/opt/spack/[padded-to-256-chars]/darwin-sequoia-aarch64/apple-clang-16.0.0/lua-lpeg-1.1.0-1-zfvdag6bm4qshhipfblhtukfj66iht5e/lib/lua/5.1/lpeg.so'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
...
     239    ld: duplicate LC_RPATH '/usr/local/spack-tools/spack/opt/spack/darw
            in-sonoma-aarch64/apple-clang-16.0.0/gcc-14.2.0-dxv7yb3pasmje7wnkq4
            eizyvyvsdl2d3/lib' in '/Users/gitlab-runner-0/builds/t1_a84b7x/0/sp
            ack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeho
            lder__/__spack_path_placeholder__/__spack_path_placeholder__/__spac
            k_path_placeholder__/__spack_path_placeholder__/__spack_path_placeh
            older__/__s/darwin-sequoia-aarch64/apple-clang-16.0.0/lua-lpeg-1.1.
            0-1-zfvdag6bm4qshhipfblhtukfj66iht5e/lib/lua/5.1/lpeg.so'
  >> 240    clang: error: linker command failed with exit code 1 (use -v to see
             invocation)
...

eugeneswalker · 2025-01-30T20:11:34Z

OpenMPI seems to hang forever:

[email protected] /nnhtzcvt2aybg3eiaffaetv7fh2xzne4 Machine Learning MPS
- At time of posting the job has been running for ~300 minutes and is still doing something, maybe?

eugeneswalker · 2025-01-31T16:38:17Z

@spackbot run pipeline

spackbot-app · 2025-01-31T16:38:23Z

I've started that pipeline for you!

eugeneswalker · 2025-02-01T18:11:33Z

OpenMPI build freezes, never completes, resulting in job timeout:

[email protected] /nnhtzcvt2aybg3eiaffaetv7fh2xzne4 Machine Learning MPS

I would disable OpenMPI until this could be figured out but many packages from the machine-learning-mps stack depend on it. @adamjstewart do we really need openmpi specifically here? Can we use a different MPI until the openmpi issue is resolved?

  packages:
   ...
    mpi:
      require: openmpi
   ...

Many if not all the clingo-bootstrap builds fail in bootstrap-aarch64-darwin. @alalazo Any idea what the issue may be?

bootstrap-aarch64-darwin-build

adamjstewart · 2025-02-01T18:19:44Z

For all ML pipelines, we can use any MPI provider. I recall having some kind of problem with whatever was being chosen by default and had to force it to be OpenMPI, but feel free to undo that. It may not even be needed on macOS.

eugeneswalker · 2025-02-03T00:27:09Z

@spackbot run pipeline

spackbot-app · 2025-02-03T00:27:15Z

I've started that pipeline for you!

eugeneswalker · 2025-02-03T02:47:45Z

The only build errors remaining here are clingo-bootstrap (see above) and the fact that numerous packages in the darwin-mps stack do seem to depend specifically on openmpi, and openmpi build hangs indefinitely for an as-yet-undetermined reason. If we can solve these, we can merge this and have our darwin pipelines back!

adamjstewart · 2025-02-03T08:00:52Z

What happens when you require mpich?

eugeneswalker · 2025-02-25T05:30:54Z

I commented out the handful of failing packages and all CI is green now. I think we should merge this now to bring the darwin pipelines back. Separately, I will follow up with a PR re-enabling the failing specs where we can iterate on fixes. Let's bring it back! @adamjstewart @kwryankrattiger

adamjstewart

Only approving my own pipelines, don't fully understand the other changes.

share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml

share/spack/gitlab/cloud_pipelines/configs/darwin/ci.yaml

eugeneswalker · 2025-02-26T20:14:51Z

All CI is going to be green here. Only one gitlab job remains and it is update-index. Any final reservations @kwryankrattiger? If not, I will turn on auto-merge and we will be back in business.

kwryankrattiger

Two minor things, both can be pulled out to follow ups I think as neither poses immediate risks and maybe worth experimenting with /tmp changes more before reverting back to the default.

share/spack/gitlab/cloud_pipelines/configs/darwin/aarch64/config.yaml

share/spack/gitlab/cloud_pipelines/stacks/ml-darwin-aarch64-mps/spack.yaml

… level config

eugeneswalker · 2025-02-26T22:22:52Z

OK. All fixed up based on recent comments. Turning on auto-merge. Will follow up with a PR to re-enable the failing py-torch* related specs. All CI is passing now.

alalazo · 2025-02-27T08:12:46Z

Is it just me, or is there no darwin pipeline that has been reactivated: https://gitlab.spack.io/spack/spack/-/pipelines/995843 ? I can't find any darwin pipeline, as instead I can see e.g. here https://gitlab.spack.io/spack/spack/-/pipelines/989714 (where they are the only ones running) 🤔

alalazo · 2025-02-27T08:28:43Z

CI was good here:

https://gitlab.spack.io/spack/spack/-/pipelines/993131

but then these changes 9b90ef5 3253a87 seem to have disabled darwin pipelines again.

* ci: darwin stacks: update tags following system updates * disable SPACK_CI_DISABLE_STACKS; only enable *darwin* stacks for testing * manually chmod u+w tmp/ before cleanup due to issue#49147 * comment out failing specs for now * re-enable logic for disabling stacks * add explanatory comment for darwin after_script additions * remove more darwin-only targetting * restore build_stage to default location * move build-job-remove out of individual darwin stacks into darwin top level config * keep build_stage in $spack/tmp for now

eugeneswalker added the macOS label Jan 7, 2025

eugeneswalker requested a review from kwryankrattiger January 7, 2025 22:21

spackbot-app bot added core PR affects Spack core functionality gitlab Issues related to gitlab integration labels Jan 7, 2025

eugeneswalker force-pushed the ci-reactivate-darwin-pipelines branch from 693f8f3 to 42d4f0e Compare January 21, 2025 15:49

eugeneswalker force-pushed the ci-reactivate-darwin-pipelines branch from 82e8eb0 to 3e9eb2c Compare January 24, 2025 19:53

eugeneswalker force-pushed the ci-reactivate-darwin-pipelines branch 2 times, most recently from 0b23a3b to c4c32bf Compare January 29, 2025 15:18

comment out failing specs for now

15c7ea0

adamjstewart previously approved these changes Feb 25, 2025

View reviewed changes

kwryankrattiger reviewed Feb 25, 2025

View reviewed changes

share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml Outdated Show resolved Hide resolved

share/spack/gitlab/cloud_pipelines/configs/darwin/ci.yaml Show resolved Hide resolved

eugeneswalker added 2 commits February 25, 2025 12:14

re-enable logic for disabling stacks

3253a87

add explanatory comment for darwin after_script additions

abbad5a

eugeneswalker dismissed adamjstewart’s stale review via abbad5a February 25, 2025 20:14

remove more darwin-only targetting

9b90ef5

adamjstewart previously approved these changes Feb 26, 2025

View reviewed changes

eugeneswalker enabled auto-merge (squash) February 26, 2025 19:10

eugeneswalker disabled auto-merge February 26, 2025 19:10

eugeneswalker requested a review from kwryankrattiger February 26, 2025 19:10

kwryankrattiger reviewed Feb 26, 2025

View reviewed changes

share/spack/gitlab/cloud_pipelines/configs/darwin/aarch64/config.yaml Show resolved Hide resolved

share/spack/gitlab/cloud_pipelines/stacks/ml-darwin-aarch64-mps/spack.yaml Outdated Show resolved Hide resolved

eugeneswalker added 2 commits February 26, 2025 12:40

restore build_stage to default location

f6df268

move build-job-remove out of individual darwin stacks into darwin top…

64bcf5d

… level config

eugeneswalker dismissed adamjstewart’s stale review via 64bcf5d February 26, 2025 20:40

adamjstewart previously approved these changes Feb 26, 2025

View reviewed changes

keep build_stage in $spack/tmp for now

95e4ebc

eugeneswalker dismissed adamjstewart’s stale review via 95e4ebc February 26, 2025 22:22

eugeneswalker enabled auto-merge (squash) February 26, 2025 22:22

eugeneswalker requested review from adamjstewart and kwryankrattiger February 26, 2025 22:22

kwryankrattiger approved these changes Feb 26, 2025

View reviewed changes

eugeneswalker merged commit 677caec into spack:develop Feb 26, 2025
37 checks passed

greenc-FNAL mentioned this pull request Mar 4, 2025

recipes/jsonnet CMake FNALssi/spack#521

Closed

Conversation

eugeneswalker commented Jan 7, 2025

Uh oh!

eugeneswalker commented Jan 10, 2025

Uh oh!

spackbot-app bot commented Jan 10, 2025

Uh oh!

eugeneswalker commented Jan 11, 2025

Uh oh!

spackbot-app bot commented Jan 11, 2025

Uh oh!

eugeneswalker commented Jan 13, 2025

Uh oh!

spackbot-app bot commented Jan 13, 2025

Uh oh!

eugeneswalker commented Jan 21, 2025

Uh oh!

eugeneswalker commented Jan 24, 2025

Uh oh!

eugeneswalker commented Jan 26, 2025

Uh oh!

spackbot-app bot commented Jan 26, 2025

Uh oh!

eugeneswalker commented Jan 26, 2025

Uh oh!

spackbot-app bot commented Jan 26, 2025

Uh oh!

eugeneswalker commented Jan 30, 2025

Uh oh!

eugeneswalker commented Jan 30, 2025

Uh oh!

eugeneswalker commented Jan 31, 2025

Uh oh!

spackbot-app bot commented Jan 31, 2025

Uh oh!

eugeneswalker commented Feb 1, 2025

Uh oh!

adamjstewart commented Feb 1, 2025

Uh oh!

eugeneswalker commented Feb 3, 2025

Uh oh!

spackbot-app bot commented Feb 3, 2025

Uh oh!

eugeneswalker commented Feb 3, 2025

Uh oh!

adamjstewart commented Feb 3, 2025

Uh oh!

eugeneswalker commented Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adamjstewart left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

eugeneswalker commented Feb 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kwryankrattiger left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

eugeneswalker commented Feb 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

alalazo commented Feb 27, 2025

Uh oh!

alalazo commented Feb 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

eugeneswalker commented Feb 25, 2025 •

edited

Loading

eugeneswalker commented Feb 26, 2025 •

edited

Loading

eugeneswalker commented Feb 26, 2025 •

edited

Loading

alalazo commented Feb 27, 2025 •

edited

Loading