gitlab ci install binary deps faster by haampie · Pull Request #33248 · spack/spack

haampie · 2022-10-12T16:18:01Z

This PR does a few things:

Parallel install of binary dependencies from buildcache (with redundant build edges pruned in the DAG)
Improve legibility:
1. Reduce verbosity installing buildcache deps, so that users get to see the build log for the source build
2. Show timers for binary installation, so we get more insight into why a job is slow.
3. Use GNU Make's output sync feature to get orderly output when installing deps in parallel
4. Remove a few spack -d flags, so we get to see the build logs before hitting the max log size.

...

This PR is built on top of these parts:

haampie · 2022-10-12T23:11:57Z

For reference:

https://gitlab.spack.io/spack/spack/-/jobs/3618684 (Duration: 22 minutes 17 seconds, before this PR)
https://gitlab.spack.io/spack/spack/-/jobs/3624534 (Duration: 8 minutes 11 seconds, after this PR)

or

https://gitlab.spack.io/spack/spack/-/jobs/3618685 (Duration: 27 minutes 53 seconds, before this PR)
https://gitlab.spack.io/spack/spack/-/jobs/3624526 (Duration: 8 minutes 53 seconds, after this PR)

haampie · 2022-10-13T16:21:29Z

@trws funnily enough we're hitting a broken gmake -Orecurse in Ubuntu 20.04 based containers :D I have no clue what they're doing... maybe they have some patches that broke it or so. The flag effectively has no effect. Fixed in a later version of Ubuntu.

haampie · 2022-10-14T09:59:31Z

Unfortunately the GNU make situation in our builders is rather sad:

macOS system make is 3.81(too old for --output-sync)
Amazon Linux 2 has gmake 3.82 (too old for --output-sync)
Ubuntu 20.04 has a broken --output-sync gmake

Bootstrapping gmake is an option...

haampie · 2022-10-14T14:01:59Z

@spackbot run pipeline

spackbot-app · 2022-10-14T14:02:03Z

I've started that pipeline for you!

trws · 2022-10-14T15:00:08Z

Unfortunately the GNU make situation in our builders is rather sad:
* macOS system make is 3.81(too old for --output-sync)

* Amazon Linux 2 has gmake 3.82 (too old for --output-sync)

* Ubuntu 20.04 has a broken --output-sync gmake
Bootstrapping gmake is an option...

lets do that, or grab a statically linked one, or upgrade it from the package managers, or something.

haampie · 2022-10-14T17:06:42Z

@spackbot run pipeline

(@zackgalbreath @scottwittenburg: seems like the above has triggered on the old HEAD of this PR -- I've force pushed and if you follow the link below, you get Merge 253ae0b into 0de1f98 but the force pushed commit was 0e8b9fa as you can see above).

spackbot-app · 2022-10-14T17:06:47Z

I've started that pipeline for you!

haampie · 2022-10-14T21:25:34Z

It's a one-liner to install julia's cross-compiled version, more people should use julia as a package manager

haampie · 2022-10-23T10:39:11Z

Now seeing https://gitlab.spack.io/spack/spack/-/jobs/3862768 which is probably #31387, added #33478 and #33477 to improve at least the error... (I don't think it's a blocker since gitlab will retry and it happens at the start)

kwryankrattiger

Two gripes that don't need to be fixed, mostly just notes for me later when we get to the CI module refactor

Downloading the make isn't my favorite, but I don't have a better solution, so end of gripe 😞
Windows CI, whenever ever it happens, will likely not work well with this. So it will eventually need to be an option instead of always on. Other options would be adding Ninja (@haampie and I talked about this briefly) or adding finer-grained parallel install support in spack itself.

Thanks for your work on this @haampie

haampie · 2022-10-24T14:16:04Z

For windows we'd need to do some refactoring anyways, since we're now generating and executing a shell script. There's already room for spack env depfile --generator ninja, (currently it only supports make as a value). Downloading make is not great, but it can be moved in the base images.

haampie · 2022-10-24T15:13:29Z

@spackbot run pipeline

spackbot-app · 2022-10-24T15:13:33Z

I've started that pipeline for you!

haampie · 2022-10-24T15:48:27Z

how do I make gitlab ci pass on this pr... Merge develop into here?

kwryankrattiger · 2022-10-24T19:03:45Z

@spackbot run pipeline

spackbot-app · 2022-10-24T19:03:49Z

I've started that pipeline for you!

haampie · 2022-10-24T19:22:22Z

merging develop s.t. hopefully the openblas hash changes by in something more favorable...

haampie · 2022-10-24T20:48:55Z

@zackgalbreath what can I do to make this PR pass? Apparently merging develop doesn't change the openblas hashes. Why is it failing in this PR?

haampie · 2022-10-25T09:28:45Z

ping @scottwittenburg / @zackgalbreath / @kwryankrattiger, no clue how to proceed here.

scottwittenburg · 2022-10-25T16:02:52Z

I'm not sure what you can do while there is a broken openblas on develop. You might get @tgamblin to merge this in spite of the failure if you think it's enough of an emergency to get this in.

haampie · 2022-10-25T18:13:28Z

It's not an emergency, but this is a ridiculous situation to be in... I don't need to fix openblas here, it's unrelated to my PR. Now my PR is blocked by it :|.

scottwittenburg · 2022-10-25T19:15:03Z

but this is a ridiculous situation to be in

I agree it's unfortunate, sorry. But what policy do you disagree with specifically? That we fail PRs that generate broken hashes? Or just that it affects this particular PR? Keep in mind that one policy is not to prune "untouched" specs from a stack if the PR touched the spack.yaml for that stack, and this PR touches every spack.yaml. So maybe you disagree with how we don't prune "untouched" specs when the affected stack's spack.yaml was touched?

@tgamblin in case you're interested in discussing whether any of these policies should be revisited.

haampie · 2022-10-25T20:19:08Z

@zackgalbreath unmarked openblas cause it's a buildmachine related issue (apparently their makefile still checks the host arch even wtih DYNAMIC_ARCH=1, and the cpu type of the build node was something weird?)

haampie · 2022-10-25T20:19:13Z

@spackbot run pipeline

spackbot-app · 2022-10-25T20:19:19Z

I've started that pipeline for you!

alalazo

We should eventually bootstrap make and tar too.

* Fast Gitlab CI job setup, and better legibility * Use a non-broken, recent GNU Make

spackbot-app bot added commands core PR affects Spack core functionality update-package labels Oct 12, 2022

spackbot-app bot requested review from chuckatkins, danlipsa and vicentebolea October 12, 2022 16:18

spackbot-app bot added the tests General test capability(ies) label Oct 12, 2022

haampie removed request for chuckatkins, danlipsa and vicentebolea October 12, 2022 16:59

haampie force-pushed the fix/gitlab-ci-faster-setup branch 4 times, most recently from 08dd758 to 4d1af00 Compare October 12, 2022 18:48

haampie mentioned this pull request Oct 12, 2022

py-pythran: customize headers attribute #33242

Merged

haampie force-pushed the fix/gitlab-ci-faster-setup branch from 4d1af00 to 5d6b6ba Compare October 12, 2022 21:58

spackbot-app bot added the python label Oct 12, 2022

haampie force-pushed the fix/gitlab-ci-faster-setup branch from e042d71 to 82cb426 Compare October 13, 2022 09:15

haampie force-pushed the fix/gitlab-ci-faster-setup branch from 253ae0b to 0e8b9fa Compare October 14, 2022 13:40

spackbot-app bot added the gitlab Issues related to gitlab integration label Oct 14, 2022

haampie force-pushed the fix/gitlab-ci-faster-setup branch from b9ec4a0 to 5b3c49b Compare October 16, 2022 09:42

haampie mentioned this pull request Oct 24, 2022

Improve legibility of Gitlab CI #33482

Merged

haampie force-pushed the fix/gitlab-ci-faster-setup branch from 100248f to 7756eab Compare October 24, 2022 11:15

kwryankrattiger previously approved these changes Oct 24, 2022

View reviewed changes

haampie added 2 commits October 25, 2022 12:31

Fast Gitlab CI job setup, and better legibility

65d2b43

Use a non-broken, recent GNU Make

83a527e

haampie dismissed kwryankrattiger’s stale review via 83a527e October 25, 2022 10:32

haampie force-pushed the fix/gitlab-ci-faster-setup branch from 881fe02 to 83a527e Compare October 25, 2022 10:32

haampie requested a review from kwryankrattiger October 25, 2022 20:20

alalazo approved these changes Oct 26, 2022

View reviewed changes

alalazo merged commit a2520e8 into spack:develop Oct 26, 2022

haampie deleted the fix/gitlab-ci-faster-setup branch October 26, 2022 08:16

becker33 pushed a commit to RikkiButler20/spack that referenced this pull request Nov 2, 2022

gitlab ci: install binary deps faster (spack#33248)

c6e69dc

* Fast Gitlab CI job setup, and better legibility * Use a non-broken, recent GNU Make

scottwittenburg mentioned this pull request Nov 14, 2022

e4s ci: use runner image with awscli tool #33855

Closed

charmoniumQ pushed a commit to charmoniumQ/spack that referenced this pull request Nov 19, 2022

gitlab ci: install binary deps faster (spack#33248)

1103d6c

* Fast Gitlab CI job setup, and better legibility * Use a non-broken, recent GNU Make

Conversation

haampie commented Oct 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haampie commented Oct 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haampie commented Oct 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haampie commented Oct 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haampie commented Oct 14, 2022

Uh oh!

spackbot-app bot commented Oct 14, 2022

Uh oh!

trws commented Oct 14, 2022

Uh oh!

haampie commented Oct 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

spackbot-app bot commented Oct 14, 2022

Uh oh!

haampie commented Oct 14, 2022

Uh oh!

haampie commented Oct 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kwryankrattiger left a comment

Choose a reason for hiding this comment

Uh oh!

haampie commented Oct 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haampie commented Oct 24, 2022

Uh oh!

spackbot-app bot commented Oct 24, 2022

Uh oh!

haampie commented Oct 24, 2022

Uh oh!

kwryankrattiger commented Oct 24, 2022

Uh oh!

spackbot-app bot commented Oct 24, 2022

Uh oh!

haampie commented Oct 24, 2022

Uh oh!

haampie commented Oct 24, 2022

Uh oh!

haampie commented Oct 25, 2022

Uh oh!

scottwittenburg commented Oct 25, 2022

Uh oh!

haampie commented Oct 25, 2022

Uh oh!

scottwittenburg commented Oct 25, 2022

Uh oh!

haampie commented Oct 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haampie commented Oct 25, 2022

Uh oh!

spackbot-app bot commented Oct 25, 2022

Uh oh!

alalazo left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

haampie commented Oct 12, 2022 •

edited

Loading

haampie commented Oct 12, 2022 •

edited

Loading

haampie commented Oct 13, 2022 •

edited

Loading

haampie commented Oct 14, 2022 •

edited

Loading

haampie commented Oct 14, 2022 •

edited

Loading

haampie commented Oct 23, 2022 •

edited

Loading

haampie commented Oct 24, 2022 •

edited

Loading

haampie commented Oct 25, 2022 •

edited

Loading