Spack in CI #32749

salotz-sitx · 2022-09-21T20:22:57Z

salotz-sitx
Sep 21, 2022

Wanted to start a general thread for discussing successes, failures, and potential solutions to leveraging Spack in Continuous Integration pipelines.

I'll start by copying some discussion that occurred in the Slack chat room recently.

@salotz-sitx / @salotz

I am looking to cache things when downloading things from mirrors in custom gitlab CI jobs using spack. So I think I would want to cache the following:
bootstrap binaries
downloaded build cache binaries
downloaded source
Where are the directories for these, spack/var/cache?

[...]

@blue42u

Sources are in spack/var/spack/cache/_source-cache, I'm not aware of a way to configure that.
The binary packages not cached by default. My PR #32136 handles the build cache case, I'm not sure if it handles the bootstrap binaries (we have a separate pipeline maintaining Docker images with those).

One issue we ran into was that relocation is awful slow, so we mix in #32583 as well.
And then if you want to use it in spack ci, have a look at #32300

This is a bit disappointing since even for a relatively small number of binaries this makes CI run very slowly.

blue42u · 2022-09-21T21:28:16Z

blue42u
Sep 21, 2022

We've been using Spack semi-successfully in HPCToolkit's CI for the majority of a month now. I say semi-successfully because I've spun out 4 outstanding PRs over the course of our CI's (continued) development that fix issues with buildcaches or spack ci.

To summarize our Spack usage, we maintain a set of environments with our software dependencies and then Spack them out in every CI job where we build HPCToolkit. We pull from the public buildcache + #32327 whenever possible (--reuse), and layer a private buildcache on top for any new packages that aren't in the public buildcache yet. So binary package caching and unpacking are critically important for our use case. (For more details, feel free to check out our code.)

Re. binary package caching and unpacking, the current incarnation of our CI mixes 4 workarounds:

We mix Local cache for buildcache files #32136 into our cloned Spack and spack config add config:binary_cache_root:.../.spack-bincache. By adding .spack-bincache/ to the cache: field in our CI jobs, we elide the time and network bandwidth cost of downloading the same binary package files repeated.
After running spack ci generate, we post-process the generated pipeline to add cache: entries to the generated jobs. This allows us to also the buildcache cache in our buildcache update jobs. (spack ci: Refine job attribute configuration #32300 alters the gitlab-ci: specification so this can be configured directly without external forces.)
We mix in [WIP] Accelerate (and simplify) buildcache relocation #32583 to speed up the unpacking (specifically relocation) step by 2x or so.
We run spack buildcache list before doing any spack install operations to refresh the binary cache indices, in our experience direct fetches tend to cause 404s which take multiple seconds each to wait through. (Update the binary index before attempting direct fetches #32137 alters the logic so this index-refresh happens automatically.)

Before #32136, we used a different approach where we cached the entire install tree instead. We had a couple hiccups with this approach that slow and difficult to operate:

GitLab's cache: mechanism stores the cache as a zip file. This messes with the ownership and permissions, so we kept it clean by saving and restoring the install tree from a cached tarball. In addition to being awkward, packing and unpacking this tarball is slow and since it's a single file GitLab's parallel fastzip support doesn't speed it up.
Any concretization with --reuse will prefer packages in buildcaches... or in the installed local cache. We use --reuse to stay up-to-date with the buildcache, if the install tree was unpacked before concretization it would end up never actually updating (unless the requirements were adjusted). So we had to add logic to concretize first and unpack the install tree second.

0 replies

blue42u · 2022-09-21T22:11:49Z

blue42u
Sep 21, 2022

Since we're griping about Spack in CI, I also wanted to mention another hiccup we ran across in HPCToolkit.

We intend to test on a matrix of 11 OS + architecture combinations (we have 6 enabled ATM), and we use 2 environments (with views) for a total of 22 environment concretizations we want to maintain packages for, all in a single buildcache. The problem is spack ci generate can only generate a pipeline for a single concrete environment at a time, meaning we would need to generate and spawn 22 pipelines. The no-specs-to-rebuild jobs alone would burn 20-30 CI-minutes.

To work around this problem, we separate the spack concretize and spack ci generate steps. We run 6 concretization jobs for the different OS + arch combinations, and then use a Python script to fuse the results into one spack.lock which is passed to a single spack ci generate job. (Feel free to check out the code that implements this.)

Obviously this is a rather atrocious setup, ideally this would all happen in a single job spanning the entire configuration matrix. But I'm not aware of any current way to do that, nor a simple extension that would improve the situation. Suggestions would be welcome.

2 replies

salotz-sitx Sep 22, 2022
Author

Ugh I didn't know about this. I do remember having issues trying to enable building against multiple architectures in my matrix, but I thought it was something with parsing the yaml file.

blue42u Oct 15, 2022

Update on this issue, about a week ago we transitioned to a new workaround for this problem, where now we spack ci generate all 22 pipelines and use an automated script to generate a "child" pipeline that triggers them as "grandchild" pipelines. The script analyzes the pipelines enough to identify whether there are any build jobs inside, and if not will skip it to save time. So in total, we only have 1 "dummy" job in the no-op case, and can support our full test matrix without too much maintenance burden.

I think it would be valuable to implement this idea in mainline spack ci generate. Not only would it support many environments (like in our use case), but it also provides a potential solution to the pipeline size issue: a large pipeline could be split and its parts triggered in sequence in the generated "child" pipeline, since each individual pipeline is under the limit GitLab is kept happy.

salotz-sitx · 2022-10-04T20:40:27Z

salotz-sitx
Oct 4, 2022
Author

salotz-sitx
Nov 9, 2022
Author

Recently I've been having some difficulty in managing my buildcache and wanted to share some experiences and maybe get some insights from anyone that has ideas on how to make buildcaches better. And specifically how the buildcaches are populated via the generated CI pipelines.

I asked before and was told that there is an intelligent algorithm that will detect when a recipe has meaningfully changed and then it will be rebuilt. In my experience this hasn't been the case. For instance I had a package that was incorrectly packaged. I accidentally left out a dependency, shipped it to the buildcache and found out much later. So I fixed the recipe but spack ci generate would not rebuild it. I think because there really isn't any logic or tools for removing packages from the buildcaches in a transactional way. In fact there is no documentation on this at all so I had to use some trial and error to figure it out.

I had to go into the GCP bucket and remove the actual .spack file and the .sig file for it. Then I have to explicitly run spack buildcache update-index before running spack ci generate. Now I actually just do this every time just to avoid flaky pipelines.

So that is to say that there really should be more tools for managing the artifacts in the buildcache and querying information on the state of the buildcache, or at least providing some more output from spack buildcache update-index to tell me what happened.

So far its been pretty scary doing anything with the build cache because I don't know what will just break it completely and I'm better off just starting from scratch.. which is getting more infeasible as users increase and the number of packages is exploding as more applications are put on it. So anything (even just documentation) would help a lot for more adoption in an industrial setting, as there is already some anxiety about moving away from things like conda.

Looking forward I think it would be great to have some kind of web UI that you can use to manage this and maybe even get support from artifact registries for the build cache (and source mirrors too!).

6 replies

salotz-sitx Nov 9, 2022
Author

Thanks @tgamblin for the more in depth description.

If you added a dependency, it would show up on the spec, and the spec would have a different hash. So it wouldn't be a rebuild -- rather you'd get a new hash of that package in the build cache.

Could it be because the concretizer is picking up that the package already exists via the reuse behavior and so not creating a job to build it? Because as I mentioned in the post once I deleted the binary from the cache it went through and built the package with the new hash.

tgamblin Nov 9, 2022
Maintainer

Definitely, if your pipeline is configured to reuse. In general for our CI pipelines in GitLab (the ones that run on PRs, release branches, and develop), we disable reuse basically for this reason. We want the "freshest" builds to happen in CI with defaults from packages, and we don't want to prefer what's already in the build cache.

salotz-sitx Nov 9, 2022
Author

I see, yes for us I am just trying to build only the packages we need which is why I have this turned on. Maybe I'll reconsider that.

blue42u Nov 10, 2022

This isn't a solution in our case, we want to --reuse the freshest from the Spack public buildcache whenever possible. So far we've gotten around this issue by using spack concretize first with only the public buildcache enabled, and then add our custom buildcache mirror(s) and running spack ci generate.

I haven't run into packaging errors yet, but I have had cases where the padded_length and such were set incorrectly and generated unusable binary packages. Would it be possible to standardize the configuration (in Spack) for building packages intended for a buildcache? That would definitely lower the barrier for entry, hunting down the particular configuration needed to make everything work as intended was by no means fun.

salotz Nov 11, 2022

That is a good idea.

Spack in CI #32749

Uh oh!

salotz-sitx Sep 21, 2022

Replies: 4 comments · 8 replies

Uh oh!

blue42u Sep 21, 2022

Uh oh!

blue42u Sep 21, 2022

Uh oh!

salotz-sitx Sep 22, 2022 Author

Uh oh!

blue42u Oct 15, 2022

Uh oh!

salotz-sitx Oct 4, 2022 Author

Uh oh!

salotz-sitx Nov 9, 2022 Author

Uh oh!

Uh oh!

salotz-sitx Nov 9, 2022 Author

Uh oh!

tgamblin Nov 9, 2022 Maintainer

Uh oh!

salotz-sitx Nov 9, 2022 Author

Uh oh!

blue42u Nov 10, 2022

Uh oh!

salotz Nov 11, 2022

salotz-sitx
Sep 21, 2022

Replies: 4 comments 8 replies

blue42u
Sep 21, 2022

blue42u
Sep 21, 2022

salotz-sitx Sep 22, 2022
Author

salotz-sitx
Oct 4, 2022
Author

salotz-sitx
Nov 9, 2022
Author

salotz-sitx Nov 9, 2022
Author

tgamblin Nov 9, 2022
Maintainer

salotz-sitx Nov 9, 2022
Author