Release workflow by scottwittenburg · Pull Request #10274 · spack/spack

scottwittenburg · 2019-01-07T23:52:12Z

This topic implements several pieces needed to support a release workflow for Spack. Some of the pieces included:

A new Spack command to create a description of packages for Gitlab-CI to build. It defines a gitlab-ci.yml file that would be placed in a repository, such that when Gitlab-CI detects a commit to the branch, it will build the packages mentioned in the description. More details on this new command are provided further down in this PR description ("Details of release-jobs command).
A build job shell script to be run by Gitlab-CI. The responsibility of this script is to be assigned a spec, determine if the package corresponding to that spec exists and is up-to-date on a configured binary mirror, and if it is not, build the package, create a buildcache for it, and push that buildcache to the binary mirror, communicating with a CDash instance along the way.
Other infrastructure and changes in support of the items above.

The command to generate this .gitlab-ci.yml file looks like this:

spack release-jobs --spec-set <path-to-release.yaml> --mirror-url "https://mirror.spack.io" --shared-runner-tag spack-k8s

A summary of the jobs/stages can be printed by the above command if you add the --print-summary command-line argument.

One approach to managing the release workflow

To avoid churn in the Spack repository, we plan not to keep the full list of release jobs under source control in a full .gitlab-ci.yml file. Rather, a separate microservice will be capable of generating the bulk of the contents of the .gitlab-ci.yml on demand, for a given commit or sha.

Here is an example of the .gitlab-ci.yml that might be maintained within the Spack repository:

include:
  - remote: 'https://internal.spack.io/glciy/${CI_COMMIT_SHA}.yaml'

Whenever the pipeline is configured to run, the file above would trigger the microservice to checkout the given sha, run the spack release-jobs ... command on the release spec set, and return the full list of staged jobs to be run.

Details of `release-jobs` command

The release-jobs command makes use of two new configuration files defined in the repository, etc/spack/defaults/release.yaml (or an arbitrary yaml file following the same schema) and 'share/spack/docker/os-container-mapping.yaml'. First the release.yaml file is used to construct a CombinatorialSpecSet, which expands the contents into a list of the entire cross-product of packages/versions/compilers expressed in that file. Next, the os-container-mapping.yaml file is used to find docker containers with compilers that "satisfy" the specs in the expanded cross-product. The os-container-mapping.yaml file is essentially a list of operating systems which each provide an image (the image should exist on DockerHub so it can be pulled by both the release-jobs command and the AWS gitlab runners), and a list of available compilers on that image. For any compiler found in one of the containers that satisfies the compiler requirement of a release spec, that spec (with the OS from os-container-mapping.yaml added to it) is added to a list that will be concretized in the appropriate docker container in order to gather all the specific dependencies of the spec. Then all the resulting specs are combined and staged.

For example, given the following release.yaml:

spec-set:
    include: []
    exclude: []
    matrix:
        - packages:
            libpng:
                versions: [1.6.34]
            ncurses:
                versions: [6.1]
            pkgconf:
                versions: [1.4.2]
            readline:
                versions: [7.0]
            zlib:
                versions: [1.2.11]
        - compilers:
            gcc:
                versions: [5.5.0]
    cdash: ["https://spack.io/cdash/submit.php?project=spack"]

and given this os-container-mapping.yaml:

containers:
  linux-ubuntu18.04-x86_64:
    image: scottwittenburg/spack_builder_ubuntu_18.04
    compilers:
      - name: [email protected]
      - name: [email protected]
  linux-centos7-x86_64:
    image: scottwittenburg/spack_builder_centos_7
    compilers:
      - name: [email protected]

then the release-jobs command will generate the following specs to be built:

$ spack release-jobs --spec-set .../spack/etc/spack/defaults/release.yaml --mirror-url https://mirror.spack.io --shared-runner-tag spack-k8s --print-summary
==> Staging summary:
==>   stage 0 (6 jobs):
==>     pkgconf/atbmtyn -> [email protected]%[email protected] arch=linux-centos7-x86_64
==>     pkgconf/bp4fmns -> [email protected]%[email protected] arch=linux-ubuntu18.04-x86_64
==>     pkgconf/qdgufjf -> [email protected]%[email protected] arch=linux-centos7-x86_64
==>     pkgconf/unoed43 -> [email protected]%[email protected] arch=linux-ubuntu18.04-x86_64
==>     zlib/ijtgxbh -> [email protected]%[email protected] arch=linux-centos7-x86_64
==>     zlib/ymics2i -> [email protected]%[email protected] arch=linux-ubuntu18.04-x86_64
==>   stage 1 (4 jobs):
==>     libpng/2vicq3d -> [email protected]%[email protected] arch=linux-ubuntu18.04-x86_64
==>     libpng/drgxai6 -> [email protected]%[email protected] arch=linux-centos7-x86_64
==>     ncurses/aqchhtc -> [email protected]%[email protected] arch=linux-centos7-x86_64
==>     ncurses/qkjccof -> [email protected]%[email protected] arch=linux-ubuntu18.04-x86_64
==>   stage 2 (2 jobs):
==>     readline/vuigree -> [email protected]%[email protected] arch=linux-ubuntu18.04-x86_64
==>     readline/xlb6j2a -> [email protected]%[email protected] arch=linux-centos7-x86_64
==> 12 build jobs generated in 3 stages

Note that although the [email protected] compiler is available in the linux-ubuntu18.04-x86_64 container, no jobs with that compiler were staged because it wasn't listed as a desired compiler in the release.yaml. Conversely, also note the additional version of pkgconf which will be built, even though it wasn't specifically mentioned in the release.yaml. This is because that particular version was a dependency of another spec in the spec set.

To avoid running the concretization of release specs (and generation of dependencies) in docker containers, the --this-machine-only argument may be passed to the release-jobs command. In this case, the current machine will be used for concretization and dependency generation, but for this to work, the os-container-mapping.yaml file must contain an entry corresponding to `spack.architecture.sys_type().

Additional Notes

Currently the Spack Gitlab instance is set up to test branches it finds on my fork (scottwittenburg/spack), so that will need to be changed before merging this topic will seem to have any effect.

This PR replaces both #8451 and #8718.

Before we can merge this, we need to address at least #10141, but there may be other prerequisites as well, hence the WIP on this PR for the time being.

Things currently holding this up:

Fix issue installing from spec.yaml using -f Installing packages using the -f option does not seem to work #10141
Fix issue concretizing xsdk-0.4.0 (just needs merge) dealii: fix concretization of xsdk package #10288
Fix omega_h regression (proposed fix here: omega-h: requires empty CMAKE_BUILD_TYPE #10297)

gartung · 2019-01-08T17:25:15Z

@scottwittenburg do you consider this complete? At FNAL we have been thinking about how to handle release builds.

gartung · 2019-01-08T17:26:26Z

Nevermind. I see this is labeled WIP.

scottwittenburg · 2019-01-08T17:35:16Z

@gartung I don't think it's necessarily precisely what we want yet, but I think we'd like to get something merged and then iterate on it from there. I think this version is pretty close to something we could merge though. Maybe you could take a look and provide your thoughts? Let me know if you have questions or want something clarified. Pinging @tgamblin as he'll probably be interested in your thoughts as well.

gartung · 2019-01-08T17:38:34Z

The idea of spec sets is appealing. We need the ability to specify a preferred set of package versions that may not be the latest. Would the spec set package versions be used in concretization?

scottwittenburg · 2019-01-08T17:51:42Z

@gartung Yes, the CombinatorialSpecSet will expand the release.yaml out to the full cross-product of packages x versions x compilers specified there. Currently each spec in the resulting list of specs is then concretized in any docker containers specified in the os-container-mapping.yaml where there is a compiler that satisfies the one in the spec. And then finally each of those (a specific pkg/version/compiler/os) becomes a job/package to be built by the Gitlab instance. The jobs are staged in such a way that for any package to be built, Gitlab has access to it's dependencies in binary form, this reduces the amount of work to be done and speeds up the builds.

Down the road I think @tgamblin would like to avoid needing to concretize each spec in a specific container in order to generate the .gitlab-ci.yml, but that's how we do things at the moment.

Hopefully I answered your question, but let me know if something is unclear.

gartung · 2019-01-08T18:20:17Z

Have you considered the case where you want to specify variants and compiler flags in the spec-set as well?

scottwittenburg · 2019-01-08T18:22:26Z

Yes, we've talked about that, and would like to get there eventually, but there's no support for it here yet.

gartung · 2019-01-08T18:23:49Z

The only way you can enforce the version, variants and flags is to manage spack/etc/spack/packages.yaml. This was problematic with different release having a different packages.yaml from the current one. The environments feature helps with this because it preserves a snapshot of the concretization.

gartung · 2019-01-09T00:40:52Z

I have determined that the buildcache creation with placeholder rpaths is bugged. The original install path is supposed to be replaced with a placeholder in the rpath list. Instead the path with the placeholder is added to the rpath list. This is why the allow root flag always has to be provided.

gartung · 2019-01-09T01:36:52Z

I take that back. The bug appears to be checking the first path in rpaths for the placeholder string. This turns out to be the compiler path for c++ libraries.

gartung · 2019-01-09T03:50:10Z

See #10287 for a fix to the problem I described.

tgamblin · 2019-01-09T09:05:35Z

@gartung: have you seen the new environments support? We have some features in the pipeline that will build on it for better spec sets -- particularly for combinatorial builds. Maybe we should chat Thursday about the work we are doing on Spack Stacks, which will basically be combinatorial environments -- "instances" of combinatorial deployments. Look at https://github.com/spack/spack/projects/7. Stacks will likely to be a good long-term way to specify a release and they'll be implemented using some of this.

scottwittenburg · 2019-01-09T17:35:14Z

@tgamblin @balay The branch associated with this PR is being tested on our AWS infrastructure (pipeline 176), and currently it's hitting a failure on both centos 7 and ubuntu 18.04 during the configuration phase of omega_h. I've created an issue there to get some help resolving it.

balay · 2019-01-09T18:52:19Z

I've filed omega-h issue at #10291

scottwittenburg · 2019-01-10T20:35:39Z

ping @opadron

scottwittenburg · 2019-01-10T20:40:21Z

ping @aashish24

scottwittenburg · 2019-01-18T20:14:58Z

@tgamblin I have removed the .gitlab-ci.yml since we're in the process of creating a microservice in the kubernetes cluster to do that. Here's the last pipeline that ran on this branch before I removed it though:

https://gitlab.spack.io/scott/spack/pipelines/179

scottwittenburg · 2019-01-18T20:17:23Z

I'm not sure why github is still showing the file in the diff... nevermind it's gone.

scottwittenburg · 2019-02-15T00:53:00Z

No, I don't think you're missing anything. At the moment we're thinking we'll have another small PR once this one is merged and do a few things all at once at that time (add that "pointer" .gitlab-ci.yml, change the gitlab instance configuration so it builds only certain branch(es) from the main repo, and a few other things).

Take a look at this branch on my fork to see an example of what that "pointer" .gitlab-ci.yml will look like (though the "included" url will have a CI variable in it which will point to the SHA the microservice should check out before running spack release-jobs ...). You can see the resulting pipeline from that testing branch on our "staging" gitlab here.

Let me know what you think. Or if my explanation wasn't very clear.

scheibelp

I have

a couple of suggestions for the future (and also checking if I'm in sync)
some documentation requests

lib/spack/spack/cmd/release_jobs.py

scheibelp · 2019-02-15T01:25:48Z

lib/spack/spack/cmd/buildcache.py

+    saveyaml.add_argument(
+        '-r', '--root-spec', default=None,
+        help='Root spec of dependent spec')
+    saveyaml.add_argument(


IMO this would make sense to move to spack spec. Since a follow-up PR is planned, I think this could be handled there. I'm curious if you agree.

Just to be sure I understand you: you're suggesting it should be a subcommand in lib/spack/spack/cmd/spec.py? Hmmm, yeah, I do agree with you if that's the case. I probably should have thought of that when you asked me to move the actual functionality that implements the command into the spack.spec module. Suddenly it looks a little out of place here.

But yes, let's hold off on this for the next PR.

lib/spack/spack/cmd/buildcache.py

scheibelp · 2019-02-15T01:55:46Z

bin/rebuild-package.sh

+    spack mirror add remote_binary_mirror ${MIRROR_URL}
+
+    # Now download it
+    spack -d buildcache download --spec-yaml "${SPEC_YAML_PATH}" --path "${BUILD_CACHE_DIR}/" --require-cdashid


Ideally the spack buildcache command wouldn't have to be aware of CDash. I think this would go hand-in-hand though with extracting CDash-awareness out of spack install (which IIRC you were planning to do later to capture more of the process of creating the tarball).

So I did remove understanding of CDash from the binary_distribution.py module, as you requested in a previous round of change requests. But at the time I didn't see how to remove it from here without forcing the caller (the user entering the command in the terminal) to understand how the buildcache entry files are named. But maybe it's as simple as just grabbing any files with the same path and prefix as the .spec.yaml file, but in that case, we'd need to download some kind of index of all files in the root of the build_cache directory to know what those were.

I hadn't thought of this as necessarily related to the CDash-awareness of spack install, except at a higher conceptual level. But maybe it is. Though you're not suggesting we re-work the way reporters are hooked into the install process in general, are you?

Didn't do anything here, as we covered this earlier today in conversation.

bin/rebuild-package.sh

lib/spack/spack/cmd/buildcache.py

lib/spack/spack/util/web.py

lib/spack/spack/test/spec_set.py

scottwittenburg · 2019-02-15T04:18:29Z

Thanks for another careful round of review and more thoughtful comments @scheibelp, I should be able to knock these all out before we talk tomorrow.

- add CombinatorialSpecSet in spack.util.spec_set module. - class is iterable and encaspulated YAML parsing and validation. - Adjust YAML format to be more generic - YAML spec-set format now has a `matrix` section, which can contain multiple lists of specs, generated different ways. Including: - specs: a raw list of specs. - packages: a list of package names and versions - compilers: a list of compiler names and versions - All of the elements of `matrix` are dimensions for the build matrix; we take the cartesian product of these lists of specs to generate a build matrix. This means we can add things like [^mpich, ^openmpi] to get builds with different MPI versions. It also means we can multiply the build matrix out with lots of different parameters. - Add a schema format for spec-sets

Adds four new sub-commands to the buildcache command: 1. save-yaml: Takes a root spec and a list of dependent spec names, along with a directory in which to save yaml files, and writes out the full spec.yaml for each of the dependent specs. This only needs to concretize the root spec once, then indexes it with the names of the dependent specs. 2. check: Checks a spec (via either an abstract spec or via a full spec.yaml) against remote mirror to see if it needs to be rebuilt. Comparies full_hash stored on remote mirror with full_hash computed locally to determine whether spec needs to be rebuilt. Can also generate list of specs to check against remote mirror by expanding the set of release specs expressed in etc/spack/defaults/release.yaml. 3. get-buildcache-name: Makes it possible to attempt to read directly the spec.yaml file on a remote or local mirror by providing the path where the file should live based on concretizing the spec. 4. download: Downloads all buildcache files associated with a spec on a remote mirror, including any .spack, .spec, and .cdashid files that might exist. Puts the files into the local path provided on the command line, and organizes them in the same hierarchy found on the remote mirror This commit also refactors lib/spack/spack/util/web.py to expose functionality allowing other modules to read data from a url.

The built images are set up with fairly recent versions of gcc and clang: - centos_7: [ [email protected] (built from src), [email protected] (spack-built from src) ] - ubuntu_18.04: [ [email protected] (system), [email protected] (system) ]

This spack command adds a new schema for a file which describes the builder containers available, along with the compilers availabe on each builder. The release-jobs command then generates the .gitlab-ci.yml file by first expanding the release spec set, concretizing each spec (in an appropriate docker container if --this-machine-only argument is not provided on command line), and then combining and staging all the concrete specs as jobs to be run by gitlab.

scheibelp · 2019-02-21T19:38:19Z

Looks good: thanks! Once you have refactored the commits, let me know and I can browse them and merge this.

scottwittenburg · 2019-02-21T19:56:41Z

Thanks for all the hard work reviewing this beast @scheibelp. I have force-pushed the rewritten history, so have a look at the commit messages when you get a chance. Those will be easy enough for me to update if you want changes.

scheibelp · 2019-02-21T21:37:48Z

Thanks!

gartung · 2019-02-28T02:56:53Z

This error was produced when trying to create a buildcache for [email protected] that was linked against system zlib which has an entry for a version 1.2.7 that does not exists in the zliz package.py

  zlib:
    paths:
      [email protected]+optimize+pic+shared arch=linux-rhel7-x86_64: /usr

==> [2019-02-27-20:52:54.622705] Warning: Missing a source id for [email protected]
Traceback (most recent call last):
  File "/build/work/spack-tools/spack/bin/spack", line 48, in <module>
    sys.exit(spack.main.main())
  File "/build/work/spack-tools/spack/lib/spack/spack/main.py", line 675, in main
    return _invoke_command(command, parser, args, unknown)
  File "/build/work/spack-tools/spack/lib/spack/spack/main.py", line 446, in _invoke_command
    return_val = command(parser, args)
  File "/build/work/spack-tools/spack/lib/spack/spack/cmd/buildcache.py", line 528, in buildcache
    args.func(args)
  File "/build/work/spack-tools/spack/lib/spack/spack/cmd/buildcache.py", line 326, in createtarball
    not args.no_rebuild_index)
  File "/build/work/spack-tools/spack/lib/spack/spack/binary_distribution.py", line 363, in build_tarball
    spec_dict['full_hash'] = spec.full_hash()
  File "/build/work/spack-tools/spack/lib/spack/spack/spec.py", line 1306, in full_hash
    package_hash = self.package.content_hash()
  File "/build/work/spack-tools/spack/lib/spack/spack/package.py", line 1092, in content_hash
    for p in self.spec.patches)
  File "/build/work/spack-tools/spack/lib/spack/llnl/util/lang.py", line 183, in __call__
    self.cache[args] = self.func(*args)
  File "/build/work/spack-tools/spack/lib/spack/spack/spec.py", line 2690, in patches
    for sha256 in self.variants['patches']._patches_in_order_of_appearance:
AttributeError: 'MultiValuedVariant' object has no attribute '_patches_in_order_of_appearance'

gartung · 2019-02-28T02:58:15Z

@scottwittenburg it appears to happen while calculating spec.full_hash()

scottwittenburg · 2019-02-28T15:22:08Z

@scheibelp I had a fix for this in my branch at one time, but took it out after some discussion that it was not required anymore. Should I dig it up and create a PR? It was just a guard against using that attribute if it didn't exist.

gartung · 2019-02-28T16:04:47Z

The fix is trivial.

diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py
index 76c06d3..2b132fa 100644
--- a/lib/spack/spack/spec.py
+++ b/lib/spack/spack/spec.py
@@ -2678,6 +2678,10 @@ class Spec(object):
         if not self.concrete:
             raise SpecError("Spec is not concrete: " + str(self))
 
+        if not hasattr(self.variants['patches'],
+                       '_patches_in_order_of_appearance'):
+            return []
+
         if 'patches' not in self.variants:
             return []

scottwittenburg · 2019-02-28T16:07:25Z

@gartung Indeed, that's what I had in my branch, but I removed because after discussing it with @scheibelp, it seemed that should not be needed.

gartung · 2019-02-28T16:12:36Z

It's a weird corner case where the package was built before
variants['patches']._patches_in_order_of_appearance
was added.

I have other packages built in another (newer) spack directory where the same [email protected] was used in packages.yaml and packages built against it do not give this error when creating the buildcache.

I added this patch to my current buildcache-placeholder-fix branch. We'll see if it passes review.

scottwittenburg · 2019-02-28T16:16:25Z

It's a weird corner case where the package was built before
variants['patches']._patches_in_order_of_appearance
was added.

Ah, then it makes sense why I seemed to need it in my local testing, but in general it seems like it shouldn't be necessary.

I have other packages built in another (newer) spack directory where the same [email protected] was used in packages.yaml and packages built against it do not give this error when creating the buildcache.

I added this patch to my current buildcache-placeholder-fix branch. We'll see if it passes review.

Perhaps a link to this conversation from your current PR would help the reviewer understand why that guard is there.

Thanks for this.

This was referenced Jan 7, 2019

WIP: Generate gitlab ci yaml #8718

Closed

Add full_hash and json index to buildcache #8451

Closed

scottwittenburg force-pushed the release-workflow branch 3 times, most recently from 98f9ffe to 2f49576 Compare January 8, 2019 16:52

scottwittenburg force-pushed the release-workflow branch from 2f49576 to 53f4c89 Compare January 8, 2019 20:39

gartung added the buildcache label Jan 9, 2019

tgamblin self-requested a review January 9, 2019 09:01

tgamblin added the WIP label Jan 9, 2019

scottwittenburg force-pushed the release-workflow branch 3 times, most recently from 89375e7 to 527e9ce Compare January 15, 2019 19:21

scheibelp reviewed Feb 15, 2019

View reviewed changes

gartung mentioned this pull request Feb 15, 2019

Buildache install fix: replace placeholder and origin install paths in rpath when relocating binaries #10287

Closed

scottwittenburg force-pushed the release-workflow branch from e9e4795 to 8f636fd Compare February 16, 2019 00:38

tgamblin and others added 5 commits February 18, 2019 12:22

release workflow: Add build scripts for jobs and means to upload pkgs

330c678

scheibelp approved these changes Feb 21, 2019

View reviewed changes

scottwittenburg force-pushed the release-workflow branch from 8f636fd to 330c678 Compare February 21, 2019 19:51

scheibelp merged commit 5600c9f into spack:develop Feb 21, 2019

obreitwi mentioned this pull request Mar 1, 2019

Add multiprocessing to buildcache command #9440

Closed

scottwittenburg mentioned this pull request Mar 18, 2019

Allow 'spack location' to take path to a spec.yaml file #10912

Closed

scheibelp mentioned this pull request Apr 21, 2019

Add container sub-command to generate container spec files #10952

Closed

tgamblin removed the WIP label Jun 30, 2019

tgamblin mentioned this pull request Jun 30, 2019

add CombinatorialSpecSet class for taking cross-products of Specs. #8574

Closed

1 task

This was referenced Aug 22, 2019

Ensure that binaries are rebuilt when changes are made to package.py files on a release branch #8387

Closed

package.py hashing #7119

Closed

Conversation

scottwittenburg commented Jan 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

One approach to managing the release workflow

Details of release-jobs command

Additional Notes

Uh oh!

gartung commented Jan 8, 2019

Uh oh!

gartung commented Jan 8, 2019

Uh oh!

scottwittenburg commented Jan 8, 2019

Uh oh!

gartung commented Jan 8, 2019

Uh oh!

scottwittenburg commented Jan 8, 2019

Uh oh!

gartung commented Jan 8, 2019

Uh oh!

scottwittenburg commented Jan 8, 2019

Uh oh!

gartung commented Jan 8, 2019

Uh oh!

gartung commented Jan 9, 2019

Uh oh!

gartung commented Jan 9, 2019

Uh oh!

gartung commented Jan 9, 2019

Uh oh!

tgamblin commented Jan 9, 2019

Uh oh!

scottwittenburg commented Jan 9, 2019

Uh oh!

balay commented Jan 9, 2019

Uh oh!

scottwittenburg commented Jan 10, 2019

Uh oh!

scottwittenburg commented Jan 10, 2019

Uh oh!

scottwittenburg commented Jan 18, 2019

Uh oh!

scottwittenburg commented Jan 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scottwittenburg commented Feb 15, 2019

Uh oh!

scheibelp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

scheibelp Feb 15, 2019

Choose a reason for hiding this comment

Uh oh!

scottwittenburg Feb 15, 2019

Choose a reason for hiding this comment

Uh oh!

scottwittenburg Feb 15, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

scheibelp Feb 15, 2019

Choose a reason for hiding this comment

Uh oh!

scottwittenburg Feb 15, 2019

Choose a reason for hiding this comment

Uh oh!

scottwittenburg Feb 15, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

scottwittenburg commented Feb 15, 2019

Uh oh!

scheibelp commented Feb 21, 2019

Uh oh!

scottwittenburg commented Feb 21, 2019

Uh oh!

scottwittenburg commented Jan 7, 2019 •

edited

Loading

Details of `release-jobs` command

scottwittenburg commented Jan 18, 2019 •

edited

Loading