spack containerize: permit to customize the base images by alalazo · Pull Request #15028 · spack/spack

alalazo · 2020-02-17T18:49:48Z

This PR reworks a few attributes in the container subsection of spack.yaml to permit the injection of custom base images both in the build and final stage. In more detail, users can still specify the base operating system and Spack version they want to use:

spack:
  container:
    images:
      os: ubuntu:18.04
      spack: develop

in which case the generated recipe will use one of the Spack images built on Docker Hub for the build stage and the base OS image in the final stage or alternatively they can specify explicitly the two base images:

spack:
  container:
    images:
      build: spack/ubuntu-bionic:latest
      final: ubuntu:18.04

and it will be up to them to ensure their consistency. The ability to install OS packages in both the build and final stage has been retained. Handles to avoid an update of the available system packages have been added to the configuration to facilitate the generation of recipes permitting deterministic builds.

Following the discussion in #14917 this PR takes at the moment a minimalist approach and thus removed the possibility to install OS packages in the generated image (if any is needed it should be the custom base image that takes care of that). In case this seems unwanted it will be sufficient to drop commit ed71483 to make this PR compatible with #14917 and #14879.

alalazo · 2020-02-17T19:12:17Z

@ChristianTackeGSI @samcmill

samcmill · 2020-02-19T16:06:47Z

My 2 cents is to retain the ability to install OS packages in the image.

The rationale is to have something in-between the canned images (zero effort but zero control) and custom images (lots of control but potentially lots of effort).

hartzell

I like this. Few inline comments.

lib/spack/docs/containers.rst

reidpr · 2020-03-05T16:46:20Z

Thanks for your hard work on this PR.

May I make a friendly request? Can everyone please use gender-neutral language when describing users? For example, instead of “a user can still specify ... Spack version he wants to use”, write “a user can still specify ... Spack version they want to use”.

Among lots of other good reasons, this would be in keeping with the Spack COC, e.g. the admonition to use “inclusive language”.

alalazo · 2020-03-05T18:03:24Z

@reidpr Edit done. I'll have a look at the docs later and adjust it too if I find anything. Thanks.

victorusu · 2020-03-06T09:52:55Z

This is great! please merge!

alalazo · 2020-03-06T11:33:36Z

@hartzell Do you have further requests wrt to docs or anything else?

alalazo · 2020-03-11T19:41:08Z

@svenevs I thought that given the positive comment in Spack's slack on spack containerize you might be interested in this PR 🙂

lib/spack/docs/containers.rst

scheibelp

I have a few minor requests but overall I think that "build"/"final" and "os"/"spack" are duplicate ways to specify the same thing, which I think is confusing. I added more detail (and what I'd like to see change) in the comment prefixed with "main concern".

share/spack/templates/container/singularity.def

lib/spack/spack/schema/container.py

hartzell

I'm not in a position to test this right now, but it looks good. Thanks!

lib/spack/docs/containers.rst

hartzell · 2020-07-09T00:23:19Z

This might be off tangent, but since I mentioned it in my experiment above....

I "want" to use CentOS and/but I need to use newer compilers. My solution to that is to include [email protected] in the build image, but I discovered that I also need to include it in the final image. If I remove it, various libraries end up missing. E.g. bcl2fastq2 depends on boost which ends up needing the libgomp in the gcc install tree. (Oddly enough, libgomp is included in the example, but even if I use the yum version I end up missing other things).

So, in spite of vary carefully doing a multistage build, I end up with a fairly large image (gcc is about 1.2GB). I could thin it by carefully trimming most of the gcc install tree (leaving just lib and lib64 might work), but that seems fragile.

Is there something else I could/should be doing (don't say CentOS8...), or is this a fact of life?

hartzell · 2020-10-13T23:51:51Z

Hi All. Any movement on this?

alalazo · 2020-10-15T13:22:26Z

@hartzell I'm keeping this PR rebased semi-frequently. Will try to have it merged for the next major release.

hartzell · 2020-10-15T18:33:02Z

Great. I'm currently running spack containerize and piping the output through sed, so I'll look forward to this landing.

FWIW, my custom builder pre-builds gcc while my custom final needs to include our internal root CA certs.

Is there a timeframe for the next major release?

alalazo · 2020-10-15T18:35:12Z

Is there a timeframe for the next major release?

Before SC next month 🙂

hartzell · 2020-10-15T18:36:10Z

Woo hoo! Thanks!

The ``base`` attribute has been renamed to ``images`` while the ``base:image`` attribute has been renamed ``images:os``. This commit is in preparation of a generalization to permit the use of custom base images in both the build and final stage.

The ``images`` attribute has now another mode of operation where users can specify both a ``build`` and a ``final`` custom base image.

Since now the base images can be customized it's redundant to also have a configure option to install os packages.

This reverts commit 05fded7.

refers spack#14802

… and final stage

Also fixed indentation as Paul suggested in another review.

The documentation has been updated according to reviews. Examples have been extended and the "Setting Base Images" section has been divided into two subsections - one for each selection method.

scheibelp

My main request is to address whitespace handling in the template files (see https://github.com/spack/spack/pull/15028/files#r399576494). The rest of the comments are very minor (generally typos).

lib/spack/docs/containers.rst

lib/spack/spack/schema/container.py

lib/spack/docs/containers.rst

…docs

* releases/v0.16: (6003 commits) update CHANGELOG.md for v0.16.0 bump version number to 0.16.0 clingo: add `master` branch version (spack#19958) cmd: add `spack mark` command (spack#16662) spack test (spack#15702) Added -level_zero -rocm -opencl flags and sha256 for TAU v2.30. (spack#19962) Improve warning message for deprecated attributes in "packages.yaml" Documentation: spack load/environments prefix inspections (spack#19961) spack load/environments: allow customization of prefix inspections (spack#18260) spack containerize: allow users to customize the base image (spack#15028) concretizer: modified weights for providers and matching for externals concretizer: maximize the number of default values used for a single variant concretizer: don't require a provider for virtual deps if spec is external concretizer: spec_clauses() shouldn't emit node_compiler_hard for rule bodies. concretizer: don't generate rules for empty version lists concretizer: add a rule to avoid cycles in the graph of dependencies External packages have a consistent hash across different concretizers Don't fail if MV variants have a tuple as default value Fixup for target preferences Added unit tests to for regressions on open concretizer bugs ... # Conflicts: # .gitignore # lib/spack/spack/binary_distribution.py # lib/spack/spack/cmd/setup.py # lib/spack/spack/modules/common.py # var/spack/repos/builtin/packages/med/package.py # var/spack/repos/builtin/packages/mofem-cephas/package.py # var/spack/repos/builtin/packages/mofem-fracture-module/package.py # var/spack/repos/builtin/packages/mofem-users-modules/package.py # var/spack/repos/builtin/packages/mpich/package.py # var/spack/repos/builtin/packages/petsc/package.py # var/spack/repos/builtin/packages/python/package.py

alalazo added feature A feature is missing in Spack containers labels Feb 17, 2020

alalazo requested review from becker33, hartzell and tgamblin February 17, 2020 19:11

hartzell suggested changes Feb 19, 2020

View reviewed changes

lib/spack/docs/containers.rst Outdated Show resolved Hide resolved

lib/spack/docs/containers.rst Show resolved Hide resolved

alalazo force-pushed the container/minimal_approach branch from ed71483 to 05fded7 Compare February 21, 2020 13:31

alalazo self-assigned this Feb 25, 2020

scheibelp self-assigned this Mar 10, 2020

scheibelp self-requested a review March 10, 2020 18:04

paulbry reviewed Mar 24, 2020

View reviewed changes

lib/spack/docs/containers.rst Show resolved Hide resolved

tgamblin self-assigned this Mar 24, 2020

scheibelp requested changes Mar 28, 2020

View reviewed changes

alalazo mentioned this pull request Apr 2, 2020

Installation issue: CUDA package create a folder in /usr/local #15797

Closed

2 tasks

hartzell approved these changes Apr 4, 2020

View reviewed changes

alalazo mentioned this pull request Apr 6, 2020

Effective Use of Spack Environments #15842

Closed

alalazo force-pushed the container/minimal_approach branch from 63333ba to 86b08ba Compare April 14, 2020 12:12

scheibelp reviewed Apr 27, 2020

View reviewed changes

lib/spack/docs/containers.rst Outdated Show resolved Hide resolved

alalazo force-pushed the container/minimal_approach branch from 86b08ba to 40c0337 Compare May 4, 2020 14:33

alalazo force-pushed the container/minimal_approach branch from 5a452de to 5e91dd9 Compare May 12, 2020 09:35

alalazo mentioned this pull request May 22, 2020

nvidia/cuda as a base image for spack #16695

Open

2 tasks

alalazo force-pushed the container/minimal_approach branch from 5e91dd9 to d8f1713 Compare June 18, 2020 12:36

alalazo force-pushed the container/minimal_approach branch from d8f1713 to d6acdd2 Compare June 26, 2020 13:07

albestro mentioned this pull request Jul 9, 2020

Testing the spack package eth-cscs/DLA-Future#259

Closed

alalazo force-pushed the container/minimal_approach branch from 65c0e52 to 63711bb Compare September 9, 2020 11:29

alalazo force-pushed the container/minimal_approach branch from 63711bb to 70f5151 Compare October 19, 2020 09:10

alalazo mentioned this pull request Oct 30, 2020

Update documentation on containers #19631

Merged

2 tasks

alalazo added 11 commits November 16, 2020 14:13

Permit to customize base images in container recipes

7e6941f

The ``images`` attribute has now another mode of operation where users can specify both a ``build`` and a ``final`` custom base image.

Remove the possibility to declare os packages to be installed

35a32c0

Since now the base images can be customized it's redundant to also have a configure option to install os packages.

Revert "Remove the possibility to declare os packages to be installed"

7889159

This reverts commit 05fded7.

spack containerize: permit to install os packages in the build stage

41047ef

refers spack#14802

Packages directly under "os_packages" are installed in both the build…

31f3142

… and final stage

Avoid for loop in favor of template manipulation

b189ca7

containerize: permit to specify the os pkg manager

6fb034d

containerize: option to avoid updating the system pkg manager cache

2994e8d

spack containerize: updated documentation

7ae38f4

containerize: updated docs according to Peter's comment

b0a3dd3

Also fixed indentation as Paul suggested in another review.

alalazo force-pushed the container/minimal_approach branch from 70f5151 to b0a3dd3 Compare November 16, 2020 13:42

containerize: updated documentation

6ccff2a

The documentation has been updated according to reviews. Examples have been extended and the "Setting Base Images" section has been divided into two subsections - one for each selection method.

scheibelp requested changes Nov 17, 2020

View reviewed changes

scheibelp reviewed Nov 17, 2020

View reviewed changes

lib/spack/docs/containers.rst Outdated Show resolved Hide resolved

containerize: improved readability of template files, fixed typos in …

30dc964

…docs

scheibelp approved these changes Nov 17, 2020

View reviewed changes

scheibelp merged commit 5f636fc into spack:develop Nov 17, 2020

alalazo deleted the container/minimal_approach branch November 17, 2020 19:25

Conversation

alalazo commented Feb 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alalazo commented Feb 17, 2020

Uh oh!

samcmill commented Feb 19, 2020

Uh oh!

hartzell left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

reidpr commented Mar 5, 2020

Uh oh!

alalazo commented Mar 5, 2020

Uh oh!

victorusu commented Mar 6, 2020

Uh oh!

alalazo commented Mar 6, 2020

Uh oh!

alalazo commented Mar 11, 2020

Uh oh!

Uh oh!

scheibelp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hartzell left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hartzell commented Jul 9, 2020

Uh oh!

hartzell commented Oct 13, 2020

Uh oh!

alalazo commented Oct 15, 2020

Uh oh!

hartzell commented Oct 15, 2020

Uh oh!

alalazo commented Oct 15, 2020

Uh oh!

hartzell commented Oct 15, 2020

Uh oh!

scheibelp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

alalazo commented Feb 17, 2020 •

edited

Loading