Generate container recipes with environments by alalazo · Pull Request #14202 · spack/spack

alalazo · 2019-12-17T23:07:34Z

depends on #13534
closes #7204

This PR adds a new command to Spack:

$ spack containerize -h
usage: spack containerize [-h] [--config CONFIG]

creates recipes to build images for different container runtimes

optional arguments:
  -h, --help       show this help message and exit
  --config CONFIG  configuration for the container recipe that will be generated

which takes an environment with an additional container section:

spack:
  specs:
  - gromacs build_type=Release 
  - mpich
  - fftw precision=float
  packages:
    all:
      target: [broadwell]

  container:
    # Select the format of the recipe e.g. docker,
    # singularity or anything else that is currently supported
    format: docker
    
    # Select from a valid list of images
    base:
      image: "ubuntu:18.04"
      spack: prerelease

    # Additional system packages that are needed at runtime
    os_packages:
    - libgomp1

and turns it into a Dockerfile or a Singularity definition file, for instance:

# Build stage with Spack pre-installed and ready to be used
FROM spack/ubuntu-bionic:prerelease as builder

# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir /opt/spack-environment \
&&  (echo "spack:" \
&&   echo "  specs:" \
&&   echo "  - gromacs build_type=Release" \
&&   echo "  - mpich" \
&&   echo "  - fftw precision=float" \
&&   echo "  packages:" \
&&   echo "    all:" \
&&   echo "      target:" \
&&   echo "      - broadwell" \
&&   echo "  config:" \
&&   echo "    install_tree: /opt/software" \
&&   echo "  concretization: together" \
&&   echo "  view: /opt/view") > /opt/spack-environment/spack.yaml

# Install the software, remove unecessary deps and strip executables
RUN cd /opt/spack-environment && spack install && spack autoremove -y
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
    xargs file -i | \
    grep 'charset=binary' | \
    grep 'x-executable\|x-archive\|x-sharedlib' | \
    awk -F: '{print $1}' | xargs strip -s


# Modifications to the environment that are necessary to run
RUN cd /opt/spack-environment && \
    spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh

# Bare OS image to run the installed executables
FROM ubuntu:18.04

COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software
COPY --from=builder /opt/view /opt/view
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh

RUN apt-get -yqq update && apt-get -yqq upgrade                                   \
 && apt-get -yqq install libgomp1 \
 && rm -rf /var/lib/apt/lists/*

ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]

victorusu · 2019-12-20T10:17:56Z

@alalazo, just to clarify... The PR will stop at the phase of creating the Dockerfile or a Singularity definition file? Or will it continue to build an image?

alalazo · 2019-12-20T10:34:25Z

@victorusu It will just create the recipe. The rationale is that a recipe can be consumed in many ways and it wouldn't make sense for Spack to force users to employ only the ones we'll wrap (e.g. you can send it to a remote builder, build it with different tools, etc.). I also think it wouldn't be that great convenience to users to have the build wrapped, while there will be some maintenance cost associated with it.

alalazo

@becker33 This should be ready for a first review, I left a couple of comments below and feel free to ask anything if things seem unclear. Thanks!

lib/spack/spack/test/container/test_cli.py

share/spack/templates/container/Dockerfile

sknigh · 2020-01-09T17:07:28Z

If I wanted my container ignore certs (e.g. spack install -k) how would I do that? Does the existing spack environment schema already support it?

lib/spack/docs/containers.rst

alalazo · 2020-01-13T10:44:09Z

If I wanted my container ignore certs (e.g. spack install -k) how would I do that? Does the existing spack environment schema already support it?

I think:

config:
  verify_ssl: false

in the Spack environment should do it.

becker33

A couple questions about the tests, and a couple comments about code clarity.

lib/spack/spack/container/__init__.py

lib/spack/spack/test/container/test_docker.py

lib/spack/spack/container/writers/__init__.py

sknigh · 2020-01-23T16:47:54Z

@alalazo This implementation can also save space by deleting .spack directories and static libraries before squashing.

Something like this...

# Remove static libraries
find /opt/software/* -type f -name \*.a | xargs rm -f
find /opt/view/* -type l -name \*.a | xargs rm -f

# Remove .spack directories
rm -rf /opt/software/*/.spack

tgamblin

I have some minor change requests, but otherwise this is nearly ready to go!

lib/spack/docs/containers.rst

lib/spack/spack/cmd/containerize.py

lib/spack/spack/container/images.json

lib/spack/docs/containers.rst

The command 'spack containerize' has been added to generate recipes to build container images from a spack.yaml file. The image format supported so far are 'docker' and 'singularity'. Recipe generation is based on Jinja2 templates, most of the logic is under the 'spack.container' package.

Before we were sourcing a file under profile.d as it is done in Docker

This adds the possibility to customize `%runscript`, `%startscript`, `%help` and `%test`.

alalazo · 2020-01-30T14:10:48Z

@tgamblin I think the requests have been addressed. Let me know if there's anything else to be done.

samcom12 · 2020-02-13T09:02:33Z

Can you please share a sample to build for singularity recipe with spack command?

alalazo · 2020-02-13T09:05:29Z

@samcom12 If you use:

format: singularity

Spack should output a valid Singularity definition file.

samcom12 · 2020-02-17T10:45:01Z

Thank you Alalazo...
Can you please share a command to create a singularity def file using spack command.
Sorry for troubling you, but I didn't get a example for same.

alalazo added feature A feature is missing in Spack environments containers labels Dec 17, 2019

alalazo requested a review from tgamblin December 17, 2019 23:07

This comment has been minimized.

Sign in to view

alalazo force-pushed the features/container_command branch 3 times, most recently from 934cf5c to 5a65ada Compare December 27, 2019 15:56

alalazo mentioned this pull request Jan 2, 2020

Spack can automatically remove unused specs #13534

Merged

alalazo force-pushed the features/container_command branch from 5a65ada to c4b306d Compare January 2, 2020 10:09

tgamblin assigned becker33 Jan 7, 2020

alalazo force-pushed the features/container_command branch 3 times, most recently from a1b8654 to b7f1fc9 Compare January 8, 2020 15:00

alalazo commented Jan 9, 2020

View reviewed changes

lib/spack/spack/test/container/test_cli.py Show resolved Hide resolved

share/spack/templates/container/Dockerfile Show resolved Hide resolved

alalazo marked this pull request as ready for review January 9, 2020 08:48

alalazo requested a review from becker33 January 9, 2020 08:48

sknigh reviewed Jan 9, 2020

View reviewed changes

lib/spack/docs/containers.rst Show resolved Hide resolved

becker33 requested changes Jan 16, 2020

View reviewed changes

alalazo force-pushed the features/container_command branch 2 times, most recently from d5dbef5 to 12fd35f Compare January 16, 2020 10:37

becker33 approved these changes Jan 27, 2020

View reviewed changes

tgamblin requested changes Jan 28, 2020

View reviewed changes

sknigh mentioned this pull request Jan 29, 2020

Proposal to extend container recipes to support base images #14673

Open

alalazo added 9 commits January 30, 2020 09:30

Added reference documentation for container images

2362a6e

Singularity env modifications rely on $SINGULARITY_ENVIRONMENT

0f3768e

Before we were sourcing a file under profile.d as it is done in Docker

Added config sections specific to Singularity

e92b575

This adds the possibility to customize `%runscript`, `%startscript`, `%help` and `%test`.

Removed leading 'test_' from test filenames

aa1b96c

Use spack.environment.config_dict instead explicit selection

ed3faf6

Updated bash completion file

226d515

Minor edit to "extra_instructions" description in docs

504bdb2

Renamed "packages" to "os_packages" in container subsection of envs

1cd80a6

Fixed cmd/gc test by adding mutable_mock_env_path fixture

2e21587

alalazo force-pushed the features/container_command branch from 2c614a2 to 2e21587 Compare January 30, 2020 10:20

alalazo added 2 commits January 30, 2020 13:14

A "container" section is not required anymore to generate a recipe

65131d3

Removed redundant argument to containerize

9bfa6d3

tgamblin approved these changes Jan 31, 2020

View reviewed changes

tgamblin merged commit 9635ff3 into spack:develop Jan 31, 2020

alalazo deleted the features/container_command branch January 31, 2020 07:28

samcmill mentioned this pull request Feb 6, 2020

Additional containerize spec file customization #14802

Closed

alalazo mentioned this pull request Feb 7, 2020

Spack determine runtime and linking dependencies from build #10489

Closed

alalazo mentioned this pull request Feb 14, 2020

spack containerize: allow copying files into the container #14917

Closed

alalazo mentioned this pull request Feb 29, 2020

Improve containerize - add input/output selection #15271

Closed

alalazo mentioned this pull request Aug 19, 2020

Add container sub-command to generate container spec files #10952

Closed

Conversation

alalazo commented Dec 17, 2019 • edited by tgamblin Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

victorusu commented Dec 20, 2019

Uh oh!

alalazo commented Dec 20, 2019

Uh oh!

alalazo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sknigh commented Jan 9, 2020

Uh oh!

Uh oh!

alalazo commented Jan 13, 2020

Uh oh!

becker33 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sknigh commented Jan 23, 2020

Uh oh!

tgamblin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alalazo commented Jan 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

samcom12 commented Feb 13, 2020

Uh oh!

alalazo commented Feb 13, 2020

Uh oh!

samcom12 commented Feb 17, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

alalazo commented Dec 17, 2019 •

edited by tgamblin

Loading

alalazo commented Jan 30, 2020 •

edited

Loading