Docker-Based Developer Resource by opadron · Pull Request #5871 · spack/spack

opadron · 2017-10-23T17:29:44Z

Hi, folks!

Here's my docker-based development setup. It should be very easy to use, all you need is the latest versions of docker and docker-compose. There's a small README that explains the basic usage and there is support for running in a container running Ubuntu and Fedora, with more choices on the way if there is interest.

The code running spack in the container is the same as your local working copy, letting you experiment in a clean environment while you make changes. Installed packages show up under your local working copy under ./opt and are shared and persist between multiple containers.

We can merge, if enough people like this, or we can keep the branch as-is. The changes stand well enough on their own that they should be easy to cherry-pick.

tgamblin · 2017-10-25T11:51:47Z

Is it worth referring to this from @ax3l's Docker workflow? Seems like this could be used to kickstart some of what is in there. @ax3l?

opadron · 2017-10-25T20:04:23Z

Another important lesson that I learned that is relevant to both @ax3l's documentation and this work:

On Mac and Windows, docker runs in a hypervisor that is allocated a low amount of memory by default (e.g.: 2 GB on my 16 GB Macbook Pro). This short supply of memory was actually the cause of the strange internal compiler errors I was running into while trying to build boost. We should definitely add a note for docker users to make sure that they increase the memory allocated to the hypervisor; ideally to as much as they can tolerate dedicating to docker. All my spack builds were running beautifully once I set my system to allocate 8 GB to docker.

ax3l · 2017-10-26T20:38:39Z

Thanks for the ping and the PR!
I have not used docker-compose before but looks intuitive :)

A few thoughts:

If you install lmod (yay!) instead of modules, you might want to change the default in modules.yaml. Per default, only tcl and dotkit modules are created. Yes, lmod is backwards compatible to tcl but I think that is not how it is implemented here (or is it, @alalazo ?).

On git clone, I would use --depth <nowNumber> or use the tar download that I had in the workflow just to save resources.

MPI won't work without openssh-server installed.

For some downloads, I think ca-certificates is useful to have installed in ubuntu.

I like your use of WORKDIR, ENTRYPOINT and CMD. I did that a bit more noob-ish :)

On Mac and Windows, [...] short supply of memory [...]

Interesting observation! Yes, please add a note in the docker workflow section on that if you like!
I also made good experiences with limiting the amount of parallism for builds (in make -j) via config.yaml, especially on virtualized environments such as travis. Might also be worth a try here.

config:
  build_jobs: 2

ax3l

just a few notorious inline comments :)
cool work!

ax3l · 2017-10-26T20:39:46Z

share/spack/docker/README.md

@@ -0,0 +1,20 @@
+Using this docker container for development.
+------------


is that the proper amount of -- for formating? :)

ax3l · 2017-10-26T20:40:11Z

share/spack/docker/README.md

+    $ docker-compose run --rm ubuntu
+
+This command should drop you into an interactive shell where you can run spack
+within an isolated docker container running ubuntu.  The copy of spack being


typos:

Ubuntu

double space

ax3l · 2017-10-26T20:41:35Z

share/spack/docker/README.md

+Using this docker container for development.
+------------
+
+    $ cd share/spack/docker


if you like you can add syntax highlighting for bash like this in markdown

ax3l · 2017-10-26T20:42:29Z

share/spack/docker/spack_ubuntu/Dockerfile

@@ -0,0 +1,25 @@
+


maybe remove this empty line?

ax3l · 2017-10-26T20:42:54Z

share/spack/docker/spack_ubuntu/Dockerfile

+    git clone git://github.com/llnl/spack.git /spack && \
+    rm -rf /spack/.git && rm -rf /var/lib/apt/lists/*
+
+


dbl empty newline :)

ax3l · 2017-10-26T20:43:21Z

share/spack/docker/spack_fedora/Dockerfile

+WORKDIR /spack
+ENTRYPOINT ["bash"]
+CMD ["-l"]
+


double instead of single EOF newline

ax3l · 2017-10-26T20:43:27Z

share/spack/docker/spack_ubuntu/Dockerfile

+WORKDIR /spack
+ENTRYPOINT ["bash"]
+CMD ["-l"]
+


double instead of single EOF newline

ax3l · 2017-10-26T20:44:51Z

share/spack/docker/README.md

+you make should be immediately reflected in the running docker container.
+
+To work within a container running a different linux distro, change the "ubuntu"
+argument to any one of the services listed under the ``docker-compose.yml``


I think the double "`" is rst not md :)

if you write this file in full .rst instead of .md one can also easier include it into the main sphinx docs into the developer workflows :)

opadron · 2017-10-27T16:15:09Z

Holy hell, thank goodness for Github's "restore branch" feature...today, I am worthy of the "bonehead of the year" award! 👎

opadron · 2017-10-27T16:15:46Z

Also, if someone can reopen, that would be great! :)

opadron · 2017-10-27T19:29:44Z

@ax3l thanks a bunch for the valuable feedback. Please have a look at my latest additions and let me know what you think. Please, keep in mind that I haven't yet had the chance to update the Fedora environment, and that there seems to be a known issue related to running Fedora on docker (see https://bugzilla.redhat.com/show_bug.cgi?id=1483553 for details).

ax3l

Very nice, thank you for the updates!

I have a few structural inline comments to reduce duplication.

I am wondering if someone wants to add a centos and arch image to the composer as well?

ax3l · 2017-10-28T11:11:30Z

lib/spack/docs/docker_for_developers.rst

+=================================
+
+This guide is intended for people who want to use our prepared docker
+environments to work on developing Spack or working on spack packages.  It is


Are you intentionally making double spaces after .s? :)

Aye; chalk it up to a habit I learned from grade school. I should write a quick script to look for these things. [:

ax3l · 2017-10-28T11:12:44Z

lib/spack/docs/workflows.rst

   # image run hook: the -l will make sure /etc/profile environments are loaded
   CMD        /bin/bash -l

+For examples of how we use docker in development, see


Just to keep this section focused, I would move this fully in the Best Practices section immediately below.

New sub-sections with """" level:

docker in development (this reference)

OSX, Windows (the paragraph as text instead of a note)

ax3l · 2017-10-28T11:16:37Z

share/spack/docker/spack_fedora/modules.yaml

+# Per-user settings (overrides default and site settings):
+#   ~/.spack/modules.yaml
+# -------------------------------------------------------------------------
+modules:


if you want to overwrite the default within user space, it should be enough to write the section

modules: enable: - lmod

in this file.

I would also check this file in only once in the directory above for all docker recipes.

Good to know. Is there a way to reference files for COPY that lie outside the Dockerfile's current directory? It was my understanding that this wasn't possible. Hence, the duplicate copy.

ax3l · 2017-10-28T11:19:41Z

share/spack/docker/spack_ubuntu/handle-ssh.sh

@@ -0,0 +1,24 @@
+uid="`id -u`"


isn't this file needed in fedora as well?

Right. Fedora et al is coming, soon (TM),

ax3l · 2017-10-28T11:19:53Z

share/spack/docker/spack_ubuntu/modules.yaml

@@ -0,0 +1,47 @@
+# -------------------------------------------------------------------------


see comment above :)

ax3l · 2017-10-30T15:30:50Z

Uh, you might be right! I usually softlink them in git, but that might cause troubles for Windows contributors.

opadron · 2017-10-31T03:46:34Z

@ax3l, I think you'll really like these latest additions.

Unfortunately, github doesn't cope with my rebases well, so you should focus on the history in the actual branch -- not as it is shown on the PR.

In addition to the changes you've suggested, I've also added arch and centos docker images, giving us 4 total distros to test and develop in. You can install spack packages from all four side-by-side within the same installation root; even simultaneously, if you really wanted to and could spare the memory!

As you review, I'd like to point out specific commits:

9811cdd~3: Contains the bulk of the work, with your suggested edits and the new images.
9811cdd~2: Is an experiment I tried with making a custom prompt to help developers keep track of when they're in a docker container. The real tricky part was trying to figure out how to get prompt updates to work asynchronously, to maximize responsiveness. I ended up firing off a subshell and communicating with it using old-school temp files. I think it's pretty neat, but it did end up being more involved than I had originally planned. Give it a try and let me know what you think.
9811cdd~1: I switched a few things around to make the prompt experiment opt-in. On this commit, you can try out the prompt by setting the CUSTOM_PROMPT environment variable within the container:

 $ docker-compose run --rm -e CUSTOM_PROMPT=1 ubuntu

9811cdd: Is a tiny and temporary commit that just works around an issue with fedora by downgrading to an older version of the image. It should be removed once the issue is fixed upstream.

It's too bad the way docker works forces us to duplicate some files, but once I've gotten over the minor headache of editing some things four times, I've come to really appreciate the convenience of developing across several distros with minimal effort. I think we're getting pretty close to a really nice development asset.

Edit: Changed HEAD references to hashes so that they still work after adding more commits.

ax3l

really nice work and well documented, thank you!

I tried it with docker-compose version 1.16.1, build 6d1ac21 + Docker version 1.6.2, build 7c8fca2 and are currently getting network errors. Is there anything I am missing?

ax3l · 2017-11-01T09:04:25Z

share/spack/docker/spack_arch/Dockerfile

@@ -0,0 +1,59 @@
+FROM pritunl/archlinux


I appreciate the minimal image, but this image does not look too popular for arch.
Although there seems not to be an official arch image, there are certainly more popular and automated ones in: https://hub.docker.com/search/?isAutomated=1&isOfficial=0&q=arch
Shall we use one of those?

Indeed, there doesn't seem to be an official arch image, and I went with the one I found mentioned in Arch's docker documentation. https://wiki.archlinux.org/index.php/Docker

I remember trying the base/archlinux image and having trouble with it, though I can't recall exactly what the issue was. In any case, I'll give these other images a try and report back.

Well, I can't remember what problems I had before, but they seem to be gone, now! I'll go ahead and push up the change for the arch image's base to base/archlinux.

ax3l · 2017-11-01T09:21:33Z

lib/spack/docs/docker_for_developers.rst

+    $ docker-compose config --services
+    fedora
+    ubuntu
+    $ docker-compose run --rm fedora


somehow I get a

Creating network "spack_default" with the default driver ERROR: 404 page not found

at this point :(

I'm not sure what the 404 means, exactly, but the version of docker you reported is almost certainly too old. You will need a relatively recent version of the docker client and server in order for the different containers to work.

For comparison, this is what I get on my Macbook using the new docker for mac engine:

$ docker version

Client: Version: 17.09.0-ce API version: 1.32 Go version: go1.8.3 Git commit: afdb6d4 Built: Tue Sep 26 22:40:09 2017 OS/Arch: darwin/amd64 Server: Version: 17.09.0-ce API version: 1.32 (minimum version 1.12) Go version: go1.8.3 Git commit: afdb6d4 Built: Tue Sep 26 22:45:38 2017 OS/Arch: linux/amd64 Experimental: true

Sorry for the long delay and thx for ping-ing!

I updated to

$ docker --version Docker version 17.09.0-ce, build afdb6d4 $ docker-compose --version docker-compose version 1.17.0, build ac53b73

now... will report back in a few minutes :)

opadron · 2017-11-13T16:07:01Z

@ax3l LGTY? 🤞

ax3l · 2017-11-14T15:45:43Z

@opadron install works so far like a charm for me!

But did you try loading the installed packages? e.g. on fedora it fails for me:

$ spack install zlib
# ok! :)

$ spack load zlib
Lmod has detected the following error:  The following module(s) are
unknown: "zlib-1.2.11-gcc-5.4.1-5nuhji2" "zlib-1.2.11-clang-3.8.1-24-crspzo2"
"zlib-1.2.11-gcc-6.3.1-dmwlk6z" "zlib-1.2.11-gcc-6.3.0-hwcs5w5"

Please check the spelling or version number. Also try "module spider ..."




[root@c37e1a9d3a96 ~]# module spider zlib
Lmod has detected the following error:  Unable to find: "zlib"

on ubuntu:

# spack load zlib

Lmod Warning: MODULEPATH is undefined


Lmod has detected the following error: 
The following module(s) are unknown: "zlib-1.2.11-clang-3.8.1-24-crspzo2" "zlib-1.2.11-gcc-5.4.1-5nuhji2" "zlib-1.2.11-gcc-6.3.1-dmwlk6z" "zlib-1.2.11-gcc-5.4.0-swly52a" "zlib-1.2.11-gcc-6.3.0-hwcs5w5"

   Please check the spelling or version number. Also try "module spider ..."

opadron · 2017-11-15T16:51:39Z

I'm inclined to call this a bug in Spack, because it runs counter to what I had come to expect as a Spack user. Considering that Spack doesn't seem to have a problem installing and reporting on installed packages from multiple distributions side-by-side (that is, within the same Spack database), it would seem to imply that Spack should be able to cope with such a situation when asked to load a module, but I have confirmed that this multi-distro scenario is what causes the various errors you are seeing.

To test this yourself, just rm -r ./opt from within the spack git dir, and then try your zlib test, again. Spack works as expected on all four distros when starting from a clean DB. When there is installation data from other distros lying around is when Spack's load function seems to break down.

There is a simple workaround that I have already prepared where every distro uses a separate Spack DB, but I think we should have a discussion about the underlying issue before deciding that we want this workaround.

/cc @tgamblin

opadron · 2017-11-15T17:02:35Z

#6245 Seems to be related to the issue I wrote about above, which tells me at least one other person thinks this is a bug in Spack.

If we're indeed willing to call this a bug in Spack, I would suggest that we continue to consider this PR for merging in spite of the bug.

ax3l

all right from my side, besides the loading issue on multi arch installs which is an orthogonal issue

alalazo · 2017-11-16T14:13:10Z

share/spack/docker/spack_arch/Dockerfile

+        python              \
+        sudo                \
+        tcl              && \
+    git clone --depth 1 git://github.com/llnl/spack.git /spack              && \


I think the urls should be updated.

alalazo

I took the time to go through the code, and I really like the idea of bind mounting volumes to get a light-weight multi-platform environment using docker.

alalazo · 2017-11-20T10:28:45Z

If you install lmod (yay!) instead of modules, you might want to change the default in modules.yaml. Per default, only tcl and dotkit modules are created. Yes, lmod is backwards compatible to tcl but I think that is not how it is implemented here (or is it, @alalazo ?).

@ax3l We have to distinguish between lmod the tool (which is completely backward compatible with environment-modules) and lmod the module file format. The latter could have been called "hierarchical lua module files", because it's what we actually generate.

So, to summarize:

lmod as a tool can be installed and used instead of environment-modules. It can interpret both tcl and lmod module files.
lmod as a module file format means we are using "hierarchical lua" module files. It's only understood by the lmod tool, and imo it's useful only if you plan to expose software directly (without any mediation layer, like spack load)

ax3l · 2017-11-20T15:15:47Z

and lmod the module file format

sorry for my short note, I was referring indeed to the lmod tool that is still able to read "legacy" ~~lua~~ tcl files from environment-modules...

alalazo · 2017-11-20T15:22:45Z

is still able to read "legacy" lua...

"legacy" tcl...

ax3l · 2018-01-24T23:16:15Z

@opadron @alalazo should we merge/finalize this PR in anyway?

tgamblin · 2018-02-27T08:36:22Z

Thanks @opadron!

@ax3l

* add docker-based development resources * incorporate feedback from @ax3l * fix docs/improve ssh handling * experiment with custom prompt * add arch/fix missing core_compilers key * add centos/minor tweaks * make prompt experiment optional * workaround problem with latest fedora docker image * add docker documentation page to index toc * try another documentation fix * switch arch linux base to base/archlinux * update the git urls in the Dockerfiles * add opensuse * switch CUSTOM_PROMPT variable to simply "PROMPT"

@ax3l

* add docker-based development resources * incorporate feedback from @ax3l * fix docs/improve ssh handling * experiment with custom prompt * add arch/fix missing core_compilers key * add centos/minor tweaks * make prompt experiment optional * workaround problem with latest fedora docker image * add docker documentation page to index toc * try another documentation fix * switch arch linux base to base/archlinux * update the git urls in the Dockerfiles * add opensuse * switch CUSTOM_PROMPT variable to simply "PROMPT"

add docker-based development resources

9384987

opadron mentioned this pull request Oct 24, 2017

Fix issue where a duplicate patch is being applied to boost #5934

Merged

tgamblin added the containers label Oct 25, 2017

tgamblin requested a review from ax3l October 25, 2017 11:52

ax3l reviewed Oct 26, 2017

View reviewed changes

opadron closed this Oct 27, 2017

opadron deleted the docker-dev-resources branch October 27, 2017 16:11

opadron restored the docker-dev-resources branch October 27, 2017 16:13

scheibelp reopened this Oct 27, 2017

incorporate feedback from @ax3l

0c25c8d

ax3l reviewed Oct 28, 2017

View reviewed changes

Omar Padron added 4 commits October 30, 2017 18:55

fix docs/improve ssh handling

6321c29

add arch/fix missing core_compilers key

e6e0c25

add centos/minor tweaks

8f52835

experiment with custom prompt

ce52da7

Omar Padron added 2 commits October 30, 2017 23:54

make prompt experiment optional

9a151aa

workaround problem with latest fedora docker image

9811cdd

opadron force-pushed the docker-dev-resources branch from 3160279 to 9811cdd Compare October 31, 2017 03:54

Omar Padron added 2 commits October 31, 2017 11:10

add docker documentation page to index toc

f06023c

try another documentation fix

89ea39c

ax3l reviewed Nov 1, 2017

View reviewed changes

switch arch linux base to base/archlinux

f83303c

ax3l approved these changes Nov 16, 2017

View reviewed changes

alalazo reviewed Nov 16, 2017

View reviewed changes

update the git urls in the Dockerfiles

642b9aa

alalazo mentioned this pull request Nov 20, 2017

dealii: add and use CudaPackage with all related variants and conflicts #6070

Merged

alalazo approved these changes Nov 20, 2017

View reviewed changes

Omar Padron added 2 commits November 27, 2017 13:11

add opensuse

26eacb4

switch CUSTOM_PROMPT variable to simply "PROMPT"

7640fd4

ax3l approved these changes Nov 27, 2017

View reviewed changes

ax3l mentioned this pull request Jan 29, 2018

initial version of Dockerfile and README #6279

Closed

tgamblin merged commit 3a9cd26 into spack:develop Feb 27, 2018

opadron deleted the docker-dev-resources branch February 27, 2018 13:48

opadron mentioned this pull request Jun 11, 2018

Docker dedupe #8441

Merged

		@@ -0,0 +1,20 @@
		Using this docker container for development.
		------------

		git clone git://github.com/llnl/spack.git /spack && \
		rm -rf /spack/.git && rm -rf /var/lib/apt/lists/*

		@@ -0,0 +1,47 @@
		# -------------------------------------------------------------------------

Conversation

opadron commented Oct 23, 2017 • edited by tgamblin Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tgamblin commented Oct 25, 2017

Uh oh!

opadron commented Oct 25, 2017

Uh oh!

ax3l commented Oct 26, 2017

Uh oh!

ax3l left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

opadron commented Oct 27, 2017

Uh oh!

opadron commented Oct 27, 2017

Uh oh!

opadron commented Oct 27, 2017

Uh oh!

ax3l left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ax3l commented Oct 30, 2017 via email

Uh oh!

opadron commented Oct 31, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ax3l left a comment

Choose a reason for hiding this comment

Uh oh!

ax3l Nov 1, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ax3l Nov 14, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

opadron commented Oct 23, 2017 •

edited by tgamblin

Loading

opadron commented Oct 31, 2017 •

edited

Loading

ax3l Nov 1, 2017 •

edited

Loading

ax3l Nov 14, 2017 •

edited

Loading

ax3l commented Nov 14, 2017 •

edited

Loading

ax3l commented Nov 20, 2017 •

edited

Loading

ax3l commented Jan 24, 2018 •

edited

Loading