@nazavode @tgamblin @opadron I think you might be interested in the above. There are a few issues to be worked out (in particular with spack buildcache, which seems not to be able to find binary caches via http on the host), but I think the initial work looks promising...

EDIT: need to check that, but it seems that --use-cache option of install doesn't follow 3xx redirects. So if I set a mirror with url:

0.0.0.0:32679

it works, if I put an extra /:

0.0.0.0:32679/

it won't. Will try to investigate this more in details.

I tried this and got an error complaining about no default GPG key.

...

==> Creating binary caches
==> 'docker' 'run' '--rm' '--mount' 'source=spack-mirror,target=/share/spack/mirror' '--mount' 'source=spack-opt,target=/home/spack/opt' '-v' '/spack/opt/spack/gpg:/home/spack/opt/spack/gpg' 'spack-builder' 'buildcache' 'create' '-r' '-d' '/share/spack/mirror' '-f' '[email protected]%[email protected]+optimize+pic+shared [email protected]%[email protected] szip'
==> adding matching spec [email protected]%[email protected]+optimize+pic+shared arch=linux-ubuntu16.04-x86_64
==> recursing dependencies
==> adding dependency [email protected]%[email protected]+optimize+pic+shared arch=linux-ubuntu16.04-x86_64
==> adding matching spec [email protected]%[email protected] arch=linux-ubuntu16.04-x86_64
==> recursing dependencies
==> adding dependency [email protected]%[email protected] arch=linux-ubuntu16.04-x86_64
==> adding matching spec [email protected]%[email protected] arch=linux-ubuntu16.04-x86_64
==> recursing dependencies
==> adding dependency [email protected]%[email protected] arch=linux-ubuntu16.04-x86_64
==> writing tarballs to /share/spack/mirror/build_cache
==> creating binary cache file for package [email protected]%[email protected] arch=linux-ubuntu16.04-x86_64
gpg: WARNING: unsafe permissions on homedir '/home/spack/opt/spack/gpg'
gpg: WARNING: unsafe permissions on homedir '/home/spack/opt/spack/gpg'
==> Error: No default key available for signing.
Use spack gpg init and spack gpg create to create a default key.
==> Error: Command exited with status 1:
'docker' 'run' '--rm' '--mount' 'source=spack-mirror,target=/share/spack/mirror' '--mount' 'source=spack-opt,target=/home/spack/opt' '-v' '/spack/opt/spack/gpg:/home/spack/opt/spack/gpg' 'spack-builder' 'buildcache' 'create' '-r' '-d' '/share/spack/mirror' '-f' '[email protected]%[email protected]+optimize+pic+shared [email protected]%[email protected] szip'
Traceback (most recent call last):
  File "/spack/lib/spack/spack/main.py", line 469, in _main
    return _invoke_spack_command(command, parser, args, unknown_args)
  File "/spack/lib/spack/spack/main.py", line 381, in _invoke_spack_command
    return_val = command(parser, args)
  File "/spack/lib/spack/spack/cmd/docker.py", line 299, in docker
    callbacks[args.subparser_name](args)
  File "/spack/lib/spack/spack/cmd/docker.py", line 284, in build
    docker_cmd(*(common_setup + buildcache_run), **verbosity)
  File "/spack/lib/spack/spack/util/executable.py", line 183, in __call__
    proc.returncode, cmd_line)
spack.util.executable.ProcessError: Command exited with status 1:
    'docker' 'run' '--rm' '--mount' 'source=spack-mirror,target=/share/spack/mirror' '--mount' 'source=spack-opt,target=/home/spack/opt' '-v' '/spack/opt/spack/gpg:/home/spack/opt/spack/gpg' 'spack-builder' 'buildcache' 'create' '-r' '-d' '/share/spack/mirror' '-f' '[email protected]%[email protected]+optimize+pic+shared [email protected]%[email protected] szip'

Even though the output of spack gpg list shows:

/spack/opt/spack/gpg/pubring.kbx
--------------------------------
pub   rsa4096/E8AECD32 2018-03-19 [SC]
uid         [ultimate] Omar Padron (GPG created for Spack) <[email protected]>

@alalazo: are you guys using this at EPFL? I think this looks pretty cool, though I might quibble about the names of the commands and some of the options. How is your build farm set up?

@alalazo: I think the question for us is how best to get buildbot to farm out container instances to VMs.
We're trying to set things up in EC2, so I'm curious whether you use a command line like this or whether you use something else to orchestrate.

@opadron I could reproduce your error. I need to dig deeper to confirm, but I suspect it's due to a change in buildcache or some of the tools / commands involved in the signature. After cleaning my working copy of Spack, and creating a new key, things worked out correctly:

$ git clean -idx

$ spack gpg list
gpg: keyring `/home/mculpo/PycharmProjects/spack/opt/spack/gpg/pubring.gpg' created
gpg: /home/mculpo/PycharmProjects/spack/opt/spack/gpg/trustdb.gpg: trustdb created

$ spack gpg create "Massimiliano Culpo" "[email protected]"
gpg: keyring `/home/mculpo/PycharmProjects/spack/opt/spack/gpg/secring.gpg' created
gpg: skipping control `%no-protection' ()
gpg: key E258F6F9 marked as ultimately trusted

$ spack -dv docker build --image=ubuntu:16.04 zlib patchelf szip
==> Reading config file /home/mculpo/PycharmProjects/spack/etc/spack/defaults/modules.yaml
==> Building image for "ubuntu:16.04"
==> 'docker' 'build' '-t' 'spack-builder' '.'
Sending build context to Docker daemon   2.56kB
Step 1/5 : FROM ubuntu:16.04
...
gpg: using "9AB057DF71F30047AFB1DEF5348CB8B1E258F6F9" as default secret key for signing
gpg: WARNING: unsafe ownership on homedir '/home/spack/opt/spack/gpg'
gpg: WARNING: unsafe ownership on homedir '/home/spack/opt/spack/gpg'
gpg: WARNING: unsafe ownership on homedir '/home/spack/opt/spack/gpg'
gpg: using "9AB057DF71F30047AFB1DEF5348CB8B1E258F6F9" as default secret key for signing
==> creating binary cache file for package [email protected]%[email protected]+optimize+pic+shared arch=linux-ubuntu16.04-x86_64  
==> creating binary cache file for package [email protected]%[email protected] arch=linux-ubuntu16.04-x86_64  
==> 'docker' 'volume' 'rm' 'spack-opt'

are you guys using this at EPFL? I think this looks pretty cool, though I might quibble about the names of the commands and some of the options. How is your build farm set up?

We are not using this in production yet - I think we still miss some features for buildcache to be bulletproof if we start distributing binaries to users (see #7237) + the PR right now works only with system compilers. On the other hand I use this PR a lot to check recipes on different operative systems.

Regarding our build farm, it's currently based on shell scripts triggered by Jenkins. In the scripts we use spack from command line directly. Probably we'll move everything to Jenkins pipeline in the near future to orchestrate the various steps better. I don't know if any of these concepts maps to buildbot though (still need to rtfm).

p.s. I am also waiting for @scheibelp environments to be merged, the idea being to build an entire set of specs into a docker image that can then be delivered to some group of users. These users could then run the image locally on their workstation or on our clusters via shifter.

Is there a specific reason for your choice of docker? Here the preference seems to be singularity containers, so I'm curious about the reasoning (really either way, as I don't know too much about the differences). If I read this PR correctly it shouldn't be too much work to do the equivalent for singularity, although we may want to think about what we can reuse if we end up supporting multiple container systems.

@healther It won't be hard to factor out common parts if we'll add a spack singularity command, so that's definitely not a concern.

Is there a specific reason for your choice of docker?

Docker seems to me more flexible and more widely used outside of the HPC world (e.g. services like Travis support docker images, I doubt they will add support for singularity images) + from Singularity homepage:

Did you already invest in Docker? The Singularity software can import your Docker images without having Docker installed or being a superuser.

so while you can build a Singularity image out of a docker one, I don't think you can do the opposite. In any case we are beginning to experiment with docker/singularity + shifter now in our group, so I am not yet aware of all the pitfalls you might get into.

Here the preference seems to be singularity containers,

Do you mean in your group, or here in this community? 😕

By here I meant our group, sorry for not being more specific. Thanks for your thoughts, I'll have to dig into the specific differences.

@tgamblin Forgot to mention:

I think this looks pretty cool, though I might quibble about the names of the commands and some of the options.

no worries changing names to options or commands. If there's already something that you want changed let me know.