Hi,

I'm currently working on a project, which aims to create a chroot / VM environment out of the box within spack. This should avoid that a compiler link against a library from the host system. This also offers the possibility to compile for different platforms through a VM. The basic usage should be something like:

spack bootstrap –isolate --extra_args_depend_on_impl bootstrap_dir

and then build a binary with the usual:

spack install libelf, glib, ...

But before I continue to complete this and submit a pull request, I would like to hear if there is a general interest in such a feature, or some thoughts / improvements from the spack community.

What I actually came up with, where five different implementations (see list below), but I consider the 3rd and 5th implementation to be the most fitting approach, which align more or less to the spack philosophy. Therefore I currently consider them for the final implementation.

1.)
A chroot environment where the files get mount binded by a whitelist.

• Pros:
  • Fine control, which files of the system should be used
• Cons:
  • A lot of work to get to a solid whitelist, which contains all basic files needed to run spack
  • Need administrator rights to run

2.)
A chroot environment based on the whitelist approach but with automatic dependency resolval by dpkg.

• Pros:
  • Fine control, which files and packages of the system should be used
  • Dependencies get resolved automatically
• Cons:
  • No control over what files get included by a dependency
  • Slow, because it results in a lot of mount bind and dpkg calls (5 - 10min on an ssd)
  • Need administrator rights to run
  • Depends on dpkg

3.)
A chroot environment, based on a distribution, which is packed into a tar and then extracted into a bootstrap dictionary. As for example distributions which got generated with mkosi or are prepacked by OpenStack or the distributer of the distribution itself.

• Pros:
  • Control over all installed packages
  • Simple to use, because it is possible to use existing packages
  • Distributions which use the standard dictionary structure can be used inside another distribution
  • Can be mounted permanently in fstab, because it need to mount /dev and /etc/resolv.conf only
  • No administrator rights required after bootstraping
• Cons:
  • Some distributions, like Arch needs an extensive preconfiguration, before they can be used
  • Currently have to be packed as tar.gz

4.)
A container system approach. I dropped this one, because it creates more problems than it solves. Docker currently switches to an community / enterprise model. I avoided Docker, because I don't know if the community edition offers all necessary features. I also tried rkt, but there philosophy is to destroy all containers after there usage, so it's not feasible with spack.

5.)
Use a VM with libvirt to create an environment. This is the most flexible solution but also the heaviest one in performance and storage.

• Pros:
  • Allows every distribution which supports spack to run
  • Multiple architectures possible
  • Can be installed easily from an ISO
  • No administrator rights required
• Cons:
  • A bit slower than the other approaches, without virtualization it is horrible slow
  • Needs an ssh server on the distribution for communication between the two spack instances
  • Compiled binaries have to be copied to the host or they must run inside the VM
  • Introduces some dependencies like Qemu-KVM libvirt-bin, virtinst, qemu-agent.