Broken python and from_sourcing_file in packages depending indirectly on python #7128
Description
When a package indirectly depends on python, any attempt to run EnvironmentModifications.from_sourcing_file inside its installation environment ends up in an error. The python command is broken as well (and this is probably the real cause of the observed issue).
Expected Result
To have that package installed.
Actual Result
$ spack install example-dependent
[lots of usual output...]
==> Executing phase: 'install'
==> Error: RuntimeError: Sourcing file /home/spack/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/example-dependent-1.0.6-ickrp7u2evbgk7l6jivxu6jexzptd5ro/SOURCE_ME returned a non-zero exit code
RuntimeError: RuntimeError: Sourcing file /home/spack/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/example-dependent-1.0.6-ickrp7u2evbgk7l6jivxu6jexzptd5ro/SOURCE_ME returned a non-zero exit code
/home/spack/spack-repo-test/packages/example-dependent/package.py:16, in install:
1 to_be_sourced = fs.join_path(prefix, 'SOURCE_ME')
2 with open(to_be_sourced, 'w') as f:
3 f.write('export SOURCE_ME_HAS_BEEN_SOURCED=1')
>> 4 EnvironmentModifications.from_sourcing_file(to_be_sourced).apply_modifications()
5 fs.touch(fs.join_path(prefix, 'INSTALL_DONE'))
See build log for details:
/home/spack/spack/var/spack/stage/example-dependent-1.0.6-ickrp7u2evbgk7l6jivxu6jexzptd5ro/bzip2-1.0.6/spack-build.outThe actual error raised by the call is:
$ cat /home/spack/spack/var/spack/stage/example-dependent-1.0.6-ickrp7u2evbgk7l6jivxu6jexzptd5ro/bzip2-1.0.6/spack-build.out
==> Executing phase: 'install'
ImportError: No module named siteSteps to reproduce the issue
I've managed to reduce the problem to a minimal set of packages that can be obtained from GitHub:
$ git clone https://github.com/nazavode/spack-repo-test.git ~/spack-repo-test
$ spack repo add ~/spack-repo-testThat Spack repo contains two packages:
example-dependencythat depends onpythonexample-dependentthat depends onexample-dependency
Having that repo added to Spack allows us to obtain this interesting spec:
$ spack spec example-dependent
Input spec
--------------------------------
example-dependent
Concretized
--------------------------------
[email protected]%[email protected] arch=linux-centos7-x86_64
^[email protected]%[email protected] arch=linux-centos7-x86_64
^[email protected]%[email protected] patches=123082ab3483ded78e86d7c809e98a804b3465b4683c96bd79a2fd799f572244 +pic+shared~tk~ucs4 arch=linux-centos7-x86_64
[a lot of other stuff python depends on...]This minimal dependency graph is enough to trigger the issue:
$ spack install example-dependent
[lots of usual output...]
==> Executing phase: 'install'
==> Error: RuntimeError: Sourcing file /home/spack/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/example-dependent-1.0.6-ickrp7u2evbgk7l6jivxu6jexzptd5ro/SOURCE_ME returned a non-zero exit code
RuntimeError: RuntimeError: Sourcing file /home/spack/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/example-dependent-1.0.6-ickrp7u2evbgk7l6jivxu6jexzptd5ro/SOURCE_ME returned a non-zero exit code
/home/spack/spack-repo-test/packages/example-dependent/package.py:16, in install:
1 to_be_sourced = fs.join_path(prefix, 'SOURCE_ME')
2 with open(to_be_sourced, 'w') as f:
3 f.write('export SOURCE_ME_HAS_BEEN_SOURCED=1')
>> 4 EnvironmentModifications.from_sourcing_file(to_be_sourced).apply_modifications()
5 fs.touch(fs.join_path(prefix, 'INSTALL_DONE'))
See build log for details:
/home/spack/spack/var/spack/stage/example-dependent-1.0.6-ickrp7u2evbgk7l6jivxu6jexzptd5ro/bzip2-1.0.6/spack-build.outThe root of the problem lies in the fact that the python interpreter inside the installation environment is broken:
$ spack install --only dependencies example-dependent
[installs just fine, output omitted...]
$ spack env example-dependent python
ImportError: No module named siteInspecting the affected environment we get:
$ spack env example-dependent bash
$ which python
/usr/bin/python
$ env | grep PYTHON
PYTHONHOME=/home/spack/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/python-2.7.14-4ipldugnhasdas2pjasdevq4ruha6biu
PYTHONPATH=
$ python
ImportError: No module named siteIt looks like ^[email protected] messed with our environment injecting that PYTHONHOME that our system's /usr/bin/python is not standing at all. The fun part is that some distro (Ubuntu 16.04 for instance) ships with a python that actually likes the broken environment and proved to be working fine (see below for details).
Wrap up
- When a package depends indirectly on
python, it ends up with:PYTHONHOME=<path to spack python prefix>PYTHONPATH=which pythonis<system python>, which is broken or not depending on its version. This in turn breaksEnvironmentModifications.from_sourcing_fileas well.
- When a package depends directly on
python, it ends up with:PYTHONHOME=<path to spack python prefix>PYTHONPATH=<path to spack python prefix>which pythonis<spack python>, which works correctly.
Information on your system
Reproduced this on a centos7 docker image that ships with python 2.7.5 while tested to be working fine on a ubuntu 16.04 docker image that ships with python 2.7.12. Spack was cd084d0.
A Dockefile that executes the whole issue reproduction procedure can be found here.