spack rebuild seems to use wrong python version when it is in the environment #17432
Description
Steps to reproduce the issue
Have a spack.yaml + .gitlab-ci.yml file like https://gitlab.com/cscs-ci/ci-testing/spack-ci/-/tree/839a2aa3794865671060c8431175f01bae8ba4b8.
- This builds
hpx. hpxhas down the line a dependency on Python- I believe this confuses spack in the
spack rebuildstep, making it fail to locate thebotocoremodule and error next as follows
To isolate the problem further, please check out this minimal setup with just the failing job, and check out the pipeline logs: https://cscs-ci.gitlab.io/-/ci-testing/spack-ci/-/jobs/630250913/artifacts/jobs_scratch_dir/logs/pipeline_log.txt.
Error Message
==> [2020-07-09-03:55:46.206507, 568] Error: ModuleNotFoundError: No module named 'botocore'
/opt/spack/lib/spack/spack/package.py:1149, in do_fetch:
1146 raise FetchError("Will not fetch %s" %
1147 self.spec.format('{name}{@version}'), ck_msg)
1148
>> 1149 self.stage.create()
1150 self.stage.fetch(mirror_only)
1151 self._fetch_time = time.time() - start_time
1152
Traceback (most recent call last):
File "/opt/spack/lib/spack/spack/build_environment.py", line 838, in child_process
return_value = function()
File "/opt/spack/lib/spack/spack/installer.py", line 1073, in build_process
pkg.do_patch()
File "/opt/spack/lib/spack/spack/package.py", line 1189, in do_patch
self.do_stage()
File "/opt/spack/lib/spack/spack/package.py", line 1174, in do_stage
self.do_fetch(mirror_only)
File "/opt/spack/lib/spack/spack/package.py", line 1150, in do_fetch
self.stage.fetch(mirror_only)
File "/opt/spack/lib/spack/spack/util/pattern.py", line 68, in getter
getattr(item, self.name)(*args, **kwargs)
File "/opt/spack/lib/spack/spack/stage.py", line 464, in fetch
self.fetcher.fetch()
File "/opt/spack/lib/spack/spack/fetch_strategy.py", line 72, in wrapper
return fun(self, *args, **kwargs)
File "/opt/spack/lib/spack/spack/fetch_strategy.py", line 1146, in fetch
_, headers, stream = web_util.read_from_url(self.url)
File "/opt/spack/lib/spack/spack/util/web.py", line 128, in read_from_url
response = _urlopen(req, timeout=_timeout, context=context)
File "/opt/spack/lib/spack/spack/util/web.py", line 476, in _urlopen
return opener(req, *args, **kwargs)
File "/scratch/snx3000/jenkssl/gitlab-runner-daint/builds/txUXd4Wq/0/cscs-ci-ci-testing-spack-ci/cscs-ci/ci-testing/spack-ci/.spack-env/view/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/scratch/snx3000/jenkssl/gitlab-runner-daint/builds/txUXd4Wq/0/cscs-ci-ci-testing-spack-ci/cscs-ci/ci-testing/spack-ci/.spack-env/view/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/scratch/snx3000/jenkssl/gitlab-runner-daint/builds/txUXd4Wq/0/cscs-ci-ci-testing-spack-ci/cscs-ci/ci-testing/spack-ci/.spack-env/view/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/opt/spack/lib/spack/spack/s3_handler.py", line 64, in s3_open
from botocore.exceptions import ClientError
ModuleNotFoundError: No module named 'botocore'
Note the paths to spack-ci/.spack-env/view/lib/python3.7/urllib/request.py, it's probably using the wrong python version.
Information on your system
I can't really provide this, it's running in CI.
Additional information
- I have run
spack debug reportand reported the version of Spack/Python/Platform - I have searched the issues of this repo and believe this is not a duplicate
- I have run the failing commands in debug mode and reported the output