poetry install fails on KeyError even for defined environment variables #3199
Description
- I am on the latest Poetry version.
- I have searched the issues of this repo and believe that this is not a duplicate.
- If an exception occurs when executing a command, I executed it again in debug mode (
-vvvoption).
- OS version and name: Debian Linux 10 Buster
- Poetry version: 1.1.2
- Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/pepastach/85b133f035fbb805cacd4d0fd75562b1
Issue
We run poetry install --remove-untracked and the process randomly fails on various "missing" environment variables. This has been reported in #3144 for the PATH variable (which happens quite a lot, more than other variables), but I have seen it for many other variables (often GitLab or Docker related). In all cases they are defined.
It fails really randomly. Different dependencies being installed, different environment variables. Sometimes it does not fail (but most often it does).
Sometimes it fails on multiple environment variables. Like KeyError 'PATH', KeyError 'CI_JOB_STAGE' or KeyError 'DOCKER_ENV_CI_DISPOSABLE_ENVIRONMENT' in the same run. It seems to me like some kind of race condition in the parallel install. I can't prove this hypothesis until we can disable parallel installation. I think it's scheduled for an upcoming release.
Here is an example output where it fails on the PATH variable.
Creating virtualenv bp-xxx-3RqKXHa2-py3.7 in /builds/xxx/.cache/poetry/pypoetry/virtualenvs
Installing dependencies from lock file
Package operations: 106 installs, 0 updates, 0 removals
• Installing attrs (20.2.0)
• Installing six (1.15.0)
• Installing cattrs (1.0.0)
...
Stack trace:
8 /usr/local/lib/python3.7/site-packages/poetry/installation/executor.py:199 in _execute_operation
197│
198│ try:
→ 199│ result = self._do_execute_operation(operation)
200│ except EnvCommandError as e:
201│ if e.e.returncode == -2:
7 /usr/local/lib/python3.7/site-packages/poetry/installation/executor.py:273 in _do_execute_operation
271│ return 0
272│
→ 273│ result = getattr(self, "_execute_{}".format(method))(operation)
274│
275│ if result != 0:
6 /usr/local/lib/python3.7/site-packages/poetry/installation/executor.py:408 in _execute_install
406│
407│ def _execute_install(self, operation): # type: (Install) -> None
→ 408│ return self._install(operation)
409│
410│ def _execute_update(self, operation): # type: (Update) -> None
5 /usr/local/lib/python3.7/site-packages/poetry/installation/executor.py:446 in _install
444│ args.insert(2, "-U")
445│
→ 446│ return self.run_pip(*args)
447│
448│ def _update(self, operation):
4 /usr/local/lib/python3.7/site-packages/poetry/installation/executor.py:297 in run_pip
295│ def run_pip(self, *args, **kwargs): # type: (...) -> int
296│ try:
→ 297│ self._env.run_pip(*args, **kwargs)
298│ except EnvCommandError as e:
299│ output = decode(e.e.output)
3 /usr/local/lib/python3.7/site-packages/poetry/utils/env.py:916 in run_pip
914│ pip = self.get_pip_command()
915│ cmd = pip + list(args)
→ 916│ return self._run(cmd, **kwargs)
917│
918│ def _run(self, cmd, **kwargs):
2 /usr/local/lib/python3.7/site-packages/poetry/utils/env.py:1183 in _run
1181│ def _run(self, cmd, **kwargs):
1182│ with self.temp_environ():
→ 1183│ os.environ["PATH"] = self._updated_path()
1184│ os.environ["VIRTUAL_ENV"] = str(self._path)
1185│
1 /usr/local/lib/python3.7/site-packages/poetry/utils/env.py:1215 in _updated_path
1213│
1214│ def _updated_path(self):
→ 1215│ return os.pathsep.join([str(self._bin_dir), os.environ["PATH"]])
1216│
1217│
KeyError
'PATH'
at /usr/local/lib/python3.7/os.py:681 in __getitem__
677│ try:
678│ value = self._data[self.encodekey(key)]
679│ except KeyError:
680│ # raise KeyError with the original key value
→ 681│ raise KeyError(key) from None
682│ return self.decodevalue(value)
683│
684│ def __setitem__(self, key, value):
685│ key = self.encodekey(key)
• Installing pluggy (0.13.1)
• Installing py (1.9.0)
• Installing wcwidth (0.2.5)
• Installing yarl (1.6.0)