Missing credentials when attempting to install dependency from private Gitlab project #5955
Description
- I am on the latest Poetry version.
- I have searched the issues of this repo and believe that this is not a duplicate.
- If an exception occurs when executing a command, I executed it again in debug mode (
-vvvoption).
- Ubuntu 20,04 WSL2:
- 1.2.0b2:
[tool.poetry]
name = "<package_name>"
version = "0.0.0"
description = "test"
authors = ["Chris Buffett <[email protected]>"]
license = "Proprietary"
[tool.poetry.dependencies]
python = "^3.7"
requests = "^2.25.1"
dask = "^2021.4.1"
fsspec = "^2021.4.0"
partd = "^1.2.0"
<private_package_name> = {git = https://<private_gitlab_repo>/<private_package>.git", rev = "master"}
[tool.poetry.extras]
dask = ["dataframe"]
[tool.poetry.dev-dependencies]
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
Issue
I have a package I'm using poetry to build and version. This private package has a dependency on another private package in a different repo. This other private package is not published to a package registry, so in order to access it, I've set up basic HTTP credentials to the git project directly (as recommended in #5567)
poetry config repositories.gitlab https://<private_gitlab_repo>/<private_package>.git
poetry config http-basic.gitlab <gitlab_deploy_token> <password>
This works fine when building/installing the package locally using poetry build and poetry install. However, when attempting to install my package as a dependency in an Airflow Docker image, the install of the private package fails due to missing authentication credentials. I'm using pip to install my package inside the Docker image, so it doesn't appear to have any notion of the Poetry credentials that were configured locally when building the package.
#12 84.81 Collecting <private_package_name>@ git+https://<private_gitlab_repo>/<private_package>.git@master
#12 84.81 Cloning https://<private_gitlab_repo>/<private_package>.git (to revision master) to /tmp/pip-resolver-gg2sgq6r/<private_package_name>
#12 84.82 Running command git clone --filter=blob:none -q https://<private_gitlab_repo>/<private_package>.git /tmp/pip-resolver-gg2sgq6r/<private_package_name>
#12 85.32 fatal: could not read Username for 'https://<private_gitlab_repo>': No such device or address
#12 85.32 Traceback (most recent call last):
#12 85.32 File "/home/airflow/.local/bin/pip-compile", line 8, in <module>
#12 85.32 sys.exit(cli())
#12 85.32 File "/home/airflow/.local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
#12 85.32 return self.main(*args, **kwargs)
#12 85.32 File "/home/airflow/.local/lib/python3.7/site-packages/click/core.py", line 782, in main
#12 85.32 rv = self.invoke(ctx)
#12 85.32 File "/home/airflow/.local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
#12 85.32 return ctx.invoke(self.callback, **ctx.params)
#12 85.32 File "/home/airflow/.local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
#12 85.32 return callback(*args, **kwargs)
#12 85.32 File "/home/airflow/.local/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
#12 85.32 return f(get_current_context(), *args, **kwargs)
#12 85.32 File "/home/airflow/.local/lib/python3.7/site-packages/piptools/scripts/compile.py", line 487, in cli
#12 85.32 results = resolver.resolve(max_rounds=max_rounds)
#12 85.32 File "/home/airflow/.local/lib/python3.7/site-packages/piptools/resolver.py", line 266, in resolve
#12 85.32 has_changed, best_matches = self._resolve_one_round()
#12 85.32 File "/home/airflow/.local/lib/python3.7/site-packages/piptools/resolver.py", line 356, in _resolve_one_round
#12 85.32 their_constraints.extend(self._iter_dependencies(best_match))
#12 85.32 File "/home/airflow/.local/lib/python3.7/site-packages/piptools/resolver.py", line 451, in _iter_dependencies
#12 85.32 dependencies = self.repository.get_dependencies(ireq)
#12 85.32 File "/home/airflow/.local/lib/python3.7/site-packages/piptools/repositories/pypi.py", line 251, in get_dependencies
#12 85.32 download_dir, ireq, wheel_cache
#12 85.32 File "/home/airflow/.local/lib/python3.7/site-packages/piptools/repositories/pypi.py", line 213, in resolve_reqs
#12 85.32 results = resolver._resolve_one(reqset, ireq)
#12 85.32 File "/home/airflow/.local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 379, in _resolve_one
#12 85.32 dist = self._get_dist_for(req_to_install)
#12 85.32 File "/home/airflow/.local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 332, in _get_dist_for
#12 85.32 dist = self.preparer.prepare_linked_requirement(req)
#12 85.32 File "/home/airflow/.local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 482, in prepare_linked_requirement
#12 85.32 return self._prepare_linked_requirement(req, parallel_builds)
#12 85.32 File "/home/airflow/.local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 528, in _prepare_linked_requirement
#12 85.32 link, req.source_dir, self._download, self.download_dir, hashes
#12 85.32 File "/home/airflow/.local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 190, in unpack_url
#12 85.32 unpack_vcs_link(link, location)
#12 85.32 File "/home/airflow/.local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 65, in unpack_vcs_link
#12 85.32 vcs_backend.unpack(location, url=hide_url(link.url))
#12 85.32 File "/home/airflow/.local/lib/python3.7/site-packages/pip/_internal/vcs/versioncontrol.py", line 598, in unpack
#12 85.32 self.obtain(location, url=url)
#12 85.32 File "/home/airflow/.local/lib/python3.7/site-packages/pip/_internal/vcs/versioncontrol.py", line 512, in obtain
#12 85.32 self.fetch_new(dest, url, rev_options)
#12 85.32 File "/home/airflow/.local/lib/python3.7/site-packages/pip/_internal/vcs/git.py", line 269, in fetch_new
#12 85.32 dest,
#12 85.32 File "/home/airflow/.local/lib/python3.7/site-packages/pip/_internal/vcs/versioncontrol.py", line 649, in run_command
#12 85.32 stdout_only=stdout_only,
#12 85.32 File "/home/airflow/.local/lib/python3.7/site-packages/pip/_internal/utils/subprocess.py", line 254, in call_subprocess
#12 85.32 raise InstallationSubprocessError(proc.returncode, command_desc)
#12 85.32 pip._internal.exceptions.InstallationSubprocessError: Command errored out with exit status 128: git clone --filter=blob:none -q https://<private_gitlab_repo>/<private_package>.git /tmp/pip-resolver-gg2sgq6r/<private_package_name> Check the logs for full command output.
I've tried setting environment variables (POETRY_HTTP_BASIC_GITLAB_USERNAME/POETRY_HTTP_BASIC_GITLAB_PASSWORD) for my private repo inside the Dockerfile, but this seems to be ignored by pip. I've tried hardcoding the private repo deploy_token/password in the git URL inside pyproject.toml, but this results in an error saying the git URL is invalid when attempting to build the package
poetry build
Building <package_name> (0.0.0)
Invalid git url "https://<deploy_token>:<password>@<private_gitlab_repo>/<private_package>.git"
This appears somewhat related to #2062. While the pre-release fix (#5567) seems to address the issue of supporting deploy tokens when adding HTTP basic authentication credentials for private gitlab projects, the absence of these credentials in the dependency URL itself prevents pip from accessing the package.