Installing private packages from GAR in Docker build fails with 500 Internal Server Error on Poetry 1.8.0 #9026
Description
- Poetry version: 1.8.0
- Python version: 3.12.0
- OS version and name: Docker image python@sha256:d0c9bf03d80a3c00963f1b973760094c0b2070638fa64dd4f67d2247c0110efc
- pyproject.toml: https://gist.github.com/ntaylorwss/69cc7b9836047c5f69fbd263191d1210
- I am on the latest stable Poetry version, installed using a recommended method.
- I have searched the issues of this repo and believe that this is not a duplicate.
- I have consulted the FAQ and blog for any relevant entries or release notes.
- If an exception occurs when executing a command, I executed it again in debug mode (
-vvvoption) and have included the output below.
Issue
I have a Python service whose pyproject.toml includes some dependencies from a private Google Artifact Registry Python repository. Until yesterday, this all worked fine. Poetry is installed in the container via:
# install poetry
RUN apt-get update \
&& apt-get install -y curl gnupg gcc \
&& curl -sSL https://install.python-poetry.org | python - \
&& poetry self add keyrings.google-artifactregistry-auth
and this keyring, given an environment with active application credentials, is able to authenticate me to my GAR repository and install the required dependencies. A build ran this morning and failed with the following error:
14.39 ValueError
14.39
14.39 Package('my-package', '0.0.0') is not in list
14.39
14.39 at /opt/poetry/venv/lib/python3.12/site-packages/poetry/repositories/legacy_repository.py:66 in package
14.39 62│ Note that this will be cached so the subsequent operations
14.39 63│ should be much faster.
14.39 64│ """
14.39 65│ try:
14.39 → 66│ index = self._packages.index(Package(name, version))
14.39 67│
14.39 68│ return self._packages[index]
14.39 69│ except ValueError:
14.39 70│ package = super().package(name, version, extras)
14.39
14.39 The following error occurred when trying to handle this error:
14.39
14.39
14.39 Stack trace:
14.39
14.39 34 /opt/poetry/venv/lib/python3.12/site-packages/cleo/application.py:327 in run
14.40 325│
14.40 326│ try:
14.40 → 327│ exit_code = self._run(io)
14.40 328│ except BrokenPipeError:
14.40 329│ # If we are piped to another process, it may close early and send a
14.40
14.40 33 /opt/poetry/venv/lib/python3.12/site-packages/poetry/console/application.py:190 in _run
14.41 188│ self._load_plugins(io)
14.41 189│
14.41 → 190│ exit_code: int = super()._run(io)
14.41 191│ return exit_code
14.41 192│
14.41
14.41 32 /opt/poetry/venv/lib/python3.12/site-packages/cleo/application.py:431 in _run
14.42 429│ io.input.interactive(interactive)
14.42 430│
14.42 → 431│ exit_code = self._run_command(command, io)
14.42 432│ self._running_command = None
14.42 433│
14.42
14.42 31 /opt/poetry/venv/lib/python3.12/site-packages/cleo/application.py:473 in _run_command
14.42 471│
14.42 472│ if error is not None:
14.42 → 473│ raise error
14.42 474│
14.42 475│ return terminate_event.exit_code
14.42
14.42 30 /opt/poetry/venv/lib/python3.12/site-packages/cleo/application.py:457 in _run_command
14.43 455│
14.43 456│ if command_event.command_should_run():
14.43 → 457│ exit_code = command.run(io)
14.43 458│ else:
14.43 459│ exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED
14.43
14.43 29 /opt/poetry/venv/lib/python3.12/site-packages/cleo/commands/base_command.py:117 in run
14.44 115│ io.input.validate()
14.44 116│
14.44 → 117│ return self.execute(io) or 0
14.44 118│
14.44 119│ def merge_application_definition(self, merge_args: bool = True) -> None:
14.44
14.44 28 /opt/poetry/venv/lib/python3.12/site-packages/cleo/commands/command.py:61 in execute
14.44 59│
14.44 60│ try:
14.44 → 61│ return self.handle()
14.44 62│ except KeyboardInterrupt:
14.44 63│ return 1
14.44
14.44 27 /opt/poetry/venv/lib/python3.12/site-packages/poetry/console/commands/install.py:153 in handle
14.45 151│ self.installer.verbose(self.io.is_verbose())
14.45 152│
14.45 → 153│ return_code = self.installer.run()
14.45 154│
14.45 155│ if return_code != 0:
14.45
14.45 26 /opt/poetry/venv/lib/python3.12/site-packages/poetry/installation/installer.py:104 in run
14.45 102│ self.verbose(True)
14.45 103│
14.45 → 104│ return self._do_install()
14.45 105│
14.45 106│ def dry_run(self, dry_run: bool = True) -> Installer:
14.45
14.45 25 /opt/poetry/venv/lib/python3.12/site-packages/poetry/installation/installer.py:241 in _do_install
14.46 239│ source_root=self._env.path.joinpath("src")
14.46 240│ ):
14.46 → 241│ ops = solver.solve(use_latest=self._whitelist).calculate_operations()
14.46 242│ else:
14.46 243│ self._io.write_line("Installing dependencies from lock file")
14.46
14.46 24 /opt/poetry/venv/lib/python3.12/site-packages/poetry/puzzle/solver.py:71 in solve
14.46 69│ with self._progress(), self._provider.use_latest_for(use_latest or []):
14.47 70│ start = time.time()
14.47 → 71│ packages, depths = self._solve()
14.47 72│ end = time.time()
14.47 73│
14.47
14.47 23 /opt/poetry/venv/lib/python3.12/site-packages/poetry/puzzle/solver.py:154 in _solve
14.47 152│
14.47 153│ try:
14.47 → 154│ result = resolve_version(self._package, self._provider)
14.47 155│
14.47 156│ packages = result.packages
14.47
14.47 22 /opt/poetry/venv/lib/python3.12/site-packages/poetry/mixology/__init__.py:18 in resolve_version
14.47 16│ solver = VersionSolver(root, provider)
14.47 17│
14.47 → 18│ return solver.solve()
14.47 19│
14.47
14.47 21 /opt/poetry/venv/lib/python3.12/site-packages/poetry/mixology/version_solver.py:175 in solve
14.48 173│ while next is not None:
14.48 174│ self._propagate(next)
14.48 → 175│ next = self._choose_package_version()
14.48 176│
14.48 177│ return self._result()
14.48
14.48 20 /opt/poetry/venv/lib/python3.12/site-packages/poetry/mixology/version_solver.py:514 in _choose_package_version
14.49 512│ package = locked
14.49 513│
14.49 → 514│ package = self._provider.complete_package(package)
14.49 515│
14.49 516│ conflict = False
14.49
14.49 19 /opt/poetry/venv/lib/python3.12/site-packages/poetry/puzzle/provider.py:489 in complete_package
14.50 487│ dependency_package = DependencyPackage(
14.50 488│ dependency,
14.50 → 489│ self._pool.package(
14.50 490│ package.pretty_name,
14.50 491│ package.version,
14.50
14.50 18 /opt/poetry/venv/lib/python3.12/site-packages/poetry/repositories/repository_pool.py:198 in package
14.51 196│ ) -> Package:
14.51 197│ if repository_name:
14.51 → 198│ return self.repository(repository_name).package(
14.51 199│ name, version, extras=extras
14.51 200│ )
14.51
14.51 17 /opt/poetry/venv/lib/python3.12/site-packages/poetry/repositories/legacy_repository.py:70 in package
14.51 68│ return self._packages[index]
14.51 69│ except ValueError:
14.51 → 70│ package = super().package(name, version, extras)
14.51 71│ package._source_type = "legacy"
14.51 72│ package._source_url = self._url
14.51
14.51 16 /opt/poetry/venv/lib/python3.12/site-packages/poetry/repositories/cached_repository.py:75 in package
14.51 73│ extras: list[str] | None = None,
14.51 74│ ) -> Package:
14.51 → 75│ return self.get_release_info(canonicalize_name(name), version).to_package(
14.51 76│ name=name, extras=extras
14.51 77│ )
14.51
14.51 15 /opt/poetry/venv/lib/python3.12/site-packages/poetry/repositories/cached_repository.py:52 in get_release_info
14.51 50│ return PackageInfo.load(self._get_release_info(name, version))
14.51 51│
14.51 → 52│ cached = self._release_cache.remember(
14.51 53│ f"{name}:{version}", lambda: self._get_release_info(name, version)
14.51 54│ )
14.51
14.51 14 /opt/poetry/venv/lib/python3.12/site-packages/poetry/utils/cache.py:147 in remember
14.52 145│ value = self.get(key)
14.52 146│ if value is None:
14.52 → 147│ value = callback() if callable(callback) else callback
14.52 148│ self.put(key, value, minutes)
14.52 149│ return value
14.52
14.52 13 /opt/poetry/venv/lib/python3.12/site-packages/poetry/repositories/cached_repository.py:53 in <lambda>
14.52 51│
14.52 52│ cached = self._release_cache.remember(
14.52 → 53│ f"{name}:{version}", lambda: self._get_release_info(name, version)
14.52 54│ )
14.52 55│
14.52
14.52 12 /opt/poetry/venv/lib/python3.12/site-packages/poetry/repositories/legacy_repository.py:123 in _get_release_info
14.52 121│ yanked = page.yanked(name, version)
14.52 122│
14.52 → 123│ return self._links_to_data(
14.52 124│ links,
14.52 125│ PackageInfo(
14.52
14.52 11 /opt/poetry/venv/lib/python3.12/site-packages/poetry/repositories/http_repository.py:360 in _links_to_data
14.53 358│
14.53 359│ # drop yanked files unless the entire release is yanked
14.53 → 360│ info = self._get_info_from_links(links, ignore_yanked=not data.yanked)
14.53 361│
14.53 362│ data.summary = info.summary
14.53
14.53 10 /opt/poetry/venv/lib/python3.12/site-packages/poetry/repositories/http_repository.py:310 in _get_info_from_links
14.53 308│ return self._get_info_from_metadata(
14.53 309│ universal_python3_wheel
14.53 → 310│ ) or self._get_info_from_wheel(universal_python3_wheel)
14.53 311│
14.53 312│ if universal_python2_wheel:
14.53
14.53 9 /opt/poetry/venv/lib/python3.12/site-packages/poetry/repositories/http_repository.py:123 in _get_info_from_wheel
14.54 121│ try:
14.54 122│ package_info = PackageInfo.from_metadata(
14.54 → 123│ metadata_from_wheel_url(link.filename, link.url, self.session)
14.54 124│ )
14.54 125│ except HTTPRangeRequestUnsupported:
14.54
14.54 8 /opt/poetry/venv/lib/python3.12/site-packages/poetry/inspection/lazy_wheel.py:77 in metadata_from_wheel_url
14.55 75│ # After context manager exit, wheel.name will point to a deleted file path.
14.55 76│ # Add `delete_backing_file=False` to disable this for debugging.
14.55 → 77│ with LazyWheelOverHTTP(url, session) as lazy_file:
14.55 78│ metadata_bytes = lazy_file.read_metadata(name)
14.55 79│
14.55
14.55 7 /opt/poetry/venv/lib/python3.12/site-packages/poetry/inspection/lazy_wheel.py:289 in __enter__
14.56 287│ def __enter__(self: U) -> U:
14.56 288│ super().__enter__()
14.56 → 289│ self._setup_content()
14.56 290│ return self
14.56 291│
14.56
14.56 6 /opt/poetry/venv/lib/python3.12/site-packages/poetry/inspection/lazy_wheel.py:357 in _setup_content
14.57 355│ if self._length is None:
14.57 356│ logger.debug("begin fetching content length")
14.57 → 357│ self._length = self._fetch_content_length()
14.57 358│ logger.debug("done fetching content length (is: %d)", self._length)
14.57 359│ # Enable us to seek and write anywhere in the backing file up to this
14.57
14.57 5 /opt/poetry/venv/lib/python3.12/site-packages/poetry/inspection/lazy_wheel.py:508 in _fetch_content_length
14.58 506│ """
14.58 507│ initial_chunk_size = self._initial_chunk_length()
14.58 → 508│ ret_length, tail = self._extract_content_length(initial_chunk_size)
14.58 509│
14.58 510│ # Need to explicitly truncate here in order to perform the write and seek
14.58
14.58 4 /opt/poetry/venv/lib/python3.12/site-packages/poetry/inspection/lazy_wheel.py:609 in _extract_content_length
14.59 607│ try:
14.59 608│ # Initial range request for just the end of the file.
14.59 → 609│ file_length, tail = self._try_initial_chunk_request(initial_chunk_size)
14.59 610│ except HTTPError as e:
14.59 611│ resp = e.response
14.59
14.59 3 /opt/poetry/venv/lib/python3.12/site-packages/poetry/inspection/lazy_wheel.py:572 in _try_initial_chunk_request
14.60 570│
14.60 571│ self._request_count += 1
14.60 → 572│ tail = self._session.get(self._url, headers=headers, stream=True)
14.60 573│ tail.raise_for_status()
14.60 574│
14.60
14.60 2 /opt/poetry/venv/lib/python3.12/site-packages/poetry/utils/authenticator.py:267 in get
14.60 265│
14.60 266│ def get(self, url: str, **kwargs: Any) -> requests.Response:
14.60 → 267│ return self.request("get", url, **kwargs)
14.60 268│
14.60 269│ def head(self, url: str, **kwargs: Any) -> requests.Response:
14.60
14.60 1 /opt/poetry/venv/lib/python3.12/site-packages/poetry/utils/authenticator.py:245 in request
14.61 243│ if resp.status_code not in STATUS_FORCELIST or is_last_attempt:
14.61 244│ if raise_for_status:
14.61 → 245│ resp.raise_for_status()
14.61 246│ return resp
14.61 247│
14.61
14.61 HTTPError
14.61
14.61 500 Server Error: Internal Server Error for url: https://us-central1-python.pkg.dev/artifacts-downloads/namespaces/atomic-uxr-operations-jcqu/repositories/python-packages/downloads/very-long-base64-string
14.61
14.61 at /opt/poetry/venv/lib/python3.12/site-packages/requests/models.py:1021 in raise_for_status
14.62 1017│ f"{self.status_code} Server Error: {reason} for url: {self.url}"
14.62 1018│ )
14.62 1019│
14.62 1020│ if http_error_msg:
14.62 → 1021│ raise HTTPError(http_error_msg, response=self)
14.62 1022│
14.62 1023│ def close(self):
14.62 1024│ """Releases the connection back to the pool. Once this method has been
14.62 1025│ called the underlying ``raw`` object must not be accessed again.
------
The image to this point in the build had installed Poetry 1.8.0. I then went back to this step in the build and added: poetry self update 1.7.1. The build completed successfully after doing this.
poetry install commands on my local machine, running Arch Linux (uname -r == 6.6.16-1-lts) with Python 3.12.1, complete successfully whether I have Poetry 1.7.1 or Poetry 1.8.0 installed, so this seems to only impact Docker.
This error occurs even if I am interacting with a container; if I cut off the Dockerfile to the instruction just before the poetry install and run the poetry install myself, I get the same 500 error. If I then take the URL given in the error message and curl it manually, I get a 200 response. Similarly I get a 200 response with requests in a Python interactive session.
I have tried this with multiple GAR repositories across multiple GCP projects under multiple accounts, so it is not related to a specific GAR repository or GCP account.
Here's a minimal version of the Dockerfile I'm using:
# python 3.12 slim
FROM python@sha256:d0c9bf03d80a3c00963f1b973760094c0b2070638fa64dd4f67d2247c0110efc AS base
ENV POETRY_HOME="/opt/poetry" \
POETRY_VIRTUALENVS_IN_PROJECT=1 \
POETRY_NO_INTERACTION=1
ENV PATH="${POETRY_HOME}/bin:${PATH}"
# install poetry
RUN apt-get update \
&& apt-get install -y curl gnupg gcc \
&& curl -sSL https://install.python-poetry.org | python - \
&& poetry self add keyrings.google-artifactregistry-auth
# install dependencies
COPY app /home/myuser/app
RUN --mount=type=secret,id=adc,target=/root/.config/gcloud/application_default_credentials.json \
cd /home/myuser/app && \
poetry install
My temporary fix is to add the following to the # install poetry instruction:
&& poetry self update 1.7.1
at which point it works fine.
To summarize:
- Inside Docker builds and Docker containers, with Poetry 1.8.0 installed,
poetry installcommands that install from private GAR sources throw 500 Internal Server Error. - In the same situation but with Poetry 1.7.1 installed, there is no error.
- On my host machine, with Poetry 1.8.0 installed OR Poetry 1.7.1 installed, there is no error.