Further SSL problems with Poetry and a private Gitlab package registry #4016
Description
- I am on the latest Poetry version.
- I have searched the issues of this repo and believe that this is not a duplicate.
- If an exception occurs when executing a command, I executed it again in debug mode (
-vvvoption).
- OS version and name: macOS 11.3
- Poetry version: 1.1.6
Issue
Closely related to #745 and #1012, I am experiencing problems working with poetry against a private (Gitlab) based package registry, on an internal project. The locked nature of this make this very hard for me to do a public replication of the problem, so apologies whilst I have to be descriptive (if a little vague for some sensitive info).
I have two poetry based python+django projects. The first (atlas-models) I have managed to package as a reusable app, and build and publish (with just poetry) to our internal Gitlab Package Registry. Presently there are two versions of the package in the registry (but I've tried with between 1 & 3 versions there, that doesn't appear to be a factor).
In my second project, when I attempt to add the first package via poetry add atlas-models -vvv, I get the following (abbreviated) output:
➜ ouh-mpages-development-repo git:(master) ✗ poetry add atlas-models -vvv
Using virtualenv: /Users/carl/Projects/pages.oxnet/documentation/ouh-mpages-development/ouh-mpages-development-repo/.venv
pages-oxnet-atlas: 2 packages found for atlas-models *
PyPI: No packages found for atlas-models *
Using version ^0.3.1 for atlas-models
Updating dependencies
Resolving dependencies...
1: fact: mpages mkdocs build is 0.1.0
1: derived: mpages mkdocs build
1: fact: mpages mkdocs build depends on Django (^3.2)
1: fact: mpages mkdocs build depends on atlas-models (^0.3.1)
1: selecting mpages mkdocs build (0.1.0)
1: derived: atlas-models (>=0.3.1,<0.4.0)
1: derived: Django (>=3.2,<4.0)
pages-oxnet-atlas: 1 packages found for atlas-models >=0.3.1,<0.4.0
PyPI: No packages found for atlas-models >=0.3.1,<0.4.0
1: fact: atlas-models (0.3.1) depends on Django (>=3.2,<4.0)
1: fact: atlas-models (0.3.1) depends on cx-oracle (>=8,<9)
1: selecting atlas-models (0.3.1)
1: derived: cx-oracle (>=8,<9)
1: fact: django (3.2) depends on asgiref (>=3.3.2,<4)
1: fact: django (3.2) depends on pytz (*)
1: fact: django (3.2) depends on sqlparse (>=0.2.2)
1: selecting django (3.2)
1: derived: sqlparse (>=0.2.2)
1: derived: pytz
1: derived: asgiref (>=3.3.2,<4)
1: selecting cx-oracle (8.1.0)
1: selecting sqlparse (0.4.1)
1: selecting pytz (2021.1)
1: selecting asgiref (3.3.4)
1: Version solving took 0.723 seconds.
1: Tried 1 solutions.
Finding the necessary packages for the current system
Package operations: 1 install, 0 updates, 0 removals, 5 skipped
• Installing asgiref (3.3.4): Pending...
• Installing asgiref (3.3.4): Skipped for the following reason: Already installed
• Installing pytz (2021.1): Pending...
• Installing pytz (2021.1): Skipped for the following reason: Already installed
• Installing sqlparse (0.4.1): Pending...
• Installing sqlparse (0.4.1): Skipped for the following reason: Already installed
• Installing cx-oracle (8.1.0): Pending...
• Installing cx-oracle (8.1.0): Skipped for the following reason: Already installed
• Installing django (3.2): Pending...
• Installing django (3.2): Skipped for the following reason: Already installed
• Installing atlas-models (0.3.1): Pending...
Retrying HTTP request in 0.5 seconds.
Retrying HTTP request in 1.0 seconds.
Retrying HTTP request in 1.5 seconds.
Retrying HTTP request in 2.0 seconds.
Retrying HTTP request in 2.5 seconds.
• Installing atlas-models (0.3.1): Failed
SSLError
HTTPSConnectionPool(host='oxnetcnsc01.oxnet.nhs.uk', port=443): Max retries exceeded with url: /api/v4/projects/202/packages/pypi/files/b278cb68cacfff2d3fd4a3d1a82c6ff12dc192c332e67db76cbdf03a02a10c2a/atlas_models-0.3.1-py3-none-any.whl (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))
at ~/.poetry/lib/poetry/_vendor/py3.9/requests/adapters.py:514 in send
510│ raise ProxyError(e, request=request)
511│
512│ if isinstance(e.reason, _SSLError):
513│ # This branch is for urllib3 v1.22 and later.
→ 514│ raise SSLError(e, request=request)
515│
516│ raise ConnectionError(e, request=request)
517│
518│ except ClosedPoolError as e:
Failed to add packages, reverting the pyproject.toml file to its original content.
The pyproject.toml reads as:
name = "MPages MkDocs Build"
version = "0.1.0"
description = "Python / Django application *snip*"
authors = ["Carl Marshall <email-redacted>"]
license = "Apache-2.0"
[tool.poetry.dependencies]
python = "^3.9"
Django = "^3.2"
[tool.poetry.dev-dependencies]
[[tool.poetry.source]]
name = "pages-oxnet-atlas"
url = "https://oxnetcnsc01.oxnet.nhs.uk/api/v4/projects/202/packages/pypi/simple"
secondary = true
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
Additionally, poetry config --list -vvv results in:
Loading configuration file /Users/carl/Library/Application Support/pypoetry/config.toml
Loading configuration file /Users/carl/Library/Application Support/pypoetry/auth.toml
cache-dir = "/Users/carl/Library/Caches/pypoetry"
certificates.oxnetcnsc01.cert = "/Users/carl/Projects/pages.oxnet/oxnet-combined.pem" # None
certificates.pages-oxnet.cert = "/Users/carl/Projects/pages.oxnet/oxnet-combined.pem" # None
certificates.pages-oxnet-atlas.cert = "/Users/carl/Projects/pages.oxnet/oxnet-combined.pem" # None
experimental.new-installer = true
installer.parallel = true
repositories.pages-oxnet.url = "https://oxnetcnsc01.oxnet.nhs.uk/api/v4/projects/202/packages/pypi"
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.path = "{cache-dir}/virtualenvs" # /Users/carl/Library/Caches/pypoetry/virtualenvs
I have also done poetry config http-basic.pages-oxnet user.name token-code for both .pages-oxnet and .pages-oxnet-atlas. This was all pretty much put in place to get the build and publish workflows to succeed. I also added certificates.oxnetcnsc01.cert and http-basic.oxnetcnsc01 as well to ensure I hadn't overlooked anything.
I can see that the process is locating the latest version of the atlas-models package from the gitlab private registry, but is clearly having a problem with downloading the wheel file.
The SSL certificates you can see in the config output above are seemingly valid as they resolved the publish SSL connection issues.
If I try to set the environment variable though (to catch a potential issue with requests as suggested in an earlier bug report), I get a whole host of different errors, i.e.:
export REQUESTS_CA_BUNDLE=/Users/carl/Projects/pages.oxnet/oxnet-combined.pemfollowed bypoetry add atlas-models -vvvresults in a much faster set of errors:
Using virtualenv: /Users/carl/Projects/pages.oxnet/documentation/ouh-mpages-development/ouh-mpages-development-repo/.venv
pages-oxnet-atlas: 2 packages found for atlas-models *
Stack trace:
23 ~/.poetry/lib/poetry/_vendor/py3.9/clikit/console_application.py:131 in run
129│ parsed_args = resolved_command.args
130│
→ 131│ status_code = command.handle(parsed_args, io)
132│ except KeyboardInterrupt:
133│ status_code = 1
22 ~/.poetry/lib/poetry/_vendor/py3.9/clikit/api/command/command.py:120 in handle
118│ def handle(self, args, io): # type: (Args, IO) -> int
119│ try:
→ 120│ status_code = self._do_handle(args, io)
121│ except KeyboardInterrupt:
122│ if io.is_debug():
21 ~/.poetry/lib/poetry/_vendor/py3.9/clikit/api/command/command.py:171 in _do_handle
169│ handler_method = self._config.handler_method
170│
→ 171│ return getattr(handler, handler_method)(args, io, self)
172│
173│ def __repr__(self): # type: () -> str
20 ~/.poetry/lib/poetry/_vendor/py3.9/cleo/commands/command.py:92 in wrap_handle
90│ self._command = command
91│
→ 92│ return self.handle()
93│
94│ def handle(self): # type: () -> Optional[int]
19 ~/.poetry/lib/poetry/console/commands/add.py:106 in handle
104│ return 0
105│
→ 106│ requirements = self._determine_requirements(
107│ packages,
108│ allow_prereleases=self.option("allow-prereleases"),
18 ~/.poetry/lib/poetry/console/commands/init.py:328 in _determine_requirements
326│ elif "version" not in requirement:
327│ # determine the best version automatically
→ 328│ name, version = self._find_best_version_for_package(
329│ requirement["name"],
330│ allow_prereleases=allow_prereleases,
17 ~/.poetry/lib/poetry/console/commands/init.py:361 in _find_best_version_for_package
359│
360│ selector = VersionSelector(self._get_pool())
→ 361│ package = selector.find_best_candidate(
362│ name, required_version, allow_prereleases=allow_prereleases, source=source
363│ )
16 ~/.poetry/lib/poetry/version/version_selector.py:32 in find_best_candidate
30│ },
31│ )
→ 32│ candidates = self._pool.find_packages(dependency)
33│ only_prereleases = all([c.version.is_prerelease() for c in candidates])
34│
15 ~/.poetry/lib/poetry/repositories/pool.py:165 in find_packages
163│ packages = []
164│ for repo in self._repositories:
→ 165│ packages += repo.find_packages(dependency)
166│
167│ return packages
14 ~/.poetry/lib/poetry/repositories/pypi_repository.py:101 in find_packages
99│
100│ try:
→ 101│ info = self.get_package_info(dependency.name)
102│ except PackageNotFound:
103│ self._log(
13 ~/.poetry/lib/poetry/repositories/pypi_repository.py:202 in get_package_info
200│ return self._get_package_info(name)
201│
→ 202│ return self._cache.store("packages").remember_forever(
203│ name, lambda: self._get_package_info(name)
204│ )
12 ~/.poetry/lib/poetry/_vendor/py3.9/cachy/repository.py:174 in remember_forever
172│ return val
173│
→ 174│ val = value(callback)
175│
176│ self.forever(key, val)
11 ~/.poetry/lib/poetry/_vendor/py3.9/cachy/helpers.py:6 in value
4│ def value(val):
5│ if callable(val):
→ 6│ return val()
7│
8│ return val
10 ~/.poetry/lib/poetry/repositories/pypi_repository.py:203 in <lambda>
201│
202│ return self._cache.store("packages").remember_forever(
→ 203│ name, lambda: self._get_package_info(name)
204│ )
205│
9 ~/.poetry/lib/poetry/repositories/pypi_repository.py:207 in _get_package_info
205│
206│ def _get_package_info(self, name): # type: (str) -> dict
→ 207│ data = self._get("pypi/{}/json".format(name))
208│ if data is None:
209│ raise PackageNotFound("Package [{}] not found.".format(name))
8 ~/.poetry/lib/poetry/repositories/pypi_repository.py:315 in _get
313│ def _get(self, endpoint): # type: (str) -> Union[dict, None]
314│ try:
→ 315│ json_response = self.session.get(self._base_url + endpoint)
316│ except requests.exceptions.TooManyRedirects:
317│ # Cache control redirect loop.
7 ~/.poetry/lib/poetry/_vendor/py3.9/requests/sessions.py:555 in get
553│
554│ kwargs.setdefault('allow_redirects', True)
→ 555│ return self.request('GET', url, **kwargs)
556│
557│ def options(self, url, **kwargs):
6 ~/.poetry/lib/poetry/_vendor/py3.9/requests/sessions.py:542 in request
540│ }
541│ send_kwargs.update(settings)
→ 542│ resp = self.send(prep, **send_kwargs)
543│
544│ return resp
5 ~/.poetry/lib/poetry/_vendor/py3.9/requests/sessions.py:677 in send
675│ # Redirect resolving generator.
676│ gen = self.resolve_redirects(r, request, **kwargs)
→ 677│ history = [resp for resp in gen]
678│ else:
679│ history = []
4 ~/.poetry/lib/poetry/_vendor/py3.9/requests/sessions.py:677 in <listcomp>
675│ # Redirect resolving generator.
676│ gen = self.resolve_redirects(r, request, **kwargs)
→ 677│ history = [resp for resp in gen]
678│ else:
679│ history = []
3 ~/.poetry/lib/poetry/_vendor/py3.9/requests/sessions.py:237 in resolve_redirects
235│ else:
236│
→ 237│ resp = self.send(
238│ req,
239│ stream=stream,
2 ~/.poetry/lib/poetry/_vendor/py3.9/requests/sessions.py:655 in send
653│
654│ # Send the request
→ 655│ r = adapter.send(request, **kwargs)
656│
657│ # Total elapsed time of the request (approximately)
1 ~/.poetry/lib/poetry/_vendor/py3.9/cachecontrol/adapter.py:53 in send
51│ request.headers.update(self.controller.conditional_headers(request))
52│
→ 53│ resp = super(CacheControlAdapter, self).send(request, **kw)
54│
55│ return resp
SSLError
HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/atlas-models/json/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))
at ~/.poetry/lib/poetry/_vendor/py3.9/requests/adapters.py:514 in send
510│ raise ProxyError(e, request=request)
511│
512│ if isinstance(e.reason, _SSLError):
513│ # This branch is for urllib3 v1.22 and later.
→ 514│ raise SSLError(e, request=request)
515│
516│ raise ConnectionError(e, request=request)
517│
518│ except ClosedPoolError as e:
When I unset REQUESTS_CA_BUNDLE and retry the add command, we're back to the original error outputs and timeout.
I am now at a loss of what else to try here. Something appears to be partly broken in the process of adding my package from the internal registry, and this is now stalling my whole development plan. I don't really want to have to remove poetry from this setup, but I do note there's very little Gitlab recognition of Poetry usage (none in their own documentation) and most examples I'm finding at either general pip or GitHub solutions.
Please let me know if there's any other practical tests, or configuration I can attempt, or if this is enough information on which to diagnose a bug.
Thank you for your time and help.