Skip to content

Encoding failure is not caught and propagated #6784

@kristang

Description

@kristang
  • Poetry version: 1.2.1 and 1.3.0.dev01
  • Python version: 3.9.13
  • OS version and name: Windows 11 Enterprise (build 22000.856)
  • pyproject.toml:
[tool.poetry]
name = "test-poetry-master"
version = "0.1.0"
description = ""
authors = ["kristang"]
readme = "README.md"

[[tool.poetry.source]]
name = "private-pypi"
url = "https://private/pypi/private-pypi/simple/"
secondary = true

[tool.poetry.dependencies]
python = "^3.9"

[tool.poetry.dev-dependencies]
pytest = "^7.1.1"
mypy = "^0.982"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
  • I am on the latest stable Poetry version, installed using a recommended method.
  • I have searched the issues of this repo and believe that this is not a duplicate.
  • I have consulted the FAQ and blog for any relevant entries or release notes.
  • If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Additionally

Issue

When adding a secondary, private source and attempting to authenticate via basic-http, poetry silently ignores a UnicodeEncodingError originating from a failed latin1 encoding of a str.

Instead, poetry moves on to the public pypi.org feed and informs the user that the searched package is not available - even if it is available on pypi.org

The error stems from loading in a user/password from Windows Certificate Manager which appears to be corrupted and cannot be encoded with 'latin1'

The source of the encoding error is poetry invoking _basic_auth_str from auth.py in the requests package:
https://github.com/psf/requests/blob/7104ad4b135daab0ed19d8e41bd469874702342b/requests/auth.py#L56-L60

The originating call in poetry is request in utils.authenticator.

if credential.username is not None or credential.password is not None:
request = requests.auth.HTTPBasicAuth(
credential.username or "", credential.password or ""
)(request)

solve() has a fairly wide try-catch with a base Exception

try:
next: str | None = self._root.name
while next is not None:
self._propagate(next)
next = self._choose_package_version()
return self._result()
except Exception:
raise
finally:
self._log(
f"Version solving took {time.time() - start:.3f} seconds.\n"
f"Tried {self._solution.attempted_solutions} solutions."
)

Reproducing

As this involves a broken str that fails to encode reproducibility is a bit hard. I will try and create tests and create a PR if my skills are sufficient enough to create something useful.

Running pipenv lock -vvv yields:

  Stack trace:

  4  ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\solver.py:157 in _solve
      155156try:
    → 157│             result = resolve_version(self._package, self._provider)
      158159│             packages = result.packages

  3  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\__init__.py:18 in resolve_version
       16│     solver = VersionSolver(root, provider)
       17│ 
    →  18return solver.solve()
       19│ 

  2  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:113 in solve
      111next: str | None = self._root.name
      112while next is not None:
    → 113self._propagate(next)
      114next = self._choose_package_version()
      115│ 

  1  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:152 in _propagate
      150# where that incompatibility will allow us to derive new assignments
      151# that avoid the conflict.152│                     root_cause = self._resolve_conflict(incompatibility)
      153154# Back jumping erases all the assignments we did at the previous

  SolveFailure

  Because test-poetry-master depends on mypy (^0.982) which doesn't exist, version solving failed.

  at ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:351 in _resolve_conflict
      347│             )
      348self._log(f'! which is caused by "{most_recent_satisfier.cause}"')
      349self._log(f"! thus: {incompatibility}")
      350│ 
    → 351raise SolveFailure(incompatibility)
      352353def _choose_package_version(self) -> str | None:
      354"""
      355│         Tries to select a version of a required package.

The following error occurred when trying to handle this error:


  Stack trace:

  11  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:329 in run
       327328try:
     → 329│                 exit_code = self._run(io)
       330except Exception as e:
       331if not self._catch_exceptions:

  10  ~\Documents\Projects\GITHUB\poetry\src\poetry\console\application.py:185 in _run
       183self._load_plugins(io)
       184│ 
     → 185│         exit_code: int = super()._run(io)
       186return exit_code
       187│ 

   9  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:423 in _run
       421│             io.input.set_stream(stream)
       422│ 
     → 423│         exit_code = self._run_command(command, io)
       424self._running_command = None
       425│ 

   8  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:465 in _run_command
       463464if error is not None:
     → 465raise error
       466467return event.exit_code

   7  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:449 in _run_command
       447448if event.command_should_run():
     → 449│                 exit_code = command.run(io)
       450else:
       451│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

   6  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\commands\base_command.py:119 in run
       117│         io.input.validate()
       118│ 
     → 119│         status_code = self.execute(io)
       120121if status_code is None:

   5  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\commands\command.py:83 in execute
        8182try:
     →  83return self.handle()
        84except KeyboardInterrupt:
        85return 1

   4  ~\Documents\Projects\GITHUB\poetry\src\poetry\console\commands\lock.py:54 in handle
        52self.installer.lock(update=not self.option("no-update"))
        53│ 
     →  54return self.installer.run()
        55│ 

   3  ~\Documents\Projects\GITHUB\poetry\src\poetry\installation\installer.py:114 in run
       112self._execute_operations = False
       113│ 
     → 114return self._do_install()
       115116def dry_run(self, dry_run: bool = True) -> Installer:

   2  ~\Documents\Projects\GITHUB\poetry\src\poetry\installation\installer.py:247 in _do_install
       245│                 source_root=self._env.path.joinpath("src")
       246│             ):
     → 247│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
       248else:
       249self._io.write_line("Installing dependencies from lock file")

   1  ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\solver.py:74 in solve
        72with self._progress(), self._provider.use_latest_for(use_latest or []):
        73│             start = time.time()
     →  74│             packages, depths = self._solve()
        75│             end = time.time()
        76│ 

  SolverProblemError

  Because test-poetry-master depends on mypy (^0.982) which doesn't exist, version solving failed.

  at ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\solver.py:163 in _solve
      159│             packages = result.packages
      160except OverrideNeeded as e:
      161return self._solve_in_compatibility_mode(e.overrides)
      162except SolveFailure as e:
    → 163raise SolverProblemError(e)
      164165│         combined_nodes = depth_first_search(PackageNode(self._package, packages))
      166│         results = dict(aggregate_package_nodes(nodes) for nodes in combined_nodes)
      167

No-where in the stacktrace is a user informed that the underlying issue is an encoding failure of credentials.

Pseduo fix

Wrapping requests.auth.HTTPBasicAuth in a try-except can catch the UnicodeEncodeError and propagate it to the user:

if credential.username is not None or credential.password is not None:
    try:
        request = requests.auth.HTTPBasicAuth(
            credential.username or "", credential.password or ""
        )(request)
    except UnicodeEncodeError as e:
        raise PoetryException(e)

The new stacktrace now contains the correct UnicodeEncodeError:

  Stack trace:

  2  ~\Documents\Projects\GITHUB\poetry\src\poetry\utils\authenticator.py:216 in request
      214if credential.username is not None or credential.password is not None:
      215try:
    → 216│                 request = requests.auth.HTTPBasicAuth(
      217│                     credential.username or "", credential.password or ""
      218│                 )(request)

  1  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\requests\auth.py:95 in __call__
       9394def __call__(self, r):
    →  95│         r.headers["Authorization"] = _basic_auth_str(self.username, self.password)
       96return r
       97│ 

  UnicodeEncodeError

  'latin-1' codec can't encode characters in position 0-5: ordinal not in range(256)

  at ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\requests\auth.py:60 in _basic_auth_str
       56if isinstance(username, str):
       57│         username = username.encode("latin1")
       5859if isinstance(password, str):
    →  60│         password = password.encode("latin1")
       6162│     authstr = "Basic " + to_native_string(
       63│         b64encode(b":".join((username, password))).strip()
       64│     )

The following error occurred when trying to handle this error:


  Stack trace:

  24  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:329 in run
       327328try:
     → 329│                 exit_code = self._run(io)
       330except Exception as e:
       331if not self._catch_exceptions:

  23  ~\Documents\Projects\GITHUB\poetry\src\poetry\console\application.py:185 in _run
       183self._load_plugins(io)
       184│ 
     → 185│         exit_code: int = super()._run(io)
       186return exit_code
       187│ 

  22  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:423 in _run
       421│             io.input.set_stream(stream)
       422│ 
     → 423│         exit_code = self._run_command(command, io)
       424self._running_command = None
       425│ 

  21  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:465 in _run_command
       463464if error is not None:
     → 465raise error
       466467return event.exit_code

  20  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:449 in _run_command
       447448if event.command_should_run():
     → 449│                 exit_code = command.run(io)
       450else:
       451│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  19  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\commands\base_command.py:119 in run
       117│         io.input.validate()
       118│ 
     → 119│         status_code = self.execute(io)
       120121if status_code is None:

  18  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\commands\command.py:83 in execute
        8182try:
     →  83return self.handle()
        84except KeyboardInterrupt:
        85return 1

  17  ~\Documents\Projects\GITHUB\poetry\src\poetry\console\commands\lock.py:54 in handle
        52self.installer.lock(update=not self.option("no-update"))
        53│ 
     →  54return self.installer.run()
        55│ 

  16  ~\Documents\Projects\GITHUB\poetry\src\poetry\installation\installer.py:114 in run
       112self._execute_operations = False
       113│ 
     → 114return self._do_install()
       115116def dry_run(self, dry_run: bool = True) -> Installer:

  15  ~\Documents\Projects\GITHUB\poetry\src\poetry\installation\installer.py:247 in _do_install
       245│                 source_root=self._env.path.joinpath("src")
       246│             ):
     → 247│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
       248else:
       249self._io.write_line("Installing dependencies from lock file")

  14  ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\solver.py:74 in solve
        72with self._progress(), self._provider.use_latest_for(use_latest or []):
        73│             start = time.time()
     →  74│             packages, depths = self._solve()
        75│             end = time.time()
        76│ 

  13  ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\solver.py:157 in _solve
       155156try:
     → 157│             result = resolve_version(self._package, self._provider)
       158159│             packages = result.packages

  12  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\__init__.py:18 in resolve_version
        16│     solver = VersionSolver(root, provider)
        17│ 
     →  18return solver.solve()
        19│ 

  11  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:114 in solve
       112while next is not None:
       113self._propagate(next)
     → 114next = self._choose_package_version()
       115116return self._result()

  10  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:413 in _choose_package_version
       411│             dependency = unsatisfied[0]
       412else:
     → 413│             dependency = min(*unsatisfied, key=_get_min)
       414415│         locked = self._provider.get_locked(dependency)

   9  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:399 in _get_min
       397398try:
     → 399│                 num_packages = len(self._dependency_cache.search_for(dependency))
       400except ValueError:
       401│                 num_packages = 0

   8  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:62 in _search_for
        60│         packages = self.cache.get(key)
        61if packages is None:
     →  62│             packages = self.provider.search_for(dependency)
        63else:
        64│             packages = [

   7  ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\provider.py:316 in search_for
        314return PackageCollection(dependency, packages)
        315│ 
     →  316│         packages = self._pool.find_packages(dependency)
        317318│         packages.sort(

   6  ~\Documents\Projects\GITHUB\poetry\src\poetry\repositories\pool.py:136 in find_packages
       134│         packages: list[Package] = []
       135for repo in self.repositories:
     → 136│             packages += repo.find_packages(dependency)
       137return packages
       138│ 

   5  ~\Documents\Projects\GITHUB\poetry\src\poetry\repositories\repository.py:42 in find_packages
        40│         ignored_pre_release_packages = []
        41│ 
     →  42for package in self._find_packages(dependency.name, constraint):
        43if package.yanked and not isinstance(constraint, Version):
        44# PEP 592: yanked files are always ignored, unless they are the only

   4  ~\Documents\Projects\GITHUB\poetry\src\poetry\repositories\legacy_repository.py:96 in _find_packages
        94│             versions = self._cache.store("matches").get(key)
        95else:
     →  96│             page = self._get_page(f"/{name}/")
        97if page is None:
        98self._log(

   3  ~\Documents\Projects\GITHUB\poetry\src\poetry\repositories\legacy_repository.py:150 in _get_page
       148def _get_page(self, endpoint: str) -> SimpleRepositoryPage | None:
       149try:
     → 150│             response = self._get_response(endpoint)
       151except UnicodeEncodeError as e:
       152raise e

   2  ~\Documents\Projects\GITHUB\poetry\src\poetry\repositories\http.py:272 in _get_response
       270│         url = self._url + endpoint
       271try:
     → 272│             response: requests.Response = self.session.get(
       273│                 url, raise_for_status=False, timeout=REQUESTS_TIMEOUT
       274│             )

   1  ~\Documents\Projects\GITHUB\poetry\src\poetry\utils\authenticator.py:274 in get
       272273def get(self, url: str, **kwargs: Any) -> requests.Response:
     → 274return self.request("get", url, **kwargs)
       275276def post(self, url: str, **kwargs: Any) -> requests.Response:

  PoetryException

  'latin-1' codec can't encode characters in position 0-5: ordinal not in range(256)

  at ~\Documents\Projects\GITHUB\poetry\src\poetry\utils\authenticator.py:220 in request
      216│                 request = requests.auth.HTTPBasicAuth(
      217│                     credential.username or "", credential.password or ""
      218│                 )(request)
      219except UnicodeEncodeError as e:
    → 220raise PoetryException(e)
      221222│         session = self.get_session(url=url)
      223│         prepared_request = session.prepare_request(request)
      224

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugSomething isn't working as expectedstatus/triageThis issue needs to be triaged

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions