Combining optional prod package with mandatory dev extra causes prod package to be non-optional #4401
Description
-
I am on the latest Poetry version.
-
I have searched the issues of this repo and believe that this is not a duplicate. (What to do in case of certain configurations of dependencies being both mandatory and optional? #2868 is similar but not the same issue.)
-
If an exception occurs when executing a command, I executed it again in debug mode (
-vvvoption). -
OS version and name: Arch Linux / unknown OS
-
Poetry version: 1.1.7
-
Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/l0b0/c0308091c7a748e6d481791c79a6bbda
Issue
In a project I'm working on we have a few different sets of production dependencies, one for each service. To avoid bloating each service with all the production packages we've marked all the production packages as optional and created extras for each of them, listing the relevant packages. The problem is when I want to introduce a mandatory dev dependency on an extra which is also an optional production dependency, like this:
[tool.poetry.dependencies]
python = "^3.9"
SQLAlchemy = {version = "*", optional = true}
[tool.poetry.dev-dependencies]
SQLAlchemy = {version = "*", extras = ["mypy"]}
[tool.poetry.extras]
foo = [
"SQLAlchemy",
]poetry lock in this case changes the sqlalchemy production package to be mandatory:
@@ -1327,7 +1327,7 @@ name = "sqlalchemy"
version = "1.4.22"
description = "Database Abstraction Library"
category = "main"
-optional = true
+optional = false
python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7"The reason I'm reporting this is that Dependabot is having trouble with the original files:
updater | ERROR <job_187255052> Error processing sqlalchemy (Dependabot::SharedHelpers::HelperSubprocessFailed)
updater | ERROR <job_187255052> Creating virtualenv foo-OyE-KPju-py3.9 in /home/dependabot/.cache/pypoetry/virtualenvs
updater | <job_187255052> Updating dependencies
updater | <job_187255052> Resolving dependencies...
updater | <job_187255052>
updater | <job_187255052> Writing lock file
updater | <job_187255052>
updater | <job_187255052> AttributeError
updater | <job_187255052>
updater | <job_187255052> 'EmptyConstraint' object has no attribute 'allows'
updater | <job_187255052>
updater | <job_187255052> at /usr/local/.pyenv/versions/3.9.6/lib/python3.9/site-packages/poetry/core/version/markers.py:291 in validate
updater | <job_187255052> 287│
updater | <job_187255052> 288│ if self._name not in environment:
updater | <job_187255052> 289│ return True
updater | <job_187255052> 290│
updater | <job_187255052> → 291│ return self._constraint.allows(self._parser(environment[self._name]))
updater | <job_187255052> 292│
updater | <job_187255052> 293│ def without_extras(self): # type: () -> MarkerTypes
updater | <job_187255052> 294│ return self.exclude("extra")
updater | <job_187255052> 295│
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-python-0.160.0/lib/dependabot/python/update_checker/poetry_version_resolver.rb:323:in `run_poetry_command'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-python-0.160.0/lib/dependabot/python/update_checker/poetry_version_resolver.rb:88:in `block (2 levels) in fetch_latest_resolvable_version_string'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-common-0.160.0/lib/dependabot/shared_helpers.rb:160:in `with_git_configured'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-python-0.160.0/lib/dependabot/python/update_checker/poetry_version_resolver.rb:76:in `block in fetch_latest_resolvable_version_string'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-common-0.160.0/lib/dependabot/shared_helpers.rb:46:in `block (2 levels) in in_a_temporary_directory'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-common-0.160.0/lib/dependabot/shared_helpers.rb:46:in `chdir'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-common-0.160.0/lib/dependabot/shared_helpers.rb:46:in `block in in_a_temporary_directory'
updater | ERROR <job_187255052> /usr/lib/ruby/2.7.0/tmpdir.rb:89:in `mktmpdir'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-common-0.160.0/lib/dependabot/shared_helpers.rb:43:in `in_a_temporary_directory'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-python-0.160.0/lib/dependabot/python/update_checker/poetry_version_resolver.rb:75:in `fetch_latest_resolvable_version_string'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-python-0.160.0/lib/dependabot/python/update_checker/poetry_version_resolver.rb:48:in `latest_resolvable_version'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-python-0.160.0/lib/dependabot/python/update_checker.rb:43:in `latest_resolvable_version'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-common-0.160.0/lib/dependabot/update_checkers/base.rb:74:in `preferred_resolvable_version'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-common-0.160.0/lib/dependabot/update_checkers/base.rb:257:in `preferred_version_resolvable_with_unlock?'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-common-0.160.0/lib/dependabot/update_checkers/base.rb:249:in `numeric_version_can_update?'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-common-0.160.0/lib/dependabot/update_checkers/base.rb:199:in `version_can_update?'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-common-0.160.0/lib/dependabot/update_checkers/base.rb:44:in `can_update?'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:429:in `requirements_to_unlock'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:224:in `check_and_create_pull_request'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:80:in `check_and_create_pr_with_error_handling'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:56:in `block in run'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:56:in `each'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:56:in `run'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/lib/dependabot/update_files_job.rb:17:in `perform_job'
updater | ERROR <job_187255052> /home/dependabot/dependabot-updater/lib/dependabot/base_job.rb:28:in `run'
updater | ERROR <job_187255052> bin/update_files.rb:21:in `<main>'
A workaround is to split the definition in two, by figuring out the name of the packages corresponding to the SQL Alchemy "mypy" extra, in this case sqlalchemy2-stubs:
@@ -66,7 +66,7 @@ pylint = "*"
pytest = "*"
pytest-randomly = "*"
pytest-subtests = "*"
-SQLAlchemy = {version = "*", optional = true, extras = ["mypy"]}
+sqlalchemy2-stubs = "*"
types-python-dateutil = "*"
types-setuptools = "*"
types-six = "*"