Skip to content

Version Incompatibility in 1.10.14 resulting from snowflake workaround #13093

@r-richmond

Description

@r-richmond

Apache Airflow version: 1.10.14
Environment: Python 3.8

  • Install tools: pipenv

What happened:

So I appear to have stumbled across a version incompatibility for airflow-1.10.14
apache-airflow[aws] was modified with this pr to restrict boto to boto3~=1.10,<1.11 with the comment for snowflake
When attempting to install the latest amazon-provider backport apache-airflow-backport-providers-amazon==2020.11.23->-r I’m faced with an incompatible version in resolved dependancies as the backport requires boto3<2.0.0,>=1.12.0

attempting to install aws extra and amazon-provide backport results in a version incompatibility error.

There are incompatible versions in the resolved dependencies:
  boto3<1.11,~=1.10 (from apache-airflow[aws]==1.10.14->-r /var/folders/dw/8s4dltks7bg5ws8kht1wfbqw0000gp/T/pipenvtfsk4ltbrequirements/pipenv-ukucfwxw-constraints.txt (line 2))
  boto3<2.0.0,>=1.12.0 (from apache-airflow-backport-providers-amazon==2020.11.23->-r /var/folders/dw/8s4dltks7bg5ws8kht1wfbqw0000gp/T/pipenvtfsk4ltbrequirements/pipenv-ukucfwxw-constraints.txt (line 3))

What you expected to happen:

Airflow has compatible versions.

How to reproduce it:

1. Create Pipfile with the following contents
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
apache-airflow = {extras = ["aws"],version = "*"}
# https://github.com/apache/airflow/blob/master/README.md#backport-packages
apache-airflow-backport-providers-amazon = "*"

[requires]
python_version = "3.8"
2. run `pipenv lock`

Anything else we need to know:

Suggested solution: move all snowflake version modifications out of related airflow extras and directly into the snowflake extra.
Slack Thread: https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1608057981159400

Metadata

Metadata

Assignees

Labels

kind:bugThis is a clearly a bug

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions