Use Pip 21.* to install airflow officially #15513
Conversation
boring-cyborg
bot
added
area:dev-tools
area:providers
kind:documentation
provider:Apache
labels
|
Hey @ashb and @kaxil -> If we get this one merged, I think we might want to release a few providers (Google, Azure) where we had some small issues with warning (I am looking at an Azure warning in a moment) + Apache.Beam operator with the updated extra and we might aim to release 2.0.3 rather quickly - with mostly "pip 21 full support". |
potiuk
force-pushed
the
attempt-to-upgrade-to-pip-21
branch
from
11ba69a to
666cb39
Compare
|
The Workflow run is cancelling this PR. Building images for the PR has failed. Follow the workflow link to check the reason. |
potiuk
force-pushed
the
attempt-to-upgrade-to-pip-21
branch
from
666cb39 to
9607eee
Compare
potiuk
force-pushed
the
attempt-to-upgrade-to-pip-21
branch
2 times, most recently
from
9fa5f49 to
7195609
Compare
potiuk
added
the
full tests needed
potiuk
force-pushed
the
attempt-to-upgrade-to-pip-21
branch
from
7195609 to
d24a9ac
Compare
potiuk
force-pushed
the
attempt-to-upgrade-to-pip-21
branch
from
d24a9ac to
28f96d8
Compare
potiuk
force-pushed
the
attempt-to-upgrade-to-pip-21
branch
3 times, most recently
from
b970cf7 to
6dc6e5e
Compare
|
The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*. |
The PIP 20.2.4 was so far the only officially supported installation mechanism for Airflow as there were some problems with conflicting dependencies (which were ignored by previous versio of PIP). This change attempts to solve this by removing a [gcp] extra from `apache-beam` which turns out to be the major source of the problem - as it contains requirements to the old version of google client libraries (but apparently only used for tests). The "apache-beam" provider migh however need the [gcp] extra for other components so in order to not break the backwards compatibility, another approach is used. Instead of adding [gcp] as extra in the apache-beam extra, the apache.beam provider's [google] extra is extended with 'apache-beam[gcp]' additional requirement so that whenever the provider is installed, the apache-beam with [gcp] extra is installed as well.
potiuk
force-pushed
the
attempt-to-upgrade-to-pip-21
branch
from
6dc6e5e to
1b1f96b
Compare
Co-authored-by: Tzu-ping Chung <[email protected]>
Co-authored-by: Tzu-ping Chung <[email protected]>
Co-authored-by: Tzu-ping Chung <[email protected]>
Co-authored-by: Tzu-ping Chung <[email protected]>
|
I'd love that one to be merged, to finally get PIP 21 support (and keep the deps under control). |
|
Just a heads-up, pip 21.1.1 will likely be released next week to fix an import issue for 3.6.0 and 3.6.1. |
Yep. I think I will add a couple of "automated" warnings (a'la dependabot). that will make us aware of the need to upgrade when newer versions of PIP/Pylint and few other tools that we are pinning. Dependabot is not good for us, because we are continuously updating automatically, and keep open dependencies, but those few dependencies we have pinned might make perfect sense to get "warnings" about. |
* Use Pip 21.* to install airflow officially The PIP 20.2.4 was so far the only officially supported installation mechanism for Airflow as there were some problems with conflicting dependencies (which were ignored by previous versio of PIP). This change attempts to solve this by removing a [gcp] extra from `apache-beam` which turns out to be the major source of the problem - as it contains requirements to the old version of google client libraries (but apparently only used for tests). The "apache-beam" provider migh however need the [gcp] extra for other components so in order to not break the backwards compatibility, another approach is used. Instead of adding [gcp] as extra in the apache-beam extra, the apache.beam provider's [google] extra is extended with 'apache-beam[gcp]' additional requirement so that whenever the provider is installed, the apache-beam with [gcp] extra is installed as well. * Update airflow/providers/apache/beam/CHANGELOG.rst Co-authored-by: Tzu-ping Chung <[email protected]> * Update airflow/providers/apache/beam/CHANGELOG.rst Co-authored-by: Tzu-ping Chung <[email protected]> * Update airflow/providers/google/CHANGELOG.rst Co-authored-by: Tzu-ping Chung <[email protected]> * Update airflow/providers/google/CHANGELOG.rst Co-authored-by: Tzu-ping Chung <[email protected]> Co-authored-by: Tzu-ping Chung <[email protected]> (cherry picked from commit e229f35)
5 participants
The PIP 20.2.4 was so far the only officially supported installation
mechanism for Airflow as there were some problems with conflicting
dependencies (which were ignored by previous versio of PIP).
This change attempts to solve this by removing a [gcp] extra
from
apache-beamwhich turns out to be the major source ofthe problem - as it contains requirements to the old version of
google client libraries (but apparently only used for tests).
The "apache-beam" provider migh however need the [gcp] extra
for other components so in order to not break the backwards
compatibility, another approach is used.
Instead of adding [gcp] as extra in the apache-beam extra,
the apache.beam provider's [google] extra is extended with
'apache-beam[gcp]' additional requirement so that whenever the
provider is installed, the apache-beam with [gcp] extra is installed
as well.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.