Prepare RC2 docs for May 2023 wave of Providers#31416
Merged
Merged
Conversation
eladkal
requested review from
Fokko,
jedcunningham,
josh-fell,
mik-laj,
o-nikolas and
potiuk
as code owners
May 19, 2023 12:41
pierrejeambrun
approved these changes
May 19, 2023
potiuk
approved these changes
May 19, 2023
Member
Makes perfect sense. |
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
May 23, 2023
In the last wave of providers apache#31416 we bumped min-airlfow-version to 2.4 and added mechanism to verify min-airflow version is ok while importing, but it turned out that there are cases where installing just old version of airflow (with no constraints) brings the latest version of those providers and causes new installation of airflow to fail. This is far too common to ignore or require to use constraints, unfortunately We do not have min-airlfow-version in the preinstalled providers for one reason only. For some tools that are NOT conforming to standards (such as bazel), having min-airflow-version for those providers causes circular dependency (even though technically dependencies in PyPI can - and often are - circular, because dependencies in Python are not DAG and can contain cycles. This was in response to this issue in 2021 apache#17795. However, this is really bazel issue, and on top of it - it's recognized as such and being fixed very recently in bazel-contrib/rules_python#1188 because there are other packages that have similar problems (pytorch and triton being popular couple). Also Bazel is not that popular in the Python world. Therefore, rather than trying to workaround the problem of bazel, we encourage them to merge and release the fix bazel-contrib/rules_python#1166 (comment) and call it out in our installation instructions, that bazel installation might lead to problems like that. If bazel does not fix it, this will only be a problem for Future installations of airflow in a few months most likely. It will not impact current bazel users installing old versions of Airflow (actually they might start having problems now if we do not fix it and yank the 5 providers released yesterday)
potiuk
added a commit
that referenced
this pull request
May 23, 2023
In the last wave of providers #31416 we bumped min-airlfow-version to 2.4 and added mechanism to verify min-airflow version is ok while importing, but it turned out that there are cases where installing just old version of airflow (with no constraints) brings the latest version of those providers and causes new installation of airflow to fail. This is far too common to ignore or require to use constraints, unfortunately We do not have min-airlfow-version in the preinstalled providers for one reason only. For some tools that are NOT conforming to standards (such as bazel), having min-airflow-version for those providers causes circular dependency (even though technically dependencies in PyPI can - and often are - circular, because dependencies in Python are not DAG and can contain cycles. This was in response to this issue in 2021 #17795. However, this is really bazel issue, and on top of it - it's recognized as such and being fixed very recently in bazel-contrib/rules_python#1188 because there are other packages that have similar problems (pytorch and triton being popular couple). Also Bazel is not that popular in the Python world. Therefore, rather than trying to workaround the problem of bazel, we encourage them to merge and release the fix bazel-contrib/rules_python#1166 (comment) and call it out in our installation instructions, that bazel installation might lead to problems like that. If bazel does not fix it, this will only be a problem for Future installations of airflow in a few months most likely. It will not impact current bazel users installing old versions of Airflow (actually they might start having problems now if we do not fix it and yank the 5 providers released yesterday)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note:
cncf.kuberneteshas changed from feature release in RC1 to major release in RC2