Skip to content

Prepare RC2 docs for May 2023 wave of Providers#31416

Merged
eladkal merged 1 commit into
apache:mainfrom
eladkal:providers
May 19, 2023
Merged

Prepare RC2 docs for May 2023 wave of Providers#31416
eladkal merged 1 commit into
apache:mainfrom
eladkal:providers

Conversation

@eladkal

@eladkal eladkal commented May 19, 2023

Copy link
Copy Markdown
Contributor

Note: cncf.kubernetes has changed from feature release in RC1 to major release in RC2

@potiuk

potiuk commented May 19, 2023

Copy link
Copy Markdown
Member

Note: cncf.kubernetes has changed from feature release in RC1 to major release in RC2

Makes perfect sense.

@eladkal
eladkal merged commit 45548b9 into apache:main May 19, 2023
@eladkal
eladkal deleted the providers branch May 19, 2023 17:36
potiuk added a commit to potiuk/airflow that referenced this pull request May 23, 2023
In the last wave of providers apache#31416 we bumped min-airlfow-version to 2.4 and
added mechanism to verify min-airflow version is ok while importing, but
it turned out that there are cases where installing just old version of
airflow (with no constraints) brings the latest version of those
providers and causes new installation of airflow to fail. This is far
too common to ignore or require to use constraints, unfortunately  We do
not have min-airlfow-version in the preinstalled providers for one
reason only. For some tools that are NOT conforming to standards (such
as bazel), having min-airflow-version for those providers causes
circular dependency (even though technically dependencies in PyPI can -
and often are - circular, because dependencies in Python are not DAG and
can contain cycles.

This was in response to this issue in 2021 apache#17795.

However, this is really  bazel issue, and on top of it - it's recognized
as such and being fixed very recently
in bazel-contrib/rules_python#1188 because there
are other packages that have similar problems (pytorch and triton being
popular couple). Also Bazel is not that popular in the Python world. 

Therefore, rather than trying to workaround the problem of bazel, we
encourage them to merge and release the fix
bazel-contrib/rules_python#1166 (comment)
and call it out in our installation instructions, that bazel
installation might lead to problems like that.

If bazel does not fix it, this will only be a problem for Future
installations of airflow in a few months most likely. It will not impact
current bazel users installing old versions of Airflow (actually they
might start having problems now if we do not fix it and yank the 5
providers released yesterday)
potiuk added a commit that referenced this pull request May 23, 2023
In the last wave of providers #31416 we bumped min-airlfow-version to 2.4 and
added mechanism to verify min-airflow version is ok while importing, but
it turned out that there are cases where installing just old version of
airflow (with no constraints) brings the latest version of those
providers and causes new installation of airflow to fail. This is far
too common to ignore or require to use constraints, unfortunately  We do
not have min-airlfow-version in the preinstalled providers for one
reason only. For some tools that are NOT conforming to standards (such
as bazel), having min-airflow-version for those providers causes
circular dependency (even though technically dependencies in PyPI can -
and often are - circular, because dependencies in Python are not DAG and
can contain cycles.

This was in response to this issue in 2021 #17795.

However, this is really  bazel issue, and on top of it - it's recognized
as such and being fixed very recently
in bazel-contrib/rules_python#1188 because there
are other packages that have similar problems (pytorch and triton being
popular couple). Also Bazel is not that popular in the Python world. 

Therefore, rather than trying to workaround the problem of bazel, we
encourage them to merge and release the fix
bazel-contrib/rules_python#1166 (comment)
and call it out in our installation instructions, that bazel
installation might lead to problems like that.

If bazel does not fix it, this will only be a problem for Future
installations of airflow in a few months most likely. It will not impact
current bazel users installing old versions of Airflow (actually they
might start having problems now if we do not fix it and yank the 5
providers released yesterday)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants