Repository navigation
Closed
Labels
kind:metaHigh-level information important to the communityHigh-level information important to the community
Description
Description
I think we are really close to release the second wave of Backport Packages. We accumulated a number of changes already and few more are about to be completed so that we are, I think, close to release the second wave.
** What will be in those packages **
Everything that is currently merged to the master branch in "providers" package will be included.
Those are open issues that we know we should merge before though:
-
Enabling- this is deferred to the next backport release as per Kubernetes - related backports needs some fixes for xcom cases #11298cncf.kubernetesprovider after 1.10.11/1.10.12 backported changes and spark-on-kubernetes operator - Implementing Impersonation for all applicable Google operators (@olchas works on it with Hooks already merged)
- DataProc operator improvements (@turbaszek works on it in Improve handling Dataproc cluster creation with ERROR state #9593 )
- Improve idempotency of BigQueryInsertJobOperator (@turbaszek works on it in Improve idempotency of BigQueryInsertJobOperator #9590)
- Release 1.10.12 of Airflow as it fixes some teething problems with Kubernetes refactor from 1.10.11
- Fix BigQuery operators problem with serialization BIgQuery operators from backport packages and DAG Serialization #10171
- AWS SES Add Amazon SES hook #10004
- Fix the DLP operators.
-
finalize and implement the POC/WIP on provider package discovery mechanism #10822 allowing to release 1.10.13 with support to auto-dicover provider capabilities such as connections and extra linksdeferred to another release -
Merge Add Kerberos Auth for Presto #10488 with Kerberos on Presto change.deferred to the next release -
Breaking changes: Email operator moved to Core: we are going to address it in 1.10.13 by adding email provider there and releasing provider package via Core operators/hooks import paths according to AIP-21 #11178
We have to make the following adjustments to the process:
- make sure that we only release the packages that have some changes #back Make sure Only changed backports get released #10375
- make sure to add changes from the previous releases - not only commits but add mechanism to add
backwards-incompatible changes akin to UPDATING.md Add mechanism to maintain UPDATING.md for each provider package separately #10376
Update:
The second wave is delayed a bit (but coming soon). The reason is two-fold
- the cncf.kubernetes package has a problem that we are solving
- we might try to get the mechanism of separate providers packages and "discovery" mechanism before we build the packages - see the POC/WIP on provider package discovery mechanism #10822 and POC/WIP on provider package discovery mechanism (1.10) #10823
Use case / motivation
We want the users of airflow 1.10 to benefit from fixes and improvements in the latest packages.
Reacted by Lucas Garcia, Nathan Hadfield, Kamil Breguła, Jimmy Xie, Shoichi Kagawa and Andrew Packer
Reacted by Lucas Garcia, r-richmond, Kamil Breguła, Shivansh Saini, Jimmy Xie, Shoichi Kagawa, Stijn De Haes, Andrew Packer and Tomek Urbaszek
Metadata
Metadata
Assignees
Labels
kind:metaHigh-level information important to the communityHigh-level information important to the community
Activity
potiuk commentedon Jul 27, 2020
@kaxil @turbaszek @dimberman @mik-laj @olchas (and other @apache/airflow-committers who are interested).
I am going for holidays today, and won't be available for a week or so, but I think this one needs some love while I am away, and maybe discussion if there are any more things we want to include in backport packages.
I think there are few small fixes needed in backport release scripts/process to make sure the "2nd wave" works on top of the "1st wave" smoothly but they're just minor adjustments (filtering out some commits from the reports and filtering out packages that are unaffected + adding the new ones that we will add).
I am happy to take care about it next week when I am back :)
potiuk commentedon Jul 27, 2020
cc: @r-richmond
mik-laj commentedon Aug 5, 2020
We should fix this issue before this packages is released.
#10171
JeffryMAC commentedon Aug 5, 2020
please also include AWS SES #10004
potiuk commentedon Aug 5, 2020
Done :)
guptamyr commentedon Aug 17, 2020
Hi, could you please also make sure to include this new operator for AWS SageMaker: #9594. Thanks!
potiuk commentedon Aug 18, 2020
Yep. It's alrready merged so it will be inclued automagically :). Baically we release everything that is now in master.
mik-laj commentedon Aug 24, 2020
We have breaking change between backport packages. - EmailOperator has been moved to core.
mik-laj commentedon Aug 24, 2020
Before releasing a new package, we should fix the operators for DLP.
bodak commentedon Aug 27, 2020
Are you planning to include GKEStartPodOperator?
https://github.com/apache/airflow/blob/master/backport_packages/refactor_backport_packages.py#L602
Removed in PR: #7908
mik-laj commentedon Aug 27, 2020
Yes. It is planned.
potiuk commentedon Aug 31, 2020
I am planning to cut the 2nd Backport package release candidate this evening. So please try to merge everything from "providers" if you want this to be part of that release.
guptamyr commentedon Sep 9, 2020
Hi,
Has the 2nd wave of backport packages been released or is there a new ETA?
potiuk commentedon Sep 11, 2020
Sorry - I should have updated it. The second wave is delayed a bit (but coming soon). The resason is two-fold
cncf.kubernetespackage has a problem that we are solvingmik-laj commentedon Sep 17, 2020
I would like this change to be included. I am working on doing a rebase.
#10488
mik-laj commentedon Oct 5, 2020
RC1 packages have been released. Feel free to test and report problems.
https://lists.apache.org/thread.html/r4dfc18d2376601667efe7dfaf0bf1c07b840f9bcd9f6311d48cb3c64%40%3Cdev.airflow.apache.org%3E
potiuk commentedon Oct 6, 2020
The Voting passed. We are going to release everythiong except "cncf.kubernetes" and we are going to have a follow-up release of "google" and "cncf.kubernetes" providers after the #11298 and #11299 has been fixed. We are going also to have some follow-up release after we release 1.10.13 with #10822, #10823 and #11178. We are also likely to release Presto operator as well with added Kerberos support then #10488.
potiuk commentedon Oct 6, 2020
Backports released.