Skip to content

Conversation

@potiuk
Copy link
Member

@potiuk potiuk commented Oct 22, 2022

As of October 11 our providers are supposed to be compatible with Airflow 2.3+ and all code for backwards compatibility with Airflow 2.2 can be removed now.


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@eladkal
Copy link
Contributor

eladkal commented Oct 22, 2022

@potiuk
Copy link
Member Author

potiuk commented Oct 22, 2022

@potiuk potiuk force-pushed the move-providers-to-min-airflow-2-2 branch from a6bbef7 to ee677fe Compare October 22, 2022 05:26
@potiuk
Copy link
Member Author

potiuk commented Oct 22, 2022

I'd love @uranusjr if you take a look as well. The biggest part of pre-2.3 was removal of dttm from "XCom.get_value" - mostly in operator links.

Also I am not sure if what I've done is correct. I assumed that after migration to 2.3 each task instance will have a "task_instance_key" and we will be able to use task instance key and get_value even for task instances created in previous Airflow versions (I believe migration takes care about it).

I would love your comments here @uranusjr if I understood correctly that we should genereally skip all the dttm usage and rely on task_instance_key being always present.

I have found however one case where it is not as straightforward and dttm is still used and I am not sure if we want to do anything about it:

class ExternalDagLink(BaseOperatorLink):
    """
    Operator link for ExternalTaskSensor and ExternalTaskMarker.
    It allows users to access DAG waited with ExternalTaskSensor or cleared by ExternalTaskMarker.
    """

    name = 'External DAG'

    def get_link(self, operator, dttm):
        ti = TaskInstance(task=operator, execution_date=dttm)
        operator.render_template_fields(ti.get_template_context())
        query = {"dag_id": operator.external_dag_id, "execution_date": dttm.isoformat()}
        return build_airflow_url_with_query(query)

In all the other cases I converted 'get_link" to

    def get_link(
        self,
        operator: BaseOperator,
        *,
        ti_key: TaskInstanceKey,

@potiuk potiuk force-pushed the move-providers-to-min-airflow-2-2 branch 2 times, most recently from f338a0e to 7c215d9 Compare October 22, 2022 11:00
@eladkal
Copy link
Contributor

eladkal commented Oct 22, 2022

We also have:

# TODO: Merge this implementation back to DbApiHook when dropping
# support for Airflow 2.2.
def test_connection(self):
"""Tests the connection by executing a select 1 from dual query"""
status, message = False, ''
try:
if self.get_first("select 1 from dual"):
status = True
message = 'Connection successfully tested'
except Exception as e:
status = False
message = str(e)
return status, message

@potiuk potiuk force-pushed the move-providers-to-min-airflow-2-2 branch 2 times, most recently from f85ba70 to c803412 Compare October 22, 2022 13:01
@potiuk
Copy link
Member Author

potiuk commented Oct 22, 2022

We also have:

# TODO: Merge this implementation back to DbApiHook when dropping
# support for Airflow 2.2.
def test_connection(self):
"""Tests the connection by executing a select 1 from dual query"""
status, message = False, ''
try:
if self.get_first("select 1 from dual"):
status = True
message = 'Connection successfully tested'
except Exception as e:
status = False
message = str(e)
return status, message

Ah cool. Indeed.

@potiuk
Copy link
Member Author

potiuk commented Oct 22, 2022

Actually, the DBApi one we could do before already after we split-out common.sql.

@potiuk potiuk force-pushed the move-providers-to-min-airflow-2-2 branch from c803412 to 04af807 Compare October 22, 2022 13:18
@potiuk
Copy link
Member Author

potiuk commented Oct 22, 2022

Fixed.

@potiuk
Copy link
Member Author

potiuk commented Oct 22, 2022

(also done the same for Trino, Snowflake and Databricks)

@potiuk potiuk force-pushed the move-providers-to-min-airflow-2-2 branch from 04af807 to 7c8c056 Compare October 22, 2022 19:48
@eladkal
Copy link
Contributor

eladkal commented Oct 22, 2022

Ahh we can also handle:

'Deferrable Operators only work starting Airflow 2.2',

'Deferrable Operators only work starting Airflow 2.2',

@potiuk potiuk force-pushed the move-providers-to-min-airflow-2-2 branch from 7c8c056 to 85cad69 Compare October 22, 2022 21:48
@potiuk potiuk mentioned this pull request Oct 23, 2022
@potiuk potiuk force-pushed the move-providers-to-min-airflow-2-2 branch 2 times, most recently from 9be69d7 to 32b060f Compare October 23, 2022 02:45
@potiuk potiuk force-pushed the move-providers-to-min-airflow-2-2 branch 3 times, most recently from 12c23d8 to f57947a Compare October 23, 2022 18:38
@potiuk
Copy link
Member Author

potiuk commented Oct 23, 2022

This one also should cut the "compatibility" test by more than hour. Currently just installing latest providers on airflow 2.2 takes > 1 hour because pip tries to resolve dependencies and it tries very hard to do so (this is because of the complex dependency resolution caused by "old" airflow and "new" providers). This is -yet another reason why the 12 months compatibility for providers is a good idea.

@potiuk potiuk force-pushed the move-providers-to-min-airflow-2-2 branch 2 times, most recently from dd5d587 to 527c5d1 Compare October 23, 2022 20:26
@potiuk
Copy link
Member Author

potiuk commented Oct 23, 2022

All right - seems I get it alll green - @uranusjr - is it possible you take a look at the XCom.get_value() changes before I merge this one?

@uranusjr
Copy link
Member

Just did, I think you did it correctly.

@potiuk potiuk force-pushed the move-providers-to-min-airflow-2-2 branch from 527c5d1 to 8ac8e56 Compare October 24, 2022 11:12
@potiuk
Copy link
Member Author

potiuk commented Oct 24, 2022

Deferrable Operators only work starting Airflow 2.2

Indeed. Thanks @eladkal for all the extra checks :)

@potiuk potiuk force-pushed the move-providers-to-min-airflow-2-2 branch from 8ac8e56 to e8a003b Compare October 24, 2022 11:16
As of October 11 our providers are supposed to be compatible with
Airflow 2.3+ and all code for backwards compatibility with Airflow 2.2
can be removed now.
@potiuk potiuk force-pushed the move-providers-to-min-airflow-2-2 branch from e8a003b to d9aa251 Compare October 24, 2022 14:04
@potiuk potiuk merged commit 78b8ea2 into apache:main Oct 24, 2022
@potiuk potiuk deleted the move-providers-to-min-airflow-2-2 branch October 24, 2022 19:06
@ephraimbuddy ephraimbuddy added the type:misc/internal Changelog: Misc changes that should appear in change log label Nov 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-tools area:providers provider:amazon AWS/Amazon - related issues type:misc/internal Changelog: Misc changes that should appear in change log

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants