Skip to content

Conversation

@turbaszek
Copy link
Member

@turbaszek turbaszek commented Sep 21, 2020

Adds new rule that will be run during airflow upgrade-check
and will look for old clasess and imports incompatible with
Airflow 2.0.

Example:

Changes in import paths of hooks, operators, sensors and others
---------------------------------------------------------------
Many hooks, operators and other classes has been renamed and moved. Those changes were part of unifying names and imports paths as described in AIP-21.
The `contrib` folder has been replaced by providers directory and packages:
https://github.com/apache/airflow#backport-packages

Problems:
  1.  Using `airflow.contrib.sensors.python_sensor.PythonSensor` will be replaced by `airflow.sensors.python.PythonSensor`. Affected file: /files/dags/test_dag_1.py
  2.  Using `airflow.operators.bash_operator.BashOperator` will be replaced by `airflow.operators.bash.BashOperator`. Affected file: /files/dags/xyz.py
  3.  Using `airflow.operators.bash_operator.BashOperator` will be replaced by `airflow.operators.bash.BashOperator`. Affected file: /files/dags/generate_dags.py

Related to #8765


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@turbaszek turbaszek added the area:upgrade Facilitating migration to a newer version of Airflow label Sep 21, 2020
@turbaszek turbaszek force-pushed the add-changed-import-rule branch from 429983a to 24ccf6a Compare September 21, 2020 12:55
@turbaszek turbaszek requested review from mik-laj and potiuk September 21, 2020 12:56
@turbaszek turbaszek force-pushed the add-changed-import-rule branch from 24ccf6a to f46c039 Compare September 21, 2020 12:57
@turbaszek turbaszek requested review from ashb and kaxil September 21, 2020 13:01
@turbaszek
Copy link
Member Author

cc @vikramkoka

@turbaszek

This comment has been minimized.

Adds new rule that will be run during airflow upgrade-check
and will look for old clasess and imports incompatible with
Airflow 2.0.
@turbaszek turbaszek force-pushed the add-changed-import-rule branch from f46c039 to 4d48116 Compare September 22, 2020 12:54
@turbaszek turbaszek marked this pull request as ready for review September 24, 2020 14:02
@turbaszek
Copy link
Member Author

The docs faliure was already fixed.

@turbaszek
Copy link
Member Author

@olchas @mik-laj @TobKed can you please take a look?

Comment on lines +28 to +33
class ImportChange(
NamedTuple(
"ImportChange",
[("old_path", str), ("new_path", str), ("providers_package", Optional[None])],
)
):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can be simplified

Suggested change
class ImportChange(
NamedTuple(
"ImportChange",
[("old_path", str), ("new_path", str), ("providers_package", Optional[None])],
)
):
class ImportChange(NamedTuple)
old_path: str
new_path: str
providers_package: Optional[None]

Copy link
Member

@kaxil kaxil Sep 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh wait, I know why you did this, for compatibility with Py 2.7 , Py 3.5 ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, not a big fan of this but we this command has to work with 2.7

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some operators did the other way.. from providers to core
#10013
this message will be wrong for it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The message will be:

Using airflow.operators.email_operator.EmailOperator will be replaced by airflow.operators.email.EmailOperator.

And I see no place for confusion here

@potiuk potiuk force-pushed the v1-10-test branch 2 times, most recently from bd4214b to 195f00d Compare October 3, 2020 22:42
@turbaszek turbaszek merged commit 4431c92 into apache:v1-10-test Oct 4, 2020
@turbaszek turbaszek deleted the add-changed-import-rule branch October 4, 2020 11:48
RaviTezu pushed a commit to RaviTezu/airflow that referenced this pull request Oct 25, 2020
Adds new rule that will be run during airflow upgrade-check
and will look for old clasess and imports incompatible with
Airflow 2.0.
kaxil pushed a commit that referenced this pull request Nov 12, 2020
Adds new rule that will be run during airflow upgrade-check
and will look for old clasess and imports incompatible with
Airflow 2.0.
@potiuk potiuk added this to the Airflow 1.10.13 milestone Nov 14, 2020
@potiuk potiuk added the type:new-feature Changelog: New Features label Nov 14, 2020
potiuk pushed a commit that referenced this pull request Nov 16, 2020
Adds new rule that will be run during airflow upgrade-check
and will look for old clasess and imports incompatible with
Airflow 2.0.
kaxil pushed a commit that referenced this pull request Nov 18, 2020
Adds new rule that will be run during airflow upgrade-check
and will look for old clasess and imports incompatible with
Airflow 2.0.
cfei18 pushed a commit to cfei18/incubator-airflow that referenced this pull request Mar 5, 2021
Adds new rule that will be run during airflow upgrade-check
and will look for old clasess and imports incompatible with
Airflow 2.0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:upgrade Facilitating migration to a newer version of Airflow type:new-feature Changelog: New Features

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants