[airflow] Add ruff rules to catch deprecated attribute access from context key for Airflow 3.0 (AIR301)#22850
Conversation
|
| *range, | ||
| ); | ||
| diagnostic.set_fix(Fix::safe_edit(Edit::range_replacement( | ||
| "\"triggering_asset_events\"".to_string(), |
There was a problem hiding this comment.
will this also check context['triggering_asset_events']?
There was a problem hiding this comment.
I think we have two functions here and they are called in different places.
1.) check_context_key_usage_in_call on Expr::Call(call_expr @ ExprCall {func, arguments, ..},)
2.) check_context_key_usage_in_subscript on Expr::Subscript(subscript_expr)
The check_context_key_usage_in_subscript will capture context['triggering_dataset_events'] and raise diagnostic to suggest update to use context['triggering_asset_events']
| ) | ||
| } | ||
| "uri" if is_removed_context_key_attribute(checker, attr_expr, "inlet_events", attr) => { | ||
| Replacement::AttrName("asset.uri") |
There was a problem hiding this comment.
I'm sorry... my description was totally wrong...
Here's what we want to check
# airflow 2.11
context["inlet_events"]["this://is-url"] # this is not ok in airflow 3+
context["inlet_events"][Dataset("this://is-url")] # this is still okWe do not support accessing inlet events through a string (used to be the URI of Dataset) now. We can either change it to Asset(uri="this://is-url") or just warn the user to use the Asset object instead of a string.
There was a problem hiding this comment.
ah I see. In this case, instead of checking the uri attribute on context["inlet_events"], we actually should check if context["inlet_events"] is being accessed again through subscript, and the key is of type str. If we detect this, we should suggest to change the str (i.e., the url) to Asset(uri="this://is-url")
also, I wanted to double-check that the implementation for external_trigger is still correct.
3750166 to
e37e963
Compare
There was a problem hiding this comment.
Hi @Lee-W , I've made the following updates to this file. Could you please have a look and let me know if the rules are implemented properly?
There was a problem hiding this comment.
The new rule added here are:
triggering_dataset_events->triggering_asset_events, useRenameto suggest a fix.context["dag_run"].external_triggeris deprecated, useMessageto warn.context["inlet_events"]["this://is-url"]is deprecated. useMessageto warn, provide a example to useAsset("this://is-url").
- and 3. above support detection for
.get()as well, or even variable bindinginlet_events = context["inlet_events"]; inlet_events["this://is-url"]
How the rules for the context keys are triggered.
check_context_key_usage_in_callis used to capture casecontext.get("key"), triggered onExpr::Callcheck_context_key_usage_in_subscriptis used to capture casecontext["key"], triggered onExpr::Subscript
In the current implementation, we need to implement the same rule in two places mentioned above. Therefore, I've done a refactor to extract the rules and the report diagnostic into the following functions.
check_deprecated_context_key(check for deprecated context key, e.g.,context["triggering_dataset_events"])check_deprecated_context_key_value_access(check for deprecated access on context key's value, e.g.,context["inlet_events"]["this://is-url"])check_removed_attribute_access_on_context_key(check for deprecated access on attribute on context key's value, e.g.,context["dag_run"].external_trigger)
In this way, we can keep our rule and diagnostic reporting in a single place, but calling these functions in the two rule trigger functions above.
I also extracted the common check logic into multiple helper functions.
is_context_variable: checks if an expression is a valid Airflow context variable. If should be either an kwarg parameter**context, or a variable fromget_current_context().is_context_key_access: checks for key access through subscriptioncontext["key"]or a callcontext.get().is_value_from_context_key: checks if an expression is or resolves to an access of a specific context key. e.g.,inlet_events = context["inlet_events"]; inlet_events.is_removed_context_key_attribute: checks if an attribute access on a specific context key; e.g.,dagrun = context["dag_run"]; dagrun.external_trigger.
e37e963 to
e12a417
Compare
|
No worries about the pings, happy to take a look! |
Lee-W
left a comment
There was a problem hiding this comment.
the error message was wrong, but everything else looks great!
| Airflow3Removal { | ||
| deprecated: "inlet_events[\"<uri>\"]".to_string(), | ||
| replacement: Replacement::Message( | ||
| "Accessing `inlet_events` via a string key is deprecated; use `inlet_events[Asset(\"<uri>\")]` instead (e.g., context[\"inlet_events\"][Asset(\"this://is-url\")]).", |
There was a problem hiding this comment.
| "Accessing `inlet_events` via a string key is deprecated; use `inlet_events[Asset(\"<uri>\")]` instead (e.g., context[\"inlet_events\"][Asset(\"this://is-url\")]).", | |
| "Accessing `inlet_events` via a string key is deprecated; use `inlet_events[Asset(uri=\"<uri>\")]` instead (e.g., context[\"inlet_events\"][\"this://is-url\"]).", |
e12a417 to
9c997ea
Compare
| | ^^^^^^^^^^^^^^^ | ||
| 198 | print(inlet_events.get("this://is-url")) | ||
| | | ||
| help: Accessing `inlet_events` via a string key is deprecated; use `context["inlet_events"][Asset(uri="this://is-url")]` instead of `context["inlet_events"]["this://is-url"]`. |
There was a problem hiding this comment.
Hi @Lee-W , thank you for caching this!! I just made an update to the message to make it bit more accurate and consistent. Can you have a look again? Thanks.
ntBre
left a comment
There was a problem hiding this comment.
Looks good to me, thank you!
* main: (45 commits) [ty] Fix wrong inlay hints for overloaded function arguments (astral-sh#23179) [ty] Respect `@no_type_check` when combined with other decorators (astral-sh#23177) [ty] Use type context when inferring constructor argument types (astral-sh#23139) [`airflow`] Add ruff rules to catch deprecated attribute access from context key for Airflow 3.0 (`AIR301`) (astral-sh#22850) Support formatting `pycon` markdown code blocks (astral-sh#23112) Markdown formatting in LSP (astral-sh#23063) Instruct Claude to use comments more sparingly (astral-sh#23181) [`flake8-gettext`] Fix false negatives for plural argument of ngettext (`INT001`, `INT002`, `INT003`) (astral-sh#21078) [ty] Invoking goto-def on parentheses of a class constructor call takes you too constructor method [ty] Make goto definition on class constructor always go to class definition [ty] Assign lower completions ranking to deprecated functions and classes (astral-sh#23089) [ty] Fix parameter references across files via keyword args (astral-sh#23012) [ty] Exclude enclosing class for base completions (astral-sh#23141) [`pyupgrade`] Fix syntax error on string with newline escape and comment (`UP037`) (astral-sh#22968) [ty] Improve documentation for `expect_single_definition` method (astral-sh#23175) [ty] Configure check mode for all projects Add `home-assistant` to ecosystem projects (astral-sh#23132) Add tabbed shell completion documentation (astral-sh#23169) Bump typing conformance-suite pin (astral-sh#23174) [ty] Fix invalid diagnostic location for a sub-call to a specialized ParamSpec (astral-sh#23036) ...
Summary
Context:
apache/airflow#41641
external_triggercheck with DagRunType apache/airflow#45961create_dagrun removed from airflow...DAG(This has already been implemented in AIR301)The existing AIR301 rules can detect when users access a removed key such as
execution_datethrough Airflow'scontext. For example,context["execution_date"], orcontext["dag_run"]. However, if an attribute is deprecated from a context key, such ascontext["dag_run"].external_trigger, the current implement will not flag it.This PR adds the logic for such check, and add two rules to flag the deprecated attribute for the
"dag_run"and"inlet_events"context key. In addition to this,"triggering_dataset_events"is a deprecated context key which can be handled by the existing rule. However, the existing rule doesn't raise a diagnostic. Hence, the rule logic is refactored a little bit, such that we can add this check and suggest aReplacement::Rename.Test Plan
The test cases have been added to
AIR301_context.py, and all the tests have been run locally and success. @Lee-W , could you please review it when you have time, thanks!Notes
In #22376, we introduced some improvements to the AIR301 code. I will re-base this PR when we are all good on that, so it can pick up those code structure improvements, and updated rules.