Skip to content

Commit 8961bab

Browse files
authored
Prevent templated field logic checks in __init__ of operators automatically (#33786)
* Implement validation of missing template fields * Update output static checks
1 parent 74baebe commit 8961bab

7 files changed

Lines changed: 386 additions & 36 deletions

File tree

‎.pre-commit-config.yaml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -293,6 +293,31 @@ repos:
293293
# changes quickly - especially when we want the early modifications from the first local group
294294
# to be applied before the non-local pre-commits are run
295295
hooks:
296+
- id: validate-operators-init
297+
name: Prevent templated field logic checks in operators' __init__
298+
language: python
299+
entry: ./scripts/ci/pre_commit/pre_commit_validate_operators_init.py
300+
pass_filenames: true
301+
files: ^airflow/providers/.*/(operators|transfers|sensors)/.*\.py$
302+
additional_dependencies: [ 'rich>=12.4.4' ]
303+
# TODO: Handle the provider-specific exclusions and remove them from the list, see:
304+
# https://github.com/apache/airflow/issues/36484
305+
exclude: |
306+
(?x)^(
307+
^.*__init__\.py$|
308+
^airflow\/providers\/google\/cloud\/operators\/bigquery\.py$|
309+
^airflow\/providers\/amazon\/aws\/transfers\/gcs_to_s3\.py$|
310+
^airflow\/providers\/databricks\/operators\/databricks\.py$|
311+
^airflow\/providers\/google\/cloud\/operators\/cloud_storage_transfer_service\.py$|
312+
^airflow\/providers\/google\/cloud\/transfers\/bigquery_to_mysql\.py$|
313+
^airflow\/providers\/google\/cloud\/operators\/vertex_ai\/auto_ml\.py$|
314+
^airflow\/providers\/amazon\/aws\/transfers\/redshift_to_s3\.py$|
315+
^airflow\/providers\/google\/cloud\/operators\/compute\.py$|
316+
^airflow\/providers\/google\/cloud\/operators\/vertex_ai\/custom_job\.py$|
317+
^airflow\/providers\/cncf\/kubernetes\/operators\/pod\.py$|
318+
^airflow\/providers\/amazon\/aws\/operators\/emr\.py$|
319+
^airflow\/providers\/amazon\/aws\/operators\/eks\.py$
320+
)$
296321
- id: ruff
297322
name: Run 'ruff' for extremely fast Python linting
298323
description: "Run 'ruff' for extremely fast Python linting"

‎STATIC_CODE_CHECKS.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -414,6 +414,8 @@ require Breeze Docker image to be built locally.
414414
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
415415
| update-version | Update version to the latest version in the documentation | |
416416
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
417+
| validate-operators-init | Prevent templated field logic checks in operators' __init__ | |
418+
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
417419
| yamllint | Check YAML files with yamllint | |
418420
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
419421

‎dev/breeze/src/airflow_breeze/pre_commit_ids.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,5 +133,6 @@
133133
"update-supported-versions",
134134
"update-vendored-in-k8s-json-schema",
135135
"update-version",
136+
"validate-operators-init",
136137
"yamllint",
137138
]

‎docs/apache-airflow/howto/custom-operator.rst‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -269,6 +269,88 @@ Currently available lexers:
269269

270270
If you use a non-existing lexer then the value of the template field will be rendered as a pretty-printed object.
271271

272+
Limitations
273+
^^^^^^^^^^^
274+
To prevent misuse, the following limitations must be observed when defining and assigning templated fields in the
275+
operator's constructor (when such exists, otherwise - see below):
276+
277+
1. Templated fields' corresponding parameters passed into the constructor must be named exactly
278+
as the fields. The following example is invalid, as the parameter passed into the constructor is not the same as the
279+
templated field:
280+
281+
.. code-block:: python
282+
283+
class HelloOperator(BaseOperator):
284+
template_fields = "field_a"
285+
286+
def __init__(field_a_id) -> None: # <- should be def __init__(field_a)-> None
287+
self.field_a = field_a_id # <- should be self.field_a = field_a
288+
289+
2. Templated fields' instance members must be assigned with their corresponding parameter from the constructor,
290+
either by a direct assignment or by calling the parent's constructor (in which these fields are
291+
defined as ``template_fields``) with explicit an assignment of the parameter.
292+
The following example is invalid, as the instance member ``self.field_a`` is not assigned at all, despite being a
293+
templated field:
294+
295+
.. code-block:: python
296+
297+
class HelloOperator(BaseOperator):
298+
template_fields = ("field_a", "field_b")
299+
300+
def __init__(field_a, field_b) -> None:
301+
self.field_b = field_b
302+
303+
304+
The following example is also invalid, as the instance member ``self.field_a`` of ``MyHelloOperator`` is initialized
305+
implicitly as part of the ``kwargs`` passed to its parent constructor:
306+
307+
.. code-block:: python
308+
309+
class HelloOperator(BaseOperator):
310+
template_fields = "field_a"
311+
312+
def __init__(field_a) -> None:
313+
self.field_a = field_a
314+
315+
316+
class MyHelloOperator(HelloOperator):
317+
template_fields = ("field_a", "field_b")
318+
319+
def __init__(field_b, **kwargs) -> None: # <- should be def __init__(field_a, field_b, **kwargs)
320+
super().__init__(**kwargs) # <- should be super().__init__(field_a=field_a, **kwargs)
321+
self.field_b = field_b
322+
323+
3. Applying actions on the parameter during the assignment in the constructor is not allowed.
324+
Any action on the value should be applied in the ``execute()`` method.
325+
Therefore, the following example is invalid:
326+
327+
.. code-block:: python
328+
329+
class HelloOperator(BaseOperator):
330+
template_fields = "field_a"
331+
332+
def __init__(field_a) -> None:
333+
self.field_a = field_a.lower() # <- assignment should be only self.field_a = field_a
334+
335+
When an operator inherits from a base operator and does not have a constructor defined on its own, the limitations above
336+
do not apply. However, the templated fields must be set properly in the parent according to those limitations.
337+
338+
Thus, the following example is valid:
339+
340+
.. code-block:: python
341+
342+
class HelloOperator(BaseOperator):
343+
template_fields = "field_a"
344+
345+
def __init__(field_a) -> None:
346+
self.field_a = field_a
347+
348+
349+
class MyHelloOperator(HelloOperator):
350+
template_fields = "field_a"
351+
352+
The limitations above are enforced by a pre-commit named 'validate-operators-init'.
353+
272354
Add template fields with subclassing
273355
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
274356
A common use case for creating a custom operator is for simply augmenting existing ``template_fields``.

0 commit comments

Comments
 (0)