Skip to content

Commit acc8121

Browse files
authored
Add coding rule for operators to avoid logic in constructors (#37035)
Related: #36484 #33786
1 parent fee7e33 commit acc8121

1 file changed

Lines changed: 42 additions & 6 deletions

File tree

‎contributing-docs/05_pull_requests.rst‎

Lines changed: 42 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ implementing them.
2525
.. contents:: :local:
2626

2727
Pull Request guidelines
28-
=======================
28+
-----------------------
2929

3030
Before you submit a Pull Request (PR) from your forked repo, check that it meets
3131
these guidelines:
@@ -101,7 +101,7 @@ these guidelines:
101101
This makes the lives of those who come after you (and your future self) a lot easier.
102102

103103
Experimental Requirement to resolve all conversations
104-
=====================================================
104+
-----------------------------------------------------
105105

106106
In December 2023 we enabled - experimentally - the requirement to resolve all the open conversations in a
107107
PR in order to make it merge-able. You will see in the status of the PR that it needs to have all the
@@ -121,14 +121,14 @@ already solved (assuming that maintainers will start treating the conversations
121121
.. _coding_style:
122122

123123
Coding style and best practices
124-
===============================
124+
-------------------------------
125125

126126
Most of our coding style rules are enforced programmatically by ruff and mypy, which are run automatically
127127
with static checks and on every Pull Request (PR), but there are some rules that are not yet automated and
128128
are more Airflow specific or semantic than style.
129129

130130
Don't Use Asserts Outside Tests
131-
-------------------------------
131+
...............................
132132

133133
Our community agreed that to various reasons we do not use ``assert`` in production code of Apache Airflow.
134134
For details check the relevant `mailing list thread <https://lists.apache.org/thread.html/bcf2d23fcd79e21b3aac9f32914e1bf656e05ffbcb8aa282af497a2d%40%3Cdev.airflow.apache.org%3E>`_.
@@ -155,7 +155,7 @@ The one exception to this is if you need to make an assert for type checking (wh
155155
156156
157157
Database Session Handling
158-
-------------------------
158+
.........................
159159

160160
**Explicit is better than implicit.** If a function accepts a ``session`` parameter it should not commit the
161161
transaction itself. Session management is up to the caller.
@@ -202,7 +202,7 @@ compatibility considerations. In most cases, ``session`` argument should be last
202202

203203

204204
Don't use time() for duration calculations
205-
-----------------------------------------
205+
..........................................
206206

207207
If you wish to compute the time difference between two events with in the same process, use
208208
``time.monotonic()``, not ``time.time()`` nor ``timezone.utcnow()``.
@@ -242,6 +242,42 @@ If the start_date of a duration calculation needs to be stored in a database, th
242242
datetime objects. In all other cases, using datetime for duration calculation MUST be avoided as creating and
243243
diffing datetime operations are (comparatively) slow.
244244

245+
Templated fields in Operator's __init__ method
246+
..............................................
247+
248+
Airflow Operators might have some fields added to the list of ``template_fields``. Such fields should be
249+
set in the constructor (``__init__`` method) of the operator and usually their values should
250+
come from the ``__init__`` method arguments. The reason for that is that the templated fields
251+
are evaluated at the time of the operator execution and when you pass arguments to the operator
252+
in the DAG, the fields that are set on the class just before the ``execute`` method is called
253+
are processed through templating engine and the fields values are set to the result of applying the
254+
templating engine to the fields (in case the field is a structure such as dict or list, the templating
255+
engine is applied to all the values of the structure).
256+
257+
That's why we expect two things in case of ``template fields``:
258+
259+
* with a few exceptions, only self.field = field should be happening in the operator's constructor
260+
* validation of the fields should be done in the ``execute`` method, not in the constructor because in
261+
the constructor, the field value might be a templated value, not the final value.
262+
263+
The exceptions are cases where we want to assign empty default value to a mutable field (list or dict)
264+
or when we have a more complex structure which we want to convert into a different format (say dict or list)
265+
but where we want to keep the original strings in the converted structure.
266+
267+
In such cases we can usually do something like this
268+
269+
.. code-block:: python
270+
271+
def __init__(self, *, my_field: list[str] = None, **kwargs):
272+
super().__init__(**kwargs)
273+
my_field = my_field or []
274+
self.my_field = my_field
275+
276+
The reason for doing it is that we are working on a cleaning up our code to have
277+
`pre-commit hook <../scripts/ci/pre_commit/pre_commit_validate_operators_init.py>`__
278+
that will make sure all the cases where logic (such as validation and complex conversion)
279+
is not done in the constructor are detected in PRs.
280+
245281
-----------
246282

247283
If you want to learn what are the options for your development environment, follow to the

0 commit comments

Comments
 (0)