task instance listener: call on_task_instance_running after rendering templates#32716
Merged
Conversation
mobuchowski
force-pushed
the
listener-move-onrunning-callback
branch
from
July 20, 2023 12:26
983dcc9 to
3c7ba73
Compare
uranusjr
previously requested changes
Jul 20, 2023
Signed-off-by: Maciej Obuchowski <[email protected]>
mobuchowski
force-pushed
the
listener-move-onrunning-callback
branch
from
July 21, 2023 10:46
3c7ba73 to
53d31fd
Compare
uranusjr
reviewed
Jul 24, 2023
Comment on lines
+1649
to
+1650
| # Is never MappedOperator at this point | ||
| self.task.pre_execute(context=context) # type: ignore[union-attr] |
Member
There was a problem hiding this comment.
Where is the task unmapped before this line is reached? How can we be sure this is the case?
Contributor
Author
Member
There was a problem hiding this comment.
The fact I wrote the code is not a particularly promising guarantee though 😛
But code here is pretty awful. I’ll add a commit to better fix this up later…
uranusjr
approved these changes
Jul 25, 2023
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously,
on_task_instance_runningwas being called in_run_raw_task. After this, it will be running close to wherepre_executecallback is called.This had disadvantage of being run before operator templates are rendered.
In some usages, this required users of this interface (like OpenLineage) to resolve templates themselves, which comes with obvious performance drawback. Now, the state of actual executed task is the same as on
execute.Another reason of moving call site is that idea of task instance listener API is to provide notification as close to when actual execution happens. This is as close as we can get AFAIK.