Skip to content

Perform "mini scheduling run" after task has finished #11498

@ashb

Description

@ashb

In order to further reduce intra-dag task scheduling lag we can add a further optimization: when a task has just finshed executing (success or failure) we can look at the downstream tasks of the task that just finished, and then make scheduling decisions for those tasks there -- we've already got the dag loaded, and we know they are likely actionable as we just finished. We should set tasks to scheduled if we can (but no further, i.e. not to queued).

Upstream_failed may also not be safe as that would run callbacks. Think about this.

We'll need a config option to disable this, as it could lead to task starvation of other dags

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions