-
Notifications
You must be signed in to change notification settings - Fork 16.3k
Closed
Labels
area:Schedulerincluding HA (high availability) schedulerincluding HA (high availability) schedulerarea:performancekind:featureFeature RequestsFeature Requests
Description
In order to further reduce intra-dag task scheduling lag we can add a further optimization: when a task has just finshed executing (success or failure) we can look at the downstream tasks of the task that just finished, and then make scheduling decisions for those tasks there -- we've already got the dag loaded, and we know they are likely actionable as we just finished. We should set tasks to scheduled if we can (but no further, i.e. not to queued).
Upstream_failed may also not be safe as that would run callbacks. Think about this.
We'll need a config option to disable this, as it could lead to task starvation of other dags
potiuk, kaxil and tooptoop4
Metadata
Metadata
Assignees
Labels
area:Schedulerincluding HA (high availability) schedulerincluding HA (high availability) schedulerarea:performancekind:featureFeature RequestsFeature Requests