Skip to content

Dispatch the highest priority tasks first in the executor - #70942

Merged
pierrejeambrun merged 1 commit into
apache:mainfrom
ignacioparicio:fix-executor-priority-order-inverted
Aug 17, 2026
Merged

Dispatch the highest priority tasks first in the executor#70942
pierrejeambrun merged 1 commit into
apache:mainfrom
ignacioparicio:fix-executor-priority-order-inverted

Conversation

@ignacioparicio

@ignacioparicio ignacioparicio commented Aug 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

BaseExecutor sorts its queued tasks by priority_weight ascending, then dispatches from the front of that list. So when more tasks are queued than there are open slots, the lowest priority tasks are the ones sent to workers, and the highest priority ones are held back.

For example, with weights 1, 5 and 10 and one slot freeing up per loop, they run in the order 1, 5, 10; the most important task runs last.

It also does not even out over time, because the queue is re-sorted on every call.

This only affects an executor that has more queued tasks than open slots, so deployments that stay under their parallelism ceiling, or that never set priority_weight, see no difference either way.

Where it happens

In airflow-core/src/airflow/executors/base_executor.py:

# order_queued_tasks_by_priority(): lowest weight first
return sorted(self.queued_tasks.items(), key=lambda x: x[1].ti.priority_weight, reverse=False)

# _get_workloads_to_schedule(): reads from the front, stops at open_slots
for task_key, task_workload in self.order_queued_tasks_by_priority():
    if len(workloads_to_schedule) >= open_slots:
        break
    workloads_to_schedule.append((task_key, task_workload))
How it got here

The sort and the way it is consumed used to match. They drifted apart in two steps:

  • Improve performance of task dequeuing #61376 (fd4fd67c0f) switched the sort from descending to ascending and, in the same commit, changed sorted_queue.pop(0) to sorted_queue.pop(). An ascending list popped from the back still gives the highest priority first, so this was correct, and O(1) instead of O(n).
  • Executor Synchronous callback workload #61153 (9fa13c82ff) refactored trigger_tasks to support callback workloads and replaced that pop() with the front-to-back loop above. The sort was left ascending.

What this PR does:

  • base_executor.py: sorts descending again, and notes on the method that consumers read from the front.
  • test_base_executor.py: adds a test with three weights and fewer open slots than tasks. The existing tests all run with more slots than tasks, so none of them reach the truncating branch.
  • Adds a bugfix newsfragment.
Was generative AI tooling used to co-author this PR?
  • Yes, Claude Code (Opus 5)

Used to trace the ordering back through the history of the two commits above, and to check that nothing else in the repo overrides the sort or depends on it being ascending.

@boring-cyborg boring-cyborg Bot added the area:Executors-core LocalExecutor & SequentialExecutor label Aug 2, 2026
@ignacioparicio
ignacioparicio marked this pull request as ready for review August 2, 2026 13:17
SameerMesiah97
SameerMesiah97 reviewed Aug 2, 2026
View reviewed changes

@SameerMesiah97 SameerMesiah97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense. The documentation has long stated that larger priority_weight values have higher priority, so this appears to correct an implementation bug. Below is the relevant section from the documentation for others who will review this PR:

Priority Weights
================

``priority_weight`` defines priorities in the executor queue. The default ``priority_weight`` is ``1``, and can be
bumped to any integer; larger numbers mean higher priority.
Moreover, each task has a true ``priority_weight`` that is calculated based on its
``weight_rule`` which defines the weighting method used for the effective total priority weight of the task.

Below are the weighting methods. By default, Airflow's weighting method is ``downstream``.

CI still needs to be triggered though.

o-nikolas
o-nikolas approved these changes Aug 11, 2026
View reviewed changes

@o-nikolas o-nikolas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a legitimate regression, nice spot. I'm not sure this needs a news fragment but it is a user facing bug so I don't mind if we keep it.

pierrejeambrun
pierrejeambrun reviewed Aug 14, 2026
View reviewed changes

@pierrejeambrun pierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR.

@ignacioparicio
ignacioparicio force-pushed the fix-executor-priority-order-inverted branch from fd454e3 to 5bfd6be Compare August 14, 2026 18:22
pierrejeambrun
pierrejeambrun approved these changes Aug 17, 2026
View reviewed changes

@pierrejeambrun pierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks.

@pierrejeambrun pierrejeambrun added this to the Airflow 3.3.2 milestone Aug 17, 2026
@pierrejeambrun
pierrejeambrun merged commit 316dc6d into apache:main Aug 17, 2026
78 checks passed
@boring-cyborg

boring-cyborg Bot commented Aug 17, 2026

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

@github-actions

github-actions Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-3-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
✅ v3-3-test PR Link

potiuk pushed a commit that referenced this pull request Aug 17, 2026
@ignacioparicio @potiuk
[v3-3-test] Dispatch the highest priority tasks first in the executor (…
227df15 
…#70942)

(cherry picked from commit 316dc6d)

Co-authored-by: Ignacio Paricio <[email protected]>
potiuk pushed a commit that referenced this pull request Aug 17, 2026
@github-actions @ignacioparicio
[v3-3-test] Dispatch the highest priority tasks first in the executor (…
a3a1c9a 
…#70942) (#71715)

(cherry picked from commit 316dc6d)

Co-authored-by: Ignacio Paricio <[email protected]>
anishgirianish added a commit to anishgirianish/airflow that referenced this pull request Aug 20, 2026
@anishgirianish
Dispatch highest priority tasks first in unified executor queues
5dbe243 
The rebase onto main brought in apache#70942, which flipped queued-task
dispatch to highest priority_weight first, but the unified sort_key
still sorted ascending and would have silently reverted that fix.
anishgirianish added a commit to anishgirianish/airflow that referenced this pull request Aug 21, 2026
@anishgirianish
Dispatch highest priority tasks first in unified executor queues
379ac76 
The rebase onto main brought in apache#70942, which flipped queued-task
dispatch to highest priority_weight first, but the unified sort_key
still sorted ascending and would have silently reverted that fix.
anishgirianish added a commit to anishgirianish/airflow that referenced this pull request Sep 3, 2026
@anishgirianish
Dispatch highest priority tasks first in unified executor queues
bf524d6 
The rebase onto main brought in apache#70942, which flipped queued-task
dispatch to highest priority_weight first, but the unified sort_key
still sorted ascending and would have silently reverted that fix.
anishgirianish added a commit to anishgirianish/airflow that referenced this pull request Sep 5, 2026
@anishgirianish
Dispatch highest priority tasks first in unified executor queues
8f78f2f 
The rebase onto main brought in apache#70942, which flipped queued-task
dispatch to highest priority_weight first, but the unified sort_key
still sorted ascending and would have silently reverted that fix.
anishgirianish added a commit to anishgirianish/airflow that referenced this pull request Sep 5, 2026
@anishgirianish
Dispatch highest priority tasks first in unified executor queues
098326a 
The rebase onto main brought in apache#70942, which flipped queued-task
dispatch to highest priority_weight first, but the unified sort_key
still sorted ascending and would have silently reverted that fix.
vatsrahul1001 pushed a commit that referenced this pull request Sep 7, 2026
@github-actions @ignacioparicio @vatsrahul1001
[v3-3-test] Dispatch the highest priority tasks first in the executor (…
dc40ee1 
…#70942) (#71715)

(cherry picked from commit 316dc6d)

Co-authored-by: Ignacio Paricio <[email protected]>
anishgirianish added a commit to anishgirianish/airflow that referenced this pull request Sep 10, 2026
@anishgirianish
Dispatch highest priority tasks first in unified executor queues
74f7f7d 
The rebase onto main brought in apache#70942, which flipped queued-task
dispatch to highest priority_weight first, but the unified sort_key
still sorted ascending and would have silently reverted that fix.
anishgirianish added a commit to anishgirianish/airflow that referenced this pull request Sep 10, 2026
@anishgirianish
Dispatch highest priority tasks first in unified executor queues
491557b 
The rebase onto main brought in apache#70942, which flipped queued-task
dispatch to highest priority_weight first, but the unified sort_key
still sorted ascending and would have silently reverted that fix.
imrichardwu pushed a commit to imrichardwu/airflow that referenced this pull request Sep 11, 2026
@ignacioparicio @imrichardwu
Dispatch the highest priority tasks first in the executor (apache#70942)
7cd4c26
anishgirianish added a commit to anishgirianish/airflow that referenced this pull request Sep 13, 2026
@anishgirianish
Dispatch highest priority tasks first in unified executor queues
e9d35ec 
The rebase onto main brought in apache#70942, which flipped queued-task
dispatch to highest priority_weight first, but the unified sort_key
still sorted ascending and would have silently reverted that fix.
anishgirianish added a commit to anishgirianish/airflow that referenced this pull request Sep 13, 2026
@anishgirianish
Dispatch highest priority tasks first in unified executor queues
f536a3e 
The rebase onto main brought in apache#70942, which flipped queued-task
dispatch to highest priority_weight first, but the unified sort_key
still sorted ascending and would have silently reverted that fix.
anishgirianish added a commit to anishgirianish/airflow that referenced this pull request Sep 14, 2026
@anishgirianish
Dispatch highest priority tasks first in unified executor queues
15065e8 
The rebase onto main brought in apache#70942, which flipped queued-task
dispatch to highest priority_weight first, but the unified sort_key
still sorted ascending and would have silently reverted that fix.
anishgirianish added a commit to anishgirianish/airflow that referenced this pull request Sep 21, 2026
@anishgirianish
Dispatch highest priority tasks first in unified executor queues
5cda926 
The rebase onto main brought in apache#70942, which flipped queued-task
dispatch to highest priority_weight first, but the unified sort_key
still sorted ascending and would have silently reverted that fix.
anishgirianish added a commit to anishgirianish/airflow that referenced this pull request Sep 21, 2026
@anishgirianish
Dispatch highest priority tasks first in unified executor queues
24b985f 
The rebase onto main brought in apache#70942, which flipped queued-task
dispatch to highest priority_weight first, but the unified sort_key
still sorted ascending and would have silently reverted that fix.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pierrejeambrun pierrejeambrun pierrejeambrun approved these changes
@o-nikolas o-nikolas o-nikolas approved these changes
@XD-DENG XD-DENG Awaiting requested review from XD-DENG XD-DENG is a code owner
@ashb ashb Awaiting requested review from ashb ashb is a code owner
@hussein-awala hussein-awala Awaiting requested review from hussein-awala hussein-awala is a code owner
@dheerajturaga dheerajturaga Awaiting requested review from dheerajturaga dheerajturaga is a code owner
+1 more reviewer
@SameerMesiah97 SameerMesiah97 SameerMesiah97 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area:Executors-core LocalExecutor & SequentialExecutor

Projects

None yet

Milestone

Airflow 3.3.2

Development

Successfully merging this pull request may close these issues.

4 participants

@ignacioparicio @pierrejeambrun @o-nikolas @SameerMesiah97