Skip to content

fix: Fix unhealthy supervisor when a priority-assigned task dies before discovery#19405

Merged
abhishekrb19 merged 4 commits into
apache:masterfrom
abhishekrb19:server_priority_orphan_tasks
May 5, 2026
Merged

fix: Fix unhealthy supervisor when a priority-assigned task dies before discovery#19405
abhishekrb19 merged 4 commits into
apache:masterfrom
abhishekrb19:server_priority_orphan_tasks

Conversation

@abhishekrb19

@abhishekrb19 abhishekrb19 commented May 4, 2026

Copy link
Copy Markdown
Contributor

Description

With #19040, when ioConfig.serverPriorityToReplicas is set, a supervisor can get stuck throwing
DruidException from computeUnassignedServerPriorities, preventing any new
task replicas with destined replicas from being created until the old tasks rollover. The
exception is as follows:

Found unassignedServerPriorities[[]] of size[0] < total replicas[1] for taskGroupId[0].
Task server priorities[[1, 0]] have already been assigned to tasks[[foo]].

The supervisor can remain in this unhealthy state unable to create additional task replicas until tasks eventually rollover. This can happen when a task spuriously fails after it's created but before it's discovered in the runInternal() loop.

Root cause

createTasksForGroup was an additional writer of group.taskIdToServerPriority,
recording the priority at task-submission time — before the task was ever observed
in group.tasks. The group's other invariant (tasks and taskIdToServerPriority
stay in sync) is enforced by discoverTasks (additions) and TaskGroup.removeTask
(removals), both of which operate on group.tasks.

If a task died after submission but before the next supervisor run picked it up from
the overlord, it never entered group.tasks, so removeTask never fired — the
priority entry was orphaned and made the next computeUnassignedServerPriorities
call throw.

Fix

This patch removes the additional writer, leaving discoverTasks and removeTask
as the sole mutators. group.taskIdToServerPriority should now stay in sync with group.tasks.
The added unit tests fail on master without the fix.

This PR has:

  • been self-reviewed.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.

The orphan scenario when tasks are discovered separately:

  1. Run N creates 2 tasks joclpbfe (prio 1) and iikfdmoa (prio 0). taskIdToServerPriority = {joclpbfe→1, iikfdmoa→0}, tasks = {}.
  2. joclpbfe fails quickly (before the next supervisor run).
  3. Run N+1: discoverTasks() iterates over getActiveTaskMap() — which excludes the already-failed joclpbfe. Only iikfdmoa gets inserted. tasks = {iikfdmoa}, taskIdToServerPriority = {joclpbfe→1, iikfdmoa→0}.
  4. joclpbfe is now orphaned: not in tasks, never iterated, so no code path calls removeTask for it. Its priority entry lives forever.
  5. Run N+2: replicas (2) > tasks.size() (1) → createTasksForGroup(0, replicas=1):
    - allRequiredPriorities = [1, 0]
    - assignedPriorities = [1, 0] (includes stale joclpbfe→1)
    - unassignedServerPriorities = [] → throws.

  The log Task server priorities[[1, 0]] have already been assigned to tasks[[iikfdmoa]] is the smoking gun — 2 priorities but 1 task id.

@aho135 aho135 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick fix on this one @abhishekrb19!

@abhishekrb19
abhishekrb19 merged commit d836a8f into apache:master May 5, 2026
63 of 64 checks passed
@abhishekrb19
abhishekrb19 deleted the server_priority_orphan_tasks branch May 5, 2026 03:24
@github-actions github-actions Bot added this to the 38.0.0 milestone May 5, 2026
317brian pushed a commit to 317brian/druid that referenced this pull request May 11, 2026
…re discovery (apache#19405)

With apache#19040, when ioConfig.serverPriorityToReplicas is set, a supervisor can get stuck throwing
DruidException from computeUnassignedServerPriorities, preventing any new
task replicas with destined replicas from being created until the old tasks rollover. The
exception is as follows:

Found unassignedServerPriorities[[]] of size[0] < total replicas[1] for taskGroupId[0].
Task server priorities[[1, 0]] have already been assigned to tasks[[foo]].
The supervisor can remain in this unhealthy state unable to create additional task replicas until tasks eventually rollover. This can happen when a task spuriously fails after it's created but before it's discovered in the runInternal() loop.

This patch removes the additional writer, leaving discoverTasks and removeTask
as the sole mutators. group.taskIdToServerPriority should now stay in sync with group.tasks.
The added unit tests fail on master without the fix.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants