fix: Fix unhealthy supervisor when a priority-assigned task dies before discovery#19405
Merged
abhishekrb19 merged 4 commits intoMay 5, 2026
Merged
Conversation
The orphan scenario when tasks are discovered separately:
1. Run N creates 2 tasks joclpbfe (prio 1) and iikfdmoa (prio 0). taskIdToServerPriority = {joclpbfe→1, iikfdmoa→0}, tasks = {}.
2. joclpbfe fails quickly (before the next supervisor run).
3. Run N+1: discoverTasks() iterates over getActiveTaskMap() — which excludes the already-failed joclpbfe. Only iikfdmoa gets inserted. tasks = {iikfdmoa}, taskIdToServerPriority = {joclpbfe→1, iikfdmoa→0}.
4. joclpbfe is now orphaned: not in tasks, never iterated, so no code path calls removeTask for it. Its priority entry lives forever.
5. Run N+2: replicas (2) > tasks.size() (1) → createTasksForGroup(0, replicas=1):
- allRequiredPriorities = [1, 0]
- assignedPriorities = [1, 0] (includes stale joclpbfe→1)
- unassignedServerPriorities = [] → throws.
The log Task server priorities[[1, 0]] have already been assigned to tasks[[iikfdmoa]] is the smoking gun — 2 priorities but 1 task id.
aho135
approved these changes
May 5, 2026
aho135
left a comment
Contributor
There was a problem hiding this comment.
Thanks for the quick fix on this one @abhishekrb19!
317brian
pushed a commit
to 317brian/druid
that referenced
this pull request
May 11, 2026
…re discovery (apache#19405) With apache#19040, when ioConfig.serverPriorityToReplicas is set, a supervisor can get stuck throwing DruidException from computeUnassignedServerPriorities, preventing any new task replicas with destined replicas from being created until the old tasks rollover. The exception is as follows: Found unassignedServerPriorities[[]] of size[0] < total replicas[1] for taskGroupId[0]. Task server priorities[[1, 0]] have already been assigned to tasks[[foo]]. The supervisor can remain in this unhealthy state unable to create additional task replicas until tasks eventually rollover. This can happen when a task spuriously fails after it's created but before it's discovered in the runInternal() loop. This patch removes the additional writer, leaving discoverTasks and removeTask as the sole mutators. group.taskIdToServerPriority should now stay in sync with group.tasks. The added unit tests fail on master without the fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
With #19040, when
ioConfig.serverPriorityToReplicasis set, a supervisor can get stuck throwingDruidExceptionfromcomputeUnassignedServerPriorities, preventing any newtask replicas with destined replicas from being created until the old tasks rollover. The
exception is as follows:
The supervisor can remain in this unhealthy state unable to create additional task replicas until tasks eventually rollover. This can happen when a task spuriously fails after it's created but before it's discovered in the
runInternal()loop.Root cause
createTasksForGroupwas an additional writer ofgroup.taskIdToServerPriority,recording the priority at task-submission time — before the task was ever observed
in
group.tasks. The group's other invariant (tasksandtaskIdToServerPrioritystay in sync) is enforced by
discoverTasks(additions) andTaskGroup.removeTask(removals), both of which operate on
group.tasks.If a task died after submission but before the next supervisor run picked it up from
the overlord, it never entered
group.tasks, soremoveTasknever fired — thepriority entry was orphaned and made the next
computeUnassignedServerPrioritiescall throw.
Fix
This patch removes the additional writer, leaving
discoverTasksandremoveTaskas the sole mutators.
group.taskIdToServerPriorityshould now stay in sync withgroup.tasks.The added unit tests fail on master without the fix.
This PR has: