Fix tasks getting stuck in Pending#2998
Merged
Merged
Conversation
Fixed a bug where tasks could get stuck in the Pending state if there was no suitable node for the task. Signed-off-by: Drew Erny <[email protected]>
dperny
force-pushed
the
fix-stalled-scheduler-tasks
branch
from
February 11, 2021 17:09
6834e7b to
807f25e
Compare
Codecov Report
@@ Coverage Diff @@
## master #2998 +/- ##
==========================================
- Coverage 61.77% 60.08% -1.69%
==========================================
Files 142 142
Lines 23005 20454 -2551
==========================================
- Hits 14211 12290 -1921
+ Misses 7303 6693 -610
+ Partials 1491 1471 -20 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
- What I did
Fixed a bug where tasks could get stuck in the Pending state if there was no suitable node for the task.
- How I did it
When a task could not be scheduled, because of something like
MaxReplicas(or, in my testing of it, the WIP Swarm Cluster Volumes functionality), it would stay in thePendingstate forever, unable to progress but never shut down.This change makes it so that if a Task has no suitable node, belongs to an older revision of the service, and is desired to be
Shutdown, then the Scheduler takes initiative to move it to stateShutdown.This is permissible because when the Task is in state
Pending, it's lifecycle is under the purview of the scheduler.- How to test it
Includes a new automated test. This test failed before the change, and succeeds now.
- Description for the changelog
Fixed a bug where tasks which could not be scheduled stayed in the Pending state forever.