Avoid active_threads changing size during iteration
#4729
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
|
I'm not entirely sure if this is sufficient. The dict comprehension should do the same thing as the copy, shouldn't it? We are acquiring the lock here during profiling as well, see distributed/distributed/worker.py Lines 2940 to 2941 in f492aa7 |
fjetter
approved these changes
Apr 22, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes dask/dask#7581. @bsesar would you mind trying this PR out to see if it resolves the error you were seeing?
Note that we could also acquire
active_threads_lockduring the heartbeat to avoidself.active_threadschanging size during iteration. However since we acquireactive_threads_lockduring task execution and heartbeat often I opted to not acquire the lock here to avoid coupling task execution and heartbeating. Happy to acquireactive_threads_lockinstead if others prefer.