Skip to content

Schedule from queue in batches to improve co-assignment#7554

Closed
gjoseph92 wants to merge 2 commits into
dask:mainfrom
gjoseph92:queue-batch-assign
Closed

Schedule from queue in batches to improve co-assignment#7554
gjoseph92 wants to merge 2 commits into
dask:mainfrom
gjoseph92:queue-batch-assign

Conversation

@gjoseph92

Copy link
Copy Markdown
Collaborator

An experimental approach to getting co-assignment along with queuing. Rather than scheduling a new task every time a thread opens up on a worker, we basically wait until all the threads are idle on a worker, then schedule a batch of nthreads queued tasks at once.

This "works" in that it improves co-assignment and reduces data transfer while maintaining low memory footprint, but it's slow.

Benchmarks: https://observablehq.com/@gjoseph92/snakebench?commits=6bb123e&commits=6215b2b&measure=duration&groupby=branch&show=passed&everygroup=true

runtime
zchart

memory
zchart (1)

data transfer
zchart (2)

  • Tests added / passed
  • Passes pre-commit run --all-files

This could slightly improve co-assignment without degrading graph traversal order or increasing parallelism
This fixes `test_deadlock_resubmit_queued_tasks_fast`. The problem:
- an "old" TaskState `f-0` is forgotten and removed from `queued` HeapSet. Internally, this removes it from a set, but a weakref is still on the heap
- a new TaskState `f-0` is added to the HeapSet. It's pushed onto the heap, but since its priority is higher (newer generation), it comes after the old `f-0` object that's still on the front of the heap
- `pop` pops old `f-0` off the heap. The weakref is still alive (for whatever reason). `value in self._data` (the set) is True, because the _hash_ of old `f-0` and new `f-0` are the same (same key). So we return the stale, old TaskState object.

Much like `WorkerState`s, there should be exactly 1 `TaskState` instance per task. If there are multiple instances with the same key, they are different tasks, and have different state. xref dask#7372, dask#7356.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant