Recent changes to the scheduling heuristics to queue tasks on scheduler were removing the ability to co-assign root tasks. This feature helped to reduce network load by assigning tasks with likely shared dependents to the same worker ahead of time.

The previous implementation of this was a stateful heuristic that cannot be easily ported to the queuing system and has known shortcomings for non-trivial topologies, see #6597

An early attempt to formulate a stateless algorithm to identify groups of tasks that should be co-assigned, in this PR called "families" can be reviewed in #7076 but failed to deliver on our expectations. This approach was ultimately rejected due to complexity, runtime concerns and lack of classification power.

An alternative approach to identify "co-assignment groups" / "cogroups" has been proposed in #7141 with an alternative implementation of the same/similar algorithm in gjoseph92#6

Goals

We want to validate the algorithm in #7141 and perform necessary benchmarks to inform future work or make a decision to enable coassignment

Technical changes required for proper validation

• The algorithm should be hooked up to queuing code path
• The worker objective function should be tested and reevaluated
• Logic needs to be implemented to bypass pure co-assign logic for large clusters to not suffer from too little parallelism

Benchmarking / Perf validation

• Workloads are identified that should benefit from co-assignment
• Metrics are identified to measure reduce network transfer. Proposal: dask_worker_transfer_incoming_count_total/dask_worker_transfer_outgoing_count_total + nbytes equivalents
• Actual impact on runtime performance for a selected set of cluster configurations is measured and documented

Nice to have but not required

• Up/Downscaling
• Full lifecycle management of cogroup management (clean up state properly on scheduler)

Non-goals

• Tune/rewrite the coassignment algorithm. If the proposed algorithm is not suitable, we should drop this effort for now