TDLR is that ensuring that launched tasks are properly cleaned up (and panics are propagated back to the main tasks) is tricky.

Tokio has a tokio::task::JoinSet which handles the cleanup part and #6507 demonstrates how to use JoinSet to propagate panics.

We should ideally also fix
datafusion/core/src/physical_plan/repartition/mod.rs to propagate panics
and remove AbortOnDrop andAbortOnDropMany with JoinSet

Originally posted by @crepererum in #6449 (comment)

Subtasks

• Migrate majority of the code base: Replace AbortOnDrop / AbortDropOnMany with tokio JoinSet #6750
• Replace impl. in RepartitionExec
• Remove AbortOnDrop / AbortDropOnMany
• Use clippy's disallowed_methods lint to prevent the usage of tokio::spawn altogether, since it led to so many cancellation and error propagation bugs.