Recently we added the ability to batch submissions with the local scheduler using chunksize as users have voiced interest ( #6220 (comment) ). Mostly this is unused (so a no-op) with the exception of the multiprocessing case where we do some batching to amortize the cost of process spin up. It's also not that visible in the documentation or elsewhere. That said, we might want to share this more broadly.
One question that comes up with this is whether we like the parameter name, chunksize (taken from concurrent.futures and used elsewhere ( dask/distributed#3650 )) or if that's confusing and we want to use something else like batchsize. Another question is this sits at the top-level config (similar to other things like pool and num_workers), is that where we want this? Lastly is this parameter useful outside the local scheduler (like would Distributed want to use this)
Recently we added the ability to batch submissions with the local scheduler using
chunksizeas users have voiced interest ( #6220 (comment) ). Mostly this is unused (so a no-op) with the exception of themultiprocessingcase where we do some batching to amortize the cost of process spin up. It's also not that visible in the documentation or elsewhere. That said, we might want to share this more broadly.One question that comes up with this is whether we like the parameter name,
chunksize(taken fromconcurrent.futuresand used elsewhere ( dask/distributed#3650 )) or if that's confusing and we want to use something else likebatchsize. Another question is this sits at the top-level config (similar to other things likepoolandnum_workers), is that where we want this? Lastly is this parameter useful outside the local scheduler (like would Distributed want to use this)