Skip to content

Local scheduler parameter chunksize #7510

Description

@jakirkham

Recently we added the ability to batch submissions with the local scheduler using chunksize as users have voiced interest ( #6220 (comment) ). Mostly this is unused (so a no-op) with the exception of the multiprocessing case where we do some batching to amortize the cost of process spin up. It's also not that visible in the documentation or elsewhere. That said, we might want to share this more broadly.

One question that comes up with this is whether we like the parameter name, chunksize (taken from concurrent.futures and used elsewhere ( dask/distributed#3650 )) or if that's confusing and we want to use something else like batchsize. Another question is this sits at the top-level config (similar to other things like pool and num_workers), is that where we want this? Lastly is this parameter useful outside the local scheduler (like would Distributed want to use this)

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs attentionIt's been a while since this was pushed on. Needs attention from the owner or a maintainer.scheduler

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions