Skip to content

Encapsulate spill buffer and memory_monitor #5891

@fjetter

Description

@fjetter

A few mechanics in worker.py are very well self-contained, and it would make a lot of sense to encapsulate them into a separate module/class in order to reduce complexity:

  • memory limit and thresholds
  • initialisation of the SpillBuffer
  • memory_monitor(), which handles pause, unpause, and force evict from spillbuffer

From the Worker's perspective, Worker.data should be treated as an opaque MutableMapping.

This is just a refactoring, with no functional impact.

Consider deprecating init arguments to Worker using target, spill, pause threshold etc. in favour of dask.config.

This change is part of the wider epic

Related issues:

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions