Skip to content

Thoughts on additional spilling layers #4629

@jakirkham

Description

@jakirkham

Have had a few thoughts about additional spilling layers we could benefit from:

  • Defragmentation (perform a memcpy into larger and larger buffers to clean up and consolidate small buffers)
  • Shared memory (also has the benefit of being accessible from other workers on the same node)
  • mmap files (in-between shared memory and a file on disk; can close when memory usage is high; also like memory when using solid state)

Would be straightforward to add these to Zict without any new dependencies

Metadata

Metadata

Assignees

No one assigned

    Labels

    discussionDiscussing a topic with no specific actions yetmemory

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions