Skip to content

feat: Implement shared memory pool for case where spark.memory.offHeap.enabled=false#1002

Closed
andygrove wants to merge 16 commits intoapache:mainfrom
andygrove:shared-mempool
Closed

feat: Implement shared memory pool for case where spark.memory.offHeap.enabled=false#1002
andygrove wants to merge 16 commits intoapache:mainfrom
andygrove:shared-mempool

Conversation

@andygrove
Copy link
Member

@andygrove andygrove commented Oct 8, 2024 •
edited
Loading

Which issue does this PR close?

Closes #996

Rationale for this change

Simplify memory configuration.

What changes are included in this PR?

Allocate one shared pool per executor, rather than one pool per native plan, when spark.memory.offHeap.enabled=false.

How are these changes tested?

  • Existing tests
  • Ran TPC-H benchmarks

@andygrove andygrove marked this pull request as ready for review October 8, 2024 06:48
@Kontinuation
Copy link
Member

Kontinuation commented Oct 8, 2024

I'm a bit worried about this approach because we are implementing greedy mode inside CometTaskMemoryManager, which is known to starve consumers frequently. I prefer using fair spill pool for "native memory management" mode. This makes spillable operators work properly without being starved but with the cost of memory pool under-utilization.

@andygrove andygrove marked this pull request as draft October 8, 2024 16:07
@andygrove
Copy link
Member Author

andygrove commented Oct 8, 2024

I'm a bit worried about this approach because we are implementing greedy mode inside CometTaskMemoryManager, which is known to starve consumers frequently. I prefer using fair spill pool for "native memory management" mode. This makes spillable operators work properly without being starved but with the cost of memory pool under-utilization.

Thanks for the feedback. I will work on a separate PR for the fair spill approach. I am moving this PR to draft for now.

@andygrove andygrove mentioned this pull request Oct 8, 2024
Closed
@andygrove
Copy link
Member Author

andygrove commented Oct 16, 2024

Closing in favor of #1021

@andygrove andygrove closed this Oct 16, 2024
@andygrove andygrove deleted the shared-mempool branch December 3, 2024 04:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Implement shared memory pool when native memory management is enabled

2 participants

@andygrove @Kontinuation