Skip to content

Profile memory usage and add guidance to the tuning guide #1315

Closed

Description

@andygrove

What is the problem the feature request solves?

We do not yet have any documentation explaining how much off-heap memory Comet uses compared to Spark. We should profile this and add documentation and potentially file issues to improve memory utilization if we find the the current usage is not ideal.

Describe the potential solution

No response

Additional context

No response

Activity

  1. modified the milestone: 0.6.0 on Jan 21, 2025
  2. modified the milestones: 0.6.0, 0.7.0, 0.8.0 on Feb 4, 2025
  3. self-assigned this
    on Mar 12, 2025
  4. modified the milestones: 0.8.0, 0.7.0 on Mar 12, 2025
  5. andygrove commented on Mar 12, 2025

    @andygrove
    MemberAuthor

    I have been testing with TPC-H @ 100 GB.

    Image

    Conclusions:

    • Spark needs at least 3 GB and so does Comet, so we double the memory requirement in on-heap mode
    • We use slightly less memory in off-heap mode
    • Users need to carefully tune Comet settings for their workloads
    • Comet can reduce memory overall because performace with Comet off-heap @ 5 GB is much faster than peak Spark performance @ 9 - 10 GB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions