Repository navigation
Unable to run TPC-H benchmarks using on-heap mode and default settings #1519
Description
Describe the bug
I use off-heap mode when running benchmarks locally, since this is the recommended approach for Comet.
Today I tried running in on-heap mode and q4 seems to hang indefinitely. I will need to see if this is a regression since the last release and try and debug the issue farther.
Steps to reproduce
No response
Expected behavior
No response
Additional context
No response
Activity
andygrove commentedon Mar 12, 2025
The first issue is that the Spark executor is configured to use 8GB, and with our default settings, Comet only gets 1.6 GB, which just isn't enough.
If I change Spark executor memory to 4GB and specify Comet memory overhead of 4GB then the job runs with good performance (close to 2x).
[-]Unable to run TPC-H benchmarks using on-heap mode[/-][+]Unable to run TPC-H benchmarks using on-heap mode and default settings[/+]andygrove commentedon Mar 12, 2025
Spark requires at least 3GB to run the benchmarks. Comet also requires at least 3GB.
In on-heap mode that means we require 6 GB total. In off-heap mode this is reduced to 5 GB.
andygrove commentedon Mar 12, 2025
See #1315 for more information.
I plan on updating the tuning guide to provide guidance on specifying Comet memory allocation to close this issue.
wForget commentedon Mar 13, 2025
I had the same confusion before. #1379 (comment) , maybe we can use executor off-heap memory (8g) as comet overhead memory.