Merged. Thanks @andygrove @comphead

Am I correct in assuming I don't have to increase spark.executor.memoryOverhead or spark.memory.offHeap.size when using Comet, only spark.comet.memoryOverhead? How does this interact with YARN memory management?

spark.comet.memoryOverhead configures the memory pool size for native execution. it is not the same as spark.executor.memoryOverhead or spark.memory.offHeap.size.

For YARN memory management, the YARN container respects the spark.executor.memoryOverhead and the Comet override this value in CometDriverPlugin

sc.conf.set(EXECUTOR_MEMORY_OVERHEAD.key, s"${execMemOverhead + cometMemOverhead}M")

If Comet CometPlugin is configured to be used, it will update Spark executor overhead based on the config value spark.comet.memoryOverhead etc. But I don't test it on YARN, not sure Spark plugins comes in before YARN's resource configuration. You probably can test it.

If YARN resource manager doesn't respect Spark plugins, you can manually set up executor overhead based on spark.comet.memoryOverhead.

@comphead Hm, interesting, I tried launching the container with and without specifying spark.comet.memoryOverhead and YARN reported the same amount of memory allocated to my application. I'll check again tomorrow with debug enabled, hopefully I'll see this: https://github.com/apache/datafusion-comet/blob/acb78280e495079312a1067b80e91671852389a5/spark/src/main/scala/org/apache/spark/Plugins.scala#L63C7-L63C14

@comphead Hm, interesting, I tried launching the container with and without specifying spark.comet.memoryOverhead and YARN reported the same amount of memory allocated to my application. I'll check again tomorrow with debug enabled, hopefully I'll see this: https://github.com/apache/datafusion-comet/blob/acb78280e495079312a1067b80e91671852389a5/spark/src/main/scala/org/apache/spark/Plugins.scala#L63C7-L63C14

@orthoxerox please try it with applying a separate plugin

--conf spark.plugins=org.apache.spark.CometPlugin