Improve memory pool configuration code, documentation, and tests #1560

What is the problem the feature request solves?

PR #1525 made improvements to the Tuning Guide but there is still incorrect and/or misleading information, and some of this was only discovered while working on that PR.

This is a follow-on issue to make improvements to the memory pool configuration code, documentation, and tests.

Here are some initial thoughts on areas that need attention:

• When running in off-heap mode we tell users to specify Comet overhead using spark.memory.offHeap.size. The original goal of using off-heap memory was to implement "unified memory management," where the native pool will interact with Spark's memory pool so that there is one unified memory pool between Spark and Comet. However, we now also have a "fair_unified" pool that can be used in off-heap mode and this does not use the "unified memory management" approach, so the name is misleading. The pool size can be configured in numerous ways and this could perhaps be simplified.
• In on-heap mode we use spark.comet.memoryOverhead to tell YARN/k8s to allocate extra memory but we do not do this when using fair_unified in off-heap mode, but still allow the user to set this instead of basing allocation off of spark.memory.offHeap.size, which is confusing.
• There are very few tests for any of the code that computes Comet memory overhead or memory pool sizes

Describe the potential solution

No response

Additional context

No response