Exempt testing-category and internal configs from the versioning policy #6088

Comet's versioning policy currently puts every spark.comet.* key inside the compatibility surface:

Configuration keys under spark.comet.*: their names, types, accepted values, default values, and semantics.

That is too broad. Two mechanisms already mark a key as not being a deployment surface, and neither is recognized by the policy.

.category(CATEGORY_TESTING) routes a key into the Development & Testing Settings table rather than in with the production settings. These exist so Comet's own suites, and contributors chasing a bug, can reach a state the rest of the code is not built to support. spark.comet.scan.enabled disables native scans to isolate a planner problem. spark.comet.explain.fallback.strict.enabled makes a declined operator throw instead of quietly reporting itself. spark.comet.testing.aggregate.partialMode.enabled runs a partial aggregate without its final counterpart. The on-heap family (spark.comet.exec.onHeap.enabled, spark.comet.exec.onHeap.memoryPool, spark.comet.memoryOverhead, spark.comet.shuffle.jvm.memoryFactor) exists because Spark's own SQL suite needs on-heap mode; the tuning guide documents off-heap only.

.internal() goes further, keeping the key out of configs.md entirely so Comet never publishes its name, its default, or what it does. Three keys are internal but not in the testing category: spark.comet.exec.broadcast.enabled ("this config is only used for unit test"), spark.comet.shuffle.jvm.spillThreshold and spark.comet.shuffle.jvm.memoryWaitTimeout (both "internal config for testing purpose or advanced tuning"). Guaranteeing a key users have no supported way to discover is a commitment with no corresponding benefit.

Taken literally, the policy as written means renaming any of these needs a withAlternative alias, changing a default needs a spark.comet.legacy.* escape hatch and an upgrade guide entry, and removing one needs a deprecation cycle and a major release. None of that is worth doing for a knob whose only callers are in spark/src/test, and it would pin the internals the knob exists to expose — the same internals the policy otherwise leaves explicitly free to change.

Proposal: state one rule that covers both mechanisms — a key is covered only if the configuration reference publishes it as a production setting. Names, types, accepted values, defaults, and semantics of the rest may change in any release, including a patch release, with no alias, no legacy config, no deprecation cycle, and no upgrade guide entry. The corollary is a rule for contributors: neither mechanism may be the only way to reach a behavior production users need, so the category and the internal flag become policy decisions rather than routing details.

Documentation to update:

• docs/source/about/versioning_policy.md — the carve-out itself, plus the sections that currently imply the opposite (patch releases, behavior changes, renames).
• docs/source/contributor-guide/config_conventions.md — what the category and internal() commit the project to.
• docs/source/user-guide/latest/configs.md — a note on the Development & Testing Settings table, and one covering keys the page does not list at all.