chore: remove remaining Spark 3.4 compatibility code - #7625

Which issue does this PR close?

Part of #4329.

Supersedes #5358 after #7521 removed the Spark 3.4 build profile and CI coverage and #7560 removed the Spark 3.4 shim source sets.

Rationale for this change

Spark 3.4 is no longer a supported build target, but compatibility branches, fallbacks, tests, generated diffs, and version-specific test assets remained in the codebase. Keeping those dead paths makes the supported Spark 3.5 and 4.x implementations harder to follow and maintain.

The separately tracked get_json_object_spark34 behavior is intentionally preserved.

What changes are included in this PR?

• Remove the remaining Spark 3.4 compatibility gates and fallback code from Comet, Delta, Iceberg, scan, shuffle, expression, and test paths.
• Delete obsolete Spark 3.4 and Iceberg 1.8.1 generated diffs and Spark 3.4-only tests and shims.
• Promote the Spark 3.5 TPC-DS plan baselines to the shared baseline and remove the now-redundant Spark 3.5 override files.
• Rename CometExec3_4PlusSuite to CometExecCompatibilitySuite and update CI suite lists and active comments to describe the supported versions.

How are these changes tested?

• ./mvnw -Pspark-3.5 -DskipTests test-compile
• ./mvnw -Pspark-4.0 -DskipTests test-compile
• ./mvnw -Pspark-4.1 -DskipTests test-compile
• ./mvnw -Pspark-4.2 -DskipTests test-compile
• python3 dev/ci/check-ci-config.py
• python3 dev/ci/check-versioned-source-parity.py
• python3 dev/ci/check-suites.py
• git diff --check

The focused Spark 3.5 SQL parser and plan-stability suites compiled locally, but the runner could not start because no compatible Spark 3.5 checkout is configured in SPARK_HOME or spark.test.home. The run-spark-3.5-tests label is requested for hosted validation.