## What changes were proposed in this pull request?

SQLConf is session-scoped and mutable. However, we do have the requirement for a static SQL conf, which is global and immutable, e.g. the `schemaStringThreshold` in `HiveExternalCatalog`, the flag to enable/disable hive support, the global temp view database in apache#14897.

Actually we've already implemented static SQL conf implicitly via `SparkConf`, this PR just make it explicit and expose it to users, so that they can see the config value via SQL command or `SparkSession.conf`, and forbid users to set/unset static SQL conf.

## How was this patch tested?

new tests in SQLConfSuite

Author: Wenchen Fan <[email protected]>

Closes apache#15295 from cloud-fan/global-conf.

…y set in SparkSession.builder.getOrCreate

### What changes were proposed in this pull request?

This PR proposes to show ignored configurations and hide the warnings for configurations that are already set when invoking `SparkSession.builder.getOrCreate`.

### Why are the changes needed?

Currently, `SparkSession.builder.getOrCreate()` is too noisy even when duplicate configurations are set. Users cannot easily tell which configurations are to fix. See the example below:

```bash
./bin/spark-shell --conf spark.abc=abc
```

```scala
import org.apache.spark.sql.SparkSession
spark.sparkContext.setLogLevel("DEBUG")
SparkSession.builder.config("spark.abc", "abc").getOrCreate
```

```
21:04:01.670 [main] WARN  org.apache.spark.sql.SparkSession - Using an existing SparkSession; some spark core configurations may not take effect.
```

It is straitforward when there are few configurations but it is difficult for users to figure out when there are too many configurations especially when these configurations are defined in a property file like 'spark-default.conf' maintained separately by system admins in production.

See also #34757 (comment).

### Does this PR introduce _any_ user-facing change?

Yes.

1. Show ignored configurations in debug level logs:

    ```bash
    ./bin/spark-shell --conf spark.abc=abc
    ```

    ```scala
    import org.apache.spark.sql.SparkSession
    spark.sparkContext.setLogLevel("DEBUG")
    SparkSession.builder
      .config("spark.sql.warehouse.dir", "2")
      .config("spark.abc", "abcb")
      .config("spark.abcd", "abcb4")
      .getOrCreate
    ```

    **Before:**

    ```
    21:13:28.360 [main] WARN  org.apache.spark.sql.SparkSession - Using an existing SparkSession; the static sql configurations will not take effect.
    21:13:28.360 [main] WARN  org.apache.spark.sql.SparkSession - Using an existing SparkSession; some spark core configurations may not take effect.
    ```

    **After**:

    ```
    20:34:30.619 [main] WARN  org.apache.spark.sql.SparkSession - Using an existing Spark session; only runtime SQL configurations will take effect.
    20:34:30.622 [main] DEBUG org.apache.spark.sql.SparkSession - Ignored static SQL configurations:
      spark.sql.warehouse.dir=2
    20:34:30.623 [main] DEBUG org.apache.spark.sql.SparkSession - Configurations that might not take effect:
      spark.abcd=abcb4
      spark.abc=abcb
    ```

2. Do not issue a warning and hide a configuration already explicitly set (with the same value) before.

    ```bash
    ./bin/spark-shell --conf spark.abc=abc
    ```

    ```scala
    import org.apache.spark.sql.SparkSession
    spark.sparkContext.setLogLevel("DEBUG")
    SparkSession.builder.config("spark.abc", "abc").getOrCreate  // **Ignore** warnings because it's already set in --conf
    SparkSession.builder.config("spark.abc.new", "abc").getOrCreate  // **Show** warnings for only configuration newly set.
    SparkSession.builder.config("spark.abc.new", "abc").getOrCreate  // **Ignore** warnings because it's set ^.
    ```

    **Before**:

    ```
    21:13:56.183 [main] WARN  org.apache.spark.sql.SparkSession - Using an existing SparkSession; some spark core configurations may not take effect.
    21:13:56.356 [main] WARN  org.apache.spark.sql.SparkSession - Using an existing SparkSession; some spark core configurations may not take effect.
    21:13:56.476 [main] WARN  org.apache.spark.sql.SparkSession - Using an existing SparkSession; some spark core configurations may not take effect.
    ```

    **After:**

    ```
    20:36:36.251 [main] WARN  org.apache.spark.sql.SparkSession - Using an existing Spark session; only runtime SQL configurations will take effect.
    20:36:36.253 [main] DEBUG org.apache.spark.sql.SparkSession - Configurations that might not take effect:
      spark.abc.new=abc
    ```

3. Do not issue a warning and hide runtime SQL configurations in debug log:

    ```bash
    ./bin/spark-shell
    ```

    ```scala
    import org.apache.spark.sql.SparkSession
    spark.sparkContext.setLogLevel("DEBUG")
    SparkSession.builder.config("spark.sql.ansi.enabled", "true").getOrCreate  // **Ignore** warnings for runtime SQL configurations
    SparkSession.builder.config("spark.buffer.size", "1234").getOrCreate  // **Show** warnings for Spark core configuration
    SparkSession.builder.config("spark.sql.source.specific", "abc").getOrCreate  // **Show** warnings for custom runtime options
    SparkSession.builder.config("spark.sql.warehouse.dir", "xyz").getOrCreate  // **Show** warnings for static SQL configurations
    ```

    **Before**:

    ```
    11:11:40.846 [main] WARN  org.apache.spark.sql.SparkSession - Using an existing SparkSession; some spark core configurations may not take effect.
    11:11:41.037 [main] WARN  org.apache.spark.sql.SparkSession - Using an existing SparkSession; some spark core configurations may not take effect.
    11:11:41.167 [main] WARN  org.apache.spark.sql.SparkSession - Using an existing SparkSession; some spark core configurations may not take effect.
    11:11:41.318 [main] WARN  org.apache.spark.sql.SparkSession - Using an existing SparkSession; the static sql configurations will not take effect.
    ```

    **After**:

    ```
    10:39:54.870 [main] WARN  org.apache.spark.sql.SparkSession - Using an existing Spark session; only runtime SQL configurations will take effect.
    10:39:54.872 [main] DEBUG org.apache.spark.sql.SparkSession - Configurations that might not take effect:
      spark.buffer.size=1234
    10:39:54.988 [main] WARN  org.apache.spark.sql.SparkSession - Using an existing Spark session; only runtime SQL configurations will take effect.
    10:39:54.988 [main] DEBUG org.apache.spark.sql.SparkSession - Configurations that might not take effect:
      spark.sql.source.specific=abc
    10:39:55.107 [main] WARN  org.apache.spark.sql.SparkSession - Using an existing Spark session; only runtime SQL configurations will take effect.
    10:39:55.108 [main] DEBUG org.apache.spark.sql.SparkSession - Ignored static SQL configurations:
      spark.sql.warehouse.dir=xyz
    ```

Note that there is no behaviour change on session state initialization when configurations are not set. For example:

```scala
import org.apache.spark.sql.SparkSession
spark.sparkContext.setLogLevel("DEBUG")
SparkSession.builder.getOrCreate
```

But the session state initialization can be triggered now for static SQL configurations set after this PR. Previously, it was not triggered. This would not introduce something user-facing or a bug but worth noting it.
For runtime SQL configurations, the session state initialization in this code path was introduced at #15295.

### How was this patch tested?

It was manually tested as shown above.

Closes #35001 from HyukjinKwon/SPARK-37727.

Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>