Repository navigation
Commit 369a1cf
committed
[SPARK-59778][SQL] Add display-only properties to the Table API
### What changes were proposed in this pull request?
Add `Table.displayProperties()` for additional display-only table properties, defaulting to an empty map. The map and its keys and values must be non-null. Spark merges these entries with non-reserved stored properties for V2 `DESCRIBE TABLE EXTENDED`, `SHOW TBLPROPERTIES`, and `SHOW TABLE EXTENDED`. Stored values take precedence, and reserved keys in the display map are ignored. Table type, location, provider, owner, and other reserved metadata continue to come from `properties()`.
Support metadata-only tables through `TableInfo.Builder.withDisplayProperties()` and `DelegatingTable`. Keep `SHOW CREATE TABLE` on `properties()`. The API documentation also guides `createTableLike` implementations to exclude display-only entries when copying source metadata, notes that implementations may compute them on demand, and explains that Spark redacts them like stored properties.
### Why are the changes needed?
Catalogs may expose transient metadata such as labels that should be visible when inspecting a table without appearing in the DDL used to recreate it. The additive API lets connectors supply these entries separately from stored properties.
JIRA: https://issues.apache.org/jira/browse/SPARK-59778
### Does this PR introduce _any_ user-facing change?
Yes. Table implementations can opt in to display-only properties. Existing connectors retain their behavior because the new method defaults to an empty map. Mockito mocks that only stub `properties()` also retain their existing results.
`spark.catalog.getTableProperties()` remains equivalent to `SHOW TBLPROPERTIES`, including in PySpark and Spark Connect, so it also returns display-only entries. The Scala, Connect Scala, and Python API documentation explains that these entries may be transient and are excluded from the DDL returned by `getCreateTableString()`.
`ALTER TABLE ... SET/UNSET TBLPROPERTIES` forwards changes to the catalog without consulting display properties. The API documentation guides catalogs to apply these changes to stored properties and keep display-only entries unchanged. After a colliding stored non-reserved property is unset, the display value is shown if it remains in the display map. Reserved display keys, including `comment`, remain ignored. The shared display helper applies the existing redaction rules and reuses the stored property map fetched for reserved metadata.
`TableInfo` carries display properties as returned metadata. Spark leaves them empty when calling the table creation and staging methods. V2 views retain their existing behavior. The SQL references for `SHOW TBLPROPERTIES` and `ALTER TABLE ... UNSET TBLPROPERTIES` explain display-only entries and stored-property precedence.
For now, `SHOW TABLE EXTENDED` omits display-only entries for the built-in session catalog and its `CatalogExtension`s. `DESCRIBE TABLE EXTENDED` and `SHOW TBLPROPERTIES` include them when an extension returns a V2 table.
### How was this patch tested?
For this revision, all 34 selected tests passed using `SPARK_LOCAL_IP=localhost build/sbt`:
- Catalyst `CatalogV2UtilSuite`: 24 tests.
- Catalyst `CatalogSuite -- -z "display properties"`: 2 tests.
- V2 `DescribeTableSuite`, `ShowTablesSuite`, `ShowTblPropertiesSuite`, `DataSourceV2MetadataTableSuite`, and `DataSourceV2DataFrameSessionCatalogSuite`, filtered by `"display properties"`: 7 tests.
- V2 `ShowCreateTableSuite -- -z "display-only properties"`: 1 test.
The added assertions cover reserved-key lookup and the session-catalog `SHOW TABLE EXTENDED` limitation. The helper redaction test checks that mutable input maps remain unchanged. External-table fixtures now use `CREATE EXTERNAL TABLE`, and display fixture setup uses the underlying live table.
Catalyst Java Checkstyle, Catalyst/SQL test Scalastyle, and `git diff --check` also passed for this revision.
The previous revision passed all 214 selected tests; one pre-existing test was ignored:
- Catalyst `CatalogSuite` and `CatalogV2UtilSuite`: 103 tests.
- V2 `DescribeTableSuite`, `ShowTblPropertiesSuite`, `ShowTablesSuite`, and `ShowCreateTableSuite`, plus `DataSourceV2MetadataTableSuite`: 100 tests.
- `DataSourceV2DataFrameSessionCatalogSuite -- -z "display properties"`: 1 test.
- `org.apache.spark.sql.internal.CatalogSuite -- -z "catalog API"`: 10 tests.
Ran with `SPARK_LOCAL_IP=localhost build/sbt`, using `catalyst/testOnly` and `sql/testOnly` for the suites above. Coverage includes stored/display collisions, SET and UNSET, every reserved display key with and without a stored location, redaction, metadata-only tables, exclusion from generated DDL, and preservation across copies, pinning, and ALTER TABLE in both V2 and session-catalog fixtures.
The previous revision also passed Catalyst Java Checkstyle, Catalyst/SQL main and test Scalastyle, and `git diff --check`. SQL API/Connect Common scalafmt validation and Python syntax checks passed in an earlier revision; those files are unchanged in this update.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-6)
Closes #59036 from szehon-ho/codex/table-display-properties.
Authored-by: Szehon Ho <[email protected]>
Signed-off-by: Szehon Ho <[email protected]>
(cherry picked from commit 918645b)
Signed-off-by: Szehon Ho <[email protected]>1 parent 3909ebe commit 369a1cf
31 files changed
Lines changed: 481 additions & 39 deletions
File tree
- docs
- python/pyspark/sql
- sql
- api/src/main/scala/org/apache/spark/sql/catalog
- catalyst/src
- main
- java/org/apache/spark/sql/connector/catalog
- scala/org/apache/spark/sql/connector/catalog
- test/scala/org/apache/spark/sql/connector/catalog
- connect/common/src/main/scala/org/apache/spark/sql/connect
- core/src
- main/scala/org/apache/spark/sql/execution/datasources/v2
- test/scala/org/apache/spark/sql
- connector
- execution/command/v2
- internal
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
27 | 35 | | |
28 | 36 | | |
29 | 37 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
278 | 278 | | |
279 | 279 | | |
280 | 280 | | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
281 | 288 | | |
282 | 289 | | |
283 | 290 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
393 | 393 | | |
394 | 394 | | |
395 | 395 | | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
396 | 402 | | |
397 | 403 | | |
398 | 404 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
846 | 846 | | |
847 | 847 | | |
848 | 848 | | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
849 | 852 | | |
850 | 853 | | |
851 | 854 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
77 | 82 | | |
78 | 83 | | |
79 | 84 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
95 | 95 | | |
96 | 96 | | |
97 | 97 | | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
98 | 130 | | |
99 | 131 | | |
100 | 132 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
19 | 20 | | |
20 | 21 | | |
21 | 22 | | |
| |||
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
27 | | - | |
| 28 | + | |
| 29 | + | |
28 | 30 | | |
29 | 31 | | |
30 | 32 | | |
| |||
34 | 36 | | |
35 | 37 | | |
36 | 38 | | |
| 39 | + | |
37 | 40 | | |
38 | 41 | | |
39 | 42 | | |
| |||
43 | 46 | | |
44 | 47 | | |
45 | 48 | | |
| 49 | + | |
46 | 50 | | |
47 | 51 | | |
48 | 52 | | |
| |||
59 | 63 | | |
60 | 64 | | |
61 | 65 | | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
62 | 82 | | |
63 | 83 | | |
64 | 84 | | |
65 | 85 | | |
66 | 86 | | |
67 | 87 | | |
68 | 88 | | |
| 89 | + | |
69 | 90 | | |
70 | 91 | | |
71 | 92 | | |
72 | 93 | | |
73 | 94 | | |
74 | 95 | | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
75 | 108 | | |
76 | 109 | | |
77 | 110 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
692 | 692 | | |
693 | 693 | | |
694 | 694 | | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
695 | 705 | | |
696 | 706 | | |
697 | 707 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
77 | 77 | | |
78 | 78 | | |
79 | 79 | | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
80 | 133 | | |
81 | 134 | | |
82 | 135 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
38 | 78 | | |
39 | 79 | | |
40 | 80 | | |
| |||
0 commit comments