Skip to content

Commit 369a1cf

Browse files
committed
[SPARK-59778][SQL] Add display-only properties to the Table API
### What changes were proposed in this pull request? Add `Table.displayProperties()` for additional display-only table properties, defaulting to an empty map. The map and its keys and values must be non-null. Spark merges these entries with non-reserved stored properties for V2 `DESCRIBE TABLE EXTENDED`, `SHOW TBLPROPERTIES`, and `SHOW TABLE EXTENDED`. Stored values take precedence, and reserved keys in the display map are ignored. Table type, location, provider, owner, and other reserved metadata continue to come from `properties()`. Support metadata-only tables through `TableInfo.Builder.withDisplayProperties()` and `DelegatingTable`. Keep `SHOW CREATE TABLE` on `properties()`. The API documentation also guides `createTableLike` implementations to exclude display-only entries when copying source metadata, notes that implementations may compute them on demand, and explains that Spark redacts them like stored properties. ### Why are the changes needed? Catalogs may expose transient metadata such as labels that should be visible when inspecting a table without appearing in the DDL used to recreate it. The additive API lets connectors supply these entries separately from stored properties. JIRA: https://issues.apache.org/jira/browse/SPARK-59778 ### Does this PR introduce _any_ user-facing change? Yes. Table implementations can opt in to display-only properties. Existing connectors retain their behavior because the new method defaults to an empty map. Mockito mocks that only stub `properties()` also retain their existing results. `spark.catalog.getTableProperties()` remains equivalent to `SHOW TBLPROPERTIES`, including in PySpark and Spark Connect, so it also returns display-only entries. The Scala, Connect Scala, and Python API documentation explains that these entries may be transient and are excluded from the DDL returned by `getCreateTableString()`. `ALTER TABLE ... SET/UNSET TBLPROPERTIES` forwards changes to the catalog without consulting display properties. The API documentation guides catalogs to apply these changes to stored properties and keep display-only entries unchanged. After a colliding stored non-reserved property is unset, the display value is shown if it remains in the display map. Reserved display keys, including `comment`, remain ignored. The shared display helper applies the existing redaction rules and reuses the stored property map fetched for reserved metadata. `TableInfo` carries display properties as returned metadata. Spark leaves them empty when calling the table creation and staging methods. V2 views retain their existing behavior. The SQL references for `SHOW TBLPROPERTIES` and `ALTER TABLE ... UNSET TBLPROPERTIES` explain display-only entries and stored-property precedence. For now, `SHOW TABLE EXTENDED` omits display-only entries for the built-in session catalog and its `CatalogExtension`s. `DESCRIBE TABLE EXTENDED` and `SHOW TBLPROPERTIES` include them when an extension returns a V2 table. ### How was this patch tested? For this revision, all 34 selected tests passed using `SPARK_LOCAL_IP=localhost build/sbt`: - Catalyst `CatalogV2UtilSuite`: 24 tests. - Catalyst `CatalogSuite -- -z "display properties"`: 2 tests. - V2 `DescribeTableSuite`, `ShowTablesSuite`, `ShowTblPropertiesSuite`, `DataSourceV2MetadataTableSuite`, and `DataSourceV2DataFrameSessionCatalogSuite`, filtered by `"display properties"`: 7 tests. - V2 `ShowCreateTableSuite -- -z "display-only properties"`: 1 test. The added assertions cover reserved-key lookup and the session-catalog `SHOW TABLE EXTENDED` limitation. The helper redaction test checks that mutable input maps remain unchanged. External-table fixtures now use `CREATE EXTERNAL TABLE`, and display fixture setup uses the underlying live table. Catalyst Java Checkstyle, Catalyst/SQL test Scalastyle, and `git diff --check` also passed for this revision. The previous revision passed all 214 selected tests; one pre-existing test was ignored: - Catalyst `CatalogSuite` and `CatalogV2UtilSuite`: 103 tests. - V2 `DescribeTableSuite`, `ShowTblPropertiesSuite`, `ShowTablesSuite`, and `ShowCreateTableSuite`, plus `DataSourceV2MetadataTableSuite`: 100 tests. - `DataSourceV2DataFrameSessionCatalogSuite -- -z "display properties"`: 1 test. - `org.apache.spark.sql.internal.CatalogSuite -- -z "catalog API"`: 10 tests. Ran with `SPARK_LOCAL_IP=localhost build/sbt`, using `catalyst/testOnly` and `sql/testOnly` for the suites above. Coverage includes stored/display collisions, SET and UNSET, every reserved display key with and without a stored location, redaction, metadata-only tables, exclusion from generated DDL, and preservation across copies, pinning, and ALTER TABLE in both V2 and session-catalog fixtures. The previous revision also passed Catalyst Java Checkstyle, Catalyst/SQL main and test Scalastyle, and `git diff --check`. SQL API/Connect Common scalafmt validation and Python syntax checks passed in an earlier revision; those files are unchanged in this update. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-6) Closes #59036 from szehon-ho/codex/table-display-properties. Authored-by: Szehon Ho <[email protected]> Signed-off-by: Szehon Ho <[email protected]> (cherry picked from commit 918645b) Signed-off-by: Szehon Ho <[email protected]>
1 parent 3909ebe commit 369a1cf

31 files changed

Lines changed: 481 additions & 39 deletions

‎docs/sql-ref-syntax-aux-show-tblproperties.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,14 @@ license: |
2424
This statement returns the value of a table property given an optional value for
2525
a property key. If no key is specified then all the properties are returned.
2626

27+
For tables in [Data Source V2](sql-v2-data-sources.html) catalogs, the output may also include
28+
display-only properties supplied by the catalog. Spark itself does not remove these entries on
29+
`ALTER TABLE ... UNSET TBLPROPERTIES`; whether they remain depends on the catalog.
30+
`SHOW CREATE TABLE` does not include them. Stored properties take precedence over display-only
31+
entries with the same key. After unsetting a stored property other than `comment`, Spark displays
32+
the display-only value with that key if the catalog still supplies it. Display-only `comment`
33+
entries are ignored, even after unsetting the stored `comment`.
34+
2735
### Syntax
2836

2937
```sql

‎docs/sql-ref-syntax-ddl-alter-table.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -278,6 +278,13 @@ ALTER TABLE table_identifier SET TBLPROPERTIES ( key1 = val1, key2 = val2, ... )
278278

279279
`ALTER TABLE UNSET` command is used to drop the table property.
280280

281+
For tables in [Data Source V2](sql-v2-data-sources.html) catalogs, catalogs may supply display-only
282+
properties in metadata commands such as `SHOW TBLPROPERTIES`. Spark itself does not remove these
283+
entries on `ALTER TABLE ... UNSET TBLPROPERTIES`; whether they remain depends on the catalog.
284+
After unsetting a stored property other than `comment`, Spark displays the display-only value
285+
with that key if the catalog still supplies it. Display-only `comment` entries are ignored,
286+
even after unsetting the stored `comment`.
287+
281288
**Note:** If the specified property key does not exist, whether specify `IF EXISTS` or not, the command will ignore it and finally succeed.
282289

283290
##### Syntax

‎python/pyspark/sql/catalog.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -393,6 +393,12 @@ def getTableProperties(self, tableName: str) -> Dict[str, str]:
393393
dict
394394
Map of property key to value.
395395
396+
Notes
397+
-----
398+
For V2 tables, the returned dict also includes display-only properties, which may be
399+
transient. These entries are excluded from the table DDL returned by
400+
:meth:`getCreateTableString`.
401+
396402
Examples
397403
--------
398404
>>> _ = spark.sql("DROP TABLE IF EXISTS tbl_props_doc")

‎sql/api/src/main/scala/org/apache/spark/sql/catalog/Catalog.scala‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -846,6 +846,9 @@ abstract class Catalog {
846846
/**
847847
* Returns all table properties as a map (same as `SHOW TBLPROPERTIES`).
848848
*
849+
* For V2 tables, the returned map also includes display-only properties, which may be
850+
* transient. These entries are excluded from the table DDL returned by `getCreateTableString`.
851+
*
849852
* @param tableName
850853
* table or view name; may be qualified with catalog and database (namespace).
851854
* @since 4.2.0

‎sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/DelegatingTable.java‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,11 @@ public Map<String, String> properties() {
7474
return Collections.unmodifiableMap(info.properties());
7575
}
7676

77+
@Override
78+
public Map<String, String> displayProperties() {
79+
return Collections.unmodifiableMap(info.displayProperties());
80+
}
81+
7782
@Override
7883
public Transform[] partitioning() {
7984
return info.partitions();

‎sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Table.java‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,38 @@ default Map<String, String> properties() {
9595
return Collections.emptyMap();
9696
}
9797

98+
/**
99+
* Returns additional, display-only table properties.
100+
*
101+
* <p>Spark adds these entries to the non-reserved properties displayed by V2 table metadata
102+
* commands and {@code Catalog.getTableProperties}. Entries in {@link #properties()} take
103+
* precedence, and reserved keys such as {@link TableCatalog#PROP_EXTERNAL},
104+
* {@link TableCatalog#PROP_LOCATION}, and {@link TableCatalog#PROP_COMMENT} are ignored in
105+
* this map. Table type and other reserved metadata are read from {@code properties()}.
106+
* After a stored non-reserved property is removed, Spark displays the entry for that key if
107+
* it remains in this map. Spark redacts these entries in the same way as stored properties.
108+
*
109+
* <p>Spark passes {@code ALTER TABLE ... SET/UNSET TBLPROPERTIES} to
110+
* {@link TableCatalog#alterTable} without consulting this map. Catalogs should apply these
111+
* changes to {@code properties()} and keep display-only entries unchanged.
112+
*
113+
* <p>For now, {@code SHOW TABLE EXTENDED} omits these entries for the built-in session catalog
114+
* and its {@link CatalogExtension}s.
115+
*
116+
* <p>These entries may be transient and are excluded from {@code SHOW CREATE TABLE}.
117+
* Implementations of {@link TableCatalog#createTableLike(Identifier, TableInfo, Table)} should
118+
* not copy them from the source table. Spark reads these entries only to display table
119+
* metadata, so implementations may compute them on demand.
120+
*
121+
* <p>The default implementation returns an empty map.
122+
*
123+
* @return a non-null map with non-null keys and values
124+
* @since 4.4.0
125+
*/
126+
default Map<String, String> displayProperties() {
127+
return Collections.emptyMap();
128+
}
129+
98130
/**
99131
* Returns the set of capabilities for this table.
100132
*/

‎sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableInfo.java‎

Lines changed: 34 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
*/
1717
package org.apache.spark.sql.connector.catalog;
1818

19+
import java.util.HashMap;
1920
import java.util.Map;
2021
import java.util.Objects;
2122

@@ -24,7 +25,8 @@
2425
import org.apache.spark.sql.types.StructType;
2526

2627
/**
27-
* Metadata describing a data-source table: its columns, properties, partitioning and constraints.
28+
* Metadata describing a data-source table: its columns, stored and display-only properties,
29+
* partitioning and constraints.
2830
* Spark realizes a {@code TableInfo} into a {@link Table} via {@link DelegatingTable}; a catalog
2931
* that has its own {@link Table} object returns that instead. Views are described by the sibling
3032
* {@link View}, which -- unlike a table -- is itself a {@link Relation} because Spark never builds
@@ -34,6 +36,7 @@ public class TableInfo {
3436

3537
private final Column[] columns;
3638
private final Map<String, String> properties;
39+
private final Map<String, String> displayProperties;
3740
private final Transform[] partitions;
3841
private final Constraint[] constraints;
3942

@@ -43,6 +46,7 @@ public class TableInfo {
4346
protected TableInfo(Builder builder) {
4447
this.columns = builder.columns;
4548
this.properties = builder.properties;
49+
this.displayProperties = builder.displayProperties;
4650
this.partitions = builder.partitions;
4751
this.constraints = builder.constraints;
4852
}
@@ -59,19 +63,48 @@ public Map<String, String> properties() {
5963
return properties;
6064
}
6165

66+
/**
67+
* Returns the additional display-only properties described by {@link Table#displayProperties()}.
68+
*
69+
* <p>These properties describe metadata returned by a catalog. Spark leaves them empty when
70+
* calling {@link TableCatalog#createTable(Identifier, TableInfo)},
71+
* {@link TableCatalog#createTableLike(Identifier, TableInfo, Table)},
72+
* {@link StagingTableCatalog#stageCreate(Identifier, TableInfo)},
73+
* {@link StagingTableCatalog#stageReplace(Identifier, TableInfo)}, and
74+
* {@link StagingTableCatalog#stageCreateOrReplace(Identifier, TableInfo)}.
75+
*
76+
* @since 4.4.0
77+
*/
78+
public Map<String, String> displayProperties() {
79+
return displayProperties;
80+
}
81+
6282
public Transform[] partitions() {
6383
return partitions;
6484
}
6585

6686
public Constraint[] constraints() { return constraints; }
6787

6888
public static class Builder extends RelationBuilder<Builder> {
89+
protected Map<String, String> displayProperties = new HashMap<>();
6990
protected Transform[] partitions = new Transform[0];
7091
protected Constraint[] constraints = new Constraint[0];
7192

7293
@Override
7394
protected Builder self() { return this; }
7495

96+
/**
97+
* Sets additional display-only properties.
98+
*
99+
* @param displayProperties a non-null map with non-null keys and values
100+
* @see TableInfo#displayProperties()
101+
* @since 4.4.0
102+
*/
103+
public Builder withDisplayProperties(Map<String, String> displayProperties) {
104+
this.displayProperties = new HashMap<>(displayProperties);
105+
return this;
106+
}
107+
75108
public Builder withPartitions(Transform[] partitions) {
76109
this.partitions = partitions;
77110
return this;

‎sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -692,6 +692,16 @@ private[sql] object CatalogV2Util {
692692
builder
693693
}
694694

695+
/** Returns redacted, non-reserved table properties with additional display-only metadata. */
696+
def tablePropertiesForDisplay(
697+
properties: util.Map[String, String],
698+
displayProperties: util.Map[String, String],
699+
conf: SQLConf): Map[String, String] = {
700+
val merged = (displayProperties.asScala.toMap ++ properties.asScala)
701+
.filterNot { case (key, _) => TABLE_RESERVED_PROPERTIES.contains(key) }
702+
conf.redactOptions(merged)
703+
}
704+
695705
def convertTableProperties(t: TableSpec): Map[String, String] = {
696706
val props = convertTableProperties(
697707
t.properties, t.options, t.serde, t.location, t.comment,

‎sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/CatalogSuite.scala‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,59 @@ class CatalogSuite extends SparkFunSuite {
7777
assert(loaded.getClass == catalog.getClass)
7878
}
7979

80+
test("TableInfo and DelegatingTable carry display-only properties separately") {
81+
val displayProperties = new util.HashMap[String, String]()
82+
displayProperties.put("catalog-label", "catalog-value")
83+
val info = new TableInfo.Builder()
84+
.withProperties(util.Map.of("persisted", "stored"))
85+
.withDisplayProperties(displayProperties)
86+
.build()
87+
displayProperties.put("catalog-label", "changed")
88+
val table = new DelegatingTable(info, "table")
89+
90+
assert(table.displayProperties() === util.Map.of("catalog-label", "catalog-value"))
91+
assert(table.properties() === util.Map.of("persisted", "stored"))
92+
intercept[UnsupportedOperationException] {
93+
table.displayProperties().put("catalog-label", "changed")
94+
}
95+
val defaultTable = new DelegatingTable(new TableInfo.Builder().build(), "default")
96+
assert(defaultTable.displayProperties().isEmpty)
97+
}
98+
99+
gridTest("display properties survive copies, pinning and alterations")(
100+
Seq("InMemoryTableCatalog", "InMemoryRowLevelOperationTableCatalog")) { catalogName =>
101+
val catalog = if (catalogName == "InMemoryRowLevelOperationTableCatalog") {
102+
new InMemoryRowLevelOperationTableCatalog
103+
} else {
104+
new InMemoryTableCatalog
105+
}
106+
// The row-level catalog always copies on load; enable this for InMemoryTableCatalog too.
107+
catalog.initialize("test", new CaseInsensitiveStringMap(util.Map.of("copyOnLoad", "true")))
108+
val table = catalog.createTable(testIdent, new TableInfo.Builder()
109+
.withColumns(columns)
110+
.withProperties(util.Map.of("persisted", "stored"))
111+
.build()).asInstanceOf[InMemoryBaseTable]
112+
table.setDisplayProperties(util.Map.of("catalog-label", "catalog-value"))
113+
table.setVersion("3")
114+
table.setValidatedVersion("2")
115+
val loaded = catalog.loadTable(testIdent).asInstanceOf[InMemoryBaseTable]
116+
assert(loaded ne table)
117+
assert(loaded.displayProperties() === util.Map.of("catalog-label", "catalog-value"))
118+
assert(loaded.version() === "3")
119+
assert(loaded.validatedVersion() === "2")
120+
catalog.pinTable(testIdent, "snapshot")
121+
122+
val altered = catalog.alterTable(testIdent, TableChange.setProperty("persisted", "updated"))
123+
.asInstanceOf[InMemoryBaseTable]
124+
assert(altered.validatedVersion() === "2")
125+
val current = catalog.loadTable(testIdent)
126+
assert(current.properties() === util.Map.of("persisted", "updated"))
127+
assert(current.displayProperties() === util.Map.of("catalog-label", "catalog-value"))
128+
val snapshot = catalog.loadTable(testIdent, "snapshot")
129+
assert(snapshot.properties() === util.Map.of("persisted", "stored"))
130+
assert(snapshot.displayProperties() === util.Map.of("catalog-label", "catalog-value"))
131+
}
132+
80133
test("listTables") {
81134
val catalog = newCatalog()
82135
val ident1 = Identifier.of(Array("ns"), "test_table_1")

‎sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/CatalogV2UtilSuite.scala‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,46 @@ import org.apache.spark.sql.util.CaseInsensitiveStringMap
3535

3636
class CatalogV2UtilSuite extends SparkFunSuite {
3737

38+
test("tablePropertiesForDisplay preserves stored properties with no display properties") {
39+
val properties = java.util.Map.of(
40+
"persisted", "stored", TableCatalog.PROP_EXTERNAL, "true")
41+
42+
assert(CatalogV2Util.tablePropertiesForDisplay(
43+
properties, java.util.Map.of[String, String](), new SQLConf) ===
44+
Map("persisted" -> "stored"))
45+
}
46+
47+
test("tablePropertiesForDisplay ignores reserved display keys and preserves stored values") {
48+
val displayProperties = new java.util.HashMap[String, String]()
49+
CatalogV2Util.TABLE_RESERVED_PROPERTIES.foreach(displayProperties.put(_, "display-value"))
50+
displayProperties.put("persisted", "display-value")
51+
displayProperties.put("catalog-label", "catalog-value")
52+
val properties = java.util.Map.of("persisted", "stored")
53+
54+
assert(CatalogV2Util.tablePropertiesForDisplay(
55+
properties, displayProperties, new SQLConf) ===
56+
Map("persisted" -> "stored", "catalog-label" -> "catalog-value"))
57+
}
58+
59+
test("tablePropertiesForDisplay redacts stored and display-only values") {
60+
val conf = new SQLConf
61+
conf.setConf(SQLConf.SQL_OPTIONS_REDACTION_PATTERN, "(?i)catalog-label".r)
62+
val properties = new java.util.HashMap[String, String](
63+
java.util.Map.of("password", "stored-password", "persisted", "stored"))
64+
val displayProperties = new java.util.HashMap[String, String](java.util.Map.of(
65+
"catalog-label", "catalog-value", "secret", "display-secret", "persisted", "display-value"))
66+
67+
assert(CatalogV2Util.tablePropertiesForDisplay(properties, displayProperties, conf) === Map(
68+
"password" -> "*********(redacted)",
69+
"catalog-label" -> "*********(redacted)",
70+
"secret" -> "*********(redacted)",
71+
"persisted" -> "stored"))
72+
assert(properties === java.util.Map.of(
73+
"password", "stored-password", "persisted", "stored"))
74+
assert(displayProperties === java.util.Map.of(
75+
"catalog-label", "catalog-value", "secret", "display-secret", "persisted", "display-value"))
76+
}
77+
3878
private def catalogWithStateOptions(keys: java.util.Set[String]): TableCatalog = {
3979
val catalog = mock(classOf[TableCatalog])
4080
when(catalog.tableStateOptionKeys()).thenReturn(keys)

0 commit comments

Comments
 (0)