Repository navigation
Spark 4.1: expose catalog labels via DESCRIBE - #18049
laskoviymishka wants to merge 2 commits into
Conversation
e393141 to
5c43066
Compare
5c43066 to
7e89da2
Compare
f985596 to
3067801
Compare
| Labels labels = ((SupportsLabels) table).labels(); | ||
| labels | ||
| .objectLabels() | ||
| .forEach((key, value) -> propsBuilder.put(LABELS_OBJECT_PREFIX + key, value)); |
There was a problem hiding this comment.
Please handle collisions with stored table properties here. A table can already have a property such as labels.object.owner; if the catalog also supplies the owner label, both entries are added to the ImmutableMap.Builder, and build() throws, breaking DESCRIBE EXTENDED and SHOW TBLPROPERTIES. The field-label prefix has the same issue. Could we define precedence, filter conflicts, and add coverage for both prefixes?
There was a problem hiding this comment.
thx @szehon-ho, good catch, you're right that feeding both into one ImmutableMap.Builder throws and breaks DESCRIBE EXTENDED / SHOW TBLPROPERTIES.
My thinking: labels and table properties are genuinely different concepts, so ideally I'd surface catalog labels as their own section in DESCRIBE, separate from Table Properties. But DescribeTableExec doesn't allow it - the sections (schema, partitioning, metadata columns, detailed info, stats) are hardcoded, each tied to a specific SPI method, and properties() is the only channel a v2 connector has for arbitrary key-values (they all land in the single Table Properties blob). A dedicated "Labels" field in DESCRIBE isn't reachable from the connector side.
So as the pragmatic fallback I went with de-duplication, so neither is lost and DESCRIBE never fails:
- a catalog label keeps its natural labels.object. / labels.field.. key when it's free;
- if a stored property already holds that key, the label is surfaced under a de-conflicted ….catalog key (numeric suffix if that's also taken);
- the stored property always keeps its own key.
Both stay visible and build() can't throw. Added coverage for the object and field prefixes.
For a properly separated, structured view, the queryable surface is the tbl.labels metadata table (scope / field_id / field_name / key / value), that's the authoritative place; the DESCRIBE properties are just at-a-glance visibility.
7e89da2 to
a812269
Compare
c2f21e0 to
7d4bf56
Compare
cb995c5 to
e2c01b6
Compare
f0a6a5b to
c59e1b2
Compare
e2c01b6 to
ef7aeda
Compare
c59e1b2 to
3cb3e3b
Compare
ef7aeda to
769261b
Compare
3cb3e3b to
7b02ad2
Compare
3e50307 to
1168368
Compare
|
Thanks for addressing the collision issue. Looking at this again, I’m working on fixing this on the Spark side by separating display-only properties: apache/spark#59036. If accepted, Iceberg can expose labels through that API for supported Spark versions without affecting generated DDL. |
1168368 to
d31b289
Compare
|
Extending the formatted DESCRIBE output of Spark tables is always a bit hard because it might break assumptions on how the content is parsed and unfortunately users aren't always doing a great job of dealing with new rows being added. Instead of appending them as table properties directly, is there a potentially better way of plumbing the output through to the DESCRIBE table AS JSON output? So that we could add the labels in a structured way? |
|
@grundprinzip interesting topic. I think @szehon-ho is already building the right Spark-side hook here: apache/spark#59036 adds That's probably the right thing to build the Iceberg side on. It would still show up in On the Iceberg side I'll hold this for now and rework it onto |
|
Yea the idea is to show them in DESCRIBE EXTENDED (not DESCRIBE). Would that also break users in your view? I think having this describe in another row in DESCRIBE AS JSON is a great idea too. |
|
@grundprinzip any thoughts on us proceeding with this approach? |
Surface catalog-provided labels as reserved table properties on the Spark table so they appear in DESCRIBE EXTENDED. The labels metadata table remains the queryable counterpart.
Catalog labels and stored table properties can compute to the same key (e.g. a stored `labels.object.owner` property and an `owner` object label). Feeding both into one ImmutableMap.Builder threw on the duplicate key, breaking DESCRIBE EXTENDED / SHOW TBLPROPERTIES. Accumulate into a mutable map and surface a colliding label under a de-conflicted `.catalog` key, so the property and the label both stay visible. Adds coverage for the object and field prefixes. Co-authored-by: Isaac <[email protected]>
d31b289 to
dfc44fd
Compare
|
Will wait for spark 4.4 support |
Part of the catalog-labels work (spec: #15750).
Surfaces catalog-provided labels as table properties on the Spark table so they appear in
DESCRIBE EXTENDED: object labels aslabels.object.<key>, field labels aslabels.field.<field-id>.<key>. If a label key collides with a stored table property, the label is shown under a de-conflicted key so both stay visible.