You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit cf2988b
Browse filesBrowse the repository at this point in the historyBrowse files
authored
[feature](fe) Support iceberg REST catalog labels as the $labels table (#69135)
### What problem does this PR solve?
Issue Number: #68916
Related PR: #69031
Problem Summary: The Iceberg REST catalog spec now lets a catalog return
`labels` with each table load (apache/iceberg#15750): catalog-owned,
per-request metadata such as ownership, classification or cost
attribution, at the table level (`object-labels`) and per field (keyed
by `field-id`). Labels are not part of the table metadata, have no
snapshots and are never written to `metadata.json`. Iceberg 1.12.0
exposes them on the loaded table via `SupportsLabels#labels()`. Doris
had no way to show them.
Add a `t$labels` system table for iceberg tables in a REST catalog:
```sql
SELECT * FROM rest_ctl.db.tbl$labels;
scope | field_id | field_name | key | value
object | NULL | NULL | owner | team-a
field | 2 | email | classification | pii
```
One row per label. `field_name` is resolved against the current table
schema and is NULL for a field id that is no longer in it (a dropped
column). The schema mirrors iceberg's proposed `LabelsTable`.
Implementation:
- Keep labels when Doris rebuilds a loaded table. iceberg's REST client
attaches them to the loaded `BaseTable` as a transient field; they are
not table metadata, so a `BaseTable` rebuilt from its `TableOperations`
must pass them explicitly.
`IcebergTableCache.TableLease#snapshotReadTable` (cached reads),
`IcebergStatementScope#snapshotReadTable` (statement reads) and
`IcebergScanPlanProvider#wrapTableForScan` (Kerberos FileIO wrap) used
constructors that reset labels to `Labels.EMPTY`, so every read saw no
labels; they now pass `baseTable.labels()`.
- `labels` is not a `MetadataTableType`, so it is handled next to the
SDK metadata tables rather than through `MetadataTableUtils`.
`IcebergConnectorMetadata` lists and resolves it only when the catalog
flavor is `rest` (no other catalog returns labels), and serves its fixed
schema and column handles without loading the base table.
- `IcebergScanPlanProvider` plans it as one split. A new split-package
helper `org.apache.iceberg.IcebergLabelsDataTask` builds an iceberg
`StaticDataTask` (package-private in the SDK) holding the label rows,
projected to the requested columns in BE's required-field order; its
metadata `DataFile` is synthesized, so no storage request is made. The
task is serialized like every other static metadata table, so BE's
`IcebergSysTableJniScanner` reads it with no BE change.
- The ordered system-table projection is extracted into
`projectSystemTableSchema` and shared with the existing path, and the
base-table load is split out of `resolveSysTable` as
`resolveSysBaseTable`.
- Labels are not versioned: like `$snapshots`/`$history`, the snapshot
pin of the handle is ignored, and `$labels` is exempt from the
resolved-empty-snapshot fence, so a table with no commit still shows its
labels.
Labels reflect the table load the statement used, so they follow the
iceberg table cache (`meta.cache.iceberg.table.ttl-second`); REFRESH
TABLE picks up new labels. Like the other system tables, a REST catalog
that requires server-side scan planning rejects `$labels`.
Co-authored-by: Isaac <[email protected]>
Co-authored-by: morningman <[email protected]>
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java
+26-3Lines changed: 26 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -533,6 +533,12 @@ public ConnectorTableSchema getTableSchema(
533
533
// parse ITS schema (e.g. t$snapshots -> committed_at/snapshot_id/...). Mirrors legacy
534
534
// IcebergSysExternalTable.getSysIcebergTable + getOrCreateSchemaCacheValue; the enable.mapping.*
535
535
// flags are threaded by the shared buildTableSchema -> parseSchema (deviation 5).
536
+
if (IcebergLabelsSysTable.isLabels(iceHandle)) {
537
+
// $labels is not an iceberg metadata table: its schema is fixed and needs no table load (the
538
+
// base table's existence was already checked by getTableHandle).
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergStatementScope.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableCache.java
0 commit comments