Skip to content

Commit cf2988b

Browse files
laskoviymishkaIsaacmorningman
authored
[feature](fe) Support iceberg REST catalog labels as the $labels table (#69135)
### What problem does this PR solve? Issue Number: #68916 Related PR: #69031 Problem Summary: The Iceberg REST catalog spec now lets a catalog return `labels` with each table load (apache/iceberg#15750): catalog-owned, per-request metadata such as ownership, classification or cost attribution, at the table level (`object-labels`) and per field (keyed by `field-id`). Labels are not part of the table metadata, have no snapshots and are never written to `metadata.json`. Iceberg 1.12.0 exposes them on the loaded table via `SupportsLabels#labels()`. Doris had no way to show them. Add a `t$labels` system table for iceberg tables in a REST catalog: ```sql SELECT * FROM rest_ctl.db.tbl$labels; scope | field_id | field_name | key | value object | NULL | NULL | owner | team-a field | 2 | email | classification | pii ``` One row per label. `field_name` is resolved against the current table schema and is NULL for a field id that is no longer in it (a dropped column). The schema mirrors iceberg's proposed `LabelsTable`. Implementation: - Keep labels when Doris rebuilds a loaded table. iceberg's REST client attaches them to the loaded `BaseTable` as a transient field; they are not table metadata, so a `BaseTable` rebuilt from its `TableOperations` must pass them explicitly. `IcebergTableCache.TableLease#snapshotReadTable` (cached reads), `IcebergStatementScope#snapshotReadTable` (statement reads) and `IcebergScanPlanProvider#wrapTableForScan` (Kerberos FileIO wrap) used constructors that reset labels to `Labels.EMPTY`, so every read saw no labels; they now pass `baseTable.labels()`. - `labels` is not a `MetadataTableType`, so it is handled next to the SDK metadata tables rather than through `MetadataTableUtils`. `IcebergConnectorMetadata` lists and resolves it only when the catalog flavor is `rest` (no other catalog returns labels), and serves its fixed schema and column handles without loading the base table. - `IcebergScanPlanProvider` plans it as one split. A new split-package helper `org.apache.iceberg.IcebergLabelsDataTask` builds an iceberg `StaticDataTask` (package-private in the SDK) holding the label rows, projected to the requested columns in BE's required-field order; its metadata `DataFile` is synthesized, so no storage request is made. The task is serialized like every other static metadata table, so BE's `IcebergSysTableJniScanner` reads it with no BE change. - The ordered system-table projection is extracted into `projectSystemTableSchema` and shared with the existing path, and the base-table load is split out of `resolveSysTable` as `resolveSysBaseTable`. - Labels are not versioned: like `$snapshots`/`$history`, the snapshot pin of the handle is ignored, and `$labels` is exempt from the resolved-empty-snapshot fence, so a table with no commit still shows its labels. Labels reflect the table load the statement used, so they follow the iceberg table cache (`meta.cache.iceberg.table.ttl-second`); REFRESH TABLE picks up new labels. Like the other system tables, a REST catalog that requires server-side scan planning rejects `$labels`. Co-authored-by: Isaac <[email protected]> Co-authored-by: morningman <[email protected]>
1 parent 122a7db commit cf2988b

13 files changed

Lines changed: 801 additions & 31 deletions

File tree

‎fe/check/checkstyle/suppressions.xml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,8 @@ under the License.
7979
<suppress files="org[\\/]apache[\\/]iceberg[\\/]DeleteFileIndex\.java" checks="[a-zA-Z0-9]*"/>
8080
<!-- Iceberg package access is required to preserve historical schema binding in DataTableScan. -->
8181
<suppress files="org[\\/]apache[\\/]iceberg[\\/]SchemaAwareDataTableScan\.java" checks="[a-zA-Z0-9]*"/>
82+
<!-- Iceberg package access is required to build a StaticDataTask for REST catalog labels; other checks apply. -->
83+
<suppress files="org[\\/]apache[\\/]iceberg[\\/]IcebergLabelsDataTask\.java" checks="ImportControl"/>
8284

8385
<!-- ignore gensrc/thrift/ExternalTableSchema.thrift -->
8486
<suppress files=".*thrift/schema/external/.*" checks=".*"/>

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java‎

Lines changed: 26 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -533,6 +533,12 @@ public ConnectorTableSchema getTableSchema(
533533
// parse ITS schema (e.g. t$snapshots -> committed_at/snapshot_id/...). Mirrors legacy
534534
// IcebergSysExternalTable.getSysIcebergTable + getOrCreateSchemaCacheValue; the enable.mapping.*
535535
// flags are threaded by the shared buildTableSchema -> parseSchema (deviation 5).
536+
if (IcebergLabelsSysTable.isLabels(iceHandle)) {
537+
// $labels is not an iceberg metadata table: its schema is fixed and needs no table load (the
538+
// base table's existence was already checked by getTableHandle).
539+
return new ConnectorTableSchema(iceHandle.getTableName(),
540+
parseSchema(IcebergLabelsSysTable.SCHEMA), "ICEBERG", Collections.emptyMap());
541+
}
536542
Table sysTable = loadSysTable(session, iceHandle);
537543
return buildTableSchema(iceHandle.getTableName(), sysTable, sysTable.schema(), false);
538544
}
@@ -833,6 +839,9 @@ public Map<String, ConnectorColumnHandle> getColumnHandles(
833839
// Mirror getTableSchema: wrap the remote load in the auth context. A sys handle resolves the
834840
// metadata-table columns (t$snapshots -> committed_at/...) so the generic scan node can look up
835841
// its pruned sys-table slots by name; a data handle resolves the base table's columns.
842+
if (IcebergLabelsSysTable.isLabels(iceHandle)) {
843+
return buildColumnHandles(IcebergLabelsSysTable.SCHEMA, false);
844+
}
836845
Table table = iceHandle.isSystemTable() ? loadSysTable(session, iceHandle) : loadTable(session, iceHandle);
837846
return buildColumnHandles(table.schema(), !iceHandle.isSystemTable());
838847
}
@@ -1919,6 +1928,9 @@ private boolean isHmsCatalog() {
19191928
* excluding it here would be a capability regression vs master. Unlike every other entry, its scan takes
19201929
* BE's native reader rather than the JNI serialized-split path (see
19211930
* {@code IcebergScanPlanProvider.doPlanPositionDeletesSystemTableScan}).
1931+
*
1932+
* <p>A REST catalog additionally exposes {@code labels} ({@link IcebergLabelsSysTable}), which is not a
1933+
* {@code MetadataTableType}: its rows are the catalog labels returned with the table load.
19221934
*/
19231935
@Override
19241936
public List<String> listSupportedSysTables(ConnectorSession session,
@@ -1927,6 +1939,9 @@ public List<String> listSupportedSysTables(ConnectorSession session,
19271939
for (MetadataTableType type : MetadataTableType.values()) {
19281940
names.add(type.name().toLowerCase(Locale.ROOT));
19291941
}
1942+
if (supportsLabels()) {
1943+
names.add(IcebergLabelsSysTable.NAME);
1944+
}
19301945
return Collections.unmodifiableList(names);
19311946
}
19321947

@@ -1969,13 +1984,16 @@ public Optional<ConnectorTableHandle> getSysTableHandle(ConnectorSession session
19691984
/**
19701985
* Whether iceberg exposes a system table named {@code sysName} (case-insensitive). Mirrors
19711986
* {@code IcebergSysTable.SUPPORTED_SYS_TABLES}: every {@code MetadataTableType}, {@code POSITION_DELETES}
1972-
* included (see {@link #listSupportedSysTables}). A {@code null} name is simply not exposed (returns
1973-
* false, not NPE).
1987+
* included (see {@link #listSupportedSysTables}), plus {@code labels} for a REST catalog. A {@code null}
1988+
* name is simply not exposed (returns false, not NPE).
19741989
*/
1975-
private static boolean isSupportedSysTable(String sysName) {
1990+
private boolean isSupportedSysTable(String sysName) {
19761991
if (sysName == null) {
19771992
return false;
19781993
}
1994+
if (IcebergLabelsSysTable.NAME.equalsIgnoreCase(sysName)) {
1995+
return supportsLabels();
1996+
}
19791997
for (MetadataTableType type : MetadataTableType.values()) {
19801998
if (type.name().equalsIgnoreCase(sysName)) {
19811999
return true;
@@ -1984,6 +2002,11 @@ private static boolean isSupportedSysTable(String sysName) {
19842002
return false;
19852003
}
19862004

2005+
/** Only a REST catalog returns catalog labels with the table load, so only it exposes {@code $labels}. */
2006+
private boolean supportsLabels() {
2007+
return IcebergCatalogProperties.TYPE_REST.equals(catalogProps.getFlavor());
2008+
}
2009+
19872010
// ========== Write / Transaction (P6.3) ==========
19882011

19892012
/**
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.connector.iceberg;
19+
20+
import org.apache.iceberg.Schema;
21+
import org.apache.iceberg.types.Types;
22+
23+
/**
24+
* The {@code t$labels} system table: the catalog-provided labels of an iceberg table loaded from a REST catalog
25+
* (apache/iceberg#15750), one row per object-level or field-level label.
26+
*
27+
* <p>Unlike every other iceberg system table it is not a {@link org.apache.iceberg.MetadataTableType}: labels
28+
* come from the REST load response, not from the table metadata, are not versioned with snapshots, and only a
29+
* REST catalog returns them. Rows reflect the table load the statement read, so they follow the iceberg table
30+
* cache like the rest of the table's state. The schema mirrors iceberg's proposed {@code LabelsTable}.
31+
*/
32+
final class IcebergLabelsSysTable {
33+
static final String NAME = "labels";
34+
35+
static final Schema SCHEMA = new Schema(
36+
Types.NestedField.required(1, "scope", Types.StringType.get()),
37+
Types.NestedField.optional(2, "field_id", Types.IntegerType.get()),
38+
Types.NestedField.optional(3, "field_name", Types.StringType.get()),
39+
Types.NestedField.required(4, "key", Types.StringType.get()),
40+
Types.NestedField.required(5, "value", Types.StringType.get()));
41+
42+
private IcebergLabelsSysTable() {
43+
}
44+
45+
/** Whether {@code handle} is a {@code $labels} handle; sys handles carry the lower-cased name. */
46+
static boolean isLabels(IcebergTableHandle handle) {
47+
return NAME.equals(handle.getSysTableName());
48+
}
49+
}

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java‎

Lines changed: 75 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -47,12 +47,14 @@
4747
import org.apache.iceberg.BatchScan;
4848
import org.apache.iceberg.ContentScanTask;
4949
import org.apache.iceberg.DataFile;
50+
import org.apache.iceberg.DataTask;
5051
import org.apache.iceberg.DeleteFile;
5152
import org.apache.iceberg.DeleteFileIndex;
5253
import org.apache.iceberg.FileContent;
5354
import org.apache.iceberg.FileFormat;
5455
import org.apache.iceberg.FileScanTask;
5556
import org.apache.iceberg.HasTableOperations;
57+
import org.apache.iceberg.IcebergLabelsDataTask;
5658
import org.apache.iceberg.ManifestContent;
5759
import org.apache.iceberg.ManifestFile;
5860
import org.apache.iceberg.ManifestFiles;
@@ -977,6 +979,9 @@ private List<ConnectorScanRange> planSystemTableScan(IcebergTableHandle handle,
977979
private List<ConnectorScanRange> doPlanSystemTableScan(IcebergTableHandle handle,
978980
List<ConnectorColumnHandle> columns, Optional<ConnectorExpression> filter,
979981
ConnectorSession session) {
982+
if (IcebergLabelsSysTable.isLabels(handle)) {
983+
return doPlanLabelsSystemTableScan(handle, columns, session);
984+
}
980985
Table metadataTable = resolveSysTable(session, handle);
981986
if (isPositionDeletesSysTable(handle)) {
982987
return doPlanPositionDeletesSystemTableScan(handle, metadataTable, columns, filter, session);
@@ -998,27 +1003,8 @@ private List<ConnectorScanRange> doPlanSystemTableScan(IcebergTableHandle handle
9981003
// hand from the requested columns.
9991004
List<String> projectedColumns = requestedColumnNames(columns);
10001005
if (!projectedColumns.isEmpty()) {
1001-
Schema metadataSchema = metadataTable.schema();
1002-
List<NestedField> projectedFields = new ArrayList<>(projectedColumns.size());
1003-
Set<Integer> seenFieldIds = new HashSet<>();
1004-
for (String columnName : projectedColumns) {
1005-
NestedField field = metadataSchema.findField(columnName);
1006-
if (field == null) {
1007-
throw new RuntimeException("Column " + columnName
1008-
+ " not found in iceberg system table schema " + metadataTable.name());
1009-
}
1010-
if (seenFieldIds.add(field.fieldId())) {
1011-
projectedFields.add(field);
1012-
}
1013-
}
1014-
// Iceberg returns split_offsets only when the last offset is smaller than file_size_in_bytes.
1015-
// Without that dependency, even valid offsets such as [4, 2048] are hidden as null. Append it
1016-
// after the requested columns so BE's positional reads keep their original order.
1017-
if (seenFieldIds.contains(DataFile.SPLIT_OFFSETS.fieldId())
1018-
&& seenFieldIds.add(DataFile.FILE_SIZE.fieldId())) {
1019-
projectedFields.add(metadataSchema.findField(DataFile.FILE_SIZE.fieldId()));
1020-
}
1021-
scan = scan.project(new Schema(projectedFields));
1006+
scan = scan.project(projectSystemTableSchema(
1007+
metadataTable.schema(), projectedColumns, metadataTable.name()));
10221008
}
10231009
List<ConnectorScanRange> ranges = new ArrayList<>();
10241010
try (CloseableIterable<FileScanTask> tasks = scan.planFiles()) {
@@ -1043,6 +1029,58 @@ private static boolean isPositionDeletesSysTable(IcebergTableHandle handle) {
10431029
&& MetadataTableType.POSITION_DELETES.name().equalsIgnoreCase(handle.getSysTableName());
10441030
}
10451031

1032+
/**
1033+
* The system-table schema projected to the requested columns, in the order BE lists them in
1034+
* required_fields: BE's {@code IcebergSysTableJniScanner} reads projected rows positionally. Built by hand
1035+
* because {@code scan.select(names)} would re-emit the fields in table-schema order. Any dependent metadata
1036+
* fields are appended after the requested columns.
1037+
*/
1038+
private static Schema projectSystemTableSchema(Schema schema, List<String> projectedColumns,
1039+
String tableName) {
1040+
List<NestedField> projectedFields = new ArrayList<>(projectedColumns.size());
1041+
Set<Integer> seenFieldIds = new HashSet<>();
1042+
for (String columnName : projectedColumns) {
1043+
NestedField field = schema.findField(columnName);
1044+
if (field == null) {
1045+
throw new RuntimeException("Column " + columnName
1046+
+ " not found in iceberg system table schema " + tableName);
1047+
}
1048+
if (seenFieldIds.add(field.fieldId())) {
1049+
projectedFields.add(field);
1050+
}
1051+
}
1052+
// Iceberg returns split_offsets only when the last offset is smaller than file_size_in_bytes.
1053+
// Without that dependency, even valid offsets such as [4, 2048] are hidden as null. Append it
1054+
// after the requested columns so BE's positional reads keep their original order.
1055+
if (seenFieldIds.contains(DataFile.SPLIT_OFFSETS.fieldId())
1056+
&& seenFieldIds.add(DataFile.FILE_SIZE.fieldId())) {
1057+
projectedFields.add(schema.findField(DataFile.FILE_SIZE.fieldId()));
1058+
}
1059+
return new Schema(projectedFields);
1060+
}
1061+
1062+
/**
1063+
* Plan {@code t$labels}: one serialized split whose {@code StaticDataTask} carries every catalog label of the
1064+
* base table, read by BE's JNI scanner like any other static metadata table. Labels are not versioned, so
1065+
* the handle's snapshot pin is ignored (as for {@code $snapshots}/{@code $history}); predicates are left to
1066+
* the scan node's conjuncts, the row count being bounded by the catalog response.
1067+
*/
1068+
private List<ConnectorScanRange> doPlanLabelsSystemTableScan(IcebergTableHandle handle,
1069+
List<ConnectorColumnHandle> columns, ConnectorSession session) {
1070+
Table base = resolveSysBaseTable(session, handle);
1071+
List<String> projectedColumns = requestedColumnNames(columns);
1072+
Schema projectedSchema = projectedColumns.isEmpty()
1073+
? IcebergLabelsSysTable.SCHEMA
1074+
: projectSystemTableSchema(IcebergLabelsSysTable.SCHEMA, projectedColumns,
1075+
base.name() + "$" + IcebergLabelsSysTable.NAME);
1076+
DataTask task = IcebergLabelsDataTask.of(
1077+
base, IcebergLabelsSysTable.SCHEMA, projectedSchema, SYS_TABLE_DUMMY_PATH);
1078+
return Collections.singletonList(new IcebergScanRange.Builder()
1079+
.path(SYS_TABLE_DUMMY_PATH)
1080+
.serializedSplit(IcebergSystemTableSerialization.serializeToBase64(task))
1081+
.build());
1082+
}
1083+
10461084
/**
10471085
* Plan a {@code $position_deletes} scan. Port of legacy
10481086
* {@code IcebergScanNode.doGetPositionDeletesSystemTableSplits} (upstream #65135).
@@ -1328,6 +1366,10 @@ private static boolean isSnapshotIndependentSystemTable(IcebergTableHandle handl
13281366
if (!handle.isSystemTable()) {
13291367
return false;
13301368
}
1369+
if (IcebergLabelsSysTable.isLabels(handle)) {
1370+
// Catalog labels come with the table load, not from a snapshot, so a table with no commit has them.
1371+
return true;
1372+
}
13311373
MetadataTableType type = MetadataTableType.from(handle.getSysTableName());
13321374
// Only metadata_log_entries has a creation row before the first data snapshot; every other metadata
13331375
// table must preserve the resolved-empty fence or a concurrent first append becomes visible.
@@ -3398,9 +3440,11 @@ Table wrapTableForScan(Table table) {
33983440
}
33993441
BaseTable baseTable = (BaseTable) table;
34003442
TableOperations rawOps = baseTable.operations();
3401-
// Keep the table's metrics reporter: every scan of a Kerberos catalog runs on this wrapper.
3443+
// Keep the table's metrics reporter and REST catalog labels: every scan of a Kerberos catalog runs on this
3444+
// wrapper.
34023445
return new BaseTable(new IcebergAuthenticatedTableOperations(
3403-
rawOps, new IcebergAuthenticatedFileIO(rawOps.io(), auth)), table.name(), baseTable.reporter());
3446+
rawOps, new IcebergAuthenticatedFileIO(rawOps.io(), auth)), table.name(), baseTable.reporter(),
3447+
baseTable.labels());
34043448
}
34053449

34063450
/**
@@ -3413,6 +3457,13 @@ Table wrapTableForScan(Table table) {
34133457
* task serialization) in ONE scope — no nested wrap here.
34143458
*/
34153459
private Table resolveSysTable(ConnectorSession session, IcebergTableHandle handle) {
3460+
return MetadataTableUtils.createMetadataTableInstance(
3461+
resolveSysBaseTable(session, handle),
3462+
MetadataTableType.from(handle.getSysTableName()));
3463+
}
3464+
3465+
/** The base table a system-table scan reads, shared with metadata binding; see {@link #resolveSysTable}. */
3466+
private Table resolveSysBaseTable(ConnectorSession session, IcebergTableHandle handle) {
34163467
IcebergCatalogOps ops = catalogOpsResolver.apply(session);
34173468
// Keep the raw base shared with metadata binding and ordinary scan properties. The caller already owns
34183469
// the auth scope, and avoiding a fresh load prevents system-table slots and rows crossing generations.
@@ -3428,9 +3479,7 @@ private Table resolveSysTable(ConnectorSession session, IcebergTableHandle handl
34283479
() -> ops.loadTable(handle.getDbName(), handle.getTableName())),
34293480
() -> ops.loadTable(handle.getDbName(), handle.getTableName()));
34303481
rejectServerSideScanPlanning(base, handle);
3431-
return MetadataTableUtils.createMetadataTableInstance(
3432-
base,
3433-
MetadataTableType.from(handle.getSysTableName()));
3482+
return base;
34343483
}
34353484

34363485
/** This catalog's engine-owned storage services (see {@link ConnectorContext#getStorageContext()}). */

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergStatementScope.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -292,7 +292,7 @@ private static Table snapshotReadTable(Table table, String dbName, String tableN
292292
TableMetadata metadata = operations.current();
293293
// Keep IO, encryption and location behavior from the raw table while pinning only metadata.
294294
return new BaseTable(new IcebergSnapshotTableOperations(operations, metadata),
295-
table.name(), baseTable.reporter());
295+
table.name(), baseTable.reporter(), baseTable.labels());
296296
}
297297

298298
/**

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableCache.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -314,7 +314,7 @@ private Table snapshotReadTable() {
314314
? operations.current()
315315
: TableMetadataParser.fromJson(snapshotMetadataLocation, snapshotMetadataJson);
316316
return new BaseTable(new IcebergSnapshotTableOperations(operations, metadata),
317-
table.name(), baseTable.reporter());
317+
table.name(), baseTable.reporter(), baseTable.labels());
318318
}
319319

320320
private TableLease tryBorrow() {

0 commit comments

Comments
 (0)