Skip to content

Commit 41f37f5

Browse files
laskoviymishkaIsaac
andcommitted
[feature](fe) Support iceberg REST catalog labels as the $labels system table
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: The Iceberg REST catalog spec now lets a catalog return `labels` with each table load (apache/iceberg#15750): catalog-owned, per-request metadata such as ownership, classification or cost attribution, at the table level (`object-labels`) and per field (keyed by `field-id`). Labels are not part of the table metadata, have no snapshots and are never written to `metadata.json`. Iceberg 1.12.0 exposes them on the loaded table via `SupportsLabels#labels()`. Doris had no way to show them. Add a `t$labels` system table for iceberg tables in a REST catalog: SELECT * FROM rest_ctl.db.tbl$labels; scope | field_id | field_name | key | value object | NULL | NULL | owner | team-a field | 2 | email | classification | pii One row per label. `field_name` is resolved against the current table schema and is NULL for a field id that is no longer in it (a dropped column). The schema mirrors iceberg's proposed `LabelsTable`. Implementation: - Keep labels when Doris rebuilds a loaded table. iceberg's REST client attaches them to the loaded `BaseTable` as a transient field; they are not table metadata, so a `BaseTable` rebuilt from its `TableOperations` must pass them explicitly. `IcebergTableCache.TableLease#snapshotReadTable` (cached reads), `IcebergStatementScope#snapshotReadTable` (statement reads) and `IcebergScanPlanProvider#wrapTableForScan` (Kerberos FileIO wrap) used constructors that reset labels to `Labels.EMPTY`, so every read saw no labels; they now pass `baseTable.labels()`. - `labels` is not a `MetadataTableType`, so it is handled next to the SDK metadata tables rather than through `MetadataTableUtils`. `IcebergConnectorMetadata` lists and resolves it only when the catalog flavor is `rest` (no other catalog returns labels), and serves its fixed schema and column handles without loading the base table. - `IcebergScanPlanProvider` plans it as one split. A new split-package helper `org.apache.iceberg.IcebergLabelsDataTask` builds an iceberg `StaticDataTask` (package-private in the SDK) holding the label rows, projected to the requested columns in BE's required-field order; its metadata `DataFile` is synthesized, so no storage request is made. The task is serialized like every other static metadata table, so BE's `IcebergSysTableJniScanner` reads it with no BE change. - The ordered system-table projection is extracted into `projectSystemTableSchema` and shared with the existing path, and the base-table load is split out of `resolveSysTable` as `resolveSysBaseTable`. - Labels are not versioned: like `$snapshots`/`$history`, the snapshot pin of the handle is ignored, and `$labels` is exempt from the resolved-empty-snapshot fence, so a table with no commit still shows its labels. Labels reflect the table load the statement used, so they follow the iceberg table cache (`meta.cache.iceberg.table.ttl-second`); REFRESH TABLE picks up new labels. Like the other system tables, a REST catalog that requires server-side scan planning rejects `$labels`. ### Release note Support querying REST catalog labels of an iceberg table through the `tbl$labels` system table. ### Check List (For Author) - Test: Unit Test - Added IcebergLabelsDataTaskTest, the $labels cases in IcebergConnectorMetadataSysTableTest, IcebergTableCacheTest#snapshotReadTableKeepsCatalogLabels and IcebergStatementScopeTest#sharedTableKeepsCatalogLabels. - `build.sh --fe` (Maven build cache disabled) passes on top of current master. Ran every iceberg and hudi FE unit test with run-fe-ut.sh: 1990 tests, 0 failures, 5 skipped. - Regression test pending a REST fixture that returns labels (apache/iceberg#18047). - Behavior changed: No (adds a new system table for REST catalogs only) - Does this need documentation: Yes (doc PR to follow) Co-authored-by: Isaac <[email protected]>
1 parent 091925a commit 41f37f5

11 files changed

Lines changed: 463 additions & 24 deletions

File tree

‎fe/check/checkstyle/suppressions.xml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,8 @@ under the License.
7979
<suppress files="org[\\/]apache[\\/]iceberg[\\/]DeleteFileIndex\.java" checks="[a-zA-Z0-9]*"/>
8080
<!-- Iceberg package access is required to preserve historical schema binding in DataTableScan. -->
8181
<suppress files="org[\\/]apache[\\/]iceberg[\\/]SchemaAwareDataTableScan\.java" checks="[a-zA-Z0-9]*"/>
82+
<!-- Iceberg package access is required to build a StaticDataTask for REST catalog labels; other checks apply. -->
83+
<suppress files="org[\\/]apache[\\/]iceberg[\\/]IcebergLabelsDataTask\.java" checks="ImportControl"/>
8284

8385
<!-- ignore gensrc/thrift/ExternalTableSchema.thrift -->
8486
<suppress files=".*thrift/schema/external/.*" checks=".*"/>

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java‎

Lines changed: 26 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -452,6 +452,12 @@ public ConnectorTableSchema getTableSchema(
452452
// parse ITS schema (e.g. t$snapshots -> committed_at/snapshot_id/...). Mirrors legacy
453453
// IcebergSysExternalTable.getSysIcebergTable + getOrCreateSchemaCacheValue; the enable.mapping.*
454454
// flags are threaded by the shared buildTableSchema -> parseSchema (deviation 5).
455+
if (IcebergLabelsSysTable.isLabels(iceHandle)) {
456+
// $labels is not an iceberg metadata table: its schema is fixed and needs no table load (the
457+
// base table's existence was already checked by getTableHandle).
458+
return new ConnectorTableSchema(iceHandle.getTableName(),
459+
parseSchema(IcebergLabelsSysTable.SCHEMA), "ICEBERG", Collections.emptyMap());
460+
}
455461
Table sysTable = loadSysTable(session, iceHandle);
456462
return buildTableSchema(iceHandle.getTableName(), sysTable, sysTable.schema(), false);
457463
}
@@ -752,6 +758,9 @@ public Map<String, ConnectorColumnHandle> getColumnHandles(
752758
// Mirror getTableSchema: wrap the remote load in the auth context. A sys handle resolves the
753759
// metadata-table columns (t$snapshots -> committed_at/...) so the generic scan node can look up
754760
// its pruned sys-table slots by name; a data handle resolves the base table's columns.
761+
if (IcebergLabelsSysTable.isLabels(iceHandle)) {
762+
return buildColumnHandles(IcebergLabelsSysTable.SCHEMA, false);
763+
}
755764
Table table = iceHandle.isSystemTable() ? loadSysTable(session, iceHandle) : loadTable(session, iceHandle);
756765
return buildColumnHandles(table.schema(), !iceHandle.isSystemTable());
757766
}
@@ -1761,6 +1770,9 @@ private boolean isHmsCatalog() {
17611770
* excluding it here would be a capability regression vs master. Unlike every other entry, its scan takes
17621771
* BE's native reader rather than the JNI serialized-split path (see
17631772
* {@code IcebergScanPlanProvider.doPlanPositionDeletesSystemTableScan}).
1773+
*
1774+
* <p>A REST catalog additionally exposes {@code labels} ({@link IcebergLabelsSysTable}), which is not a
1775+
* {@code MetadataTableType}: its rows are the catalog labels returned with the table load.
17641776
*/
17651777
@Override
17661778
public List<String> listSupportedSysTables(ConnectorSession session,
@@ -1769,6 +1781,9 @@ public List<String> listSupportedSysTables(ConnectorSession session,
17691781
for (MetadataTableType type : MetadataTableType.values()) {
17701782
names.add(type.name().toLowerCase(Locale.ROOT));
17711783
}
1784+
if (supportsLabels()) {
1785+
names.add(IcebergLabelsSysTable.NAME);
1786+
}
17721787
return Collections.unmodifiableList(names);
17731788
}
17741789

@@ -1811,13 +1826,16 @@ public Optional<ConnectorTableHandle> getSysTableHandle(ConnectorSession session
18111826
/**
18121827
* Whether iceberg exposes a system table named {@code sysName} (case-insensitive). Mirrors
18131828
* {@code IcebergSysTable.SUPPORTED_SYS_TABLES}: every {@code MetadataTableType}, {@code POSITION_DELETES}
1814-
* included (see {@link #listSupportedSysTables}). A {@code null} name is simply not exposed (returns
1815-
* false, not NPE).
1829+
* included (see {@link #listSupportedSysTables}), plus {@code labels} for a REST catalog. A {@code null}
1830+
* name is simply not exposed (returns false, not NPE).
18161831
*/
1817-
private static boolean isSupportedSysTable(String sysName) {
1832+
private boolean isSupportedSysTable(String sysName) {
18181833
if (sysName == null) {
18191834
return false;
18201835
}
1836+
if (IcebergLabelsSysTable.NAME.equalsIgnoreCase(sysName)) {
1837+
return supportsLabels();
1838+
}
18211839
for (MetadataTableType type : MetadataTableType.values()) {
18221840
if (type.name().equalsIgnoreCase(sysName)) {
18231841
return true;
@@ -1826,6 +1844,11 @@ private static boolean isSupportedSysTable(String sysName) {
18261844
return false;
18271845
}
18281846

1847+
/** Only a REST catalog returns catalog labels with the table load, so only it exposes {@code $labels}. */
1848+
private boolean supportsLabels() {
1849+
return IcebergCatalogProperties.TYPE_REST.equals(catalogProps.getFlavor());
1850+
}
1851+
18291852
// ========== Write / Transaction (P6.3) ==========
18301853

18311854
/**
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.connector.iceberg;
19+
20+
import org.apache.iceberg.Schema;
21+
import org.apache.iceberg.types.Types;
22+
23+
/**
24+
* The {@code t$labels} system table: the catalog-provided labels of an iceberg table loaded from a REST catalog
25+
* (apache/iceberg#15750), one row per object-level or field-level label.
26+
*
27+
* <p>Unlike every other iceberg system table it is not a {@link org.apache.iceberg.MetadataTableType}: labels
28+
* come from the REST load response, not from the table metadata, are not versioned with snapshots, and only a
29+
* REST catalog returns them. Rows reflect the table load the statement read, so they follow the iceberg table
30+
* cache like the rest of the table's state. The schema mirrors iceberg's proposed {@code LabelsTable}.
31+
*/
32+
final class IcebergLabelsSysTable {
33+
static final String NAME = "labels";
34+
35+
static final Schema SCHEMA = new Schema(
36+
Types.NestedField.required(1, "scope", Types.StringType.get()),
37+
Types.NestedField.optional(2, "field_id", Types.IntegerType.get()),
38+
Types.NestedField.optional(3, "field_name", Types.StringType.get()),
39+
Types.NestedField.required(4, "key", Types.StringType.get()),
40+
Types.NestedField.required(5, "value", Types.StringType.get()));
41+
42+
private IcebergLabelsSysTable() {
43+
}
44+
45+
/** Whether {@code handle} is a {@code $labels} handle; sys handles carry the lower-cased name. */
46+
static boolean isLabels(IcebergTableHandle handle) {
47+
return NAME.equals(handle.getSysTableName());
48+
}
49+
}

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java‎

Lines changed: 67 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -47,12 +47,14 @@
4747
import org.apache.iceberg.BatchScan;
4848
import org.apache.iceberg.ContentScanTask;
4949
import org.apache.iceberg.DataFile;
50+
import org.apache.iceberg.DataTask;
5051
import org.apache.iceberg.DeleteFile;
5152
import org.apache.iceberg.DeleteFileIndex;
5253
import org.apache.iceberg.FileContent;
5354
import org.apache.iceberg.FileFormat;
5455
import org.apache.iceberg.FileScanTask;
5556
import org.apache.iceberg.HasTableOperations;
57+
import org.apache.iceberg.IcebergLabelsDataTask;
5658
import org.apache.iceberg.ManifestContent;
5759
import org.apache.iceberg.ManifestFile;
5860
import org.apache.iceberg.ManifestFiles;
@@ -956,6 +958,9 @@ private List<ConnectorScanRange> planSystemTableScan(IcebergTableHandle handle,
956958
private List<ConnectorScanRange> doPlanSystemTableScan(IcebergTableHandle handle,
957959
List<ConnectorColumnHandle> columns, Optional<ConnectorExpression> filter,
958960
ConnectorSession session) {
961+
if (IcebergLabelsSysTable.isLabels(handle)) {
962+
return doPlanLabelsSystemTableScan(handle, columns, session);
963+
}
959964
Table metadataTable = resolveSysTable(session, handle);
960965
if (isPositionDeletesSysTable(handle)) {
961966
return doPlanPositionDeletesSystemTableScan(handle, metadataTable, columns, filter, session);
@@ -977,20 +982,8 @@ private List<ConnectorScanRange> doPlanSystemTableScan(IcebergTableHandle handle
977982
// hand from the requested columns.
978983
List<String> projectedColumns = requestedLowerNames(columns);
979984
if (!projectedColumns.isEmpty()) {
980-
Schema metadataSchema = metadataTable.schema();
981-
List<NestedField> projectedFields = new ArrayList<>(projectedColumns.size());
982-
Set<Integer> seenFieldIds = new HashSet<>();
983-
for (String columnName : projectedColumns) {
984-
NestedField field = metadataSchema.findField(columnName);
985-
if (field == null) {
986-
throw new RuntimeException("Column " + columnName
987-
+ " not found in iceberg system table schema " + metadataTable.name());
988-
}
989-
if (seenFieldIds.add(field.fieldId())) {
990-
projectedFields.add(field);
991-
}
992-
}
993-
scan = scan.project(new Schema(projectedFields));
985+
scan = scan.project(projectSystemTableSchema(
986+
metadataTable.schema(), projectedColumns, metadataTable.name()));
994987
}
995988
List<ConnectorScanRange> ranges = new ArrayList<>();
996989
try (CloseableIterable<FileScanTask> tasks = scan.planFiles()) {
@@ -1015,6 +1008,50 @@ private static boolean isPositionDeletesSysTable(IcebergTableHandle handle) {
10151008
&& MetadataTableType.POSITION_DELETES.name().equalsIgnoreCase(handle.getSysTableName());
10161009
}
10171010

1011+
/**
1012+
* The system-table schema projected to exactly the requested columns, in the order BE lists them in
1013+
* required_fields: BE's {@code IcebergSysTableJniScanner} reads projected rows positionally. Built by hand
1014+
* because {@code scan.select(names)} would re-emit the fields in table-schema order.
1015+
*/
1016+
private static Schema projectSystemTableSchema(Schema schema, List<String> projectedColumns,
1017+
String tableName) {
1018+
List<NestedField> projectedFields = new ArrayList<>(projectedColumns.size());
1019+
Set<Integer> seenFieldIds = new HashSet<>();
1020+
for (String columnName : projectedColumns) {
1021+
NestedField field = schema.findField(columnName);
1022+
if (field == null) {
1023+
throw new RuntimeException("Column " + columnName
1024+
+ " not found in iceberg system table schema " + tableName);
1025+
}
1026+
if (seenFieldIds.add(field.fieldId())) {
1027+
projectedFields.add(field);
1028+
}
1029+
}
1030+
return new Schema(projectedFields);
1031+
}
1032+
1033+
/**
1034+
* Plan {@code t$labels}: one serialized split whose {@code StaticDataTask} carries every catalog label of the
1035+
* base table, read by BE's JNI scanner like any other static metadata table. Labels are not versioned, so
1036+
* the handle's snapshot pin is ignored (as for {@code $snapshots}/{@code $history}); predicates are left to
1037+
* the scan node's conjuncts, the row count being bounded by the catalog response.
1038+
*/
1039+
private List<ConnectorScanRange> doPlanLabelsSystemTableScan(IcebergTableHandle handle,
1040+
List<ConnectorColumnHandle> columns, ConnectorSession session) {
1041+
Table base = resolveSysBaseTable(session, handle);
1042+
List<String> projectedColumns = requestedLowerNames(columns);
1043+
Schema projectedSchema = projectedColumns.isEmpty()
1044+
? IcebergLabelsSysTable.SCHEMA
1045+
: projectSystemTableSchema(IcebergLabelsSysTable.SCHEMA, projectedColumns,
1046+
base.name() + "$" + IcebergLabelsSysTable.NAME);
1047+
DataTask task = IcebergLabelsDataTask.of(
1048+
base, IcebergLabelsSysTable.SCHEMA, projectedSchema, SYS_TABLE_DUMMY_PATH);
1049+
return Collections.singletonList(new IcebergScanRange.Builder()
1050+
.path(SYS_TABLE_DUMMY_PATH)
1051+
.serializedSplit(IcebergSystemTableSerialization.serializeToBase64(task))
1052+
.build());
1053+
}
1054+
10181055
/**
10191056
* Plan a {@code $position_deletes} scan. Port of legacy
10201057
* {@code IcebergScanNode.doGetPositionDeletesSystemTableSplits} (upstream #65135).
@@ -1301,6 +1338,10 @@ private static boolean isSnapshotIndependentSystemTable(IcebergTableHandle handl
13011338
if (!handle.isSystemTable()) {
13021339
return false;
13031340
}
1341+
if (IcebergLabelsSysTable.isLabels(handle)) {
1342+
// Catalog labels come with the table load, not from a snapshot, so a table with no commit has them.
1343+
return true;
1344+
}
13041345
MetadataTableType type = MetadataTableType.from(handle.getSysTableName());
13051346
// Only metadata_log_entries has a creation row before the first data snapshot; every other metadata
13061347
// table must preserve the resolved-empty fence or a concurrent first append becomes visible.
@@ -3202,9 +3243,11 @@ Table wrapTableForScan(Table table) {
32023243
}
32033244
BaseTable baseTable = (BaseTable) table;
32043245
TableOperations rawOps = baseTable.operations();
3205-
// Keep the table's metrics reporter: every scan of a Kerberos catalog runs on this wrapper.
3246+
// Keep the table's metrics reporter and REST catalog labels: every scan of a Kerberos catalog runs on this
3247+
// wrapper.
32063248
return new BaseTable(new IcebergAuthenticatedTableOperations(
3207-
rawOps, new IcebergAuthenticatedFileIO(rawOps.io(), auth)), table.name(), baseTable.reporter());
3249+
rawOps, new IcebergAuthenticatedFileIO(rawOps.io(), auth)), table.name(), baseTable.reporter(),
3250+
baseTable.labels());
32083251
}
32093252

32103253
/**
@@ -3217,6 +3260,13 @@ Table wrapTableForScan(Table table) {
32173260
* task serialization) in ONE scope — no nested wrap here.
32183261
*/
32193262
private Table resolveSysTable(ConnectorSession session, IcebergTableHandle handle) {
3263+
return MetadataTableUtils.createMetadataTableInstance(
3264+
resolveSysBaseTable(session, handle),
3265+
MetadataTableType.from(handle.getSysTableName()));
3266+
}
3267+
3268+
/** The base table a system-table scan reads, shared with metadata binding; see {@link #resolveSysTable}. */
3269+
private Table resolveSysBaseTable(ConnectorSession session, IcebergTableHandle handle) {
32203270
IcebergCatalogOps ops = catalogOpsResolver.apply(session);
32213271
// Keep the raw base shared with metadata binding and ordinary scan properties. The caller already owns
32223272
// the auth scope, and avoiding a fresh load prevents system-table slots and rows crossing generations.
@@ -3232,9 +3282,7 @@ private Table resolveSysTable(ConnectorSession session, IcebergTableHandle handl
32323282
() -> ops.loadTable(handle.getDbName(), handle.getTableName())),
32333283
() -> ops.loadTable(handle.getDbName(), handle.getTableName()));
32343284
rejectServerSideScanPlanning(base, handle);
3235-
return MetadataTableUtils.createMetadataTableInstance(
3236-
base,
3237-
MetadataTableType.from(handle.getSysTableName()));
3285+
return base;
32383286
}
32393287

32403288
/** This catalog's engine-owned storage services (see {@link ConnectorContext#getStorageContext()}). */

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergStatementScope.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -287,7 +287,7 @@ private static Table snapshotReadTable(Table table, String dbName, String tableN
287287
TableMetadata metadata = operations.current();
288288
// Keep IO, encryption and location behavior from the raw table while pinning only metadata.
289289
return new BaseTable(new IcebergSnapshotTableOperations(operations, metadata),
290-
table.name(), baseTable.reporter());
290+
table.name(), baseTable.reporter(), baseTable.labels());
291291
}
292292

293293
/**

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableCache.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -309,7 +309,7 @@ private Table snapshotReadTable() {
309309
? operations.current()
310310
: TableMetadataParser.fromJson(snapshotMetadataLocation, snapshotMetadataJson);
311311
return new BaseTable(new IcebergSnapshotTableOperations(operations, metadata),
312-
table.name(), baseTable.reporter());
312+
table.name(), baseTable.reporter(), baseTable.labels());
313313
}
314314

315315
private TableLease tryBorrow() {

0 commit comments

Comments
 (0)