Repository navigation
REST Spec: add labels field for catalog metadata enrichment - #15750
Conversation
acdb80a to
a80fc1a
Compare
7daf93c to
cfcd777
Compare
a4dd7be to
e8c9e33
Compare
laskoviymishka
changed the title
e8c9e33 to
b44678b
Compare
Reference client implementation for the IRC labels read-path spec change (apache#15750). Without it, RESTObjectMapper (which sets FAIL_ON_UNKNOWN_PROPERTIES = false) silently drops the labels field on deserialization, so labels returned by a catalog are invisible to the Java client. - New rest.labels package (mirroring rest.credentials): Labels (object + fields sub-scopes) and FieldLabels (per-field, keyed by field-id) value types as immutables interfaces, each with a JSON parser. FieldLabels validates field-id >= 1 and a non-empty labels map, matching Credential. - Wire an optional labels field into LoadTableResponse / LoadViewResponse and their parsers. Labels are omitted from the wire when absent (or empty), so the change is additive and backward compatible. - Accept the LoadViewResponse::labels() interface addition in .palantir/revapi.yml (the interface is only implemented by the generated ImmutableLoadViewResponse). - Tests: parser round-trips (object-only, fields-only, both, empty), optional-when-absent, field-id validation from the wire, and a full RESTObjectMapper (Jackson) round-trip for table labels.
Reference client implementation for the IRC labels read-path spec change (apache#15750). Without it, RESTObjectMapper (which sets FAIL_ON_UNKNOWN_PROPERTIES = false) silently drops the labels field on deserialization, so labels returned by a catalog are invisible to the Java client. - New rest.labels package (mirroring rest.credentials): Labels (object + fields sub-scopes) and FieldLabels (per-field, keyed by field-id) value types as immutables interfaces, each with a JSON parser. FieldLabels validates field-id >= 1 and a non-empty labels map, matching Credential. - Wire an optional labels field into LoadTableResponse / LoadViewResponse and their parsers. On LoadViewResponse it is a @Value.Default default method so no interface API break is introduced. Labels are omitted from the wire when absent (or empty), so the change is additive and backward compatible. - Tests: parser round-trips (object-only, fields-only, both, empty), optional-when-absent, field-id validation from the wire, and a full RESTObjectMapper (Jackson) round-trip for table labels.
Reference client implementation for the IRC labels read-path spec change (apache#15750). Without it, RESTObjectMapper (which sets FAIL_ON_UNKNOWN_PROPERTIES = false) silently drops the labels field on deserialization, so labels returned by a catalog are invisible to the Java client. - New rest.labels package (mirroring rest.credentials): Labels (object + fields sub-scopes) and FieldLabels (per-field, keyed by field-id) value types as immutables interfaces, each with a JSON parser. FieldLabels validates field-id >= 1 and a non-empty labels map, matching Credential. - Wire an optional labels field into LoadTableResponse / LoadViewResponse and their parsers. On LoadViewResponse it is a @Value.Default default method so no interface API break is introduced. Labels are omitted from the wire when absent (or empty), so the change is additive and backward compatible. - Tests: parser round-trips (object-only, fields-only, both, empty), optional-when-absent, field-id validation from the wire, and a full RESTObjectMapper (Jackson) round-trip for table labels.
There was a problem hiding this comment.
Thanks for the work @laskoviymishka !
Posted couple of feedbacks on spec, which i believe we should resolve !
On an orthogonal note: I have some doubts on the POC that was linked to the VOTE thread, specially this section
Governance: Trusted Engine Architecture
labels travel through the standard IRC protocol, they enable a trusted engine architecture: the catalog carries classification, each engine implements and enforces its own policies based on the same labels
This assumes classification is stored in catalog and policies are stored in engine ? i have personally not seen systems like this generally policies are stored along with the catalog (which can mounted with a policy store like RANGER / OPA),
can you please elaborate which catalog + engine pair is this,
in the POC an explicit view is created here
Also why can't the policy store be behind catalog and why can't one use ReadRestrictions spec directly to returning these restriction rather than calling the policy store again from engine and creating a governed view locally ?
| labels: | ||
| $ref: '#/components/schemas/Labels' |
There was a problem hiding this comment.
can you please elaborate the use case of when does one has labels on view ?
we had similar debate on read restriction on this and we decided to not have this for views
There was a problem hiding this comment.
Views can carry inherited labels (from the underlying tables / namespace), and surfacing that on the view has real value: a consumer reading the view sees the classification/context without re-deriving it from the sources.
The view use case came up multiple times during the sync discussions and in the design doc.
It's a different call than ReadRestrictions because labels are informational exposure, not an access decision that has to be enforced consistently, so the reason to drop views there doesn't carry over one-to-one. The most important use case is informational exchange for the reader: for example, added AI semantic context.
| description: Field ID from the current schema of the table or view | ||
| labels: | ||
| type: object | ||
| description: Flat key-value labels for this field |
There was a problem hiding this comment.
I am fairly concerned on how feasible / practical it for platforms to rely on this,
since these are sometimes well defined object which have their own inheritance mechanism.
I checked some of the catalogs which actually do supports tags (unfortunately Apache Polaris is not a good example which is used in POC as it neither supports ABAC nor does it support TAGs (there is proposal just published)) and it feels pretty standard to have such kind of info carried for example :
i believe we should think this through more on how are we gonna converge this behaviour for example consider a label attached to a namespace and gets inherited to the table, it flows to another catalog which mounts this as an external obj... how will the external catalog know this was inherited ? and if in the external catalog platform if some one attaches direct label to the obj it would override the namespace one ?
These things make me nervous in having this in the protocol without we agreeing on the attachment experience.
I understand the proposal with exposing things that catalog tracks and should not be part of table props but this is getting more involved if we include governance.
not dictating this behaviour can also be possible but i believe folks who exchange this info will be forced to stuff more in this kv pair and end up making custom adapters around it and i feel it defeats the purpose of having this in the interoperable spec in the first place
There was a problem hiding this comment.
Agree here, that inheritance and attachment are real, well-defined concepts in the catalogs that have a tag/entity model, and that's exactly why I'd keep them out of this field.
Baking an attachment/inheritance model into the read path would force every catalog to reason about it, including the simpler catalogs and engines that have no such concept today. They'd end up with a more complicated story than they need just to produce or consume a flat value.
So the intent here is the lowest common denominator: the catalog resolves whatever it resolves (inheritance applied or not, its own way) and returns the resolved values. A catalog with a rich tag model can still populate it; one without doesn't have to grow one.
The structured side you're describing (attachment, inheritance provenance, cross-catalog convergence) is important, but it's a heavier layer that should build on top of this. I'm planning to expand into that as a follow-up (the structured classification layer, see in original doc: here).
I plan to continue pushing this once simple read-path landed.
There was a problem hiding this comment.
I agree with @laskoviymishka's points here and we should be careful to not try to infer how the labels are applied (i.e. direct vs. inheritance) as we want to be flexible to support any model catalogs implement. Importantly, the protocol is only reflecting what is applied to the catalog object at the point of access. How the catalog arrived at that is immaterial.
| fields: | ||
| type: array | ||
| description: >- | ||
| Field-level labels. Each entry identifies its field by field-id. |
There was a problem hiding this comment.
what happens in case of nested fields ? can the outer struct say {'sensitivity': high} and the inner struct say {'sensistivity': low } .... which is correct then ?
how would one treat this info, if we are saying this is info what one should use for governing access to the enitities, we should be explicit on the interpretation
There was a problem hiding this comment.
Every field-id is its own entry: the outer struct and the nested field are two separate entries keyed by their own field-ids, so on the wire there's no conflict to resolve. Both values are just carried as-is.
The question "which one wins" only comes up if you read a label as an enforced access decision, and that's the part the spec deliberately doesn't claim. Labels are informational; they don't carry precedence semantics. If a catalog does derive an effective value across nested fields, that resolution rule lives in the catalog and it can return the resolved value on the field-id it applies to.
Defining precedence/interpretation in the spec would mean specifying a tag model, which is the heavier structured layer I'd rather build on top as a follow-up (design doc here) than bake into the read path here.
| type: object | ||
| description: Flat key-value labels for this field | ||
| additionalProperties: | ||
| type: string |
There was a problem hiding this comment.
if labels names can be renamed then key name is not sufficient for uniqueness in case of cross platform syncing, as a user might have taken a lot of action based on these synced label for example attaching a PII policy to the label.
How do we expect the system to be reconcile in this scenario if we have things like uuid missing from these ?
There was a problem hiding this comment.
Agree here, but I don't think we could solve this on the read path.
Stable identity across rename (so a consumer that bound a PII policy to a synced label can reconcile when the value or name changes) is an entity property. It needs a first-class tag with its own id and lifecycle, which is the structured write layer, not a flat resolved read value.
Adding a uuid to this field would pull that entity model onto the read path, which is exactly the flat-vs-structured line I'm trying to keep clean. So rather than half-modeling it here, I'd carry it as a first-class part of the structured classification layer (design doc here), which calls out stable identity and rename/delete lifecycle explicitly.
This is the case we went deep on offline and it's the main driver for the structured follow up track.
Reference client implementation for the IRC labels read-path spec change (apache#15750). Without it, RESTObjectMapper (which sets FAIL_ON_UNKNOWN_PROPERTIES = false) silently drops the labels field on deserialization, so labels returned by a catalog are invisible to the Java client. - New rest.labels package (mirroring rest.credentials): Labels (object + fields sub-scopes) and FieldLabels (per-field, keyed by field-id) value types as immutables interfaces, each with a JSON parser. FieldLabels validates field-id >= 1 and a non-empty labels map, matching Credential. - Wire an optional labels field into LoadTableResponse / LoadViewResponse and their parsers. On LoadViewResponse it is a @Value.Default default method so no interface API break is introduced. Labels are omitted from the wire when absent (or empty), so the change is additive and backward compatible. - Tests: parser round-trips (object-only, fields-only, both, empty), optional-when-absent, field-id validation from the wire, and a full RESTObjectMapper (Jackson) round-trip for table labels.
eed5f8c to
e701a0d
Compare
|
Thanks @singhpk234 for review! Let me unpack the POC, i think that section is oversold and it's dragging the wrong framing onto the thread. Disclaimer: this isn't a governance proposal and labels don't mandate any enforcement model. the POC is one thing a consumer can do with labels, not the way to govern iceberg. The pair is an IRC catalog → ClickHouse. The explicit view you found is there because ClickHouse enforces access control natively: Engine-enforcement isn't specific to ClickHouse: Postgres (RLS + column grants) and MySQL (grants, view-based row filtering) enforce the same way and have no per-query external-authorizer path either. Any of them consuming a catalog would act on classification with its own primitives rather than a catalog-side decision. Which is also the honest answer to "why not ReadRestrictions directly". I don't see these as competing:
A catalog can do both. Where the policy store sits behind the catalog and identity resolves cleanly catalog-side (Trino+Ranger, Spark+catalog), ClickHouse just isn't that case - its SQL identities don't map 1:1 to catalog principals so the catalog can't resolve whose decision to return, and it queries well beyond iceberg so it wants one policy layer across all its sources, not a per-catalog decision for the iceberg slice. so for that engine, catalog-provides-classification + engine-enforces is the thing that actually fits, not a claim that it's better in general. Stepping back: governance is one consumer of labels, next to discovery, cost attribution, semantic/AI context. |
Reference client implementation for the IRC labels read-path spec change (apache#15750). Without it, RESTObjectMapper (which sets FAIL_ON_UNKNOWN_PROPERTIES = false) silently drops the labels field on deserialization, so labels returned by a catalog are invisible to the Java client. - New rest.labels package (mirroring rest.credentials): Labels (object + fields sub-scopes) and FieldLabels (per-field, keyed by field-id) value types as immutables interfaces, each with a JSON parser. FieldLabels validates field-id >= 1 and a non-empty labels map, matching Credential. - Wire an optional labels field into LoadTableResponse / LoadViewResponse and their parsers. On LoadViewResponse it is a @Value.Default default method so no interface API break is introduced. Labels are omitted from the wire when absent (or empty), so the change is additive and backward compatible. - Tests: parser round-trips (object-only, fields-only, both, empty), optional-when-absent, field-id validation from the wire, and a full RESTObjectMapper (Jackson) round-trip for table labels.
Reference client implementation for the IRC labels read-path spec change (apache#15750). Without it, RESTObjectMapper (which sets FAIL_ON_UNKNOWN_PROPERTIES = false) silently drops the labels field on deserialization, so labels returned by a catalog are invisible to the Java client. - New rest.labels package (mirroring rest.credentials): Labels (object + fields sub-scopes) and FieldLabels (per-field, keyed by field-id) value types as immutables interfaces, each with a JSON parser. FieldLabels validates field-id >= 1 and a non-empty labels map, matching Credential. - Wire an optional labels field into LoadTableResponse / LoadViewResponse and their parsers. On LoadViewResponse it is a @Value.Default default method so no interface API break is introduced. Labels are omitted from the wire when absent (or empty), so the change is additive and backward compatible. - Expose labels on the loaded table via a SupportsLabels mixin implemented by BaseTable, populated from the load response in RESTSessionCatalog. Labels are ephemeral catalog enrichment, not table state, and are not preserved across table serialization. - Tests: parser round-trips, optional-when-absent, field-id validation, a full RESTObjectMapper (Jackson) round-trip, and BaseTable label exposure.
a135461 to
d36ffd0
Compare
Implement the client read path for catalog labels (added to the REST spec in #15750): parse the labels a catalog returns on load table/view responses so they are visible to the Java client. Until now RESTObjectMapper (FAIL_ON_UNKNOWN_PROPERTIES = false) silently dropped the field, so any labels a catalog returned were lost. Add Labels and FieldLabel value types with JSON parsers, and wire an optional labels field into LoadTableResponse and LoadViewResponse. Labels default to absent and are omitted from the wire when empty, so the change is additive and backward compatible.
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Upgrade the Iceberg Java SDK from 1.11.0 to 1.12.0. 1.12.0 brings the client side of REST catalog labels (`Labels`, `SupportsLabels`, labels on `LoadTableResponse`, see apache/iceberg#15750, apache#18045, apache#18046), which Doris needs before it can expose catalog-provided labels for iceberg tables. This change only upgrades the dependency and adapts Doris to it. - Bump `iceberg.version` 1.11.0 -> 1.12.0, and `avro.version` 1.12.1 -> 1.12.2 to stay consistent with what iceberg 1.12.0 is built against (the pom requires the two to move together). - Bump the explicit caffeine pin in fe-connector-iceberg 2.9.3 -> 3.2.4. Iceberg 1.12.0 moved to caffeine 3. The pin is a direct dependency, so it wins Maven mediation; leaving it at 2.9.3 would package an older caffeine into the iceberg plugin than iceberg-core is built against. - Re-vendor `org.apache.iceberg.DeleteFileIndex` from 1.12.0 and re-apply the Doris changes (public class/builder methods, schemasById comment, equality-field-id sets). The vendored class shadows the SDK class in the same package, and 1.12.0 changed it (an extra `specsById` constructor argument and delete/data file partition-match validation), so keeping the 1.11.0 copy would break the SDK's own callers at runtime. Since Doris passes `specsById` to the builder, the cached scan planning path now also rejects delete files whose partition spec or tuple does not match the data file, the same as upstream iceberg. - Pass `table.specs()` to `ManifestFiles.read` in `IcebergManifestCache`. Iceberg 1.12.0 removed the deprecated `read(ManifestFile, FileIO)` overload, which read the partition specs from the manifest file's own metadata (only possible for Avro manifests). This matches the delete manifest path next to it (`readDeleteManifest(..., table.specs())`) and `IcebergRemoveOrphanFilesAction`. - Keep the AWS SDK at 2.29.52 and default iceberg-aws's HTTP client to `urlconnection`. In 1.12.0, iceberg-aws's default `http-client.type` (`apache`) builds an `Apache5HttpClient` from `software.amazon.awssdk:apache5-client`, which first shipped in AWS SDK 2.32. Doris pins 2.29.52 on purpose (apache#58566: newer SDKs add default S3 integrity checksums that break S3-compatible services), so with the default every S3FileIO, Glue and STS client iceberg builds failed with `NoClassDefFoundError: software/amazon/awssdk/http/apache5/Apache5HttpClient$Builder`. `IcebergCatalogFactory.buildBaseCatalogProperties`, the base of every catalog flavor, now sets `http-client.type=urlconnection` unless the catalog sets it. S3FileIO keeps the setting when it is serialized, so the BE iceberg metadata scanner, which rebuilds the FileIO from it, uses the same client; `url-connection-client` is added to that scanner, which only bundled `apache-client` and `netty-nio-client`. An explicit `http-client.type=apache` is rejected at CREATE/ALTER CATALOG instead of failing on the first S3 or Glue call. `SchemaAwareDataTableScan` (the other vendored class) needs no change: `DataTableScan`, `BaseTableScan`, `SnapshotScan` and `TableScanContext` are identical between 1.11.0 and 1.12.0. ### Release note Upgrade the Iceberg Java SDK to 1.12.0. Iceberg's AWS clients (S3FileIO, Glue, STS) now use the URLConnection HTTP client by default, and the `http-client.type=apache` catalog property is no longer accepted. ### Check List (For Author) - Test: Unit Test - Added IcebergCatalogFactoryTest#buildBaseDefaultsIcebergAwsHttpClientToUrlConnection (builds a real S3 client builder with the default) and #buildBaseKeepsAnExplicitHttpClientType, and IcebergCatalogPropertiesTest#createTimeRulesRejectTheApacheHttpClientType; IcebergCatalogOptionsSnapshotTest snapshots now include the default. - `build.sh --fe` passes. Ran every iceberg FE unit test with run-fe-ut.sh (fe-connector-iceberg, fe-connector-metastore-iceberg, be-java-extensions/iceberg-metadata-scanner): 1630 tests, 0 failures, 5 skipped. - Iceberg regression suites not run. - Behavior changed: Yes. - Iceberg's AWS clients use the URLConnection HTTP client by default, so `http-client.apache.*` properties no longer apply, and `http-client.type=apache` is rejected at CREATE/ALTER CATALOG. - Iceberg scans whose delete files have a partition spec or partition tuple that does not match the data file now fail with a ValidationException (iceberg 1.12.0 behavior) instead of applying the deletes. - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Upgrade the Iceberg Java SDK from 1.11.0 to 1.12.0. 1.12.0 brings the client side of REST catalog labels (`Labels`, `SupportsLabels`, labels on `LoadTableResponse`, see apache/iceberg#15750, apache#18045, apache#18046), which Doris needs before it can expose catalog-provided labels for iceberg tables. This change only upgrades the dependency and adapts Doris to it. - Bump `iceberg.version` 1.11.0 -> 1.12.0, and `avro.version` 1.12.1 -> 1.12.2 to stay consistent with what iceberg 1.12.0 is built against (the pom requires the two to move together). - Bump the explicit caffeine pin in fe-connector-iceberg 2.9.3 -> 3.2.4. Iceberg 1.12.0 moved to caffeine 3. The pin is a direct dependency, so it wins Maven mediation; leaving it at 2.9.3 would package an older caffeine into the iceberg plugin than iceberg-core is built against. - Re-vendor `org.apache.iceberg.DeleteFileIndex` from 1.12.0 and re-apply the Doris changes (public class/builder methods, schemasById comment, equality-field-id sets). The vendored class shadows the SDK class in the same package, and 1.12.0 changed it (an extra `specsById` constructor argument and delete/data file partition-match validation), so keeping the 1.11.0 copy would break the SDK's own callers at runtime. Since Doris passes `specsById` to the builder, the cached scan planning path now also rejects delete files whose partition spec or tuple does not match the data file, the same as upstream iceberg. - Pass `table.specs()` to `ManifestFiles.read` in `IcebergManifestCache`. Iceberg 1.12.0 removed the deprecated `read(ManifestFile, FileIO)` overload, which read the partition specs from the manifest file's own metadata (only possible for Avro manifests). This matches the delete manifest path next to it (`readDeleteManifest(..., table.specs())`) and `IcebergRemoveOrphanFilesAction`. - Keep the AWS SDK at 2.29.52 and default iceberg-aws's HTTP client to `urlconnection`. In 1.12.0, iceberg-aws's default `http-client.type` (`apache`) builds an `Apache5HttpClient` from `software.amazon.awssdk:apache5-client`, which first shipped in AWS SDK 2.32. Doris pins 2.29.52 on purpose (apache#58566: newer SDKs add default S3 integrity checksums that break S3-compatible services), so with the default every S3FileIO, Glue and STS client iceberg builds failed with `NoClassDefFoundError: software/amazon/awssdk/http/apache5/Apache5HttpClient$Builder`. `IcebergCatalogFactory.buildBaseCatalogProperties`, the base of every catalog flavor, now sets `http-client.type=urlconnection` unless the catalog sets it. S3FileIO keeps the setting when it is serialized, so the BE iceberg metadata scanner, which rebuilds the FileIO from it, uses the same client; `url-connection-client` is added to that scanner, which only bundled `apache-client` and `netty-nio-client`. An explicit `http-client.type=apache` is rejected at CREATE/ALTER CATALOG instead of failing on the first S3 or Glue call. - Report the mixed-version case clearly. FE sends iceberg system-table splits to the BE as Java-serialized iceberg objects, and iceberg 1.12.0 changed the serialized form of classes every split carries (`BaseFile`, `PartitionData`, their anonymous inner classes; 24 serializable classes changed `serialVersionUID`). Unlike 1.11.0, where only `Schema` changed and is shimmed, this cannot be bridged by remapping UIDs, so while FE and BE run different iceberg versions (a rolling upgrade) iceberg system tables cannot be read. `IcebergSerializationCompat` now turns the bare `InvalidClassException` / `ClassNotFoundException` into an error naming the cause and the BE's iceberg version, and the shim comments no longer claim a shared wire across versions. `SchemaAwareDataTableScan` (the other vendored class) needs no change: `DataTableScan`, `BaseTableScan`, `SnapshotScan` and `TableScanContext` are identical between 1.11.0 and 1.12.0. ### Release note Upgrade the Iceberg Java SDK to 1.12.0. Iceberg's AWS clients (S3FileIO, Glue, STS) now use the URLConnection HTTP client by default, and the `http-client.type=apache` catalog property is no longer accepted. During a rolling upgrade from a release on Iceberg 1.11, Iceberg system tables (`tbl$snapshots`, `tbl$files`, ...) are unavailable until FE and BE are both upgraded. ### Check List (For Author) - Test: Unit Test - Added IcebergCatalogFactoryTest#buildBaseDefaultsIcebergAwsHttpClientToUrlConnection (builds a real S3 client builder with the default) and #buildBaseKeepsAnExplicitHttpClientType, and IcebergCatalogPropertiesTest#createTimeRulesRejectTheApacheHttpClientType; IcebergCatalogOptionsSnapshotTest snapshots now include the default. - Added IcebergSerializationCompatTest#reportsATaskFromAnotherIcebergVersion (a DataFile carrying Iceberg 1.11's BaseFile UID) and #reportsAClassThisIcebergDoesNotHave. - Verified the mixed-version break outside the build: a DataFile written with iceberg 1.11.0 cannot be read with 1.12.0 and vice versa, even after rewriting every changed class UID in the stream. - `build.sh --fe` passes. Ran every iceberg FE unit test with run-fe-ut.sh (fe-connector-iceberg, fe-connector-metastore-iceberg, be-java-extensions/iceberg-metadata-scanner): 1650 tests, 0 failures, 5 skipped. - Iceberg regression suites not run. - Behavior changed: Yes. - Iceberg's AWS clients use the URLConnection HTTP client by default, so `http-client.apache.*` properties no longer apply, and `http-client.type=apache` is rejected at CREATE/ALTER CATALOG. - Iceberg scans whose delete files have a partition spec or partition tuple that does not match the data file now fail with a ValidationException (iceberg 1.12.0 behavior) instead of applying the deletes. - While FE and BE run different iceberg versions (rolling upgrade from iceberg 1.11), iceberg system tables fail with an explicit "different Iceberg version" error until both sides are upgraded. - Does this need documentation: No
…em table ### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: The Iceberg REST catalog spec now lets a catalog return `labels` with each table load (apache/iceberg#15750): catalog-owned, per-request metadata such as ownership, classification or cost attribution, at the table level (`object-labels`) and per field (keyed by `field-id`). Labels are not part of the table metadata, have no snapshots and are never written to `metadata.json`. Iceberg 1.12.0 exposes them on the loaded table via `SupportsLabels#labels()`. Doris had no way to show them. Add a `t$labels` system table for iceberg tables in a REST catalog: SELECT * FROM rest_ctl.db.tbl$labels; scope | field_id | field_name | key | value object | NULL | NULL | owner | team-a field | 2 | email | classification | pii One row per label. `field_name` is resolved against the current table schema and is NULL for a field id that is no longer in it (a dropped column). The schema mirrors iceberg's proposed `LabelsTable`. Implementation: - Keep labels when Doris rebuilds a loaded table. iceberg's REST client attaches them to the loaded `BaseTable` as a transient field; they are not table metadata, so a `BaseTable` rebuilt from its `TableOperations` must pass them explicitly. `IcebergTableCache.TableLease#snapshotReadTable` (cached reads), `IcebergStatementScope#snapshotReadTable` (statement reads) and `IcebergScanPlanProvider#wrapTableForScan` (Kerberos FileIO wrap) used constructors that reset labels to `Labels.EMPTY`, so every read saw no labels; they now pass `baseTable.labels()`. - `labels` is not a `MetadataTableType`, so it is handled next to the SDK metadata tables rather than through `MetadataTableUtils`. `IcebergConnectorMetadata` lists and resolves it only when the catalog flavor is `rest` (no other catalog returns labels), and serves its fixed schema and column handles without loading the base table. - `IcebergScanPlanProvider` plans it as one split. A new split-package helper `org.apache.iceberg.IcebergLabelsDataTask` builds an iceberg `StaticDataTask` (package-private in the SDK) holding the label rows, projected to the requested columns in BE's required-field order; its metadata `DataFile` is synthesized, so no storage request is made. The task is serialized like every other static metadata table, so BE's `IcebergSysTableJniScanner` reads it with no BE change. - The ordered system-table projection is extracted into `projectSystemTableSchema` and shared with the existing path, and the base-table load is split out of `resolveSysTable` as `resolveSysBaseTable`. - Labels are not versioned: like `$snapshots`/`$history`, the snapshot pin of the handle is ignored, and `$labels` is exempt from the resolved-empty-snapshot fence, so a table with no commit still shows its labels. Labels reflect the table load the statement used, so they follow the iceberg table cache (`meta.cache.iceberg.table.ttl-second`); REFRESH TABLE picks up new labels. Like the other system tables, a REST catalog that requires server-side scan planning rejects `$labels`. ### Release note Support querying REST catalog labels of an iceberg table through the `tbl$labels` system table. ### Check List (For Author) - Test: Unit Test - Added IcebergLabelsDataTaskTest, the $labels cases in IcebergConnectorMetadataSysTableTest, IcebergTableCacheTest#snapshotReadTableKeepsCatalogLabels and IcebergStatementScopeTest#sharedTableKeepsCatalogLabels. - `build.sh --fe` (Maven build cache disabled) passes on top of current master. Ran every iceberg and hudi FE unit test with run-fe-ut.sh: 1990 tests, 0 failures, 5 skipped. - Regression test pending a REST fixture that returns labels (apache/iceberg#18047). - Behavior changed: No (adds a new system table for REST catalogs only) - Does this need documentation: Yes (doc PR to follow) Co-authored-by: Isaac <[email protected]>
…em table ### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: The Iceberg REST catalog spec now lets a catalog return `labels` with each table load (apache/iceberg#15750): catalog-owned, per-request metadata such as ownership, classification or cost attribution, at the table level (`object-labels`) and per field (keyed by `field-id`). Labels are not part of the table metadata, have no snapshots and are never written to `metadata.json`. Iceberg 1.12.0 exposes them on the loaded table via `SupportsLabels#labels()`. Doris had no way to show them. Add a `t$labels` system table for iceberg tables in a REST catalog: SELECT * FROM rest_ctl.db.tbl$labels; scope | field_id | field_name | key | value object | NULL | NULL | owner | team-a field | 2 | email | classification | pii One row per label. `field_name` is resolved against the current table schema and is NULL for a field id that is no longer in it (a dropped column). The schema mirrors iceberg's proposed `LabelsTable`. Implementation: - Keep labels when Doris rebuilds a loaded table. iceberg's REST client attaches them to the loaded `BaseTable` as a transient field; they are not table metadata, so a `BaseTable` rebuilt from its `TableOperations` must pass them explicitly. `IcebergTableCache.TableLease#snapshotReadTable` (cached reads), `IcebergStatementScope#snapshotReadTable` (statement reads) and `IcebergScanPlanProvider#wrapTableForScan` (Kerberos FileIO wrap) used constructors that reset labels to `Labels.EMPTY`, so every read saw no labels; they now pass `baseTable.labels()`. - `labels` is not a `MetadataTableType`, so it is handled next to the SDK metadata tables rather than through `MetadataTableUtils`. `IcebergConnectorMetadata` lists and resolves it only when the catalog flavor is `rest` (no other catalog returns labels), and serves its fixed schema and column handles without loading the base table. - `IcebergScanPlanProvider` plans it as one split. A new split-package helper `org.apache.iceberg.IcebergLabelsDataTask` builds an iceberg `StaticDataTask` (package-private in the SDK) holding the label rows, projected to the requested columns in BE's required-field order; its metadata `DataFile` is synthesized, so no storage request is made. The task is serialized like every other static metadata table, so BE's `IcebergSysTableJniScanner` reads it with no BE change. - The ordered system-table projection is extracted into `projectSystemTableSchema` and shared with the existing path, and the base-table load is split out of `resolveSysTable` as `resolveSysBaseTable`. - Labels are not versioned: like `$snapshots`/`$history`, the snapshot pin of the handle is ignored, and `$labels` is exempt from the resolved-empty-snapshot fence, so a table with no commit still shows its labels. Labels reflect the table load the statement used, so they follow the iceberg table cache (`meta.cache.iceberg.table.ttl-second`); REFRESH TABLE picks up new labels. Like the other system tables, a REST catalog that requires server-side scan planning rejects `$labels`. ### Release note Support querying REST catalog labels of an iceberg table through the `tbl$labels` system table. ### Check List (For Author) - Test: Unit Test - Added IcebergLabelsDataTaskTest, the $labels cases in IcebergConnectorMetadataSysTableTest, IcebergTableCacheTest#snapshotReadTableKeepsCatalogLabels and IcebergStatementScopeTest#sharedTableKeepsCatalogLabels. - `build.sh --fe` (Maven build cache disabled) passes on top of current master. Ran every iceberg and hudi FE unit test with run-fe-ut.sh: 1990 tests, 0 failures, 5 skipped. - Regression test pending a REST fixture that returns labels (apache/iceberg#18047). - Behavior changed: No (adds a new system table for REST catalogs only) - Does this need documentation: Yes (doc PR to follow) Co-authored-by: Isaac <[email protected]>
#69135) ### What problem does this PR solve? Issue Number: #68916 Related PR: #69031 Problem Summary: The Iceberg REST catalog spec now lets a catalog return `labels` with each table load (apache/iceberg#15750): catalog-owned, per-request metadata such as ownership, classification or cost attribution, at the table level (`object-labels`) and per field (keyed by `field-id`). Labels are not part of the table metadata, have no snapshots and are never written to `metadata.json`. Iceberg 1.12.0 exposes them on the loaded table via `SupportsLabels#labels()`. Doris had no way to show them. Add a `t$labels` system table for iceberg tables in a REST catalog: ```sql SELECT * FROM rest_ctl.db.tbl$labels; scope | field_id | field_name | key | value object | NULL | NULL | owner | team-a field | 2 | email | classification | pii ``` One row per label. `field_name` is resolved against the current table schema and is NULL for a field id that is no longer in it (a dropped column). The schema mirrors iceberg's proposed `LabelsTable`. Implementation: - Keep labels when Doris rebuilds a loaded table. iceberg's REST client attaches them to the loaded `BaseTable` as a transient field; they are not table metadata, so a `BaseTable` rebuilt from its `TableOperations` must pass them explicitly. `IcebergTableCache.TableLease#snapshotReadTable` (cached reads), `IcebergStatementScope#snapshotReadTable` (statement reads) and `IcebergScanPlanProvider#wrapTableForScan` (Kerberos FileIO wrap) used constructors that reset labels to `Labels.EMPTY`, so every read saw no labels; they now pass `baseTable.labels()`. - `labels` is not a `MetadataTableType`, so it is handled next to the SDK metadata tables rather than through `MetadataTableUtils`. `IcebergConnectorMetadata` lists and resolves it only when the catalog flavor is `rest` (no other catalog returns labels), and serves its fixed schema and column handles without loading the base table. - `IcebergScanPlanProvider` plans it as one split. A new split-package helper `org.apache.iceberg.IcebergLabelsDataTask` builds an iceberg `StaticDataTask` (package-private in the SDK) holding the label rows, projected to the requested columns in BE's required-field order; its metadata `DataFile` is synthesized, so no storage request is made. The task is serialized like every other static metadata table, so BE's `IcebergSysTableJniScanner` reads it with no BE change. - The ordered system-table projection is extracted into `projectSystemTableSchema` and shared with the existing path, and the base-table load is split out of `resolveSysTable` as `resolveSysBaseTable`. - Labels are not versioned: like `$snapshots`/`$history`, the snapshot pin of the handle is ignored, and `$labels` is exempt from the resolved-empty-snapshot fence, so a table with no commit still shows its labels. Labels reflect the table load the statement used, so they follow the iceberg table cache (`meta.cache.iceberg.table.ttl-second`); REFRESH TABLE picks up new labels. Like the other system tables, a REST catalog that requires server-side scan planning rejects `$labels`. Co-authored-by: Isaac <[email protected]> Co-authored-by: morningman <[email protected]>
Add the Iceberg REST Catalog `labels` wire types from apache/iceberg#15750 and an optional `labels` member on `LoadTableResult` and `LoadViewResult`, ahead of exposing Governance Tags as labels (lakekeeper#2078). - `Labels { object-labels, fields }` and `FieldLabels { field-id, labels }` in `iceberg_ext::catalog::rest`, matching the `Labels` / `FieldLabels` / `CatalogObjectLabels` schemas in the vendored spec. `field-id` and `labels` are required on a field entry; both `Labels` members are optional. - `Labels::from_maps` builds the canonical form: keys ordered, fields sorted by field id with one entry each, empty entries dropped, and `None` when nothing is left. Equal label sets built this way serialize to the same bytes, which a label-aware `loadTable` ETag will rely on. - Empty members and an absent `labels` are omitted on serialize and default on deserialize, so responses without labels still parse. - Every producer sets `labels: None`, so no response body changes and the `loadTable` ETag needs no new input yet. Downstream: `LoadTableResult` and `LoadViewResult` gain a public field. Code building them with struct-literal syntax must add `labels: None`. Co-authored-by: Isaac <[email protected]>
`loadTable` and `loadView` now return Governance Tags as Iceberg REST Catalog labels (apache/iceberg#15750), so query engines receive them without the Management API (lakekeeper#2078). - Object labels are the object's effective tags, the set the Management API lists with `effective=true`: inherited from namespaces and warehouse, nearest attachment winning. Read with the same `list_effective_tag_candidates` + `resolve_effective_tags` path, so there is no second inheritance algorithm. - Field labels (tables only) are the direct column tags, keyed by field id, so they follow a renamed column. A label is returned while some schema in the table metadata has its field id. - Tag names are label keys unchanged. Markers map to "true"; free-text and enumerated tags keep their value. No tags, no `labels`. - Visibility follows `effective=true`: whoever may load the object, including through a view that reads the table on the caller's behalf, sees its labels. A failed tag read fails the load. - Create and register responses carry no labels, and neither do their idempotent replays: `load_table_with_flags` takes a `TableLabels` mode, and replays pass `Omit`, reading nothing, so a replay keeps the original body and tag. Commit responses have no `labels` member. Reads: two queries per load (effective tags, column tags; one for a view), run concurrently on the read pool before the metadata transaction opens, so no second connection is held alongside it. The count does not depend on the number of ancestors, tags or columns. They also run on a load answered with 304, which needs them. Conditional `loadTable`: a tag change on the table, a column or an ancestor changes the body while the metadata location stays, so the labels are read before the 304 check and their digest is a `TableResponseShape` input. Absent labels hash nothing: a table without labels keeps its existing `lk3` tag and body, so the prefix stays and an upgrade forces no fleet-wide reload. The digest is computed from the canonical `Labels::from_maps` value before the schema filter, which depends only on the metadata the tag already covers. `load_table_with_flags` stays under its pre-change length: the warehouse refetch and the storage-config generation move into helpers. Tests: projection and schema-filter unit tests; ETag tests pinning the unlabelled tag, a labelled tag, and digest separation; a 304 test for label changes; Postgres integration tests for inheritance, override, value kinds, column grouping, rename and drop of a tagged column, views, a `createTable` replay, and the ETag across namespace, table and column tag changes and their removal. Co-authored-by: Isaac <[email protected]>
Adds an optional
labelsfield to the REST catalog spec for catalog-owned metadata enrichment.Motivation
A catalog maintains operational context about the objects it serves — ownership, classification, cost attribution, semantic hints — but the spec gives it no standard place to return that context. Today catalogs work around this by extending load responses with vendor-specific fields that OSS engines can't consume. Labels provide one standard channel instead.
Labels are catalog-owned and ephemeral: generated per request, never written to
metadata.json, no commits or snapshots, no versioning with table history. They are not table state, and different catalogs serving the same object may legitimately return different labels. Catalogs may populate them; engines may use them or ignore them entirely.This is deliberately kept out of properties. Properties are table state — writer-owned, persisted, and they travel with the table across catalogs. Catalog enrichment is the opposite, so putting it in properties would conflate authority and force it to travel where it shouldn't. The boundary is the table spec (properties) vs. the IRC spec (labels).
What changed
New
Labels(tableflat key-values +columnskeyed byfield-idfor stability across schema evolution) andColumnLabelsschemas;labelsfield added toLoadTableResultandLoadViewResult.Read-only and backward compatible (
labelsis optional; older clients ignore it). Namespace labels and the write path (UpdateLabels) are follow-ups.Proposal and design doc: #15521.