SigV4 Auth Support for Catalog Federation - Part 2: Connection Config Persistence by XJDKC · Pull Request #2190 · apache/polaris

XJDKC · 2025-07-28T01:50:36Z

Milestones

This is Part 2 of the [Splitting] Initial SigV4 Auth Support for Catalog Federation. Upcoming parts will build on this system:

~~SigV4 Auth Support for Catalog Federation - Part 1: Entity Transformation System #1899~~
SigV4 Auth Support for Catalog Federation - Part 2: Connection Config Persistence #2190
SigV4 Auth Support for Catalog Federation - Part 3: Service Identity Info Injection
SigV4 Auth Support for Catalog Federation - Part 4: Connection Credential Manager

Introduction

This PR introduces DPOs (data persistence objects) that allow Polaris to persist SigV4 authentication parameters and service identity references associated with remote catalog connections.

The core idea is to persist a reference to Polaris's own service credentials (e.g. AWS IAM user) rather than the credentials themselves. This enables secure, pluggable credential resolution from external secret stores (e.g. a vault or secret manager) and supports the ability to assume user-specified roles at runtime via SigV4.

Design Overview

Each ConnectionConfigInfoDpo (used for remote catalog federation) now contains a ServiceIdentityInfoDpo, which in turn holds a ServiceSecretReference. This design allows:

Polaris to store only references to its service identity (e.g. AWS IAM user)
The actual credentials to be stored securely in a vault or secret manager
Runtime resolution of credentials based on these references
Role assumption using SigV4AuthenticationParametersDpo (supplied by the user)

This separation of identity metadata and authentication parameters provides a secure and flexible foundation for credential management.

Key Components

SigV4AuthenticationParametersDpo: Holds user-supplied role assumption parameters like:
- roleArn
- roleSessionName: optional
- externalId: optional
ServiceIdentityInfoDpo: Stores metadata about the Polaris-side service identity, including a secret reference.
AwsIamServiceIdentityInfoDpo:
ServiceSecretReference: Specialized version of ServiceIdentityInfoDpo for AWS IAM. It includes:
- iamArn: Polaris's AWS user or role
- ServiceSecretReference: (points to credentials in a vault)
ServiceSecretReference: Points to a credential (e.g., in a vault). This is a logical URN or identifier — Polaris never persists raw secrets.

Flowchart

XJDKC · 2025-07-28T21:47:19Z

polaris-core/src/main/java/org/apache/polaris/core/secrets/ServiceSecretReference.java

+ * specific to an implementation of the secret storage and which is needed "unwrap" the actual
+ * secret in combination with whatever is stored in the remote secrets storage.
+ */
+public class ServiceSecretReference extends UserSecretReference {


I extended UserSecretReference and introduced a new class ServiceSecretReference, to better reflect the purpose of the reference. I'm open to renaming it.

The URN format check remains the same (urn:polaris-secret:xxx:xxx). To make this more flexible and reusable, we could introduce a base class called SecretReference. WDYT?

If the members will be exactly the same as UserSecretReference and this only differs by type, we should probably just rename UserSecretReference to SecretReference without any subclasses (or only extend in the future if we really need extended or modified functionality for one of the types).

dennishuo

Overall LGTM, just the couple small comments

dennishuo · 2025-08-14T17:20:12Z

polaris-core/src/main/java/org/apache/polaris/core/connection/ConnectionConfigInfoDpo.java

      int connectionTypeCode,
      @Nonnull String uri,
      @Nonnull AuthenticationParametersDpo authenticationParameters,
+      @Nonnull ServiceIdentityInfoDpo serviceIdentity,


Is this actually @Nullable? Here and above

dimas-b

LGTM with some minor comments.

dimas-b · 2025-08-14T22:36:23Z

polaris-core/src/main/java/org/apache/polaris/core/secrets/SecretReference.java

 * encryption.
 */
-public class UserSecretReference {
+public class SecretReference {


nit: Is this rename critical to adding SigV4 support? It would have been nicer to do it in a separate PR without functional changes (to reduce distraction) 🙂

Ack, will remove this to another PR!

No worries, I already read through all these changes :)

dimas-b · 2025-08-14T22:47:07Z

polaris-core/src/main/java/org/apache/polaris/core/identity/ServiceIdentityType.java

+  public static @Nonnull ServiceIdentityType fromCode(int identityTypeCode) {
+    // ensure it is within bounds
+    if (identityTypeCode < 0 || identityTypeCode >= REVERSE_MAPPING_ARRAY.length) {
+      return NULL_TYPE;


What is the purpose of NULL_TYPE? It has a specific type code (0) but we map to it from other specific type codes... how is that meant to work end-to-end? 🤔

I understand that this code follows an existing pattern, so I'm not asking to redo the pattern... but I still do not understand the purpose of this enum value 😅

It's just a enum value for all the out-of-bounds codes

With this, we don't need to check serviceIdentityType != null since it will always return a nonnull value!

I do not see any Polaris code where serviceIdentityType is a target of method calls at all (where NPE could be possible) :)

My point in the original method is that if the idea is to keep working without error, then we're losing information in fromCode(). What is the benefit of returning a non-null value, when the original type code's meaning is lost?

Right now, ServiceIdentityType is not used. But In part 3, ServiceIdentityInfo will be assigned and injected to catalog entity and then serviceIdentityType is not null and will be used for the further identity resolution (resolve the identity, get the credential and then make a remote call)

To put it another way: why bother with an enum in this case when the type code is an int and code's behaviour appears to be based on ServiceIdentityInfoDpo sub-types' details ?

Just to make it slightly easier for downstream implementations, without needing to check whether the returned value is null. But yes, the original type code's meaning is lost.

But we should be able to original code if the returned type is NULL right, since the caller passes the code to this method

This enum is a bridge between serialized (int) and runtime (class) representation (we don't want to compare the int with 1/2/3, the code is very confusing and hard to read).

We will use this ServiceIdentityType in both dpo and the runtime ResolvedServiceIdentity class (will be added in part 3).

polaris-core/src/main/java/org/apache/polaris/core/entity/CatalogEntity.java

polaris-core/src/main/java/org/apache/polaris/core/connection/ConnectionConfigInfoDpo.java

dimas-b · 2025-08-14T23:23:59Z

@XJDKC : looks like my previous comments are totally optional... Please fix conflicts, though.

…ion-sigv4-part-2

dimas-b

I'm ok with merging this. We can deal with the ServiceIdentityType issue later (it's not critical, IMHO)

dimas-b · 2025-08-15T18:03:40Z

@dennishuo : waiting for your approval :)

dennishuo

LGTM! Sorry for the delay, forgot to hit submit

* Fix Keycloak getting-started example (apache#2349) The `polaris-setup` container was erroneously including a non-existent scope when fetching a token from Keycloak. * fix(deps): update dependency com.nimbusds:nimbus-jose-jwt to v10.4.2 (apache#2350) * Use PolarisTaskConstants (apache#2346) * Add a regression test for Catalog Federation (apache#2286) * Add a regression test for Catalog Federation * Install jq dependency * Fix token issues * Update regtests/README.md Co-authored-by: Eric Maynard <[email protected]> * Update README.md --------- Co-authored-by: Eric Maynard <[email protected]> * Modularize federation (Option 2) (apache#2332) * Modularize federation (Option 2) * Move polaris-extensions-federation-hadoop dependency * Change identifier to lowerCase * Change identifiers to constants * Replace CallContext with RealmConfig in enforceFeatureEnabledOrThrow (apache#2348) * Replace CallContext with RealmConfig in CatalogEntity (apache#2336) * chore(deps): update registry.access.redhat.com/ubi9/openjdk-21-runtime docker tag to v1.23-6 (apache#2353) * fix(deps): update dependency com.gradleup.shadow:shadow-gradle-plugin to v9.0.2 (apache#2358) * chore(deps): update postgres docker tag to v17.6 (apache#2354) * Add integration tests with Keycloak (apache#2343) * Fix REST responses for failed Admin operations (apache#2291) * Fix REST responses for failed Admin operations the `boolean` return values of many methods in `PolarisAdminService` were often simply not getting used at all, thus the REST api returned success in those cases even though the `PrivilegeResult` was marked as failed. due to this fix a silently failing test now needs to be adjusted. we return the `PrivilegeResult` instead of a `boolean` to give the client at least some indication of what has gone wrong on the server side. note that some of the other operations were throwing Expcetions already, which are already reported back correctly to the client. * review: use http 400 BAD_REQUEST * Make PolarisAuthorizer RequestScoped (apache#2340) all methods in `PolarisAuthorizer` currently have a `CallContext` parameter. in its only implementation only `CallContext.getRealmConfig` is getting used. so since `PolarisAuthorizer` cant be used outside a request, we can simply make it request-scoped and inject the request-scoped `RealmConfig` directly. * fix(deps): update mockito monorepo to v5.19.0 (apache#2360) * Fix soft-merge conflict on `main` (apache#2364) * feat(docs): Add Getting Stated guide for MinIO (apache#2227) * feat(docs): Add Getting Stated guide for MinIO A simple page of step-by-step instructions for setting up a local environment with Polaris, MinIO and Spark. Closes apache#1530 * IntelliJ: fix project icon in IJ project list (apache#2366) ... copy source has changed * Use asMap property helpers (apache#2347) seems like these helpers existed for a long time but were just not getting used consistently * SigV4 Auth Support for Catalog Federation - Part 2: Connection Config Persistence (apache#2190) * Add SigV4 related DPOs * Rename UserSecretReference to SecretReference and fix some small issues * fix(deps): update dependency software.amazon.awssdk:bom to v2.32.24 (apache#2371) * Rat-check: exclude venv, cleanup excludes, include .svg (apache#2363) * `.svg` files are XML files and can contain a license header * Re-grouped the exclusion rat patterns * Added exclude for `.venv` * Added exclude for `.ruff_cache` * NoSQL: Async-impls: add some safeguards + javadoc spelling * NoSQL: spelling * NoSQL: dependency updates * Last merged commit: 5a7686b --------- Co-authored-by: Alexandre Dutra <[email protected]> Co-authored-by: Mend Renovate <[email protected]> Co-authored-by: Christopher Lambert <[email protected]> Co-authored-by: Pooja Nilangekar <[email protected]> Co-authored-by: Eric Maynard <[email protected]> Co-authored-by: Dmitri Bourlatchkov <[email protected]> Co-authored-by: Rulin Xing <[email protected]>

github-project-automation bot added this to Basic Kanban Board Jul 28, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Jul 28, 2025

XJDKC force-pushed the rxing-catalog-federation-sigv4-part-2 branch 4 times, most recently from 515b458 to e3113f6 Compare July 28, 2025 20:03

Add SigV4 related DPOs

063440f

XJDKC force-pushed the rxing-catalog-federation-sigv4-part-2 branch from e3113f6 to 063440f Compare July 28, 2025 21:46

XJDKC commented Jul 28, 2025

View reviewed changes

XJDKC mentioned this pull request Jul 29, 2025

SigV4 Auth Support for Catalog Federation - Part 1: Entity Transformation System #1899

Closed

dennishuo previously approved these changes Aug 14, 2025

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Aug 14, 2025

Rename UserSecretReference to SecretReference and fix some small issues

fef363f

XJDKC dismissed dennishuo’s stale review via fef363f August 14, 2025 17:58

dimas-b reviewed Aug 14, 2025

View reviewed changes

Merge remote-tracking branch 'apache/main' into rxing-catalog-federat…

601e132

…ion-sigv4-part-2

dimas-b approved these changes Aug 14, 2025

View reviewed changes

dennishuo approved these changes Aug 16, 2025

View reviewed changes

dennishuo merged commit e45be14 into apache:main Aug 16, 2025
12 checks passed

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Aug 16, 2025

XJDKC mentioned this pull request Sep 8, 2025

SigV4 Auth Support for Catalog Federation - Part 3: Service Identity Info Injection #2523

Merged

This was referenced Oct 4, 2025

SigV4 Auth Support for Catalog Federation - Part 4: Connection Credential Manager #2759

Merged

[Catalog Federation] Delete Unused ServiceSecretReference #2768

Merged

Conversation

XJDKC commented Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Milestones

Introduction

Design Overview

Key Components

Flowchart

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dennishuo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

XJDKC Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dimas-b commented Aug 14, 2025

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

dimas-b commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dennishuo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

XJDKC commented Jul 28, 2025 •

edited

Loading

XJDKC Aug 14, 2025 •

edited

Loading

dimas-b Aug 14, 2025 •

edited

Loading

dimas-b Aug 14, 2025 •

edited

Loading

dimas-b commented Aug 15, 2025 •

edited

Loading