Skip to content

[Catalog Federation] Add Connection Credential Vendors for Other Auth Types#2782

Merged
HonahX merged 10 commits intoapache:mainfrom
XJDKC:rxing-add-connection-credential-vendors-for-other-auth-types
Oct 14, 2025
Merged

[Catalog Federation] Add Connection Credential Vendors for Other Auth Types#2782
HonahX merged 10 commits intoapache:mainfrom
XJDKC:rxing-add-connection-credential-vendors-for-other-auth-types

Conversation

@XJDKC
Copy link
Member

@XJDKC XJDKC commented Oct 9, 2025

Add Connection Credential Vendors for Other Auth Types

This change is a prerequisite for enabling connection credential caching.
By making PolarisCredentialManager the central entry point for obtaining connection credentials, we can introduce caching cleanly and manage all credential flows in a consistent way.

PolarisCredentialManager::getConnectionCredentials will delegate the call to the appropriate credential vendor based on the authentication type:

  • SigV4 -> SigV4ConnectionCredentialVendor
    • ServiceIdentityProvider: Uses ServiceIdentityProvider to retrieve the IAM user’s AWS credential that represents Polaris’s service identity.
    • StsClient: Then use StsClient to assume the user-provided iam role and obtain temporary AWS credentials.
  • OAuth -> OAuthConnectionCredentialVendor
    • UserSecretsManager: Uses UserSecretsManager to retrieve the OAuth2 credential from the secret manager.
  • Bearer -> BearerConnectionCredentialVendor
    • UserSecretsManager: Uses UserSecretsManager to retrieve the bearer token from the secret manager.
  • Implicit -> ImplicitConnectionCredentialVendor
    • A no-op vendor used when no authentication credentials are required.

With this structure, we no longer need to use UserSecretsManager directly inside IcebergCatalogPropertiesProvider::asIcebergCatalogProperties.

This unifies credential retrieval across all authentication types and simplifies the overall credential management flow.

Copy link
Contributor

@dimas-b dimas-b left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall 👍 Some comments below.

Copy link
Contributor

@HonahX HonahX left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this! It looks great!

"Catalog federation is enabled but no ConnectionCredentialVendor found for "
+ "authentication type '%s'. External catalog connections using this "
+ "authentication type will fail.",
type),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious, would this check be too strong that we only consider it to be production ready if all auth types need the vendor, or this is more like a warning despite it is called "Error"?

Copy link
Contributor

@dimas-b dimas-b Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, naming needs improvement here. An error production readiness check will not block startup. Only severe cases break startup.

Copy link
Contributor

@dimas-b dimas-b Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure the check is logically correct. A particular deployment only needs to provide ConnectionCredentialVendor implementations for auth types that are actually used by created catalogs.

I think the vendor availability is implicitly checked on catalog creation already.

On startup, I think we only need to check types for existing catalogs, not for all types listed in the enum. The latter would put an undue burden on custom deployments.

Copy link
Member Author

@XJDKC XJDKC Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, I will switch to severe.

A particular deployment only needs to provide ConnectionCredentialVendor implementations for auth types that are actually used by created catalogs. I think we only need to check types for existing catalogs, not for all types listed in the enum

The logic is: if we enable the catalog federation then all the enabled auth types must have a corresponding ConnectionCredentialVendor. IIRC, we don't want to defer the check during the runtime based on the feedback in the previous PR.

If vendors only implement one / two auth types for catalog federation, they need to only enable them explicitly, right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM 👍

Copy link
Contributor

@HonahX HonahX left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@github-project-automation github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Oct 14, 2025
@HonahX
Copy link
Contributor

HonahX commented Oct 14, 2025

Seems we have all the comments resolved. Thanks @dimas-b for reviewing this!

@HonahX HonahX merged commit 6b957ec into apache:main Oct 14, 2025
15 checks passed
@github-project-automation github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Oct 14, 2025
@XJDKC
Copy link
Member Author

XJDKC commented Oct 14, 2025

@HonahX @dimas-b Thanks for reviewing the PR!

snazy added a commit to snazy/polaris that referenced this pull request Nov 20, 2025
* Build: remove code to post-process generated Quarkus jars (apache#2667)

Before Quarkus 3.28, the Quarkus generated jars used the "current" timestamp for all ZIP entries, which made the jars not-reproducible.
Since Quarkus 3.28, the generated jars use a fixed timestamp for all ZIP entries, so the custom code is no longer necessary.

This PR depends on Quarkus 3.28.

* Update docker.io/jaegertracing/all-in-one Docker tag to v1.74.0 (apache#2751)

* Updating metastore documentation with Aurora postgres example (apache#2706)

* added Aurora postgres to metastore documentation

* Service: Add events for APIs awaiting API changes (apache#2712)

* fix(enhancement): add .idea, .vscode, .venv to top level .gitignore (apache#2718)

fix(enhancement): add .idea, .vscode, .venv to top level .gitignore

* Fix javadocs of `PolarisPrincipal.getPrincipalRoles()` (apache#2752)

* fix(enhancement): squash commits (apache#2643)

* fix(deps): update dependency io.smallrye.config:smallrye-config-core to v3.14.1 (apache#2755)

* Extract interface for RequestIdGenerator (apache#2720)

Summary of changes:

1. Extracted an interface from `RequestIdGenerator`.
2. The `generateRequestId` method now returns a `Uni<String>` in case custom implementations need to perform I/O or other blocking calls during request ID generation.
3. Also addressed comments in apache#2602.

* JDBC: Handle schema evolution (apache#2714)

* Deprecate legacy management endpoints for removal (apache#2749)

* Deprecate LegacyManagementEndpoints for removal

* Add PolarisResolutionManifestCatalogView.getResolvedCatalogEntity helper (apache#2750)

this centralizes some common code and simplifies some test setups

* Enforce that S3 credentials are vended when requested (apache#2711)

This is a follow-up change to apache#2672 striving to improve user-facing error reporting for S3 storage systems without STS.

* Add property to `AccessConfig` to indicate whether the backing storage integration can produce credentials.

* Add a check to `IcebergCatalogHandler` (leading to 400) that storage credentials are vended when requested and the backend is capable of vending credentials in principle.

* Update `PolarisStorageIntegrationProviderImpl` to indicate that FILE storage does not support credential vending (requesitng redential vending with FILE storage does not produce any credentials and does not flag an error, which matches current Polaris behaviour).

* Only those S3 systems where STS is not available (or disabled / not permitted) are affected.

* Other storage integrations are not affected by this PR.

* [Catalog Federation] Ignore JIT entities when deleting federated catalogs, add integration test for namespace/table-level RBAC (apache#2690)

When enabling table/namespace level RBAC in federated catalog, JIT entities will be created during privilege grant. In the short term, we should ignore them when dropping the catalog. In the long term, we will clean-up those entities when deleting the catalog.

This will be the first step towards JIT entity clean-up:

1. Ignore JIT entities when dropping federated catalog (orphan entities)
2. Register tasks/in-place cleanup JIT entities during catalog drop
3. Add new functionality to PolarisMetastoreManager to support atomic delete non-used JIT entities during revoke.
4. Global Garbage Collector to clean-up unreachable entities (entities with non-existing catalog path/parent)

* SigV4 Auth Support for Catalog Federation - Part 3: Service Identity Info Injection (apache#2523)

This PR introduces service identity management for SigV4 Auth Support for Catalog Federation. Unlike user-supplied parameters, the service identity represents the identity of the Polaris service itself and should be managed by Polaris.

* Service Identity Injection

* Return injected service identity info in response

* Use AwsCredentialsProvider to retrieve the credentials

* Move some logic to ServiceIdentityConfiguration

* Rename ServiceIdentityRegistry to ServiceIdentityProvider

* Rename ResolvedServiceIdentity to ServiceIdentityCredential

* Simplify the logic and add more tests

* Use SecretReference and fix some small issues

* Disable Catalog Federation

* Update actions/stale digest to 5f858e3 (apache#2758)

* Service: RealmContextFilter test refactor (apache#2747)

* Update dependency software.amazon.awssdk:bom to v2.35.0 (apache#2760)

* Update apache/spark Docker tag to v3.5.7 (apache#2727)

* Update eric-maynard Team entry (apache#2763)

I'm no longer affiliated with Snowflake, so we should update this page accordingly

* Refactor resolutionManifest handling in PolarisAdminService (apache#2748)

- remove mutable `resolutionManifest` field in favor of letting the
  "authorize" methods return their `PolarisResolutionManifest`
- replace "find" helpers with "get" helpers that have built-in error
  handling

* Implement Finer Grained Operations and Privileges For Update Table (apache#2697)

This implements finer grained operations and privileges for update table in a backwards compatible way as discussed on the mailing list.

The idea is that all the existing privileges and operations will work and continue to work even after this change. (i.e. TABLE_WRITE_PROPERTIES will still ensure update table is authorized even after these changes).

However, because Polaris will now be able to identify each operation within an UpdateTable request and has a privilege model with inheritance that maps to each operation, users will now have the option of restricting permissions at a finer level if desired.

* [Python CLI][CI Failure] Pin pydantic version to < 2.12.0 to fix CI failure (apache#2770)

* Delete ServiceSecretReference (apache#2768)

* JDBC: Fix Bootstrap with schema options (apache#2762)

* Site: Add puppygraph integration (apache#2753)

* Update Changelog with finer grained authz (apache#2775)

* Add Arguments to Various Event Records (apache#2765)

* Update immutables to v2.11.5 (apache#2776)

* Client: add support for policy management (apache#2701)

Implementation for policy management via Polaris CLI (apache#1867).

Here are the subcommands to API mapping:

attach
 - PUT /polaris/v1/{prefix}/namespaces/{namespace}/policies/{policy-name}/mappings
create
 - POST /polaris/v1/{prefix}/namespaces/{namespace}/policies/{policy-name}/mappings
delete
 - DELETE /polaris/v1/{prefix}/namespaces/{namespace}/policies/{policy-name}
detach
 - POST /polaris/v1/{prefix}/namespaces/{namespace}/policies/{policy-name}/mappings
get
 - GET /polaris/v1/{prefix}/namespaces/{namespace}/policies/{policy-name}
list
 - GET /polaris/v1/{prefix}/namespaces/{namespace}/policies 
   - This is default for `list` operation
 - GET /polaris/v1/{prefix}/applicable-policies
   - This is when we have `--applicable` option provided
update
 - PUT /polaris/v1/{prefix}/namespaces/{namespace}/policies/{policy-name}

* Update dependency com.google.cloud:google-cloud-storage-bom to v2.58.1 (apache#2764)

* Update dependency org.jboss.weld:weld-junit5 to v5.0.3.Final (apache#2777)

* Update the LICENSE and NOTICE files in the runtime (apache#2779)

* SigV4 Auth Support for Catalog Federation - Part 4: Connection Credential Manager (apache#2759)

This PR introduces a flexible credential management system for Polaris. Building on Part 3's service identity management, this system combines Polaris service identities with user-provided authentication parameters to generate credentials for remote catalog access.
The core of this PR is the new ConnectionCredentialVendor interface, which:

Generates connection credentials by combining service identity with user auth parameters
Supports different authentication types (AWS SIGV4, AZURE Entra, GCP IAM) through CDI, currently only supports SigV4.
Provides on-demand credential generation
Enables easy extension for new authentication types
In the long term, we should move the storage credential management logic out of PolarisMetastoreManager, PolarisMetastoreManager should only provide persistence interfaces.

* Extract IcebergCatalog.getAccessConfig to a separate class AccessConfigProvider (apache#2736)

This PR extracts credential vending entrypoint getAccessConfig from IcebergCatalog into a new centralized AccessConfigProvider class, decoupling credential generation from catalog implementations.

The old SupportsCredentialVending is removed in this PR upon discussion

* Update immutables to v2.11.6 (apache#2780)

* Enhance Release docs (apache#2787)

* Spark: Remove unnecessary dependency (apache#2789)

* Update Pull Request Template (apache#2788)

* Freeze 1.2 change log (apache#2783)

* [Catalog Federation] Enable Credential Vending for Passthrough Facade Catalog (apache#2784)

This PR introduces credential vending support for passthrough-facade catalogs.

When creating a passthrough-facade catalog, the configuration currently requires two components:

StorageConfig – specifies the storage info for the remote catalog.
ConnectionInfo – defines connection parameters for the underlying remote catalog.

With this change, the StorageConfig is now also used to vend temporary credentials for user requests.
Credential vending honors table-level RBAC policies to determine whether to issue read-only or read-write credentials, ensuring access control consistency with Polaris authorization semantics.

A new test case validates the credential vending workflow, verifying both read and write credential vending.

Note: the remote catalog referenced by the passthrough-facade does not need to support IRC

* Site: Add docs for catalog federation (apache#2761)

* Python client: update CHANGELOG.MD for recent changes (apache#2796)

* Python client: remove Python 3.9 support (apache#2795)

* Update dependency software.amazon.awssdk:bom to v2.35.5 (apache#2799)

* FIX REG tests with cloud providers (apache#2793)

* [Catalog Federation] Block credential vending for remote tables outside allowed location list (apache#2791)

* Correct invalid example in management service OpenAPI spec (apache#2801)

The `example` was incorrectly placed as a sibling of `$ref` within a `schema` object in `polaris-management-service.yml`. According to the OpenAPI specification, properties that are siblings of a `$ref` are ignored.

This was causing a `NullPointerException` in OpenAPI Generator v7.13.0+ due to a change in how examples are processed. The generator now expects all `examples` to be valid and non-empty, and a misplaced `example` can lead to a null reference when the generator tries to access it (we are not yet using v7.13.0+, thus not a problem at the moment).

This commit moves the `example` to be a sibling of the `schema` object, which is the correct placement according to the OpenAPI specification.

Reference error when using newer version of openapi-generator-cli:
```
openapi-generator-cli generate -i spec/polaris-catalog-service.yaml -g python -o client/python --additional-properties=packageName=polaris.catalog --additional-properties=apiNameSuffix="" --skip-validate-spec --additional-properties=pythonVersion=3.13 --ignore-file-override /local/client/python/.openapi-generator-ignore 
...
  Exception: Cannot invoke "io.swagger.v3.oas.models.examples.Example.getValue()" because the return value of "java.util.Map.get(Object)" is null
	at org.openapitools.codegen.DefaultGenerator.processOperation(DefaultGenerator.java:1606)
	at org.openapitools.codegen.DefaultGenerator.processPaths(DefaultGenerator.java:1474)
	at org.openapitools.codegen.DefaultGenerator.generateApis(DefaultGenerator.java:663)
	at org.openapitools.codegen.DefaultGenerator.generate(DefaultGenerator.java:1296)
	at org.openapitools.codegen.cmd.Generate.execute(Generate.java:535)
	at org.openapitools.codegen.cmd.OpenApiGeneratorCommand.run(OpenApiGeneratorCommand.java:32)
	at org.openapitools.codegen.OpenAPIGenerator.main(OpenAPIGenerator.java:66)
Caused by: java.lang.NullPointerException: Cannot invoke "io.swagger.v3.oas.models.examples.Example.getValue()" because the return value of "java.util.Map.get(Object)" is null
	at org.openapitools.codegen.utils.ExamplesUtils.unaliasExamples(ExamplesUtils.java:75)
	at org.openapitools.codegen.DefaultCodegen.unaliasExamples(DefaultCodegen.java:2343)
	at org.openapitools.codegen.DefaultCodegen.fromResponse(DefaultCodegen.java:4934)
	at org.openapitools.codegen.DefaultCodegen.fromOperation(DefaultCodegen.java:4575)
	at org.openapitools.codegen.DefaultGenerator.processOperation(DefaultGenerator.java:1574)
	... 6 more
```

* Update dependency io.opentelemetry:opentelemetry-bom to v1.55.0 (apache#2804)

* Update dependency io.micrometer:micrometer-bom to v1.15.5 (apache#2806)

* Bump version for python deps (apache#2800)

* bump version for python deps

* bump version for python deps

* bump version for python deps

* Update openapi-generatr-cli from 7.11.0.post0 to 7.12.0

* Pin poetry version

* Pin poetry version

* Update dependency io.projectreactor.netty:reactor-netty-http to v1.2.11 (apache#2809)

* [Catalog Federation] Add Connection Credential Vendors for Other Auth Types (apache#2782)

Add Connection Credential Vendors for Other Auth Types

This change is a prerequisite for enabling connection credential caching.
By making PolarisCredentialManager the central entry point for obtaining connection credentials, we can introduce caching cleanly and manage all credential flows in a consistent way.

* Last merged commit 6b957ec

---------

Co-authored-by: Mend Renovate <[email protected]>
Co-authored-by: fabio-rizzo-01 <[email protected]>
Co-authored-by: Adnan Hemani <[email protected]>
Co-authored-by: Artur Rakhmatulin <[email protected]>
Co-authored-by: Alexandre Dutra <[email protected]>
Co-authored-by: Prashant Singh <[email protected]>
Co-authored-by: Christopher Lambert <[email protected]>
Co-authored-by: Dmitri Bourlatchkov <[email protected]>
Co-authored-by: Honah (Jonas) J. <[email protected]>
Co-authored-by: Rulin Xing <[email protected]>
Co-authored-by: Eric Maynard <[email protected]>
Co-authored-by: Travis Bowen <[email protected]>
Co-authored-by: Jaz Ku <[email protected]>
Co-authored-by: Yong Zheng <[email protected]>
Co-authored-by: JB Onofré <[email protected]>
Co-authored-by: Yufei Gu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants