Spark Iceberg REST Catalog refresh token #12363

Query engine

Spark

Question

When I set up Spark to connect to the REST Catalog (Lakekeeper) using Keycloak as the OAuth2 server with the client credentials flow, I configured the token to expire after 5 minutes on Keycloak. Once the 5 minutes elapsed, the REST Catalog returned an "Unauthorized" exception. It appears that Spark's REST Catalog does not refresh the token from the OAuth2 server.

spark.sql.catalog.rest=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.rest.type=rest
spark.sql.catalog.rest.uri=...
spark.sql.catalog.rest.credential={{ .Env.CLIENT_ID }}:{{ .Env.CLIENT_SECRET }}
spark.sql.catalog.rest.warehouse=my_warehouse
spark.sql.catalog.rest.scope=...
spark.sql.catalog.rest.oauth2-server-uri=...
spark.sql.catalog.rest.io-impl=org.apache.iceberg.aws.s3.S3FileIO

A similar setup on Trino works correctly.