Skip to content

Commit 3a75387

Browse files
[fix](catalog) re-auth Iceberg REST catalog on 401 instead of wedging until FE restart (apache#64966)
### What problem does this PR solve? Problem Summary: An Iceberg REST catalog using oauth2 client credentials obtains its token when the catalog client is built and keeps it refreshed in the background. If a refresh permanently fails (for example the auth server is briefly unreachable or overloaded right when the refresh fires), the client is left holding a stale token, and every subsequent request fails with `NotAuthorizedException: Not authorized` (HTTP 401) — forever. The client has no re-authentication path of its own, and neither `REFRESH CATALOG` nor metadata-cache invalidation rebuilds it, so the catalog stays unusable until the FE process is restarted. We hit this in production against an Apache Polaris REST catalog. The failure mode is nasty to diagnose because the 401 is swallowed: `getDbNullable`/`buildDbForInit` report `Database [<db>] does not exist`, `SHOW DATABASES` returns a partial list, and since DDL statements (`CREATE VIEW`/`CREATE MTMV`/CTAS/`INSERT`) are `ForwardToMaster`, one wedged master FE breaks all DDL cluster-wide while plain SELECTs on follower FEs keep working. Only an FE restart recovered. ### What is changed? Recovery now lives entirely at the REST layer (per review feedback — thanks @CalvinKirs): a new `ReauthenticatingRestSessionCatalog` (a `BaseViewSessionCatalog`) that `IcebergRestProperties` wraps around the `RESTSessionCatalog` it already builds and owns. On a 401 raised by a request made under the catalog's own identity, the wrapper rebuilds the delegate through the same build seam (fresh HTTP client, fresh OAuth2 token fetch), closes the wedged client, and retries the operation once. If the retry also fails, the error propagates unchanged. Design notes: - **REST-only.** Only `IcebergRestProperties` constructs the wrapper; no other catalog type sees this logic. `IcebergMetadataOps` is untouched apart from widening two declared types from `RESTSessionCatalog` to `BaseViewSessionCatalog`. - **All paths covered, no call-site changes, no stale references.** The default catalog (`asCatalog(empty)`), the view catalog (`asViewCatalog(empty)`), and per-user delegated sessions are all thin views that call back into the session catalog, so they inherit the recovery automatically — and because the swap happens below them, no holder of any of those references ever points at a dead client after recovery. - **Reads and mutations both retried.** A 401 is rejected by the server before the request is processed, so retrying after re-authentication cannot double-apply an operation. - **Per-user delegated sessions are excluded.** A 401 on a request carrying a delegated (per-user) credential means that user's token is invalid; rebuilding the shared client cannot fix it and must not be triggered by it. Likewise, when catalog initialization itself runs under a delegated credential (`iceberg.rest.session=user`), no wrapper is installed, so the rebuild path can never capture a user token into the shared client. - **Typed detection only.** `NotAuthorizedException` anywhere in the cause chain; no message-string matching. - **Bounded.** One rebuild + one retry per failed request; concurrent 401s coalesce into a single rebuild (a request that raced with a completed rebuild just retries on the fresh client). Boundary: recovery triggers on catalog-level operations. Objects already handed out (a loaded `Table`/`View`, a `buildTable`/`buildView` builder) keep the client they were created with; if the token expires between obtaining and using one, that single use can still 401, and the next catalog-level operation heals the client. ### Behavior changed: - Iceberg REST catalogs recover automatically from an expired/rejected OAuth2 token (401) by re-authenticating and retrying once, instead of failing every request until the FE restarts. Other catalog types and non-401 failures are unaffected. Signed-off-by: Raghvendra Singh <[email protected]> Co-authored-by: Claude <[email protected]>
1 parent 14331e4 commit 3a75387

6 files changed

Lines changed: 508 additions & 13 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@
6969
import org.apache.iceberg.TableProperties;
7070
import org.apache.iceberg.UpdatePartitionSpec;
7171
import org.apache.iceberg.UpdateSchema;
72+
import org.apache.iceberg.catalog.BaseViewSessionCatalog;
7273
import org.apache.iceberg.catalog.Catalog;
7374
import org.apache.iceberg.catalog.Namespace;
7475
import org.apache.iceberg.catalog.SessionCatalog;
@@ -79,7 +80,6 @@
7980
import org.apache.iceberg.expressions.Expressions;
8081
import org.apache.iceberg.expressions.Literal;
8182
import org.apache.iceberg.expressions.Term;
82-
import org.apache.iceberg.rest.RESTSessionCatalog;
8383
import org.apache.iceberg.types.Type;
8484
import org.apache.iceberg.types.Types;
8585
import org.apache.iceberg.types.Types.NestedField;
@@ -134,7 +134,7 @@ public IcebergMetadataOps(ExternalCatalog dorisCatalog, Catalog catalog) {
134134
// properties. For any other catalog type this is null (session disabled).
135135
this.userSessionCatalog = dorisCatalog instanceof IcebergUserSessionCatalog
136136
? (IcebergUserSessionCatalog) dorisCatalog : null;
137-
RESTSessionCatalog restSessionCatalog =
137+
BaseViewSessionCatalog restSessionCatalog =
138138
userSessionCatalog == null ? null : userSessionCatalog.getRestSessionCatalog();
139139
IcebergRestProperties.DelegatedTokenMode delegatedTokenMode =
140140
userSessionCatalog == null ? IcebergRestProperties.DelegatedTokenMode.ACCESS_TOKEN
@@ -1383,7 +1383,7 @@ private Optional<ViewCatalog> viewCatalog(SessionContext ctx) {
13831383
return defaultViewCatalog;
13841384
}
13851385

1386-
private Optional<ViewCatalog> resolveDefaultViewCatalog(Catalog catalog, RESTSessionCatalog restSessionCatalog,
1386+
private Optional<ViewCatalog> resolveDefaultViewCatalog(Catalog catalog, BaseViewSessionCatalog restSessionCatalog,
13871387
boolean viewEnabled) {
13881388
// Branch on whether this is a REST (session-aware) catalog, not on whether restSessionCatalog happens to
13891389
// be built: for REST the default Catalog (asCatalog) is not a ViewCatalog, so views must come from the

‎fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergRestExternalCatalog.java‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@
2929
import org.apache.doris.datasource.property.metastore.MetastoreProperties;
3030

3131
import com.google.common.collect.Lists;
32-
import org.apache.iceberg.rest.RESTSessionCatalog;
32+
import org.apache.iceberg.catalog.BaseViewSessionCatalog;
3333
import org.apache.logging.log4j.LogManager;
3434
import org.apache.logging.log4j.Logger;
3535

@@ -226,7 +226,7 @@ public boolean useSessionCatalog(SessionContext ctx) {
226226
}
227227

228228
@Override
229-
public RESTSessionCatalog getRestSessionCatalog() {
229+
public BaseViewSessionCatalog getRestSessionCatalog() {
230230
IcebergRestProperties props = restProperties();
231231
return props == null ? null : props.getRestSessionCatalog();
232232
}

‎fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUserSessionCatalog.java‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@
2020
import org.apache.doris.datasource.SessionContext;
2121
import org.apache.doris.datasource.property.metastore.IcebergRestProperties;
2222

23-
import org.apache.iceberg.rest.RESTSessionCatalog;
23+
import org.apache.iceberg.catalog.BaseViewSessionCatalog;
2424

2525
/**
2626
* Capability interface for an Iceberg catalog that supports per-user dynamic session
@@ -47,7 +47,7 @@ public interface IcebergUserSessionCatalog {
4747
boolean useSessionCatalog(SessionContext ctx);
4848

4949
/** The session-aware Iceberg REST catalog backing this catalog (may be null before initialization). */
50-
RESTSessionCatalog getRestSessionCatalog();
50+
BaseViewSessionCatalog getRestSessionCatalog();
5151

5252
/** The delegated-token mode used when attaching the user's credential to session requests. */
5353
IcebergRestProperties.DelegatedTokenMode getDelegatedTokenMode();
Lines changed: 283 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,283 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.datasource.iceberg;
19+
20+
import com.google.common.annotations.VisibleForTesting;
21+
import org.apache.commons.lang3.exception.ExceptionUtils;
22+
import org.apache.iceberg.Schema;
23+
import org.apache.iceberg.Table;
24+
import org.apache.iceberg.catalog.BaseViewSessionCatalog;
25+
import org.apache.iceberg.catalog.Catalog;
26+
import org.apache.iceberg.catalog.Namespace;
27+
import org.apache.iceberg.catalog.SessionCatalog.SessionContext;
28+
import org.apache.iceberg.catalog.TableIdentifier;
29+
import org.apache.iceberg.exceptions.NotAuthorizedException;
30+
import org.apache.iceberg.rest.RESTSessionCatalog;
31+
import org.apache.iceberg.view.View;
32+
import org.apache.iceberg.view.ViewBuilder;
33+
import org.apache.logging.log4j.LogManager;
34+
import org.apache.logging.log4j.Logger;
35+
36+
import java.io.Closeable;
37+
import java.io.IOException;
38+
import java.util.List;
39+
import java.util.Map;
40+
import java.util.Set;
41+
import java.util.function.Supplier;
42+
43+
/**
44+
* A session catalog that transparently recovers from an expired/rejected credential on an Iceberg REST
45+
* catalog, instead of failing every request until the FE restarts.
46+
*
47+
* <p><b>Why this exists.</b> {@link RESTSessionCatalog} obtains an OAuth2 token at {@code initialize()} and
48+
* keeps it refreshed in the background. If a refresh permanently fails (e.g. the auth server was briefly
49+
* unreachable or overloaded at refresh time), the client is left holding a stale token and every subsequent
50+
* request fails with {@link NotAuthorizedException} (HTTP 401) forever — the client has no re-authentication
51+
* path of its own, and neither {@code REFRESH CATALOG} nor metadata-cache invalidation rebuilds it. The only
52+
* recovery is building a fresh client, which is exactly what this wrapper does: on a 401 from a request made
53+
* under the catalog's own identity, it rebuilds the delegate {@link RESTSessionCatalog} (forcing a fresh
54+
* OAuth2 token fetch), closes the wedged one, and retries the operation once. If the retry also fails, the
55+
* error propagates unchanged.
56+
*
57+
* <p><b>Why a wrapper at the session-catalog level.</b> Everything Doris hands out for a REST catalog — the
58+
* default {@code asCatalog(empty)} used by {@code IcebergMetadataOps}, the {@code asViewCatalog(empty)} view
59+
* path, and per-user delegated sessions — is a thin view that calls back into this class (see
60+
* {@link org.apache.iceberg.catalog.BaseSessionCatalog.AsCatalog}). Wrapping here means no holder of any of
61+
* those references ever sees a stale client after recovery, and non-REST catalogs are untouched because only
62+
* {@code IcebergRestProperties} constructs this class.
63+
*
64+
* <p><b>What is retried.</b> Both reads and mutations: a 401 is rejected by the server before the request is
65+
* processed, so retrying after re-authentication cannot double-apply an operation. Requests that carry a
66+
* per-user delegated credential are <b>not</b> recovered — a 401 there means that user's token is invalid,
67+
* and rebuilding the shared client cannot (and must not) fix it.
68+
*
69+
* <p><b>Boundary.</b> Recovery triggers on catalog-level operations. Objects already handed out (a loaded
70+
* {@link Table}/{@link View}, a builder from {@code buildTable}/{@code buildView}) keep their own reference
71+
* to the client they were created with; if the credential expires between obtaining such an object and using
72+
* it, that one use can still fail with a 401. The next catalog-level operation rebuilds the client, after
73+
* which reloading the object succeeds.
74+
*/
75+
public class ReauthenticatingRestSessionCatalog extends BaseViewSessionCatalog implements Closeable {
76+
77+
private static final Logger LOG = LogManager.getLogger(ReauthenticatingRestSessionCatalog.class);
78+
79+
private final Supplier<RESTSessionCatalog> delegateBuilder;
80+
private volatile RESTSessionCatalog delegate;
81+
82+
public ReauthenticatingRestSessionCatalog(RESTSessionCatalog initialDelegate,
83+
Supplier<RESTSessionCatalog> delegateBuilder) {
84+
this.delegate = initialDelegate;
85+
this.delegateBuilder = delegateBuilder;
86+
}
87+
88+
@VisibleForTesting
89+
RESTSessionCatalog currentDelegate() {
90+
return delegate;
91+
}
92+
93+
private <T> T withAuthRecovery(SessionContext context, Supplier<T> op) {
94+
RESTSessionCatalog attemptedOn = delegate;
95+
try {
96+
return op.get();
97+
} catch (RuntimeException e) {
98+
if (!isAuthExpired(e) || !usesCatalogIdentity(context)) {
99+
throw e;
100+
}
101+
reauthenticate(attemptedOn, e);
102+
return op.get();
103+
}
104+
}
105+
106+
private void runWithAuthRecovery(SessionContext context, Runnable op) {
107+
withAuthRecovery(context, () -> {
108+
op.run();
109+
return null;
110+
});
111+
}
112+
113+
/**
114+
* Rebuilds the delegate (fresh client, fresh token) and closes the wedged one. Synchronized so that
115+
* concurrent 401s coalesce into a single rebuild: a thread whose failed attempt ran against an
116+
* already-replaced delegate skips the rebuild and just retries on the fresh one.
117+
*/
118+
private synchronized void reauthenticate(RESTSessionCatalog attemptedOn, RuntimeException cause) {
119+
if (delegate != attemptedOn) {
120+
return;
121+
}
122+
LOG.warn("Iceberg REST catalog {} rejected its cached credential (401 Not Authorized) and the client "
123+
+ "cannot recover it internally. Rebuilding the REST client to force re-authentication, "
124+
+ "then retrying the request once.", name(), cause);
125+
RESTSessionCatalog replacement = delegateBuilder.get();
126+
RESTSessionCatalog wedged = delegate;
127+
delegate = replacement;
128+
try {
129+
wedged.close();
130+
} catch (IOException | RuntimeException e) {
131+
LOG.warn("Failed to close the replaced Iceberg REST client of catalog {}", name(), e);
132+
}
133+
}
134+
135+
/**
136+
* A request made with a per-user delegated credential authenticates as that user, not as the catalog;
137+
* a 401 there is that token's problem and must not rebuild (or leak into) the shared client.
138+
*/
139+
private static boolean usesCatalogIdentity(SessionContext context) {
140+
return context == null || context.credentials() == null || context.credentials().isEmpty();
141+
}
142+
143+
private static boolean isAuthExpired(Throwable t) {
144+
return ExceptionUtils.getThrowableList(t).stream().anyMatch(c -> c instanceof NotAuthorizedException);
145+
}
146+
147+
// ---- SessionCatalog ----
148+
149+
@Override
150+
public void initialize(String name, Map<String, String> properties) {
151+
super.initialize(name, properties);
152+
delegate.initialize(name, properties);
153+
}
154+
155+
@Override
156+
public String name() {
157+
return delegate.name();
158+
}
159+
160+
@Override
161+
public Map<String, String> properties() {
162+
return delegate.properties();
163+
}
164+
165+
@Override
166+
public List<TableIdentifier> listTables(SessionContext context, Namespace ns) {
167+
return withAuthRecovery(context, () -> delegate.listTables(context, ns));
168+
}
169+
170+
@Override
171+
public Catalog.TableBuilder buildTable(SessionContext context, TableIdentifier ident, Schema schema) {
172+
return withAuthRecovery(context, () -> delegate.buildTable(context, ident, schema));
173+
}
174+
175+
@Override
176+
public Table registerTable(SessionContext context, TableIdentifier ident, String metadataFileLocation) {
177+
return withAuthRecovery(context, () -> delegate.registerTable(context, ident, metadataFileLocation));
178+
}
179+
180+
@Override
181+
public boolean tableExists(SessionContext context, TableIdentifier ident) {
182+
return withAuthRecovery(context, () -> delegate.tableExists(context, ident));
183+
}
184+
185+
@Override
186+
public Table loadTable(SessionContext context, TableIdentifier ident) {
187+
return withAuthRecovery(context, () -> delegate.loadTable(context, ident));
188+
}
189+
190+
@Override
191+
public boolean dropTable(SessionContext context, TableIdentifier ident) {
192+
return withAuthRecovery(context, () -> delegate.dropTable(context, ident));
193+
}
194+
195+
@Override
196+
public boolean purgeTable(SessionContext context, TableIdentifier ident) {
197+
return withAuthRecovery(context, () -> delegate.purgeTable(context, ident));
198+
}
199+
200+
@Override
201+
public void renameTable(SessionContext context, TableIdentifier from, TableIdentifier to) {
202+
runWithAuthRecovery(context, () -> delegate.renameTable(context, from, to));
203+
}
204+
205+
@Override
206+
public void invalidateTable(SessionContext context, TableIdentifier ident) {
207+
runWithAuthRecovery(context, () -> delegate.invalidateTable(context, ident));
208+
}
209+
210+
@Override
211+
public void createNamespace(SessionContext context, Namespace namespace, Map<String, String> metadata) {
212+
runWithAuthRecovery(context, () -> delegate.createNamespace(context, namespace, metadata));
213+
}
214+
215+
@Override
216+
public List<Namespace> listNamespaces(SessionContext context, Namespace namespace) {
217+
return withAuthRecovery(context, () -> delegate.listNamespaces(context, namespace));
218+
}
219+
220+
@Override
221+
public Map<String, String> loadNamespaceMetadata(SessionContext context, Namespace namespace) {
222+
return withAuthRecovery(context, () -> delegate.loadNamespaceMetadata(context, namespace));
223+
}
224+
225+
@Override
226+
public boolean dropNamespace(SessionContext context, Namespace namespace) {
227+
return withAuthRecovery(context, () -> delegate.dropNamespace(context, namespace));
228+
}
229+
230+
@Override
231+
public boolean updateNamespaceMetadata(SessionContext context, Namespace namespace,
232+
Map<String, String> updates, Set<String> removals) {
233+
return withAuthRecovery(context, () -> delegate.updateNamespaceMetadata(context, namespace, updates,
234+
removals));
235+
}
236+
237+
@Override
238+
public boolean namespaceExists(SessionContext context, Namespace namespace) {
239+
return withAuthRecovery(context, () -> delegate.namespaceExists(context, namespace));
240+
}
241+
242+
// ---- ViewSessionCatalog ----
243+
244+
@Override
245+
public List<TableIdentifier> listViews(SessionContext context, Namespace namespace) {
246+
return withAuthRecovery(context, () -> delegate.listViews(context, namespace));
247+
}
248+
249+
@Override
250+
public View loadView(SessionContext context, TableIdentifier identifier) {
251+
return withAuthRecovery(context, () -> delegate.loadView(context, identifier));
252+
}
253+
254+
@Override
255+
public boolean viewExists(SessionContext context, TableIdentifier identifier) {
256+
return withAuthRecovery(context, () -> delegate.viewExists(context, identifier));
257+
}
258+
259+
@Override
260+
public ViewBuilder buildView(SessionContext context, TableIdentifier identifier) {
261+
return withAuthRecovery(context, () -> delegate.buildView(context, identifier));
262+
}
263+
264+
@Override
265+
public boolean dropView(SessionContext context, TableIdentifier identifier) {
266+
return withAuthRecovery(context, () -> delegate.dropView(context, identifier));
267+
}
268+
269+
@Override
270+
public void renameView(SessionContext context, TableIdentifier from, TableIdentifier to) {
271+
runWithAuthRecovery(context, () -> delegate.renameView(context, from, to));
272+
}
273+
274+
@Override
275+
public void invalidateView(SessionContext context, TableIdentifier identifier) {
276+
runWithAuthRecovery(context, () -> delegate.invalidateView(context, identifier));
277+
}
278+
279+
@Override
280+
public void close() throws IOException {
281+
delegate.close();
282+
}
283+
}

0 commit comments

Comments
 (0)