Repository navigation
Commit e545f1a
authored
[feature](fe) Support OIDC session credentials for Iceberg REST catalog (#63068)
This PR supports Iceberg REST catalog user-session authentication with
delegated credentials from Doris authentication.
access could fail or use non-user credentials.
This PR adds a connection-scoped datasource session context, forwards it
across FE-to-master execution, and uses Iceberg `SessionCatalog` when
`iceberg.rest.session = user` is enabled.
#### Overview
The full flow is:
1. Client logs in with OAuth/OIDC/JWT/SAML credential.
2. Doris authentication validates the credential.
3. Doris stores a delegated datasource credential in the connection
`ConnectContext`.
4. If execution is forwarded to master FE, the credential and session id
are carried in `TMasterOpRequest`.
5. Iceberg REST user-session catalogs use the delegated credential
through Iceberg `SessionCatalog`.
6. Doris bypasses shared Iceberg metadata caches for user-session
metadata paths.
```mermaid
flowchart TD
Client["Client login credential"]
Auth["Doris authentication validates credential"]
Resp["Authentication success"]
DC["DelegatedCredential"]
SC["SessionContext in ConnectContext"]
SQL["SQL, SHOW, or DDL"]
Check{"Use Iceberg REST user session?"}
Normal["Existing non-session catalog path"]
Cache["Existing shared metadata caches"]
IcebergSC["Iceberg SessionCatalog SessionContext"]
SessionCatalog["Iceberg REST SessionCatalog"]
Rest["Iceberg REST server authorizes request"]
Result["Return metadata or query result"]
Client --> Auth
Auth --> Resp
Resp --> DC
DC --> SC
SC --> SQL
SQL --> Check
Check -->|No| Normal
Normal --> Cache
Cache --> Result
Check -->|Yes| IcebergSC
IcebergSC --> SessionCatalog
SessionCatalog --> Rest
Rest --> Result
```
#### Goals
- Capture authenticated user credentials as datasource delegated
credentials.
- Keep delegated credentials connection-scoped and in-memory only.
- Preserve delegated credentials when statements are forwarded between
FEs.
- Use Iceberg REST `SessionCatalog` for user-session metadata
operations.
- Support both table and view metadata paths.
- Avoid sharing user-authenticated metadata through global Doris
metadata caches.
- Keep existing behavior unchanged unless `iceberg.rest.session = user`
is configured.
#### Non-goals
- This PR does not change Doris privilege checking.
- This PR does not persist delegated credentials.
- This PR does not change BE scan execution.
- This PR does not implement Doris-side token refresh.
- This PR does not enable user-session mode for non-OAuth2 Iceberg REST
security.
#### Configuration
User-session mode is enabled with:
```sql
'iceberg.rest.security.type' = 'oauth2',
'iceberg.rest.session' = 'user',
'iceberg.rest.oauth2.delegated-token-mode' = 'access_token'
```
`iceberg.rest.oauth2.delegated-token-mode` supports:
- `access_token`: pass the delegated credential as Iceberg OAuth bearer
token.
- `token_exchange`: pass the original token type to Iceberg REST OAuth
so Iceberg can perform token exchange.
Rules:
- `iceberg.rest.session = user` requires `iceberg.rest.security.type =
oauth2`.
- If user-session mode is disabled, Doris uses the existing non-session
Iceberg REST path.
- If no delegated credential exists, Doris uses the existing non-session
path.
#### Authentication Flow
Doris introduces two runtime objects:
- `DelegatedCredential`: credential type, token, and optional
expiration.
- `SessionContext`: datasource-facing session state stored in
`ConnectContext`.
```mermaid
sequenceDiagram
participant Client
participant AuthPlugin as Doris Auth Plugin
participant AuthMgr as AuthenticatorManager
participant Ctx as ConnectContext
Client->>AuthPlugin: Login credential
AuthPlugin-->>AuthMgr: Authentication success
AuthMgr->>AuthMgr: Build DelegatedCredential
AuthMgr->>Ctx: Set SessionContext
Note over Ctx: In-memory and connection-scoped only
```
Supported delegated credential types:
- OAuth access token
- OIDC id token
- JWT token
- SAML assertion
The delegated credential is not persisted to edit log or catalog
metadata.
#### FE Forwarding
A client may connect to follower/observer FE while execution is
forwarded to master FE. The master FE must restore the same datasource
session context before
planning and execution.
```mermaid
sequenceDiagram
participant Client
participant ObserverFE
participant MasterFE
participant IcebergREST
Client->>ObserverFE: SQL with authenticated session
MasterFE->>MasterFE: Restore SessionContext before execution
MasterFE->>IcebergREST: Metadata request with user credential
IcebergREST-->>MasterFE: Metadata response
MasterFE-->>ObserverFE: Query result
ObserverFE-->>Client: Query result
```
New optional thrift fields are added to `TMasterOpRequest`:
- `delegated_credential_type`
- `delegated_credential_token`
- `delegated_credential_expires_at_millis`
- `delegated_credential_session_id`
The session id is required because Iceberg
`SessionCatalog.SessionContext.sessionId()` is used as a session-scoped
auth/cache key. Normal FE forwarding
preserves the original session id. The fallback random id only exists
for compatibility with old or manually constructed forwarded requests
that carry a
delegated token without a session id.
#### Iceberg SessionCatalog Integration
Doris normally keeps an Iceberg `Catalog` instance. Iceberg REST also
provides an internal `SessionCatalog` that can create a session-bound
catalog from
`SessionCatalog.SessionContext`.
`IcebergSessionCatalogAdapter` bridges Doris `SessionContext` to Iceberg
`SessionCatalog.SessionContext`.
```mermaid
flowchart TD
DorisSC["Doris SessionContext"]
Mode{"delegated-token-mode"}
Access["credentials token equals delegated token"]
Exchange["credentials id token jwt or saml equals delegated token"]
IcebergSC["Iceberg SessionCatalog SessionContext"]
BaseSessionCatalog["Iceberg BaseSessionCatalog"]
ActiveCatalog["Session-bound Catalog or ViewCatalog"]
DorisSC --> Mode
Mode -->|access_token| Access
Mode -->|token_exchange| Exchange
Access --> IcebergSC
Exchange --> IcebergSC
IcebergSC --> BaseSessionCatalog
BaseSessionCatalog --> ActiveCatalog
```
Mapping behavior:
- `access_token`: Doris maps the delegated credential to Iceberg OAuth
`token`.
- `token_exchange`: Doris maps the delegated credential to the matching
Iceberg token type key, such as id token, JWT, or SAML.
#### Metadata Operation Routing
When user-session mode is active, Doris routes Iceberg metadata
operations through session-aware methods.
```mermaid
flowchart TD
Op["Iceberg metadata operation"]
UseSession{"Use session catalog?"}
Shared["Existing global metadata cache"]
Direct["Direct session-aware load"]
CatalogOps["Catalog operations"]
ViewOps["View operations"]
NamespaceOps["Namespace operations"]
UseSession -->|Yes| Direct
Direct --> CatalogOps
Direct --> ViewOps
Direct --> NamespaceOps
```
Session-aware paths include:
- `listDatabaseNames`
- `listTableNames`
- `tableExist`
- `loadTable`
- `listViewNames`
- `viewExists`
- `loadView`
- view schema loading
- snapshot/schema paths that depend on a user-session loaded Iceberg
table
#### Cache Isolation
The existing Doris Iceberg metadata cache is shared by catalog/table
metadata. That is correct for static catalog credentials, but unsafe for
user-session
access because the Iceberg REST server may return different metadata for
different users.
For user-session mode, Doris bypasses shared metadata caches for
credential-dependent metadata.
```mermaid
flowchart TD
Request["Metadata request"]
UserSession{"REST user session with delegated credential?"}
GlobalCache["Use shared Doris metadata cache"]
DirectLoad["Load with current SessionContext"]
UserA["User A authorized metadata"]
UserB["User B authorized metadata"]
Risk["Cross-user metadata reuse risk"]
Request --> UserSession
UserSession -->|No| GlobalCache
UserSession -->|Yes| DirectLoad
DirectLoad --> UserA
DirectLoad --> UserB
GlobalCache -.-> Risk
```
This avoids reusing user A's authenticated Iceberg table/view metadata
for user B.
#### Table And View Coverage
Both table and view metadata paths use the same session context.
Table paths include:
- table existence check
- table loading
- table list
- snapshot/schema loading based on session-loaded table
View paths include:
- view existence check
- view loading
- view list
- view schema loading
This is necessary because Iceberg REST user-session authorization may
apply to both tables and views.
#### Token Expiration
Authentication may return optional credential expiration metadata. Doris
preserves this metadata in `DelegatedCredential` and forwards it between
FEs with the
session context.
The Iceberg REST server remains the authority for whether the delegated
credential is valid for an external metadata request. Doris should not
reject every
SQL statement only because the datasource delegated credential is
expired, because many SQL statements do not access that external Iceberg
REST catalog.
#### Compatibility
All new thrift fields are optional.
Existing behavior is preserved for:
- catalogs without `iceberg.rest.session = user`
- non-Iceberg catalogs
- Iceberg non-REST catalogs
- Iceberg REST catalogs using static OAuth credential or static OAuth
token
- requests without delegated credentials1 parent a6ab604 commit e545f1a
42 files changed
Lines changed: 2625 additions & 136 deletions
File tree
- fe
- fe-authentication
- fe-authentication-api
- src
- main/java/org/apache/doris/authentication
- test/java/org/apache/doris/authentication
- fe-authentication-handler/src
- main/java/org/apache/doris/authentication/handler
- test/java/org/apache/doris/authentication/handler
- fe-core/src
- main/java/org/apache/doris
- authentication
- datasource
- iceberg
- property/metastore
- mysql/authenticate
- integration
- plugin
- qe
- test/java/org/apache/doris
- authentication
- datasource
- iceberg
- property/metastore
- mysql/authenticate
- plugin
- qe
- gensrc/thrift
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
37 | 47 | | |
38 | 48 | | |
39 | 49 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
67 | 67 | | |
68 | 68 | | |
69 | 69 | | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
70 | 73 | | |
71 | 74 | | |
72 | 75 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
| |||
52 | 53 | | |
53 | 54 | | |
54 | 55 | | |
| 56 | + | |
55 | 57 | | |
56 | 58 | | |
57 | 59 | | |
| |||
82 | 84 | | |
83 | 85 | | |
84 | 86 | | |
| 87 | + | |
85 | 88 | | |
86 | 89 | | |
87 | 90 | | |
88 | 91 | | |
89 | 92 | | |
90 | 93 | | |
91 | 94 | | |
92 | | - | |
| 95 | + | |
| 96 | + | |
93 | 97 | | |
94 | 98 | | |
95 | 99 | | |
96 | 100 | | |
97 | 101 | | |
98 | 102 | | |
| 103 | + | |
99 | 104 | | |
100 | 105 | | |
101 | 106 | | |
| |||
119 | 124 | | |
120 | 125 | | |
121 | 126 | | |
122 | | - | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
123 | 158 | | |
124 | 159 | | |
125 | 160 | | |
| |||
130 | 165 | | |
131 | 166 | | |
132 | 167 | | |
133 | | - | |
| 168 | + | |
134 | 169 | | |
135 | 170 | | |
136 | 171 | | |
| |||
142 | 177 | | |
143 | 178 | | |
144 | 179 | | |
145 | | - | |
| 180 | + | |
146 | 181 | | |
147 | 182 | | |
148 | 183 | | |
| |||
220 | 255 | | |
221 | 256 | | |
222 | 257 | | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
223 | 269 | | |
224 | 270 | | |
225 | 271 | | |
| |||
270 | 316 | | |
271 | 317 | | |
272 | 318 | | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
273 | 323 | | |
274 | 324 | | |
275 | 325 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
68 | 68 | | |
69 | 69 | | |
70 | 70 | | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
71 | 104 | | |
72 | 105 | | |
73 | 106 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
| 34 | + | |
34 | 35 | | |
35 | 36 | | |
36 | 37 | | |
| |||
232 | 233 | | |
233 | 234 | | |
234 | 235 | | |
235 | | - | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
236 | 241 | | |
237 | 242 | | |
238 | 243 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
154 | 154 | | |
155 | 155 | | |
156 | 156 | | |
157 | | - | |
| 157 | + | |
| 158 | + | |
158 | 159 | | |
159 | 160 | | |
160 | 161 | | |
| |||
181 | 182 | | |
182 | 183 | | |
183 | 184 | | |
184 | | - | |
| 185 | + | |
| 186 | + | |
185 | 187 | | |
186 | 188 | | |
187 | 189 | | |
| |||
194 | 196 | | |
195 | 197 | | |
196 | 198 | | |
| 199 | + | |
| 200 | + | |
197 | 201 | | |
198 | 202 | | |
199 | 203 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
| 40 | + | |
40 | 41 | | |
41 | 42 | | |
42 | 43 | | |
| |||
251 | 252 | | |
252 | 253 | | |
253 | 254 | | |
254 | | - | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
255 | 260 | | |
256 | 261 | | |
257 | 262 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
364 | 364 | | |
365 | 365 | | |
366 | 366 | | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
367 | 380 | | |
368 | 381 | | |
369 | 382 | | |
| |||
0 commit comments