feat: support user identity forwarding to tasks via TaskAuthContext by jtuglu1 · Pull Request #19236 · apache/druid

jtuglu1 · 2026-03-30T23:30:20Z

Description

Create a mechanism to propagate user identity context from authenticator to tasks. This allows tasks to safely operate with a sort of per-task identity, allowing tasks to access information dynamically that would be hard to setup in the current state of things. This allows for auth mechanisms like catalog credential vending (through Iceberg Catalog, etc.) and allows Druid to operate more like a standalone engine that can integrate with other things within a larger data ecosystem.

Design

Constraints

This currently does NOT support task restarts.

Release note

Create a mechanism to propagate user identity context from authenticator to tasks to support things like credential vending, etc.

This PR has:

+   * @return TaskAuthContext to inject into the task, or null to skip injection
+   */
+  @Nullable
+  TaskAuthContext createTaskAuthContext(AuthenticationResult authenticationResult, Task task);


+   * @return TaskAuthContext to inject into the task, or null to skip injection
+   */
+  @Nullable
+  TaskAuthContext createTaskAuthContext(AuthenticationResult authenticationResult, Task task);


cecemei · 2026-04-07T18:03:54Z

@jtuglu1 whats the status for this PR? should we move this to 38?

jtuglu1 · 2026-04-07T18:16:23Z

@jtuglu1 whats the status for this PR? should we move this to 38?

This one I wanted to get into v37 – I was hoping to get this merged today.

gianm

I reviewed the core changes only, focusing on trying to understand the security properties of the changes.

I wonder what your thoughts are on an alternate design that keeps the vended credentials inside the input source itself:

Add a new method to InputSource like scopeForUser(AuthenticationResult authResult). Default implementation is return this
Whenever a task is submitted, call scopeForUser on all of its input sources at whichever service initially accepts the task (either Broker [for SQL DML] or Overlord [for anything else]).
The IcebergInputSource would implement scopeForUser to fetch the vended credentials and transform itself into an input source that bakes in the vended credentials. It would use a PasswordProvider so it is redactable.

The idea would be to avoid the need for a new credential vending system in core, putting most of the changes inside the Iceberg extension instead.

gianm · 2026-04-08T07:31:05Z

+
+  /**
+   * Returns sensitive credentials (e.g., OAuth tokens, API keys).
+   * This method MUST be redacted during serialization via {@link TaskAuthContextRedactionMixIn}.


This is over-eager, isn't it? The credentials must be serialized in some cases. When we send a task spec from Overlord to the server that will actually run the task, the credentials must be included. In some cases the nonredacted task file will need to end up on disk, so it can be run. Perhaps this should say "MUST be redacted when written to log files, the metadata store, or returned from user-facing APIs".

Yeah the comment here is a bit aggressive, but that's currently what is already done. The field is not redacted when being submitted from Overlord to MM/Peon/Indexer. It is only redacted currently on logging, persistence (to disk or db).

gianm · 2026-04-08T07:32:27Z

+ *
+ * <p><b>Credential lifetime:</b> Vended credentials (OAuth tokens, STS session tokens) have
+ * limited lifetimes, typically 1-12 hours. There is currently no credential refresh mechanism,
+ * so long-running tasks may fail when credentials expire. Task restarts are also not supported


Any thoughts on how to handle long-running tasks? I expect it will be a problem. It's not uncommon for tasks to take hours.

I was going to add this in a separate change. Using a task auth context, inputSource could conceivably just use that with some implementable revend() method. The only issue I would want to avoid is having the subtasks refresh the credentials (and not the driver task), since this might cause thundering herd effect where many subtasks all attempt to refresh at once. Spark does a better job of this by letting executor handle credential refresh on the driver, then propagate that to the executors.

gianm · 2026-04-08T07:34:39Z

+   *
+   * @return the identity string
+   */
+  String getIdentity();


The auth context is a @JsonProperty so I believe that means people can set it explicitly when they submit tasks. Does anything bad happen if someone sets a context and sets the identity to someone else? What do you think about clearing the user-provided auth context, if any, in OverlordResource?

Does anything bad happen if someone sets a context and sets the identity to someone else?

If we want to scope the auth context to Druid-settable only, then yes we should. IMO, I think that's a safer option but I could see people wanting to use this apart from Authorization result.

gianm · 2026-04-08T07:38:01Z


-    FutureUtils.getUnchecked(overlordClient.runTask(taskId, controllerTask), true);
+    // Propagate auth context headers to Overlord for consumption
+    if (plannerContext.getAuthenticationResult() != null && plannerContext.getAuthenticationResult().getContext() != null) {


What is the purpose of this? Authentication context is meant to be an arbitrary extension-defined in-memory map. It may not in general take well to being stuffed into a header. It may also include sensitive information that shouldn't be sent in a header.

We need a way to propagate AuthenticationResult from the broker to the overlord. I could make an overridable serialization method/class for AuthenticationResult to ensure we only pass-through what's needed (and let the user specify that), but propagating the headers here was the least intrusive approach for that change. We still need to pass the authentication result somehow.

gianm · 2026-04-08T07:47:27Z


+    // Inject auth context if provider is configured
+    if (taskAuthContextProvider != null) {
+      final AuthenticationResult authenticationResult = AuthorizationUtils.authenticationResultFromRequest(req);


How will this work in the SQL DML path, where the user submits a task to /druid/v2/sql/task/ and the Broker then submits the task using its own credentials? The current design is that the Broker authenticates the user, authorizes the DML operation, and then submits it to the Overlord using a service account (not the user's own credentials).

IMO it would make more sense in this case for the Broker to get the vended credentials and pass them along to the Overlord.

How will this work in the SQL DML path, where the user submits a task to /druid/v2/sql/task/ and the Broker then submits the task using its own credentials?

That's the thing. In our implementation, the broker passes through the user auth context and not its own credentials, so there's no use of the Broker credentials in the task payload (only used for validating the request came from the Brokers).

gianm · 2026-04-08T07:51:56Z

+ */
+@ExtensionPoint
+@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type")
+public interface TaskAuthContext


I didn't see an implementation of TaskAuthContext or TaskAuthContextProvider in this PR. Are they meant to be added later? What would they look like? I was wondering in particular what taskAuthContextProvider.createTaskAuthContext would do exactly with the authenticationResult that is passed in.

They are meant for users to implement. I have internal versions of these classes which take our internal identity tokens and propagate them through to the Iceberg input source, to be then used for vending S3 credentials to read data from Iceberg.

jtuglu1 · 2026-04-08T17:29:46Z

I reviewed the core changes only, focusing on trying to understand the security properties of the changes.

I wonder what your thoughts are on an alternate design that keeps the vended credentials inside the input source itself:

Add a new method to InputSource like scopeForUser(AuthenticationResult authResult). Default implementation is return this

Whenever a task is submitted, call scopeForUser on all of its input sources at whichever service initially accepts the task (either Broker [for SQL DML] or Overlord [for anything else]).

The IcebergInputSource would implement scopeForUser to fetch the vended credentials and transform itself into an input source that bakes in the vended credentials. It would use a PasswordProvider so it is redactable.

The idea would be to avoid the need for a new credential vending system in core, putting most of the changes inside the Iceberg extension instead.

Are you proposing pushing the vending of credentials using an identity to the broker/overlord prior to task submission? I'd ideally like to propagate the auth context to the task and have it vend the credentials at runtime, not at submit time.

Currently, the way this works is:

We extract user identity
We pass to an Iceberg catalog client which uses this identity to then vend S3 credentials
These S3 credentials are then attached to the subtasks that spawn from the driver task

jtuglu1 · 2026-04-09T20:13:57Z

@gianm any further thoughts on the above?

gianm · 2026-04-10T17:29:24Z

Are you proposing pushing the vending of credentials using an identity to the broker/overlord prior to task submission? I'd ideally like to propagate the auth context to the task and have it vend the credentials at runtime, not at submit time.

Yes, that's what I was proposing. There are a couple of reasons. First, because doing it at submit-time needs fewer changes to core. Second, because one of the proposed changes to core— the propagation of the user's own credentials between Druid services— is not something we've done before, and I feel this requires extra care from a security perspective. I was hoping to avoid the need for it completely.

Maybe a hybrid approach would work? We could introduce scopeForUser in core and run it at submit time. In your custom extension, rather than applying vended credentials at scope/submit time, you could use scopeForUser to embed the user's own credentials in the input source. We could add a PasswordProvider field to IcebergInputSource to support that. Then you could use them at runtime in the task to acquire vended credentials.

jtuglu1 · 2026-04-10T17:31:48Z

Maybe a hybrid approach would work? We could introduce scopeForUser in core and run it at submit time. In your custom extension, rather than applying vended credentials at scope/submit time, you could use scopeForUser to embed the user's own credentials in the input source. We could add a PasswordProvider field to IcebergInputSource to support that. Then you could use them at runtime in the task to acquire vended credentials.

I think this makes sense – my only concern is, for example, being able to reliably "touch-up" these inputSource fields in arbitrary specs (both on overlord and on broker) to apply scopeForUser(). For example, you can have an MSQ query that queries multiple iceberg tables and joins against a Druid table.

In general, having the identity at the task-level also opens up ability for Druid to do auth with other resources that tasks might read from (e.g. Kafka), so generally want to lean in direction of having identity/credentials exposable at the task-level (not just seeing the end credentials for whatever is needed).

jtuglu1 · 2026-04-13T18:36:13Z

Maybe a hybrid approach would work? We could introduce scopeForUser in core and run it at submit time. In your custom extension, rather than applying vended credentials at scope/submit time, you could use scopeForUser to embed the user's own credentials in the input source. We could add a PasswordProvider field to IcebergInputSource to support that. Then you could use them at runtime in the task to acquire vended credentials.

I think this makes sense – my only concern is, for example, being able to reliably "touch-up" these inputSource fields in arbitrary specs (both on overlord and on broker) to apply scopeForUser(). For example, you can have an MSQ query that queries multiple iceberg tables and joins against a Druid table.

In general, having the identity at the task-level also opens up ability for Druid to do auth with other resources that tasks might read from (e.g. Kafka), so generally want to lean in direction of having identity/credentials exposable at the task-level (not just seeing the end credentials for whatever is needed).

Another thing to add here is I'd like to (if possible) avoid putting the burden on the caller for "injecting" this user identity. IMO, if we can always do it at the overlord level (and put the burden on the task type implementor to actually do something with a valid, provided identity), that should be more maintainable. Especially in a world with multiple input sources and IMO code related to input sources (e.g. Kafka, Iceberg, Delta) should not be involved on anything but a supervisor thread and the task process itself. I think the (one?) exception to this might be with Dart on broker-only (but I'm not sure if accepts reading an external inputSource on broker-only w/o creating a task).

This pattern is seen in other master/worker ETL systems like Spark, Ray, Trino, etc.

jtuglu1 · 2026-04-15T06:08:54Z

@gianm @clintropolis any more thoughts here?

cecemei · 2026-04-17T00:26:55Z

@jtuglu1 looks like there's a merge conflict that needs to be resolved.

jtuglu1 · 2026-04-17T00:47:51Z

@cecemei I'm unsure this will get into v37 as we have not settled on design. I've removed this from v37 milestone.

FrankChen021 · 2026-04-25T05:13:24Z

-    FutureUtils.getUnchecked(overlordClient.runTask(taskId, controllerTask), true);
+    // Propagate auth context headers to Overlord for consumption
+    if (plannerContext.getAuthenticationResult() != null && plannerContext.getAuthenticationResult().getContext() != null) {
+      final Map<String, String> extraHeaders = CollectionUtils.mapValues(plannerContext.getAuthenticationResult().getContext(), String::valueOf);


[P2] Do not forward authenticator context as raw headers

AuthenticationResult.getContext() is authenticator-defined metadata, not a map of HTTP headers. Forwarding every entry directly into the Broker-to-Overlord request can leak arbitrary auth context to the internal RPC path and can also break MSQ task submission when a provider stores non-header-safe keys or values. Please propagate only an explicit allowlisted header/token shape or use a typed inter-service auth-context carrier.

jtuglu1 · 2026-05-04T18:27:50Z

We could add a PasswordProvider field to IcebergInputSource to support that. Then you could use them at runtime in the task to acquire vended credentials.

@gianm one other risk here is that this runs the risk of credential injection. For example, it does not guarantee that the identity of the user POSTing the task is the same as that which is embedded in the field in the spec. By making this a public, settable field by the user, an authorized user (who can WRITE to that datasource) may be able to submit a spec containing the credentials of another user (who has access to that input source). There is no mechanism to check whether the identity of the user submitting the task is the same as the identity accessing the resources (this could be enforced client-side but IMO Druid should be able to propagate and handle setting of this field). Users should not need to implement any of this logic client-side IMO. All they should need to worry about is translating a Druid identity into something their input source can understand/auth against.

Whether we want to support the odd case where a user might want to support calling with an inputSource-specific password provider is a valid consideration here, but I think we need to have some mechanism in-place to ensure that we do the natural thing which users expect by default (take my identity and pass it through to the task, using my identity to auth against any external resources).

jtuglu1 changed the title ~~Support user identity forwarding to tasks via TaskAuthContext~~ feat: support user identity forwarding to tasks via TaskAuthContext Mar 30, 2026

github-actions Bot added Area - Batch Ingestion Area - Dependencies Area - Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels Mar 30, 2026

jtuglu1 force-pushed the task-auth-context branch from f68a5ba to 3255f0b Compare March 30, 2026 23:31

jtuglu1 added the Design Review label Mar 30, 2026

github-advanced-security AI found potential problems Mar 31, 2026

View reviewed changes

jtuglu1 force-pushed the task-auth-context branch 3 times, most recently from f2012ec to 99c006c Compare March 31, 2026 02:13

github-advanced-security AI found potential problems Mar 31, 2026

View reviewed changes

jtuglu1 force-pushed the task-auth-context branch 4 times, most recently from 1f4e9ff to 09f57a7 Compare April 1, 2026 03:27

jtuglu1 added this to the 37.0.0 milestone Apr 1, 2026

jtuglu1 force-pushed the task-auth-context branch from 33be042 to 81fae04 Compare April 2, 2026 01:35

github-actions Bot added the Area - Querying label Apr 2, 2026

jtuglu1 force-pushed the task-auth-context branch from 81fae04 to 6ac3139 Compare April 6, 2026 18:25

Support user identity forwarding to tasks via TaskAuthContext

74d8369

jtuglu1 force-pushed the task-auth-context branch from 6ac3139 to 74d8369 Compare April 7, 2026 18:35

gianm reviewed Apr 8, 2026

View reviewed changes

jtuglu1 requested review from abhishekrb19 and gianm April 8, 2026 18:27

jtuglu1 requested a review from clintropolis April 16, 2026 22:59

jtuglu1 removed this from the 37.0.0 milestone Apr 17, 2026

FrankChen021 reviewed Apr 25, 2026

View reviewed changes

Uh oh!

Conversation

jtuglu1 commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Design

Constraints

Release note

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check notice

Check notice

Uh oh!

cecemei commented Apr 7, 2026

Uh oh!

jtuglu1 commented Apr 7, 2026

Uh oh!

gianm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jtuglu1 Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jtuglu1 Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jtuglu1 commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jtuglu1 commented Apr 9, 2026

Uh oh!

gianm commented Apr 10, 2026

Uh oh!

jtuglu1 commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jtuglu1 commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jtuglu1 commented Apr 15, 2026

Uh oh!

cecemei commented Apr 17, 2026

Uh oh!

jtuglu1 commented Apr 17, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jtuglu1 commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

jtuglu1 commented Mar 30, 2026 •

edited

Loading

jtuglu1 Apr 8, 2026 •

edited

Loading

jtuglu1 Apr 8, 2026 •

edited

Loading

jtuglu1 commented Apr 8, 2026 •

edited

Loading

jtuglu1 commented Apr 10, 2026 •

edited

Loading

jtuglu1 commented Apr 13, 2026 •

edited

Loading

jtuglu1 commented May 4, 2026 •

edited

Loading