Broker Config and Query Blocklist by mshahid6 · Pull Request #19011 · apache/druid

mshahid6 · 2026-02-11T05:59:57Z

Description

Added a new BrokerDynamicConfig to enable dynamic, runtime configuration of Druid brokers without restarts and a query blocklist for dynamically blocking queries in the situation there is a rogue app/user spamming the cluster without relying on static configs/restarts.

QueryBlocklistRule

Enforced early in QueryLifecycle (after init) and throws DruidException when a query matches a rule i.e. if ALL specified criteria match (AND logic). Null or empty criteria act as wildcards (match everything):

dataSources: Matches if ANY datasource in the query intersects with the rule's datasources
queryTypes: Matches if the query type is in the rule's query types
contextMatches: Matches if ALL key-value pairs in the rule match the query context (exact string match)

Query blocklist is managed via the existing coordinator config APIs:

GET /druid/coordinator/v1/config/broker

POST /druid/coordinator/v1/config/broker
  {
    "queryBlocklist": [
      {
        "ruleName": "block-wikipedia-groupbys",
        "dataSources": ["wikipedia"],
        "queryTypes": ["groupBy"],
        "contextMatches": {"priority": "0"}
      }
    ]
  }

GET /druid/coordinator/v1/config/broker/history

Considerations

Creating a separate broker-level dynamic config was considered which can also be used for other future features such as datasource aliasing, routing rules or feature flags. However, Coordinator handles communications with the metadata store and syncs to brokers so that there are fewer servers communicating with the metadata store and future built-in metadata store operations go through central leaders

The query blocklist follows a "best effort" approach:

If BrokerViewOfCoordinatorConfig is null, blocklist check is skipped
If BrokerDynamicConfig is null (config not yet loaded), queries are allowed to proceed
Once config is loaded, blocklist enforcement begins

Rationale: The query blocklist is an operational safety feature, not a security control. Blocking all queries when config is unavailable would be more disruptive than the problem it's trying to solve. Operators can still use coordinator/historical unavailability to enforce hard blocks if needed.

Release note

Added query blocklist feature for dynamically blocking queries without restarts. Operators can block queries by datasource, query type, or query context using the new /druid/coordinator/v1/config/broker API. Rules use AND logic (all criteria must match) and are stored in the metadata database.

Key changed/added classes in this PR

BrokerDynamicConfig - New broker-specific dynamic config with queryBlocklist field
BrokerDynamicConfigSyncer - Coordinator component that syncs config to brokers
BrokerViewOfBrokerConfig - Broker component that fetches config from coordinator
BaseDynamicConfigSyncer - Base class for config syncers (extracted)
BaseBrokerViewOfConfig - Base class for broker config views (extracted)
QueryBlocklistRule - Rule-based query matching with datasource/type/context criteria
CoordinatorBrokerConfigsResource - REST endpoints for managing broker config
QueryLifecycle - Added checkQueryBlocklist() method to enforce rules
QueryLifecycleFactory - Injects BrokerViewOfBrokerConfig for accessing blocklist
CoordinatorDynamicConfigSyncer - Refactored to extend BaseDynamicConfigSyncer
BrokerViewOfCoordinatorConfig - Refactored to extend BaseBrokerViewOfConfig
CliBroker - Added Guice binding for BrokerViewOfBrokerConfig
CliCoordinator - Added Guice binding for BrokerDynamicConfigSyncer
DruidCoordinator - Added lifecycle management for BrokerDynamicConfigSyncer
BrokerDynamicConfigTest
BrokerDynamicConfigResourceTest
CoordinatorBrokerConfigsResourceTest
QueryBlocklistRuleTest
QueryLifecycleTest
BrokerViewOfBrokerConfigTest

This PR has:

gianm

This is a useful concept, although I think it should be done in such a way that the Coordinator handles communications with the metadata store. Other things that work this way include lookup definitions, users and roles for basic authentication and authorization, centralized schema, etc.

The rationale for why we've done it this way, in these other cases, is three-fold:

Operationally, it's better if a smaller number of servers is communicating with the metadata store, to avoid overloading it and to make it easier to configure security for the metadata store.
Syncing dynamic configurations from a central leader (Coordinator) allows updates to propagate more quickly.
Someday we may want to make an option to use a builtin metadata store. This will be simplest to implement if metadata store operations go through central leaders. Those leaders would naturally become the place where

The way it can work is:

Move the user-facing POST and GET APIs to the Coordinator.
Brokers pull configs from the Coordinator on startup.
When the Coordinator receives an update to the config, it should push it out to an internal POST API on the Brokers.
Brokers also pull configs from the Coordinator periodically, in case they missed a push.

There are possible race conditions surrounding the push and pull: it's possible a new config is getting pushed out around the same time that the Broker is doing a scheduled pull. To solve these you can include a timestamp in the config object, and have the Brokers reject configs with older timestamps than the current one.

Please also consider what guarantees should exist around availability of the Broker config. Is it "required", i.e., if the Broker can't fetch it on startup then startup will fail? Or is it "best effort", i.e., if the Broker can't fetch it on startup, we proceed with a null config or default config for some time, until the next scheduled pull succeeds?

…ttern

jtuglu1

Did a first pass – overall looking good!

jtuglu1 · 2026-02-13T05:07:17Z

Another thing I forgot to mention is docs + UI, namely:

Docs:

Add the field to the coordinator dynamic config description, explaining what it does.
Separate documentation section for the rulesets. What the rule format looks like (description of each field, its default, etc.). The AND logic between the fields' predicates.

UI:
For completeness since coordinator config already in UI, do we want to add this as a field? Should be pretty simple to do with an AI tool.

…g JSON

jtuglu1

Overall, LGTM. Left a few last comments.

jtuglu1

LGTM, approved. Left a few nits, but overall 👍.

jtuglu1

One last thing. Looks like there's some straggler ImmutableMap – I'll merge once those are fixed up.

jtuglu1 · 2026-02-26T01:20:46Z

@gianm any final thoughts here? Otherwise, I'm planning to merge this as-is. We can address any subsequent comments in a follow-up.

maytasm

LGTM.
One thing Im wondering is if we should make a new API (on the coordinator) for this blocklist rules and store it separately from the Coordinator Dynamic configs. For example, in Basic Security, you can POST(/druid-ext/basic-security/authentication/db/{authenticatorName}/users/{userName}) on the Coordinator to create a new User. Basically we would have a new set of APIs for GET, POST, etc for blocklist resource. I think this is what @gianm is thinking of in #19011 (review). Although reusing the Coordinator Dynamic configs gives us for 'free' all the existing framework for syncing configs between Broker and Coordinator. Let's wait for @gianm

mshahid6 · 2026-02-26T17:10:21Z

@maytasm oh interesting, just curious, what are the advantages of doing it that way?

jtuglu1 · 2026-02-26T17:17:23Z

LGTM. One thing Im wondering is if we should make a new API (on the coordinator) for this blocklist rules and store it separately from the Coordinator Dynamic configs. For example, in Basic Security, you can POST(/druid-ext/basic-security/authentication/db/{authenticatorName}/users/{userName}) on the Coordinator to create a new User. Basically we would have a new set of APIs for GET, POST, etc for blocklist resource. I think this is what @gianm is thinking of in #19011 (review). Although reusing the Coordinator Dynamic configs gives us for 'free' all the existing framework for syncing configs between Broker and Coordinator. Let's wait for @gianm

I'm not sure this is needed right now. 1) IMO we don't have a need programmatically (yet) for this that we cannot solve with a get/put of dynamic config API. 2) you can make the same argument for a good portion of the overlord dynamic config as well, yet that also is a single, dynamic config API. I think it keeps things simple for now, and if we want to expand it later, we can. You get all the built-in updating/propagation/auditing support for something that (hopefully) shouldn't be updated that often. However, if we were to extend this to datasource aliasing (which I don't think would fit in here), then I can imagine a higher write tput necessitating a separate API. The litmus test, IMO, for whether to extract something from dynamic config should be: how frequently is it read/written to (by clients), how "large" is the config (does it scale linearly with some sort of cluster resource – datasource, user, node, etc.), and how long do these configs tend to stay set to a specific value (e.g. is it truly "dynamic").

maytasm · 2026-02-27T06:59:14Z

@maytasm oh interesting, just curious, what are the advantages of doing it that way?

I agree on all of @jtuglu1 points above. I don't have strong opinion but just feel a little weird for this config to be in Coordinator Dynamic config as it has nothing to do with Coordination. All the other configs in Coordinator Dynamic config are related to Coordinator Duties like moving segments, loading segments, killing segments, replicating segments, etc.

gianm

I had been imagining a separate config for this, not making it part of CoordinatorDynamicConfig. Something like BrokerDynamicConfig that is still managed on the Coordinator, but that is also synced to all the Brokers.

It should be straightforward to do, because it should be closely analogous to CoordinatorDynamicConfig. I recommend doing it. It would allow us to keep the Coordinator and Broker dynamic configs cleanly separated, and add more Broker configs over time without overly complicating the single object.

jtuglu1

LGTM – thank you! Few non-blocking comments and think we can merge on Monday. It might be nice to add an embedded test that exercises both the coordinator and broker dynamic config read/write paths, but that can also be added later.

maytasm

LGTM. Thanks!!

gianm

The latest version LGTM.

abhishekrb19 · 2026-03-10T05:13:40Z

I think something from this PR is causing one of the embedded tests to consistently fail in master and on all recent PRs:

[Execute: .github/scripts/run_unit-tests -Dtest=!QTest,'I*,A*,U*' -Dmaven.test.failure.ignore=true](https://github.com/apache/druid/actions/runs/22832980666/job/66354459774#annotation:6:8087)
Process completed with exit code 1.

Error:  Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.5.4:test (default-test) on project druid-embedded-tests: 
Error:  
Error:  See /home/runner/work/druid/druid/embedded-tests/target/surefire-reports for the individual test results.
Error:  See dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
Error:  The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
Error:  Command was /bin/sh -c cd '/home/runner/work/druid/druid/embedded-tests' && '/opt/hostedtoolcache/Java_Zulu_jdk/21.0.10-7/x64/bin/java' '--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED' '--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED' '--add-opens=java.base/java.nio=ALL-UNNAMED' '--add-opens=java.base/sun.nio.ch=ALL-UNNAMED' '--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED' '--add-opens=java.base/java.io=ALL-UNNAMED' '--add-opens=java.base/java.lang=ALL-UNNAMED' '--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED' '--add-opens=java.base/java.util=ALL-UNNAMED' '-Djava.security.manager=allow' '-Xmx2048m' '-XX:MaxDirectMemorySize=2500m' '-XX:+ExitOnOutOfMemoryError' '-XX:+HeapDumpOnOutOfMemoryError' '-Duser.language=en' '-Duser.GroupByQueryRunnerTest.javacountry=US' '-Dfile.encoding=UTF-8' '-Duser.timezone=UTC' '-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager' '-Daws.region=us-east-1' '-Ddruid.test.stupidPool.poison=true' '-XX:OnOutOfMemoryError=/home/runner/work/druid/druid/dev/chmod-heap-dumps.sh' '-XX:HeapDumpPath=/home/runner/work/druid/druid/target' '-Ddruid.indexing.doubleStorage=double' '-javaagent:/home/runner/work/druid/druid/jfr-profiler.jar' '-Djfr.profiler.http.username=druid-ci' '-Djfr.profiler.http.***' '-Djfr.profiler.tags.project=druid' '-Djfr.profiler.tags.jvm_version=21.0.10' '-Djfr.profiler.tags.run_id=22832980666' '-Djfr.profiler.tags.run_number=19767' '-Djfr.profiler.tags.run_attempt=2' '-Djfr.profiler.tags.key=test-jdk21-[I*,A*,U*]' '-Djfr.profiler.tags.event_ref=pull_request-19011' '-Djfr.profiler.tags.run_url=https://github.com/apache/druid/actions/runs/22832980666' '-jar' '/home/runner/work/druid/druid/embedded-tests/target/surefire/surefirebooter-20260309201804026_2049.jar' '/home/runner/work/druid/druid/embedded-tests/target/surefire' '2026-03-09T20-00-18_955-jvmRun1' 'surefire-20260309201804026_2047tmp' 'surefire_23-20260309201804026_2048tmp'
Error:  Error occurred in starting fork, check output in log
Error:  Process Exit Code: 1
Error:  org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
Error:  Command was /bin/sh -c cd '/home/runner/work/druid/druid/embedded-tests' && '/opt/hostedtoolcache/Java_Zulu_jdk/21.0.10-7/x64/bin/java' '--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED' '--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED' '--add-opens=java.base/java.nio=ALL-UNNAMED' '--add-opens=java.base/sun.nio.ch=ALL-UNNAMED' '--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED' '--add-opens=java.base/java.io=ALL-UNNAMED' '--add-opens=java.base/java.lang=ALL-UNNAMED' '--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED' '--add-opens=java.base/java.util=ALL-UNNAMED' '-Djava.security.manager=allow' '-Xmx2048m' '-XX:MaxDirectMemorySize=2500m' '-XX:+ExitOnOutOfMemoryError' '-XX:+HeapDumpOnOutOfMemoryError' '-Duser.language=en' '-Duser.GroupByQueryRunnerTest.javacountry=US' '-Dfile.encoding=UTF-8' '-Duser.timezone=UTC' '-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager' '-Daws.region=us-east-1' '-Ddruid.test.stupidPool.poison=true' '-XX:OnOutOfMemoryError=/home/runner/work/druid/druid/dev/chmod-heap-dumps.sh' '-XX:HeapDumpPath=/home/runner/work/druid/druid/target' '-Ddruid.indexing.doubleStorage=double' '-javaagent:/home/runner/work/druid/druid/jfr-profiler.jar' '-Djfr.profiler.http.username=druid-ci' '-Djfr.profiler.http.***' '-Djfr.profiler.tags.project=druid' '-Djfr.profiler.tags.jvm_version=21.0.10' '-Djfr.profiler.tags.run_id=22832980666' '-Djfr.profiler.tags.run_number=19767' '-Djfr.profiler.tags.run_attempt=2' '-Djfr.profiler.tags.key=test-jdk21-[I*,A*,U*]' '-Djfr.profiler.tags.event_ref=pull_request-19011' '-Djfr.profiler.tags.run_url=https://github.com/apache/druid/actions/runs/22832980666' '-jar' '/home/runner/work/druid/druid/embedded-tests/target/surefire/surefirebooter-20260309201804026_2049.jar' '/home/runner/work/druid/druid/embedded-tests/target/surefire' '2026-03-09T20-00-18_955-jvmRun1' 'surefire-20260309201804026_2047tmp' 'surefire_23-20260309201804026_2048tmp'
Error:  Error occurred in starting fork, check output in log
Error:  Process Exit Code: 1
Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:643)

https://github.com/apache/druid/actions/runs/22832980666/job/66354459774

abhishekrb19 · 2026-03-10T06:27:43Z

Just downloaded the test-report artifacts from GitHub and I see org.apache.druid.testing.embedded.docker.IngestionBackwardCompatibilityDockerTest was the last embedded test and the test JVM crashes.

Since the test uses an old Coordinator from 31.0.0, would some of the changes here be compatible with that? I’m wondering if the test might actually be uncovering a legitimate backwards compatibility issue with an old Coordinator + new Brokers.

cc: @mshahid6 @jtuglu1

jtuglu1 · 2026-03-10T06:31:40Z

IngestionBackwardCompatibilityDockerTest.java

Yes, I saw that too – looking. My guess is brokers are calling coordinator config endpoint on boot (which doesn't exist on old coordinators).

abhishekrb19 · 2026-03-10T06:40:11Z

+  public ListenableFuture<BrokerDynamicConfig> getBrokerDynamicConfig()
+  {
+    return FutureUtils.transform(
+        client.asyncRequest(
+            new RequestBuilder(HttpMethod.GET, "/druid/coordinator/v1/broker/config"),
+            new BytesFullResponseHandler()
+        ),
+        holder -> JacksonUtils.readValue(


Looks like this change is causing org.apache.druid.testing.embedded.docker.IngestionBackwardCompatibilityDockerTest to fail consistently (which seems like it would be a legit issue during rolling upgrades since the Coordinators are typically upgraded in the end).

I haven’t looked closely at the surrounding code yet, but either this code or the caller likely needs to account for the new /druid/coordinator/v1/broker/config endpoint not being available on Coordinators during rolling upgrades in a backward compatible manner.

Yes, that must be the reason.

Probably just catching the not found exception and continuing with the Broker start up should suffice.

Yeah – I have a test branch that I'll raise in a bit to fix this

Will summarize the changes there

Thanks for the fix, @jtuglu1! (And to @kfaraz for adding the backward compatibility embedded test 🙂)

Broker Dynamic Config and Query Blocklist

a92b6fb

github-actions Bot added the Area - Querying label Feb 11, 2026

gianm reviewed Feb 11, 2026

View reviewed changes

mshahid6 added 2 commits February 12, 2026 10:18

Add query blocklist to CoordinatorDynamicConfig following existing pa…

37810cd

…ttern

Fix SqlModuleTest

24ad226

jtuglu1 self-requested a review February 12, 2026 19:06

jtuglu1 reviewed Feb 12, 2026

View reviewed changes

Comment thread server/src/main/java/org/apache/druid/server/QueryBlocklistRule.java

mshahid6 changed the title ~~Broker Dynamic Config and Query Blocklist~~ Query Blocklist Feb 12, 2026

update from comments

928fb85

add query blocklist documentation and UI in coordinator dynamic confi…

25da56e

…g JSON

github-actions Bot added Area - Documentation Area - Web Console labels Feb 19, 2026

add blocklist to spellcheck exceptions

ab49ff0

mshahid6 requested review from gianm and jtuglu1 February 20, 2026 17:19

jtuglu1 requested review from vogievetsky February 20, 2026 18:01

jtuglu1 reviewed Feb 20, 2026

View reviewed changes

mshahid6 added 2 commits February 20, 2026 14:29

updates from comments

acfe8c3

add map import

f7ca704

jtuglu1 self-requested a review February 23, 2026 22:44

jtuglu1 approved these changes Feb 23, 2026

View reviewed changes

Comment thread server/src/test/java/org/apache/druid/segment/metadata/SegmentMetadataCacheTestBase.java Outdated

Comment thread server/src/main/java/org/apache/druid/server/QueryLifecycle.java Outdated

comment updates

42b2f0e

jtuglu1 approved these changes Feb 25, 2026

View reviewed changes

Comment thread server/src/test/java/org/apache/druid/server/QueryResourceTest.java Outdated

replace ImmutableMap.of with Map.of()

06eb3b3

maytasm approved these changes Feb 26, 2026

View reviewed changes

gianm reviewed Mar 3, 2026

View reviewed changes

separate Broker Dynamic Config and updates from comments

a32cb18

github-advanced-security AI found potential problems Mar 3, 2026

View reviewed changes

Comment thread server/src/main/java/org/apache/druid/server/http/CoordinatorBrokerConfigsResource.java Fixed

mshahid6 added 3 commits March 3, 2026 12:46

small fixes

3b174fb

improve test coverage

deb129c

more test coverage

5c1c37e

maytasm requested changes Mar 4, 2026

View reviewed changes

updates from comments

d6bbe89

mshahid6 changed the title ~~Query Blocklist~~ Broker Config and Query Blocklist Mar 5, 2026

jtuglu1 self-requested a review March 7, 2026 00:35

jtuglu1 reviewed Mar 7, 2026

View reviewed changes

comment updates

8b15f3f

mshahid6 requested a review from jtuglu1 March 7, 2026 05:25

maytasm approved these changes Mar 8, 2026

View reviewed changes

Comment thread server/src/main/java/org/apache/druid/server/http/BrokerDynamicConfigSyncer.java Outdated

use builder

e0f3e3d

jtuglu1 approved these changes Mar 9, 2026

View reviewed changes

gianm approved these changes Mar 9, 2026

View reviewed changes

jtuglu1 added the Release Notes label Mar 9, 2026

jtuglu1 merged commit 74342b2 into apache:master Mar 9, 2026
61 of 63 checks passed

mshahid6 deleted the broker-config-query-blocklist branch March 9, 2026 21:22

github-actions Bot added this to the 37.0.0 milestone Mar 9, 2026

abhishekrb19 mentioned this pull request Mar 10, 2026

Update Broker segment metadata cache more often in builtin configs. #19115

Merged

abhishekrb19 reviewed Mar 10, 2026

View reviewed changes

Uh oh!

Conversation

mshahid6 commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

QueryBlocklistRule

Considerations

Release note

Key changed/added classes in this PR

Uh oh!

gianm left a comment

Choose a reason for hiding this comment

Uh oh!

jtuglu1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jtuglu1 commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jtuglu1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jtuglu1 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jtuglu1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jtuglu1 commented Feb 26, 2026

Uh oh!

maytasm left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mshahid6 commented Feb 26, 2026

Uh oh!

jtuglu1 commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maytasm commented Feb 27, 2026

Uh oh!

gianm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jtuglu1 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mshahid6 commented Feb 11, 2026 •

edited

Loading

jtuglu1 commented Feb 13, 2026 •

edited

Loading

jtuglu1 left a comment •

edited

Loading

maytasm left a comment •

edited

Loading

jtuglu1 commented Feb 26, 2026 •

edited

Loading

jtuglu1 left a comment •

edited

Loading

abhishekrb19 commented Mar 10, 2026 •

edited

Loading

jtuglu1 commented Mar 10, 2026 •

edited

Loading

abhishekrb19 Mar 10, 2026 •

edited

Loading