Skip to content

Commit 6669fb6

Browse files
morningmanclaude
andcommitted
[fix](ranger) Load the Ranger plugin on its own thread, so an FE starts without waiting for Ranger
RangerBasePlugin.init() downloads the service's roles and policies before it returns - and, now that the policies carry the user store enricher, the user store too. The plugin behind access_controller_type=ranger-doris is built in the Env constructor, so those three REST calls, bounded only by the plugin's REST timeouts and retries, were time the FE could not start for. BackgroundLoadedRangerPlugin (ranger-common) is now the base of both Doris plugins: init() starts RangerBasePlugin.init() on a RangerPluginLoader thread and returns; every answer the plugin gives, and the groups a request is built with, wait for that first load to end (awaitLoaded), so a check made before it has ended is answered after it - never out of an empty engine - and RangerAccessController waits before taking its lifecycle fence, so a slow admin is waited for with no lock held. cleanup() during the load returns at once and the loader stops the plugin once the load has ended. The user store enricher and the start-up log line move from the two plugins into the base. Measured on a local FE against the docker Ranger: the controller is built 1 ms after the plugin, the load takes 861 ms on its own thread and ends before the FE is master. Against an unreachable admin the FE is master after 7 s and opens its query port at 40 s while the load runs for 70 s (three calls, each retried, then the local cache); a SHOW DATABASES issued at 40 s by a user in a Ranger group waits 36 s and then answers out of the cached policies and user store, and SELECT 1 answers at once. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent c6b681a commit 6669fb6

14 files changed

Lines changed: 602 additions & 98 deletions

File tree

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,245 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.catalog.authorizer.ranger;
19+
20+
import com.google.common.annotations.VisibleForTesting;
21+
import org.apache.logging.log4j.LogManager;
22+
import org.apache.logging.log4j.Logger;
23+
import org.apache.ranger.plugin.policyengine.RangerAccessRequest;
24+
import org.apache.ranger.plugin.policyengine.RangerAccessResult;
25+
import org.apache.ranger.plugin.policyengine.RangerAccessResultProcessor;
26+
import org.apache.ranger.plugin.service.RangerBasePlugin;
27+
import org.apache.ranger.plugin.util.ServicePolicies;
28+
29+
import java.util.Collection;
30+
import java.util.concurrent.CountDownLatch;
31+
import java.util.concurrent.TimeUnit;
32+
import java.util.concurrent.atomic.AtomicBoolean;
33+
import java.util.concurrent.atomic.AtomicReference;
34+
35+
/**
36+
* The plugin a Doris Ranger source answers out of: a {@link RangerBasePlugin} that loads without holding up
37+
* whoever built it, and whose requests can carry the groups Ranger keeps for a user.
38+
*
39+
* <p><b>The first load runs on a thread of its own.</b> {@code RangerBasePlugin.init()} does not return until
40+
* it has downloaded the service's roles and its policies from the Ranger admin - and, because the policies
41+
* arrive with the user store enricher on them (below), the user store as well: three REST calls, bounded
42+
* only by the plugin's REST timeouts and retries ({@code policy.rest.client.connection.timeoutMs} is two
43+
* minutes by default, {@code read.timeoutMs} thirty seconds). The plugin behind
44+
* {@code access_controller_type=ranger-doris} is built in the {@code Env} constructor, on the thread starting
45+
* the FE, so that used to be time an FE could not start for: a user store the size of an enterprise
46+
* directory, or an admin that is slow or unreachable, held the whole process. Here {@link #init()} starts
47+
* that load and returns, and {@link #awaitLoaded()} is where
48+
* whoever needs its outcome waits - which every answer this plugin gives does first, so a check made before
49+
* the load has ended is answered after it, never out of an engine that has nothing in it yet. Whatever else
50+
* the FE does to start runs meanwhile, and the check that pays for Ranger's latency is the first one that
51+
* needs Ranger, not every process that embeds it.
52+
*
53+
* <p>What a check waits for is exactly what the constructor used to guarantee: that the load has
54+
* <em>ended</em> - with the policies from the admin, or from the local cache when the admin could not be
55+
* reached, or with nothing at all, in which case the engine is null and {@code RangerAccessController}
56+
* refuses, as it always has. It is not shortened by a timeout of its own: the load is bounded by the REST
57+
* timeouts the operator already tunes, and answering out of an empty engine before it has ended would be
58+
* refusing checks the policies are about to allow - and, worse, passing ones a policy written against a
59+
* group is about to deny.
60+
*
61+
* <p>Stopping it is {@link #cleanup()}, as before. Stopped while still loading - a {@code CREATE CATALOG}
62+
* dry run, the loser of a race in a factory - it finishes the load first and stops itself then, on the
63+
* loading thread: what is running is a REST call, which an interrupt does not cut short, so waiting for it
64+
* here would put the whole REST timeout back onto the thread closing a catalog.
65+
*
66+
* <p><b>Requests carry Ranger's own groups.</b> Doris has none to offer, and a policy item written against a
67+
* group matches nothing without them; the store they are read from is asked for with the policies, see
68+
* {@link RangerUserStoreGroups}.
69+
*
70+
* <p>A plugin that is never {@link #init() initialized} - a test's, answering out of overrides of its own -
71+
* has nothing to wait for, and {@link #awaitLoaded()} returns at once.
72+
*/
73+
public abstract class BackgroundLoadedRangerPlugin extends RangerBasePlugin {
74+
private static final Logger LOG = LogManager.getLogger(BackgroundLoadedRangerPlugin.class);
75+
76+
/** Released once the first load has ended, however it ended. */
77+
private final CountDownLatch loaded = new CountDownLatch(1);
78+
/** The thread running the first load, from the moment {@link #init()} starts it. */
79+
private final AtomicReference<Thread> loader = new AtomicReference<>();
80+
/** Whether {@link #cleanup()} has been asked for; read by the loader when the load ends. */
81+
private volatile boolean stopRequested;
82+
/** Whether the stop has run. It runs once, from whichever of {@link #cleanup()} and the loader is last. */
83+
private final AtomicBoolean stopped = new AtomicBoolean();
84+
85+
protected BackgroundLoadedRangerPlugin(String serviceType, String serviceName, String appId) {
86+
super(serviceType, serviceName, appId);
87+
}
88+
89+
/**
90+
* Starts the first load - {@code RangerBasePlugin.init()}, roles, policies and user store - on a thread
91+
* of its own, and returns at once. Once per plugin.
92+
*/
93+
@Override
94+
public void init() {
95+
Thread thread = new Thread(this::load, "RangerPluginLoader(serviceType=" + getServiceType()
96+
+ ", serviceName=" + getServiceName() + ")");
97+
if (!loader.compareAndSet(null, thread)) {
98+
throw new IllegalStateException("Ranger plugin for service " + getServiceName()
99+
+ " has already been initialized");
100+
}
101+
LOG.info(RangerUserStoreGroups.describe(getConfig()));
102+
// A daemon: it ends with the load, and a load still running when the FE exits is not worth waiting
103+
// for. What it inherits, and what the refresher threads it goes on to start inherit from it, is the
104+
// context classloader of the thread calling this - the plugin's own, set by the engine around the
105+
// factory call - which is what keeps Ranger's class-name lookups working after that call returns.
106+
thread.setDaemon(true);
107+
thread.start();
108+
}
109+
110+
private void load() {
111+
long startedAtNanos = System.nanoTime();
112+
try {
113+
if (stopRequested) {
114+
// Stopped before the load began; there is nothing to load for.
115+
return;
116+
}
117+
firstLoad();
118+
LOG.info("Ranger service {} loaded in {} ms: policies version {}, roles version {}, user store"
119+
+ " version {}", getServiceName(),
120+
TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startedAtNanos), getPoliciesVersion(),
121+
getRolesVersion(), getUserStoreVersion());
122+
} catch (Throwable e) {
123+
// Everything RangerBasePlugin.init() does about an admin it cannot reach is logged and survived
124+
// inside it, so this is something else - a broken audit configuration, say. It used to fail the
125+
// FE's start; now it fails every check against this service until the FE is restarted, which
126+
// RangerAccessController reports on each one as an engine that is not initialized.
127+
LOG.error("Ranger service {} failed to load; every check against it is refused until the FE is"
128+
+ " restarted", getServiceName(), e);
129+
} finally {
130+
loaded.countDown();
131+
if (stopRequested) {
132+
stopOnce();
133+
}
134+
}
135+
}
136+
137+
/**
138+
* The load itself: {@code RangerBasePlugin.init()}, which downloads everything this plugin answers out
139+
* of before it returns. A test's plugin overrides it to stand in for the Ranger admin.
140+
*/
141+
protected void firstLoad() {
142+
super.init();
143+
}
144+
145+
/**
146+
* Waits until the first load has ended, if one was started; see the class comment for what that means.
147+
*
148+
* <p>Interrupted, it returns with the interrupt restored rather than throwing: what follows it is a
149+
* question to an engine that may not be there yet, whose null answer the caller already reads as a
150+
* refusal.
151+
*/
152+
public void awaitLoaded() {
153+
if (loader.get() == null || isLoaded()) {
154+
return;
155+
}
156+
try {
157+
loaded.await();
158+
} catch (InterruptedException e) {
159+
Thread.currentThread().interrupt();
160+
}
161+
}
162+
163+
/**
164+
* Whether the first load has ended - not whether it found anything, see {@link #awaitLoaded()} - and so
165+
* false for a plugin that was never initialized, which has no load to end.
166+
*/
167+
public boolean isLoaded() {
168+
return loaded.getCount() == 0;
169+
}
170+
171+
/**
172+
* Stops the plugin's threads, or - while the first load is still running - arranges for the loader to
173+
* stop them once it has ended, and returns at once.
174+
*/
175+
@Override
176+
public void cleanup() {
177+
if (Thread.currentThread() == loader.get()) {
178+
// RangerBasePlugin.init() begins by clearing whatever state a plugin has, through this method; on
179+
// the loader that is what a call to it is, not a stop.
180+
super.cleanup();
181+
return;
182+
}
183+
stopRequested = true;
184+
if (loader.get() == null || isLoaded()) {
185+
stopOnce();
186+
}
187+
// Otherwise the loader stops the plugin when the load ends, see load(): between its countDown and its
188+
// read of stopRequested, and this method's write of stopRequested and its read of the latch, at
189+
// least one side sees the other's write, and stopOnce keeps it to one stop.
190+
}
191+
192+
private void stopOnce() {
193+
if (stopped.compareAndSet(false, true)) {
194+
super.cleanup();
195+
}
196+
}
197+
198+
@VisibleForTesting
199+
boolean isStopped() {
200+
return stopped.get();
201+
}
202+
203+
/**
204+
* Takes the policies Ranger downloaded, and asks for the user store with them, so that the requests the
205+
* source over this plugin builds can carry the groups Ranger keeps for a user; see
206+
* {@link RangerUserStoreGroups}. Done on the way in rather than left to Ranger, because Ranger only does
207+
* it on its own from 2.5 on, and behind a property.
208+
*/
209+
@Override
210+
public void setPolicies(ServicePolicies policies) {
211+
RangerUserStoreGroups.addUserStoreEnricher(getConfig(), policies);
212+
super.setPolicies(policies);
213+
}
214+
215+
// Every answer waits for the first load to end. These four are the ones Doris asks; the one-argument
216+
// isAccessAllowed overloads arrive here through RangerBasePlugin.
217+
218+
@Override
219+
public RangerAccessResult isAccessAllowed(RangerAccessRequest request,
220+
RangerAccessResultProcessor resultProcessor) {
221+
awaitLoaded();
222+
return super.isAccessAllowed(request, resultProcessor);
223+
}
224+
225+
@Override
226+
public Collection<RangerAccessResult> isAccessAllowed(Collection<RangerAccessRequest> requests,
227+
RangerAccessResultProcessor resultProcessor) {
228+
awaitLoaded();
229+
return super.isAccessAllowed(requests, resultProcessor);
230+
}
231+
232+
@Override
233+
public RangerAccessResult evalDataMaskPolicies(RangerAccessRequest request,
234+
RangerAccessResultProcessor resultProcessor) {
235+
awaitLoaded();
236+
return super.evalDataMaskPolicies(request, resultProcessor);
237+
}
238+
239+
@Override
240+
public RangerAccessResult evalRowFilterPolicies(RangerAccessRequest request,
241+
RangerAccessResultProcessor resultProcessor) {
242+
awaitLoaded();
243+
return super.evalRowFilterPolicies(request, resultProcessor);
244+
}
245+
}

‎fe/fe-authorization/fe-authorization-plugins/fe-authorization-plugin-ranger-common/src/main/java/org/apache/doris/catalog/authorizer/ranger/RangerAccessController.java‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,6 @@
3535
import org.apache.ranger.plugin.policyengine.RangerAccessResourceImpl;
3636
import org.apache.ranger.plugin.policyengine.RangerAccessResult;
3737
import org.apache.ranger.plugin.policyengine.RangerAccessResultProcessor;
38-
import org.apache.ranger.plugin.service.RangerBasePlugin;
3938

4039
import java.util.Collection;
4140
import java.util.HashMap;
@@ -58,6 +57,11 @@
5857
* give - its policy engine not yet initialized, or being cleaned up - is a refusal too, not an empty policy
5958
* set. Getting the second one wrong is silent: the engine reads "no row filter, no column mask" as licence to
6059
* read the table whole and in the clear.
60+
*
61+
* <p>A plugin still on its first load is neither: it is about to have an answer, and every question here
62+
* waits for it ({@link BackgroundLoadedRangerPlugin#awaitLoaded()}) before anything else - before the
63+
* lifecycle fence above all, so that a load the Ranger admin is slow to serve is waited for with no lock
64+
* held, and closing this controller meanwhile is not queued behind it.
6165
*/
6266
public abstract class RangerAccessController implements AuthorizationPlugin {
6367
private static final Logger LOG = LogManager.getLogger(RangerAccessController.class);
@@ -181,6 +185,7 @@ protected interface PluginCheck {
181185
@Override
182186
public final void checkPrivilege(AuthorizedSubject subject, AuthorizedResource resource,
183187
AccessRequirement requirement, AccessContext context) throws AccessDeniedException {
188+
getPlugin().awaitLoaded();
184189
checkWhileOpen(resource, () -> checkPrivilegeInternal(subject, resource, requirement, context));
185190
}
186191

@@ -334,6 +339,7 @@ private static String policyIdent(RangerAccessResult policy) {
334339
@Override
335340
public List<RowFilterSpec> getRowFilters(AuthorizedSubject subject, AuthorizedResource.Table table,
336341
AccessContext context) {
342+
getPlugin().awaitLoaded();
337343
return whileOpen(() -> evalRowFilterPolicies(subject, table, context));
338344
}
339345

@@ -383,6 +389,7 @@ private List<RowFilterSpec> evalRowFilterPolicies(AuthorizedSubject subject,
383389
@Override
384390
public Map<String, DataMaskSpec> getDataMasks(AuthorizedSubject subject, AuthorizedResource.Table table,
385391
Set<String> columns, AccessContext context) {
392+
getPlugin().awaitLoaded();
386393
return whileOpen(() -> {
387394
Map<String, DataMaskSpec> masks = new HashMap<>();
388395
for (String column : columns) {
@@ -520,7 +527,7 @@ protected String dataMaskExpressionOf(RangerAccessResult policy, String maskType
520527

521528
protected abstract RangerAccessResourceImpl createResource(String ctl, String db, String tbl, String col);
522529

523-
protected abstract RangerBasePlugin getPlugin();
530+
protected abstract BackgroundLoadedRangerPlugin getPlugin();
524531

525532
protected abstract RangerAccessResultProcessor getAccessResultProcessor();
526533
}

‎fe/fe-authorization/fe-authorization-plugins/fe-authorization-plugin-ranger-common/src/main/java/org/apache/doris/catalog/authorizer/ranger/RangerUserStoreGroups.java‎

Lines changed: 13 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,6 @@
2424
import org.apache.ranger.plugin.contextenricher.RangerUserStoreEnricher;
2525
import org.apache.ranger.plugin.policyengine.RangerPluginContext;
2626
import org.apache.ranger.plugin.service.RangerAuthContext;
27-
import org.apache.ranger.plugin.service.RangerBasePlugin;
2827
import org.apache.ranger.plugin.util.RangerUserStoreUtil;
2928
import org.apache.ranger.plugin.util.ServiceDefUtil;
3029
import org.apache.ranger.plugin.util.ServicePolicies;
@@ -81,11 +80,11 @@ public static boolean enabledFor(RangerPluginConfig config) {
8180
* Puts a user store enricher on the service definition Ranger just downloaded, unless one is there or
8281
* this is switched off, so that the plugin fetches and refreshes the user store {@link #groupsOf} reads.
8382
*
84-
* <p>To be called by a plugin's {@code setPolicies} before handing the policies on, on every call and
85-
* not only the first: a download of policy deltas comes with its own copy of the service definition,
86-
* which is why {@code RangerBasePlugin} re-adds the enricher on deltas too. The retriever class and the
87-
* refresh interval are read under the option names Ranger itself reads them under, so that an operator
88-
* who has tuned them for Ranger's own {@code use.rangerGroups} has tuned them here.
83+
* <p>Called by {@link BackgroundLoadedRangerPlugin#setPolicies} before handing the policies on, on every
84+
* call and not only the first: a download of policy deltas comes with its own copy of the service
85+
* definition, which is why {@code RangerBasePlugin} re-adds the enricher on deltas too. The retriever
86+
* class and the refresh interval are read under the option names Ranger itself reads them under, so that
87+
* an operator who has tuned them for Ranger's own {@code use.rangerGroups} has tuned them here.
8988
*/
9089
public static void addUserStoreEnricher(RangerPluginConfig config, ServicePolicies policies) {
9190
if (policies == null || config == null || !enabledFor(config)) {
@@ -117,15 +116,18 @@ public static String describe(RangerPluginConfig config) {
117116
/**
118117
* The groups the user store {@code plugin} has downloaded puts {@code user} in.
119118
*
120-
* <p>Empty when this is switched off, when no user store has arrived yet - the plugin has not loaded its
121-
* policies, or Ranger Admin could not be reached - and when the store does not know the user, which is
122-
* the case for every account that exists in Doris only. Empty and not null on purpose: a request with an
123-
* empty group set matches items written against users and roles exactly as it did before.
119+
* <p>Read once the plugin's first load has ended, since the store arrives with it: asked earlier, this
120+
* would say "no groups" about a user the store is about to put in several. Empty when this is switched
121+
* off, when no store arrived - Ranger Admin could not be reached, and nothing was cached - and when the
122+
* store does not know the user, which is the case for every account that exists in Doris only. Empty
123+
* and not null on purpose: a request with an empty group set matches items written against users and
124+
* roles exactly as it did before.
124125
*/
125-
public static Set<String> groupsOf(RangerBasePlugin plugin, String user) {
126+
public static Set<String> groupsOf(BackgroundLoadedRangerPlugin plugin, String user) {
126127
if (user == null || !enabledFor(plugin.getConfig())) {
127128
return Collections.emptySet();
128129
}
130+
plugin.awaitLoaded();
129131
// The auth context the policy engine publishes is where Ranger's own request processing reads the
130132
// store from; it is replaced together with the engine, and the store is carried over when it is.
131133
RangerPluginContext pluginContext = plugin.getPluginContext();

0 commit comments

Comments
 (0)