Skip to content

Commit 99016ab

Browse files
morningmanclaude
andcommitted
[fix](ranger) Refuse an unusable configuration before the load starts, and replace a plugin whose load failed
Review finding F-01 on 400194c. Since the plugin loads on its own thread, an exception out of RangerBasePlugin.init() no longer reached whoever built the plugin: a CREATE CATALOG whose fe/conf lacks ranger-<type>-security.xml (an empty policy.rest.url), a REST timeout that is not a number, or an audit destination Doris does not ship succeeded where the dry run used to fail it with the cause, every check on the catalog was then refused with a generic "no answer" message, and - for ranger-doris - the failed plugin was kept as the shared one and handed to every later binding, so fixing the configuration took a restart of every FE. init() now does the part of the load that needs no admin on the calling thread, before it starts the load (preflight): the audit subsystem, when this plugin is the first to need it, and the admin client - RangerAdminRESTClient fails on exactly the configuration errors above and dials nothing until its first use, and PolicyRefresher and the user store retriever reuse the client found in the plugin context. So those errors fail the constructor again, which is what the dry run and an FE start see; a refused init() leaves the plugin as it was. What only the load can find - a chained plugin that fails to start, a refresher setting only the refresher reads - still fails the load, but the failed plugin is no longer kept: isFailed() is public, and both factories let go of a failed plugin on the way in, evict the controllers over it (a re-attach must not be handed it again; the bindings still holding them refuse as before and stop nothing when they let go), stop what is left of it, and build a new one for the binding being made. A catalog recovers on ALTER CATALOG once the cause is fixed; the instance-scope source, bound once at start, on a restart - which is what the ERROR line now says, instead of promising a restart for ranger-hive too. The two engine comments that described the dry run as proving the source usable now say what it proves: what the constructor checks. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_0191LreyBauJV7K7sZAbrB27
1 parent 400194c commit 99016ab

11 files changed

Lines changed: 332 additions & 42 deletions

File tree

‎fe/fe-authorization/fe-authorization-plugins/fe-authorization-plugin-ranger-common/src/main/java/org/apache/doris/catalog/authorizer/ranger/BackgroundLoadedRangerPlugin.java‎

Lines changed: 74 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
import com.google.common.annotations.VisibleForTesting;
2121
import org.apache.logging.log4j.LogManager;
2222
import org.apache.logging.log4j.Logger;
23+
import org.apache.ranger.audit.provider.AuditProviderFactory;
2324
import org.apache.ranger.plugin.policyengine.RangerAccessRequest;
2425
import org.apache.ranger.plugin.policyengine.RangerAccessResult;
2526
import org.apache.ranger.plugin.policyengine.RangerAccessResultProcessor;
@@ -54,16 +55,24 @@
5455
* <p>What a check waits for is exactly what the constructor used to guarantee: that the load has
5556
* <em>ended</em> - with the policies from the admin, or from the local cache when the admin could not be
5657
* reached, or with nothing at all, in which case the engine is null and {@code RangerAccessController}
57-
* refuses, as it always has. A load that <em>threw</em> ends the same way, whatever it had installed by
58-
* then: what {@code RangerBasePlugin.init()} does about an admin it cannot reach is logged and survived
59-
* inside it, so a throw is something else - a broken audit configuration, a chained plugin that could not
60-
* start - and where the constructor used to fail the FE's start with it, this plugin stops what the load
61-
* did publish and refuses every check until the FE is restarted, see {@link #load()}. The wait is not
62-
* shortened by a timeout of its own: the load is bounded by the REST timeouts the operator already tunes,
63-
* and answering out of an empty engine before it has ended would be refusing checks the policies are about
64-
* to allow - and, worse, passing ones a policy written against a group is about to deny. What does cut it
65-
* short is the caller having no use for the answer any more: a controller closed while its check waits
66-
* refuses, and {@link #awaitLoaded(BooleanSupplier)} is how it stops waiting.
58+
* refuses, as it always has. What the load could not have got anywhere with is not left to it: before the
59+
* load starts, {@link #init()} does the part of it that needs no admin on the calling thread - the audit
60+
* subsystem and the admin client, see {@link #preflight()} - so a configuration that fails there (no
61+
* {@code policy.rest.url}, a REST timeout that is not a number, an audit destination Doris does not ship)
62+
* fails the constructor with its cause, as it did before the load had a thread of its own; that is what a
63+
* {@code CREATE CATALOG} dry run and an FE start refuse. A load that <em>threw</em> past that ends like the
64+
* others, whatever it had installed by then: what {@code RangerBasePlugin.init()} does about an admin it
65+
* cannot reach is logged and survived inside it, so such a throw is something else - a chained plugin that
66+
* could not start, a refresher setting only the refresher reads - and this plugin stops what the load did
67+
* publish, refuses every check, and says so through {@link #isFailed()}, see {@link #load()}. How long that
68+
* lasts is up to whoever holds the plugin: the factories hand a failed one to no further binding and build a
69+
* new one in its place, so a catalog bound to it recovers on the next {@code ALTER CATALOG} once the cause
70+
* is fixed, while the instance-scope source, bound once at start, recovers when the FE is restarted. The
71+
* wait is not shortened by a timeout of its own: the load is bounded by the REST timeouts the operator
72+
* already tunes, and answering out of an empty engine before it has ended would be refusing checks the
73+
* policies are about to allow - and, worse, passing ones a policy written against a group is about to deny.
74+
* What does cut it short is the caller having no use for the answer any more: a controller closed while
75+
* its check waits refuses, and {@link #awaitLoaded(BooleanSupplier)} is how it stops waiting.
6776
*
6877
* <p>Stopping it is {@link #cleanup()}, as before. Stopped while still loading - a {@code CREATE CATALOG}
6978
* dry run, the loser of a race in a factory - it finishes the load first and stops itself then, on the
@@ -100,6 +109,10 @@ protected BackgroundLoadedRangerPlugin(String serviceType, String serviceName, S
100109
/**
101110
* Starts the first load - {@code RangerBasePlugin.init()}, roles, policies and user store - on a thread
102111
* of its own, and returns at once. Once per plugin.
112+
*
113+
* <p>Not before {@link #preflight()} has passed, on this thread: a configuration the load could not have
114+
* got anywhere with fails this call with its cause instead of failing the load, and a call this refuses
115+
* has started nothing - the plugin is as it was before it.
103116
*/
104117
@Override
105118
public void init() {
@@ -109,6 +122,14 @@ public void init() {
109122
throw new IllegalStateException("Ranger plugin for service " + getServiceName()
110123
+ " has already been initialized");
111124
}
125+
try {
126+
preflight();
127+
} catch (RuntimeException | Error e) {
128+
// Nothing has been started, so nothing is waited for: a plugin this leaves behind - none in
129+
// production, where the constructor throws it away with the exception - has no load to end.
130+
loader.set(null);
131+
throw e;
132+
}
112133
LOG.info(RangerUserStoreGroups.describe(getConfig()));
113134
// A daemon: it ends with the load, and a load still running when the FE exits is not worth waiting
114135
// for. What it inherits, and what the refresher threads it goes on to start inherit from it, is the
@@ -142,17 +163,22 @@ && getUserStoreVersion() < 0) {
142163
}
143164
} catch (Throwable e) {
144165
// Everything RangerBasePlugin.init() does about an admin it cannot reach is logged and survived
145-
// inside it, so this is something else - a broken audit configuration, a chained plugin that
146-
// could not start. It used to fail the FE's start; now it fails every check against this
147-
// service until the FE is restarted, which RangerAccessController reports on each one as an
148-
// engine that is not initialized. Refusing takes both of the following: the flag, read by every
149-
// answer, and stopping what the load had installed before it threw - RangerBasePlugin.init()
150-
// publishes the policy engine, refresher and all, before it initializes the chained plugins, and
151-
// left running that engine would answer checks with part of the configured authorization
152-
// missing, whatever the line below says.
166+
// inside it, and what the load could not have got anywhere with the preflight refused before it
167+
// started, so this is something else - a chained plugin that could not start, a refresher
168+
// setting only the refresher reads. It used to fail the FE's start, or the CREATE CATALOG; now
169+
// it fails every check against this plugin, which RangerAccessController reports on each one as
170+
// an engine that is not initialized, for as long as the plugin is held: the factories build a
171+
// new one for the next binding of a source whose plugin has failed (isFailed), which is how a
172+
// fixed configuration reaches a catalog - through ALTER CATALOG - and does not reach the
173+
// instance-scope source, bound once at start, short of a restart. Refusing takes both of the
174+
// following: the flag, read by every answer, and stopping what the load had installed before it
175+
// threw - RangerBasePlugin.init() publishes the policy engine, refresher and all, before it
176+
// initializes the chained plugins, and left running that engine would answer checks with part
177+
// of the configured authorization missing, whatever the line below says.
153178
failed = true;
154-
LOG.error("Ranger service {} failed to load; every check against it is refused until the FE is"
155-
+ " restarted", getServiceName(), e);
179+
LOG.error("Ranger service {} failed to load; every check against it is refused. Once the cause"
180+
+ " is fixed, ALTER CATALOG binds a catalog governed by it to a new plugin; an FE governed"
181+
+ " by it through access_controller_type has to be restarted", getServiceName(), e);
156182
stopOnce();
157183
} finally {
158184
loaded.countDown();
@@ -170,6 +196,28 @@ protected void firstLoad() {
170196
super.init();
171197
}
172198

199+
/**
200+
* The part of the load that needs no Ranger admin, done on the thread calling {@link #init()} before the
201+
* load starts: the audit subsystem, when this is the first plugin in the process to need it, and the
202+
* admin client the load goes on to poll through. Both are what {@code RangerBasePlugin.init()} does
203+
* first and would otherwise do on the loader, and both fail on configuration alone - the client refuses
204+
* an empty {@code policy.rest.url} and a {@code policy.rest.client.*} setting that is not a number, the
205+
* audit factory a destination Doris does not ship - so done here they fail whoever is building the
206+
* plugin, with the cause, the way the constructor did before the load had a thread of its own. Neither
207+
* dials anything: the client opens its connection on first use, and it is kept in the plugin context,
208+
* where the refresher the load creates finds it rather than building a second one.
209+
*
210+
* <p>A test plugin standing in for the admin, whose {@link #firstLoad()} never polls, needs a URL for
211+
* this all the same, or overrides it.
212+
*/
213+
protected void preflight() {
214+
AuditProviderFactory auditProviderFactory = AuditProviderFactory.getInstance();
215+
if (!auditProviderFactory.isInitDone() && getConfig().getProperties() != null) {
216+
auditProviderFactory.init(getConfig().getProperties(), getAppId());
217+
}
218+
getPluginContext().createAdminClient(getConfig());
219+
}
220+
173221
/**
174222
* Waits until the first load has ended, if one was started; see the class comment for what that means.
175223
*
@@ -216,9 +264,12 @@ public boolean isLoaded() {
216264
return loaded.getCount() == 0;
217265
}
218266

219-
/** Whether the first load threw, after which every answer is a refusal; see {@link #load()}. */
220-
@VisibleForTesting
221-
boolean isFailed() {
267+
/**
268+
* Whether the first load threw, after which every answer is a refusal and the plugin has stopped itself;
269+
* see {@link #load()}. A factory sharing this plugin reads it on the way in, to hand a failed one to
270+
* nobody else and build a new one instead.
271+
*/
272+
public boolean isFailed() {
222273
return failed;
223274
}
224275

‎fe/fe-authorization/fe-authorization-plugins/fe-authorization-plugin-ranger-doris/src/main/java/org/apache/doris/catalog/authorizer/ranger/doris/RangerDorisAccessController.java‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -131,7 +131,9 @@ public void close() {
131131
return;
132132
}
133133
// Built directly rather than through the factory - a test, or an embedding that owns its own plugin -
134-
// so there is nobody else to account for and the plugin is ours to stop.
134+
// so there is nobody else to account for and the plugin is ours to stop. Or the factory has let go of
135+
// this controller with the failed plugin it is over, which has stopped itself and takes this as a
136+
// no-op.
135137
if (markClosed()) {
136138
stopPlugin();
137139
}

‎fe/fe-authorization/fe-authorization-plugins/fe-authorization-plugin-ranger-doris/src/main/java/org/apache/doris/catalog/authorizer/ranger/doris/RangerDorisAccessControllerFactory.java‎

Lines changed: 46 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,9 @@ public class RangerDorisAccessControllerFactory implements AuthorizationPluginFa
6262
* policy refresher and joins it without a timeout - and then the whole first load on the way back up:
6363
* roles, policies and user store downloaded again, on the plugin's own thread, before the catalog answers
6464
* a single check.
65+
*
66+
* <p>The one exception is a plugin whose first load threw: it has stopped itself and refuses every
67+
* check, and the next binding lets go of it and builds a new one, see {@link #letGoOfAFailedPlugin()}.
6568
*/
6669
private static BackgroundLoadedRangerPlugin sharedPlugin;
6770
private static final Map<Map<String, String>, Held> byConfiguration = new LinkedHashMap<>();
@@ -101,6 +104,15 @@ private static RangerDorisAccessController acquire(Map<String, String> propertie
101104
RangerAccessController.validateProperties(properties);
102105
Map<String, String> configuration = normalize(properties);
103106

107+
BackgroundLoadedRangerPlugin failed;
108+
synchronized (LOCK) {
109+
failed = letGoOfAFailedPlugin();
110+
}
111+
if (failed != null) {
112+
// With no lock held, like every stop. A plugin whose load threw has stopped itself, so this is a
113+
// no-op that waits for nothing; but what the factory lets go of, it stops.
114+
failed.cleanup();
115+
}
104116
BackgroundLoadedRangerPlugin built = null;
105117
BackgroundLoadedRangerPlugin toStop = null;
106118
try {
@@ -150,15 +162,47 @@ private static RangerDorisAccessController acquire(Map<String, String> propertie
150162
}
151163
}
152164

165+
/**
166+
* Lets go of the shared plugin when its first load threw, so that the binding being made gets a new one;
167+
* returns the plugin for the caller to stop, or null.
168+
*
169+
* <p>A plugin whose load threw has stopped itself and refuses every check, see
170+
* {@link BackgroundLoadedRangerPlugin#isFailed()}. The load fails on what its preflight could not check
171+
* for it, and what the operator then fixes in {@code fe/conf} reaches this source through its next
172+
* binding - a {@code CREATE CATALOG}, or the {@code ALTER CATALOG} that detaches and re-attaches a catalog
173+
* bound to the failed one - which is why a failed plugin is not shared any further: kept, it would be
174+
* handed to every binding until the FE was restarted. Its controllers go with it, because a re-attach
175+
* that found one in {@link #byConfiguration} would be handed the failed plugin again. The bindings
176+
* still holding them go on refusing, as they have since the load failed, and when they let go
177+
* {@link #release} has nothing here to account for, so they stop the plugin themselves - which a plugin
178+
* that has stopped itself takes as a no-op.
179+
*
180+
* <p>Caller holds {@link #LOCK}.
181+
*/
182+
private static BackgroundLoadedRangerPlugin letGoOfAFailedPlugin() {
183+
if (sharedPlugin == null || !sharedPlugin.isFailed()) {
184+
return null;
185+
}
186+
LOG.warn("The Ranger plugin of {} failed its first load; a new one is built for this binding, and the"
187+
+ " {} controller(s) over the failed one refuse until what holds them is bound again"
188+
+ " (a catalog by ALTER CATALOG, the instance by an FE restart).",
189+
RangerDorisAccessController.NAME, byConfiguration.size());
190+
BackgroundLoadedRangerPlugin failed = sharedPlugin;
191+
sharedPlugin = null;
192+
byConfiguration.clear();
193+
return failed;
194+
}
195+
153196
/**
154197
* Gives up one binding's hold on a controller, fencing the controller off once nothing holds it any more.
155198
*
156199
* <p>The shared plugin stays up either way - see {@link #sharedPlugin} for why the alternative is worse
157200
* than an idle policy download timer. What has to stop here is this controller: a query may still be
158201
* holding it, and from here it must refuse rather than answer.
159202
*
160-
* @return whether this factory owned {@code controller}; false means it was built some other way and its
161-
* caller has to stop it itself.
203+
* @return whether this factory owned {@code controller}; false means it was built some other way, or the
204+
* factory has let go of it with the failed plugin it is over, and its caller has to stop it
205+
* itself.
162206
*/
163207
static boolean release(RangerDorisAccessController controller) {
164208
if (controller == null) {

0 commit comments

Comments
 (0)