Skip to content

Commit 2d15b1b

Browse files
morningmanclaude
andcommitted
fix: FIX-JDBC-DRIVER-URL — resolve+alias driver_url for BE; validate at CREATE (B-8a + B-8b)
rereview2 #4. JDBC-metastore-flavor paimon catalogs only. Connector-only, zero new SPI. Root cause: - B-8a (functional BLOCKER): PaimonScanPlanProvider.getBackendPaimonOptions forwarded driver_url to BE RAW and its `key.startsWith("jdbc.")` filter dropped the `paimon.jdbc.*` alias. A bare `jdbc.driver_url=mysql.jar` reached BE, where JdbcDriverUtils.registerDriver does `new URL(value)` -> MalformedURLException; a `paimon.jdbc.driver_url` alias was dropped outright. Legacy PaimonJdbcMetaStoreProperties.getBackendPaimonOptions emits `jdbc.driver_url=JdbcResource.getFullDriverUrl(driverUrl)` (resolved) + `jdbc.driver_class`. - B-8b (security): driver_url was loaded into the FE JVM (URLClassLoader) and shipped to BE with no format / jdbc_driver_url_white_list / jdbc_driver_secure_path validation, plus a stale "paimon is not in SPI_READY_TYPES" disclaimer (false since the B7 cutover added paimon to CatalogFactory SPI_READY_TYPES). Solution (reuses existing hooks; no new SPI surface): - B-8a: getBackendPaimonOptions now reads driver_url via firstNonBlank(JDBC_DRIVER_URL) (honors both the jdbc.* and paimon.jdbc.* alias) and emits the canonical `jdbc.driver_url` RESOLVED to a scheme-bearing URL plus `jdbc.driver_class` (BE accepts both alias forms). Resolution is extracted to a shared static PaimonCatalogFactory.resolveDriverUrl(driverUrl, env) so FE driver registration and the BE-bound options resolve a given driver_url identically. - B-8b: PaimonConnector overrides Connector.preCreateValidation to route a configured driver_url (either alias) through ConnectorValidationContext.validateAndResolveDriverPath at CREATE CATALOG (format/whitelist/secure-path; throws -> CREATE fails before the jar loads). Mirrors JdbcDorisConnector. Stale disclaimer replaced with an accurate note. Scope (user-signed D-050; see DV-028/DV-029): validation is CREATE-time only — parity with the JDBC reference connector. The FE-restart-reload / ALTER-CATALOG / scan-time re-validation gap is a pre-existing fe-core limitation shared by all plugin connectors (default config is permissive); accepted, with a cross-connector follow-up filed. BE-side paimon.jdbc.{user,password,uri} alias-drop is out of scope (BE deserializes the table from serialized_table; only driver_url/driver_class are consumed by registerDriverIfNeeded). Tests: PaimonScanPlanProviderTest +5 (resolve bare name, honor paimon.jdbc.* alias, both-aliases priority+override, preserve scheme-bearing, non-jdbc empty); new PaimonConnectorPreCreateValidationTest +5 (validate jdbc/alias, skip non-jdbc/no-driver_url, propagate rejection). Module 232/0/0 (1 CI-gated skip); fail-before verified (5/9 new tests red when neutered); checkstyle 0; connector import-gate clean. Live e2e (JDBC flavor + remote jar) is CI-gated. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
1 parent 667f779 commit 2d15b1b

10 files changed

Lines changed: 575 additions & 29 deletions

File tree

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCatalogFactory.java‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -136,6 +136,41 @@ public static String firstNonBlank(Map<String, String> props, String... keys) {
136136
return null;
137137
}
138138

139+
/**
140+
* Resolves a JDBC {@code driver_url} to a full, scheme-bearing URL string. A value already
141+
* carrying a scheme ({@code "://"}) is used as-is; an absolute path (starting with {@code "/"})
142+
* is returned unchanged; otherwise it is treated as a bare jar file name and resolved against
143+
* the engine's configured {@code jdbc_drivers_dir} (defaulting to
144+
* {@code $DORIS_HOME/plugins/jdbc_drivers}), mirroring the minimal {@code JdbcResource.getFullDriverUrl}
145+
* resolution (no file-existence / legacy old-dir / cloud-download handling).
146+
*
147+
* <p>Shared by {@code PaimonConnector} (FE {@code URLClassLoader} driver registration) and
148+
* {@code PaimonScanPlanProvider.getBackendPaimonOptions} (the BE-bound options, where BE does
149+
* {@code new URL(value)} and a bare {@code "mysql.jar"} would throw {@code MalformedURLException})
150+
* so BOTH sides resolve a given {@code driver_url} <em>identically</em>. Security validation
151+
* (format / {@code jdbc_driver_url_white_list} / {@code jdbc_driver_secure_path}) is enforced
152+
* separately at CREATE CATALOG via {@code PaimonConnector.preCreateValidation}.
153+
*
154+
* @param driverUrl the raw driver_url; must be non-null and non-blank (the caller's responsibility —
155+
* both call sites guard with {@code firstNonBlank}/non-null checks before calling)
156+
* @param env the engine environment map (e.g. {@code jdbc_drivers_dir}, {@code doris_home}); never null
157+
*/
158+
public static String resolveDriverUrl(String driverUrl, Map<String, String> env) {
159+
if (driverUrl.contains("://")) {
160+
return driverUrl;
161+
}
162+
if (driverUrl.startsWith("/")) {
163+
// Absolute path, no scheme: legacy returns it as-is (no driversDir prepend).
164+
return driverUrl;
165+
}
166+
String driversDir = env.get("jdbc_drivers_dir");
167+
if (StringUtils.isBlank(driversDir)) {
168+
String dorisHome = env.getOrDefault("doris_home", ".");
169+
driversDir = dorisHome + "/plugins/jdbc_drivers";
170+
}
171+
return "file://" + driversDir + "/" + driverUrl;
172+
}
173+
139174
/**
140175
* Fail-fast validation, mirroring the legacy per-flavor rules. Throws
141176
* {@link IllegalArgumentException} (style consistent with MaxCompute), which the caller

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java‎

Lines changed: 37 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
import org.apache.doris.connector.api.ConnectorCapability;
2222
import org.apache.doris.connector.api.ConnectorMetadata;
2323
import org.apache.doris.connector.api.ConnectorSession;
24+
import org.apache.doris.connector.api.ConnectorValidationContext;
2425
import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider;
2526
import org.apache.doris.connector.spi.ConnectorContext;
2627

@@ -38,6 +39,7 @@
3839
import java.net.MalformedURLException;
3940
import java.net.URL;
4041
import java.net.URLClassLoader;
42+
import java.util.Collections;
4143
import java.util.EnumSet;
4244
import java.util.Map;
4345
import java.util.Set;
@@ -195,6 +197,28 @@ private Catalog createCatalogFromContext(CatalogContext catalogContext, String f
195197
}
196198
}
197199

200+
/**
201+
* Enforces JDBC driver-url security at CREATE CATALOG (rereview2 B-8b). For the JDBC flavor a
202+
* configured {@code driver_url} — read from either the {@code jdbc.driver_url} or the
203+
* {@code paimon.jdbc.driver_url} alias — is routed through the engine's
204+
* {@link ConnectorValidationContext#validateAndResolveDriverPath} hook, which applies the FE
205+
* format / {@code jdbc_driver_url_white_list} / {@code jdbc_driver_secure_path} gates (legacy
206+
* {@code JdbcResource.getFullDriverUrl}). A rejected url throws here, so CREATE CATALOG fails
207+
* before the jar is ever loaded into the FE JVM by {@link #maybeRegisterJdbcDriver}. Mirrors
208+
* {@code JdbcDorisConnector.preCreateValidation}; non-JDBC flavors are a no-op.
209+
*/
210+
@Override
211+
public void preCreateValidation(ConnectorValidationContext validationContext) throws Exception {
212+
if (!PaimonConnectorProperties.JDBC.equals(PaimonCatalogFactory.resolveFlavor(properties))) {
213+
return;
214+
}
215+
String driverUrl = PaimonCatalogFactory.firstNonBlank(
216+
properties, PaimonConnectorProperties.JDBC_DRIVER_URL);
217+
if (StringUtils.isNotBlank(driverUrl)) {
218+
validationContext.validateAndResolveDriverPath(driverUrl);
219+
}
220+
}
221+
198222
/**
199223
* If a JDBC driver_url is configured, dynamically load + register the driver before creating
200224
* the catalog. {@link java.sql.DriverManager#getConnection} does not consult the thread context
@@ -216,34 +240,22 @@ private void maybeRegisterJdbcDriver() {
216240
}
217241

218242
/**
219-
* Resolves a driver_url to a full URL string. If it is already a URL (contains {@code "://"})
220-
* it is used as-is; an absolute path (starting with {@code "/"}) is returned unchanged;
221-
* otherwise it is treated as a bare jar file name and resolved against the engine's configured
222-
* {@code jdbc_drivers_dir} (defaulting to {@code $DORIS_HOME/plugins/jdbc_drivers}), mirroring
223-
* the minimal {@code JdbcResource.getFullDriverUrl} behavior.
243+
* Resolves a driver_url to a full, scheme-bearing URL string for FE driver registration,
244+
* delegating to the shared {@link PaimonCatalogFactory#resolveDriverUrl} so the FE registration
245+
* path and the BE-bound scan options ({@code PaimonScanPlanProvider.getBackendPaimonOptions})
246+
* resolve a given driver_url identically.
224247
*
225-
* <p>NOTE (B1/cutover-blocker): legacy JdbcResource.getFullDriverUrl enforced FE security
226-
* allow-lists (jdbc_driver_url_white_list, jdbc_driver_secure_path) + jar-name format
227-
* validation. Those gates are NOT enforced here (the connector cannot import fe-core).
228-
* Before the jdbc driver_url path goes live at cutover (P5-B7), driver-url validation
229-
* must be routed through a ConnectorContext hook (cf. sanitizeJdbcUrl). Until then,
230-
* paimon is not in SPI_READY_TYPES so this path is not user-reachable.
248+
* <p>FE security validation (format / {@code jdbc_driver_url_white_list} /
249+
* {@code jdbc_driver_secure_path}) is enforced at CREATE CATALOG by {@link #preCreateValidation}
250+
* via the engine's {@code ConnectorValidationContext.validateAndResolveDriverPath} hook — a
251+
* rejected url fails catalog creation before this path is ever reached. Like the JDBC reference
252+
* connector ({@code JdbcDorisConnector}), validation is CREATE-time only; catalogs reloaded after
253+
* an FE restart or reconfigured via ALTER CATALOG are not re-validated against a since-tightened
254+
* allow-list (a pre-existing fe-core gap shared by all plugin connectors — see deviations-log).
231255
*/
232256
private String resolveFullDriverUrl(String driverUrl) {
233-
if (driverUrl.contains("://")) {
234-
return driverUrl;
235-
}
236-
if (driverUrl.startsWith("/")) {
237-
// Absolute path, no scheme: legacy returns it as-is (no driversDir prepend).
238-
return driverUrl;
239-
}
240-
Map<String, String> env = context.getEnvironment();
241-
String driversDir = env.get("jdbc_drivers_dir");
242-
if (StringUtils.isBlank(driversDir)) {
243-
String dorisHome = env.getOrDefault("doris_home", ".");
244-
driversDir = dorisHome + "/plugins/jdbc_drivers";
245-
}
246-
return "file://" + driversDir + "/" + driverUrl;
257+
Map<String, String> env = context != null ? context.getEnvironment() : Collections.emptyMap();
258+
return PaimonCatalogFactory.resolveDriverUrl(driverUrl, env);
247259
}
248260

249261
private void registerJdbcDriver(String driverUrl, String driverClassName) {

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -608,7 +608,8 @@ private String getTableLocation(Table table) {
608608
return table.options().get("path");
609609
}
610610

611-
private Map<String, String> getBackendPaimonOptions() {
611+
// Package-private for direct unit testing (PaimonScanPlanProviderTest).
612+
Map<String, String> getBackendPaimonOptions() {
612613
String metastoreType = properties.get("paimon.catalog.type");
613614
if (!"jdbc".equalsIgnoreCase(metastoreType)) {
614615
return Collections.emptyMap();
@@ -623,6 +624,24 @@ private Map<String, String> getBackendPaimonOptions() {
623624
options.put(key, entry.getValue());
624625
}
625626
}
627+
// FIX-JDBC-DRIVER-URL (B-8a): the loop above forwards driver_url RAW and only matches the
628+
// "jdbc.*" form, so a bare "jdbc.driver_url=mysql.jar" reaches BE unresolved (BE does
629+
// new URL(value) -> MalformedURLException, JdbcDriverUtils.registerDriver) and a
630+
// "paimon.jdbc.driver_url" alias is dropped entirely. Emit the canonical, RESOLVED keys the
631+
// BE reader accepts (PaimonJdbcDriverUtils reads both aliases): honor either alias and resolve
632+
// a bare jar name to a full file:// URL. Mirrors legacy
633+
// PaimonJdbcMetaStoreProperties.getBackendPaimonOptions (getFullDriverUrl + driver_class).
634+
String driverUrl = PaimonCatalogFactory.firstNonBlank(
635+
properties, PaimonConnectorProperties.JDBC_DRIVER_URL);
636+
if (driverUrl != null) {
637+
Map<String, String> env = context != null ? context.getEnvironment() : Collections.emptyMap();
638+
options.put("jdbc.driver_url", PaimonCatalogFactory.resolveDriverUrl(driverUrl, env));
639+
String driverClass = PaimonCatalogFactory.firstNonBlank(
640+
properties, PaimonConnectorProperties.JDBC_DRIVER_CLASS);
641+
if (driverClass != null) {
642+
options.put("jdbc.driver_class", driverClass);
643+
}
644+
}
626645
return options;
627646
}
628647

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.connector.paimon;
19+
20+
import org.apache.doris.connector.api.ConnectorValidationContext;
21+
22+
import org.junit.jupiter.api.Assertions;
23+
import org.junit.jupiter.api.Test;
24+
25+
import java.util.ArrayList;
26+
import java.util.Collections;
27+
import java.util.HashMap;
28+
import java.util.List;
29+
import java.util.Map;
30+
31+
/**
32+
* Tests for {@link PaimonConnector#preCreateValidation} (rereview2 B-8b): a JDBC-flavor catalog
33+
* with a {@code driver_url} must route it through the engine's
34+
* {@link ConnectorValidationContext#validateAndResolveDriverPath} security gate at CREATE CATALOG,
35+
* before the jar is ever loaded into the FE JVM. Mirrors {@code JdbcDorisConnector.preCreateValidation}.
36+
*
37+
* <p>Offline: a hand-written {@link RecordingValidationContext} fake records each validated url and
38+
* can simulate a rejected url. {@link RecordingConnectorContext} supplies the (unused-by-this-path)
39+
* {@code ConnectorContext}.
40+
*/
41+
public class PaimonConnectorPreCreateValidationTest {
42+
43+
private static PaimonConnector connector(Map<String, String> props) {
44+
return new PaimonConnector(props, new RecordingConnectorContext());
45+
}
46+
47+
@Test
48+
public void validatesJdbcDriverUrl() throws Exception {
49+
Map<String, String> props = new HashMap<>();
50+
props.put("paimon.catalog.type", "jdbc");
51+
props.put("jdbc.driver_url", "mysql.jar");
52+
RecordingValidationContext ctx = new RecordingValidationContext();
53+
54+
connector(props).preCreateValidation(ctx);
55+
56+
// WHY (BLOCKER B-8b): a jdbc driver_url is loaded into the FE JVM (URLClassLoader); CREATE
57+
// CATALOG must route it through the engine's format / white-list / secure-path gate. MUTATION:
58+
// dropping the preCreateValidation override -> validateAndResolveDriverPath never called -> red.
59+
Assertions.assertEquals(Collections.singletonList("mysql.jar"), ctx.validatedDriverUrls);
60+
}
61+
62+
@Test
63+
public void validatesPaimonJdbcDriverUrlAlias() throws Exception {
64+
Map<String, String> props = new HashMap<>();
65+
props.put("paimon.catalog.type", "jdbc");
66+
props.put("paimon.jdbc.driver_url", "mysql.jar");
67+
RecordingValidationContext ctx = new RecordingValidationContext();
68+
69+
connector(props).preCreateValidation(ctx);
70+
71+
Assertions.assertEquals(Collections.singletonList("mysql.jar"), ctx.validatedDriverUrls,
72+
"the paimon.jdbc.driver_url alias must also be validated");
73+
}
74+
75+
@Test
76+
public void skipsValidationForNonJdbcFlavor() throws Exception {
77+
Map<String, String> props = new HashMap<>();
78+
props.put("paimon.catalog.type", "filesystem");
79+
props.put("jdbc.driver_url", "mysql.jar");
80+
RecordingValidationContext ctx = new RecordingValidationContext();
81+
82+
connector(props).preCreateValidation(ctx);
83+
84+
Assertions.assertTrue(ctx.validatedDriverUrls.isEmpty(),
85+
"non-JDBC flavors must not trigger driver-url validation");
86+
}
87+
88+
@Test
89+
public void skipsValidationWhenNoDriverUrl() throws Exception {
90+
Map<String, String> props = new HashMap<>();
91+
props.put("paimon.catalog.type", "jdbc");
92+
RecordingValidationContext ctx = new RecordingValidationContext();
93+
94+
connector(props).preCreateValidation(ctx);
95+
96+
Assertions.assertTrue(ctx.validatedDriverUrls.isEmpty(),
97+
"a jdbc catalog without a driver_url uses the platform driver -> nothing to validate");
98+
}
99+
100+
@Test
101+
public void propagatesRejectedDriverUrl() {
102+
Map<String, String> props = new HashMap<>();
103+
props.put("paimon.catalog.type", "jdbc");
104+
props.put("jdbc.driver_url", "http://evil.test/x.jar");
105+
RecordingValidationContext ctx = new RecordingValidationContext();
106+
ctx.reject = true;
107+
108+
// WHY (BLOCKER B-8b): a disallowed url must FAIL CREATE CATALOG — the hook throws and the
109+
// connector must let it propagate, not swallow it. MUTATION: catching the exception -> no
110+
// throw -> red.
111+
Assertions.assertThrows(IllegalArgumentException.class,
112+
() -> connector(props).preCreateValidation(ctx));
113+
}
114+
115+
/** Hand-written {@link ConnectorValidationContext} test double (no Mockito). */
116+
private static final class RecordingValidationContext implements ConnectorValidationContext {
117+
final List<String> validatedDriverUrls = new ArrayList<>();
118+
boolean reject;
119+
120+
@Override
121+
public long getCatalogId() {
122+
return 0;
123+
}
124+
125+
@Override
126+
public String getProperty(String key) {
127+
return null;
128+
}
129+
130+
@Override
131+
public void storeProperty(String key, String value) {
132+
}
133+
134+
@Override
135+
public String validateAndResolveDriverPath(String driverUrl) throws Exception {
136+
validatedDriverUrls.add(driverUrl);
137+
if (reject) {
138+
throw new IllegalArgumentException("disallowed driver url: " + driverUrl);
139+
}
140+
return "file:///resolved/" + driverUrl;
141+
}
142+
143+
@Override
144+
public String computeDriverChecksum(String driverUrl) {
145+
return "deadbeef";
146+
}
147+
148+
@Override
149+
public void requestBeConnectivityTest(byte[] serializedDescriptor, int connectionTypeValue,
150+
String testQuery) {
151+
}
152+
}
153+
}

0 commit comments

Comments
 (0)