Skip to content

Commit 82646c3

Browse files
morningmanclaude
andauthored
[feat](connector) give each connector plugin its own conf file (apache#66347)
### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: Giving one connector a new deployment-level setting currently costs two edits in the engine: an `@ConfField` in fe-common's `Config`, and a line in fe-core's `DefaultConnectorContext.buildEnvironment()` that forwards it. A connector plugin cannot read `Config` itself — it loads child-first, so its own bundled copy shadows the engine's and every field reads back as a code default — so the engine has to carry each key by name. The result is that `fe-core` knows the config key names of connectors it is otherwise entirely agnostic about, and that list only grows. **This PR gives a connector plugin a configuration file of its own.** The engine reads `<pluginDir>/<name>.conf` (where `<name>` is `ConnectorProvider.name()`), parses it generically, and serves it back through the new `ConnectorContext.getConnectorConfig()`. No key name of any connector reaches `fe-core`, and **a new connector needs no engine change at all** to add a deployment-level setting. Connectors read a setting through one entry point: ```java ConnectorConf.get(context, "drivers_dir", "jdbc_drivers_dir", null) // ^ key in <name>.conf ^ the fe.conf key it used to live under ``` Resolution is **plugin conf → fe.conf → default**, with blank treated as "not set" at each step (an operator who writes `key=` has not configured it, and reading it as a set empty string would let one stray line mask the fe.conf value actually in effect). Six settings across five connectors are moved onto the new channel. **Every `@ConfField` is kept** and stays the fallback, so an existing deployment upgrades with nothing to edit and behaves exactly as before: | connector (`name()`) | conf file | key | fe.conf fallback | | --- | --- | --- | --- | | `trino-connector` | `trino-connector.conf` | `plugin_dir` | `trino_connector_plugin_dir` | | `hms` (hive) | `hms.conf` | `default_file_format` | `hive_default_file_format` | | `hms` (hive) | `hms.conf` | `enable_create_bucket_table` | `enable_create_hive_bucket_table` | | `jdbc` | `jdbc.conf` | `drivers_dir` | `jdbc_drivers_dir` | | `jdbc` | `jdbc.conf` | `force_sqlserver_encrypt_false` | `force_sqlserver_jdbc_encrypt_false` | | `iceberg` / `paimon` | `iceberg.conf` / `paimon.conf` | `drivers_dir` | `jdbc_drivers_dir` | | `iceberg` / `paimon` | `iceberg.conf` / `paimon.conf` | `metastore_client_timeout_second` | `hive_metastore_client_timeout_second` | Two of these are shared by several connectors at the fe.conf end (one `jdbc_drivers_dir` serves jdbc, iceberg and paimon). A per-plugin file cannot express that, so a deployment that moves them sets the value in each plugin's conf; leaving them commented out keeps the single shared fe.conf value, which is what every existing deployment gets. Both templates say so. `doris_home` and `doris_version` stay in `getEnvironment()` — they are not connector settings. `jdbc_driver_secure_path` is dropped from it: no connector ever read it (the JDBC allow-list is enforced in fe-core by `JdbcResource`, which reads `Config` directly), so it was a dead key that read like a connector setting. The `Config` field itself is unchanged. **Packaging.** A plugin ships `src/main/resources/<name>.conf.template`; `build.sh` seeds the live `<name>.conf` from it with `cp -n`, globbing `*.conf.template` with no connector named, so a new connector needs no `build.sh` change either. The live `.conf` is deliberately **not** in the plugin zip, so the ordinary upgrade — unzipping a newer plugin build over the deployed directory — refreshes the jars and the template but never the administrator's file. Note the conf file is named after `ConnectorProvider.name()`, **not** after the plugin directory: `plugins/connector/hive/` holds `hms.conf` and `plugins/connector/trino/` holds `trino-connector.conf`. The directory name is the deployer's choice and cannot be what the engine keys on. Each connector carries a test asserting its shipped template name still tracks `name()`. `connector.plugin.api.version` stays at **1.0**. The added `ConnectorContext.getConnectorConfig()` is a `default` method, nothing outside fe-connector-spi implements `ConnectorContext`, and the only decorators of it extend the parent-first `ForwardingConnectorContext` — loaded from the FE's own classpath, so they inherit the new forward without being rebuilt. A connector plugin built before this change loads and behaves exactly as it did; it simply never reads the new map. ### Release note Connector plugins can now carry their own deployment-level configuration file, `<DORIS_HOME>/plugins/connector/<dir>/<name>.conf`, seeded from a template shipped with each plugin. Settings there take precedence over the corresponding `fe.conf` keys, which keep working unchanged — no action is required when upgrading. The file must be maintained on every FE node and takes effect after an FE restart. ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [x] Unit Test - [x] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> **Unit tests** — new: `ConnectorConfFileTest`, `ConnectorConfTest` (spi), `ConnectorPluginConfTest` (fe-core, drives `loadPlugins` + `createConnector` on real plugin directories), `IcebergConnectorConfTest`, `PaimonConnectorConfTest`, plus per-connector cases and a template-name guard in `TrinoBootstrapTest`, `HiveConnectorMetadataDdlTest`, `JdbcUrlNormalizerTest`. Existing `ForwardingConnectorContextTest` covers the new forward by reflection. Each was mutation-checked (precedence reversed, forward removed, surface baseline reverted, conf not keyed per provider, conf never attached, template renamed) and confirmed to fail. **Manual test** — `sh build.sh --fe`, then verified in `output/fe/plugins/connector/`: - `hive/hms.conf`, `trino/trino-connector.conf`, `iceberg/iceberg.conf`, `jdbc/jdbc.conf`, `paimon/paimon.conf` are created and byte-identical to their templates; `es/`, `hudi/`, `maxcompute/` ship no template and get no file, with no error. - Every seeded file has **0 active settings** (all commented out), so a fresh deployment behaves exactly as before. - Upgrade safety: hand-edited `hms.conf`, replayed the deploy step (`unzip -o` + the `cp -n` loop) from the real plugin zip — the edit survives and only the template is refreshed. - Behavior changed: - [x] Yes. Two, both narrow: 1. A deployment-level setting now resolves from the plugin's `<name>.conf` before `fe.conf`. For a deployment that does not edit the seeded (all-commented) file, nothing changes. 2. A blank `hive_default_file_format` in fe.conf now falls through to `orc` instead of reaching the metastore create as an empty format string. An empty file format was never a working configuration. - Does this need documentation? - [x] Yes. <!-- doris-website PR not filed yet --> Each shipped `.conf.template` documents its own keys inline, and `fe/fe-connector/README.md` plus `fe-connector-api/package-info.java` (Rule 7) are updated for connector authors. A doris-website page for operators is still to be written. --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
1 parent 8b0cafd commit 82646c3

54 files changed

Lines changed: 2040 additions & 134 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.licenserc.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,16 @@ header:
5959
- "fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt"
6060
- "fe/fe-authentication/fe-authentication-spi/src/test/resources/authentication-plugin-surface.txt"
6161
- "fe/fe-core/src/test/resources/lineage-plugin-surface.txt"
62+
# Connector plugin settings templates. build.sh seeds each connector's live
63+
# <name>.conf from its template verbatim (cp -n), so the template's content IS
64+
# the file an administrator edits in plugins/connector/<dir>/. Matched by name
65+
# rather than by a **/*.conf.template glob, so a new one is a deliberate entry
66+
# here rather than something a wildcard silently absorbs.
67+
- "fe/fe-connector/fe-connector-hive/src/main/resources/hms.conf.template"
68+
- "fe/fe-connector/fe-connector-iceberg/src/main/resources/iceberg.conf.template"
69+
- "fe/fe-connector/fe-connector-jdbc/src/main/resources/jdbc.conf.template"
70+
- "fe/fe-connector/fe-connector-paimon/src/main/resources/paimon.conf.template"
71+
- "fe/fe-connector/fe-connector-trino/src/main/resources/trino-connector.conf.template"
6272
# Golden 4.1.3 upgrade fixtures, emitted by Gen413Fixtures running real 4.1.3
6373
# bytecode (see the sibling PROVENANCE.txt) and stamped "do not edit by hand".
6474
# These are the generator's only non-binary outputs -- its .bin files are skipped

‎build.sh‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1082,8 +1082,17 @@ if [[ "${BUILD_FE}" -eq 1 ]]; then
10821082
fi
10831083
mkdir -p "${conn_plugin_target}"
10841084
unzip -o "${conn_zip}" -d "${conn_plugin_target}/"
1085+
# A connector's own settings file. The zip carries only <name>.conf.template; the live
1086+
# <name>.conf is seeded from it here and never overwritten, so an upgrade that unzips a new
1087+
# plugin build over this directory refreshes the jars and the template but leaves whatever the
1088+
# administrator configured. Deliberately generic (globbed on *.conf.template, no connector
1089+
# named): a new connector ships a template and needs no change here.
1090+
for conn_conf_tpl in "${conn_plugin_target}"/*.conf.template; do
1091+
[ -e "${conn_conf_tpl}" ] || continue
1092+
cp -n "${conn_conf_tpl}" "${conn_conf_tpl%.template}"
1093+
done
10851094
done
1086-
unset CONN_PLUGIN_DIR conn_module conn_plugin_target conn_module_dir conn_zip
1095+
unset CONN_PLUGIN_DIR conn_module conn_plugin_target conn_module_dir conn_zip conn_conf_tpl
10871096

10881097
# RC-4: self-contain the paimon connector plugin for OSS. The connector sets
10891098
# fs.oss.impl=com.aliyun.jindodata.oss.JindoOssFileSystem; that impl lives in the jindofs jars,

‎fe/fe-connector/README.md‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -211,7 +211,23 @@ metastore/shade/cache). For a write path, the richest example is
211211
(example: `RecordingConnectorContext`). Never touch
212212
`connector-metadata-methods.txt` unless you changed the shared SPI
213213
surface itself.
214-
14. **Packaging.** Add `src/main/assembly/plugin-zip.xml` (copy from es or
214+
14. **Deployment-level settings** (if any). A value that is one-per-FE rather
215+
than one-per-catalog goes in your plugin's own settings file, NOT in
216+
fe.conf: ship `src/main/resources/<name>.conf.template`, add it to your
217+
assembly's `<files>` at the zip root, and read it with
218+
`ConnectorConf.get(context, "<key>", null, "<default>")`. `<name>` is
219+
`ConnectorProvider.name()`, which is **not** necessarily your plugin
220+
directory name — `plugins/connector/hive/` holds `hms.conf` and
221+
`plugins/connector/trino/` holds `trino-connector.conf`. Guard that with a
222+
test asserting `name() + ".conf.template"` is on the classpath (copy
223+
`IcebergConnectorConfTest#theConfTemplateIsNamedAfterTheProvider`); a
224+
template under any other name deploys a file the engine never opens.
225+
`build.sh` seeds the live `.conf` from the template generically, so it
226+
needs no change. Do NOT add a key to `Config.java` or to
227+
`DefaultConnectorContext.buildEnvironment` — that is an engine change per
228+
setting, and the keys still there are only the ones several connectors
229+
share plus the fe.conf fallbacks kept for existing deployments.
230+
15. **Packaging.** Add `src/main/assembly/plugin-zip.xml` (copy from es or
215231
paimon). Verify your module through `package`/`install`, not just
216232
`test-compile` — shades and the plugin zip only materialize then.
217233
15. **Gates and e2e.** Your module must pass the forbidden-import gate (runs

‎fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/package-info.java‎

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -160,8 +160,8 @@
160160
*
161161
* <h2>Rule 7 — where a connector's tunable knobs live</h2>
162162
*
163-
* <p>Pick the channel by the SCOPE of the value. Only the second one obliges anyone to touch the engine, so
164-
* a knob that can be catalog-scoped should be.</p>
163+
* <p>Pick the channel by the SCOPE of the value. None of them requires an engine change any more, so a
164+
* knob that can be catalog-scoped should be, and one that cannot belongs in the plugin's own conf file.</p>
165165
*
166166
* <ul>
167167
* <li><b>Per catalog</b> &rarr; a key in {@code CREATE CATALOG ... PROPERTIES(...)}. The engine hands the
@@ -174,9 +174,22 @@
174174
* declared in {@code HiveConnectorProperties} and read in {@code HiveScanPlanProvider} /
175175
* {@code HiveConnector}; those key strings appear nowhere in {@code fe-core}.</li>
176176
* <li><b>Per FE process</b> (one deployment-level value for every catalog, e.g. a driver directory)
177-
* &rarr; an {@code fe.conf} field forwarded through {@code ConnectorContext.getEnvironment()} by
178-
* {@code DefaultConnectorContext.buildEnvironment}. This is the one knob shape that requires an engine
179-
* change per key, so use it only when the value genuinely is not per catalog.</li>
177+
* &rarr; a key in the plugin's own {@code <name>.conf}, read with {@code ConnectorConf.get}. The engine
178+
* locates and parses {@code <pluginDir>/<name>.conf} generically
179+
* ({@code ConnectorPluginManager.loadPlugins}) and serves it back through
180+
* {@code ConnectorContext.getConnectorConfig()}, so no key name of yours reaches {@code fe-core} and
181+
* adding one costs the engine nothing. Do <b>not</b> prefix these keys — the file name already
182+
* namespaces them. Ship {@code src/main/resources/<name>.conf.template} and add it to your assembly's
183+
* {@code <files>} at the zip root; {@code build.sh} seeds the live {@code .conf} from it. {@code <name>}
184+
* is {@code ConnectorProvider.name()} and need not equal your plugin directory name
185+
* ({@code plugins/connector/hive/} holds {@code hms.conf}).</li>
186+
* <li><b>Per FE process, legacy</b> &rarr; an {@code fe.conf} field forwarded through
187+
* {@code ConnectorContext.getEnvironment()} by {@code DefaultConnectorContext.buildEnvironment}. This is
188+
* the one shape that requires an engine change per key, and it is <b>closed to new keys</b>. What is
189+
* still there is either shared by several connectors or not a connector setting at all
190+
* ({@code doris_home}, {@code doris_version}); the connector settings among them are kept as the
191+
* fallback {@code ConnectorConf.get} consults after the plugin conf, so deployments configured before
192+
* the conf files existed keep working untouched.</li>
180193
* <li><b>Per session</b> &rarr; read the query's session variables from
181194
* {@link ConnectorSession#getSessionProperties()}. The connector does not declare them; it looks up the
182195
* names it cares about.</li>

‎fe/fe-connector/fe-connector-hive/src/main/assembly/plugin-zip.xml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,17 @@ under the License.
4040
<source>${project.build.directory}/${project.build.finalName}.jar</source>
4141
<outputDirectory>/</outputDirectory>
4242
</file>
43+
<!--
44+
The connector's settings template. Only the .template ships; build.sh seeds the live hms.conf
45+
from it with cp -n, so unzipping a newer plugin build over a deployed directory refreshes this
46+
file but never the administrator's copy. The name must equal ConnectorProvider.name() +
47+
".conf.template" - note that name() is "hms" while this plugin's directory is "hive";
48+
HiveConnectorMetadataDdlTest fails if the two drift apart.
49+
-->
50+
<file>
51+
<source>src/main/resources/hms.conf.template</source>
52+
<outputDirectory>/</outputDirectory>
53+
</file>
4354
</files>
4455

4556
<dependencySets>

‎fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,7 @@
6767
import org.apache.doris.connector.hms.HmsPartitionInfo;
6868
import org.apache.doris.connector.hms.HmsTableInfo;
6969
import org.apache.doris.connector.hms.HmsTypeMapping;
70+
import org.apache.doris.connector.spi.ConnectorConf;
7071
import org.apache.doris.connector.spi.ConnectorContext;
7172
import org.apache.doris.connector.spi.ConnectorStorageContext;
7273
import org.apache.doris.filesystem.FileSystem;
@@ -1618,7 +1619,8 @@ public void createTable(ConnectorSession session, ConnectorCreateTableRequest re
16181619
}
16191620
Map<String, String> env = context.getEnvironment();
16201621
String fileFormat = userProps.getOrDefault(HiveConnectorProperties.CREATE_FILE_FORMAT,
1621-
env.getOrDefault(HiveConnectorProperties.ENV_HIVE_DEFAULT_FILE_FORMAT,
1622+
ConnectorConf.get(context, HiveConnectorProperties.CONF_DEFAULT_FILE_FORMAT,
1623+
HiveConnectorProperties.ENV_HIVE_DEFAULT_FILE_FORMAT,
16221624
HiveConnectorProperties.DEFAULT_FILE_FORMAT));
16231625

16241626
// Metastore table parameters: lower-case every key and stamp the file_format / location keys under a
@@ -1673,11 +1675,14 @@ public void createTable(ConnectorSession session, ConnectorCreateTableRequest re
16731675
// enable gate first, then the hash requirement.
16741676
ConnectorBucketSpec bucketSpec = request.getBucketSpec();
16751677
if (bucketSpec != null) {
1676-
boolean bucketEnabled = Boolean.parseBoolean(env.getOrDefault(
1678+
boolean bucketEnabled = Boolean.parseBoolean(ConnectorConf.get(context,
1679+
HiveConnectorProperties.CONF_ENABLE_CREATE_BUCKET_TABLE,
16771680
HiveConnectorProperties.ENV_ENABLE_CREATE_HIVE_BUCKET_TABLE, "false"));
16781681
if (!bucketEnabled) {
1679-
throw new DorisConnectorException(
1680-
"Create hive bucket table need set enable_create_hive_bucket_table to true");
1682+
throw new DorisConnectorException("Create hive bucket table need set '"
1683+
+ HiveConnectorProperties.CONF_ENABLE_CREATE_BUCKET_TABLE + "' in hms.conf (or "
1684+
+ HiveConnectorProperties.ENV_ENABLE_CREATE_HIVE_BUCKET_TABLE
1685+
+ " in fe.conf) to true");
16811686
}
16821687
if (HiveConnectorProperties.BUCKET_ALGO_RANDOM.equals(bucketSpec.getAlgorithm())) {
16831688
throw new DorisConnectorException("External hive table only supports hash bucketing");

‎fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProperties.java‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,13 @@ private HiveConnectorProperties() {
7373
new HashSet<>(Arrays.asList(CREATE_FILE_FORMAT, CREATE_LOCATION)));
7474
public static final String DORIS_PROP_PREFIX = "doris.";
7575

76+
// -- deployment-level settings, read from this plugin's own hms.conf --
77+
// The file is named after ConnectorProvider.name() ("hms"), NOT after the plugin directory
78+
// ("plugins/connector/hive"). Each key falls back to the ENV_ name below it, which is the fe.conf key
79+
// it used to live under and still works.
80+
public static final String CONF_DEFAULT_FILE_FORMAT = "default_file_format";
81+
public static final String CONF_ENABLE_CREATE_BUCKET_TABLE = "enable_create_bucket_table";
82+
7683
// -- environment keys threaded from fe-core DefaultConnectorContext (must stay byte-identical there) --
7784
public static final String ENV_HIVE_DEFAULT_FILE_FORMAT = "hive_default_file_format";
7885
public static final String ENV_ENABLE_CREATE_HIVE_BUCKET_TABLE = "enable_create_hive_bucket_table";
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Hive (HMS) connector plugin configuration.
2+
#
3+
# build.sh seeds hms.conf from this file on first deploy and never overwrites it, so an upgrade that
4+
# unzips a new plugin build over this directory keeps whatever you configured here.
5+
# This file must exist on EVERY FE node -- it is not replicated through Doris metadata.
6+
# Changes take effect after an FE restart.
7+
#
8+
# The file name is the connector's name (ConnectorProvider.name() == "hms"), NOT the directory name:
9+
# this plugin is deployed under plugins/connector/hive/ but its settings file is hms.conf.
10+
#
11+
# Every setting below is optional. Left commented out, each falls back to the fe.conf key named
12+
# after it, and then to the built-in default -- so an existing deployment needs no change here.
13+
14+
# Storage format for a CREATE TABLE that does not name one itself.
15+
# Falls back to fe.conf's hive_default_file_format, whose default is orc. A CREATE TABLE may override
16+
# it per table with the file_format property, which wins over both.
17+
# default_file_format=orc
18+
19+
# Whether CREATE TABLE may create a bucketed hive table. Off by default.
20+
# Falls back to fe.conf's enable_create_hive_bucket_table.
21+
# enable_create_bucket_table=false

‎fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/FakeConnectorContext.java‎

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,9 +24,10 @@
2424
import java.util.Map;
2525

2626
/**
27-
* Minimal {@link ConnectorContext} test double: carries a fixed catalog identity and an environment map (the
28-
* channel through which fe-core threads the FE-global CREATE TABLE defaults). Everything else uses the
29-
* interface defaults.
27+
* Minimal {@link ConnectorContext} test double: carries a fixed catalog identity plus the two maps a
28+
* deployment-level setting can arrive in — this plugin's own {@code hms.conf}
29+
* ({@link #getConnectorConfig()}) and fe.conf as forwarded by the engine ({@link #getEnvironment()}).
30+
* Everything else uses the interface defaults.
3031
*/
3132
public class FakeConnectorContext implements ConnectorContext, ConnectorStorageContext {
3233

@@ -40,6 +41,7 @@ public ConnectorStorageContext getStorageContext() {
4041
private final String catalogName;
4142
private final long catalogId;
4243
private final Map<String, String> environment;
44+
private final Map<String, String> connectorConfig;
4345

4446
public FakeConnectorContext() {
4547
this("test_catalog", 0L, Collections.emptyMap());
@@ -50,9 +52,15 @@ public FakeConnectorContext(Map<String, String> environment) {
5052
}
5153

5254
public FakeConnectorContext(String catalogName, long catalogId, Map<String, String> environment) {
55+
this(catalogName, catalogId, environment, Collections.emptyMap());
56+
}
57+
58+
public FakeConnectorContext(String catalogName, long catalogId, Map<String, String> environment,
59+
Map<String, String> connectorConfig) {
5360
this.catalogName = catalogName;
5461
this.catalogId = catalogId;
5562
this.environment = environment == null ? Collections.emptyMap() : environment;
63+
this.connectorConfig = connectorConfig == null ? Collections.emptyMap() : connectorConfig;
5664
}
5765

5866
@Override
@@ -69,4 +77,9 @@ public long getCatalogId() {
6977
public Map<String, String> getEnvironment() {
7078
return environment;
7179
}
80+
81+
@Override
82+
public Map<String, String> getConnectorConfig() {
83+
return connectorConfig;
84+
}
7285
}

‎fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataDdlTest.java‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,67 @@ public void createTableFallsBackToOrcWhenEnvMissing() {
112112
Assertions.assertEquals("orc", client.lastCreateTable.getFileFormat());
113113
}
114114

115+
// ==================== createTable: the plugin's own hms.conf ====================
116+
117+
@Test
118+
public void createTableFileFormatFromPluginConfBeatsFeConf() {
119+
RecordingHmsClient client = new RecordingHmsClient();
120+
// WHY: the point of the plugin conf channel. An administrator who sets default_file_format in
121+
// hms.conf must get it even though fe.conf still names the old value -- reverse the precedence
122+
// and migrating a deployment to the new file silently does nothing.
123+
Map<String, String> conf = Collections.singletonMap(
124+
HiveConnectorProperties.CONF_DEFAULT_FILE_FORMAT, "parquet");
125+
Map<String, String> env = Collections.singletonMap(
126+
HiveConnectorProperties.ENV_HIVE_DEFAULT_FILE_FORMAT, "orc");
127+
128+
metadataWithConf(client, conf, env).createTable(session(), request().build());
129+
130+
Assertions.assertEquals("parquet", client.lastCreateTable.getFileFormat());
131+
}
132+
133+
@Test
134+
public void createTableUserFileFormatStillBeatsThePluginConf() {
135+
RecordingHmsClient client = new RecordingHmsClient();
136+
// WHY: the new channel is deployment-level; it must not outrank a per-catalog CREATE TABLE
137+
// property. Only the two deployment channels reordered relative to each other.
138+
Map<String, String> conf = Collections.singletonMap(
139+
HiveConnectorProperties.CONF_DEFAULT_FILE_FORMAT, "parquet");
140+
141+
metadataWithConf(client, conf, Collections.emptyMap()).createTable(session(),
142+
request().properties(Collections.singletonMap("file_format", "orc")).build());
143+
144+
Assertions.assertEquals("orc", client.lastCreateTable.getFileFormat());
145+
}
146+
147+
@Test
148+
public void createTableBucketGateCanBeOpenedFromThePluginConfAlone() {
149+
RecordingHmsClient client = new RecordingHmsClient();
150+
// WHY: the second migrated setting, and the one where getting the channel wrong is expensive --
151+
// a deployment that opts in through hms.conf but is still gated by fe.conf's default 'false'
152+
// would find bucketed creates rejected with no indication which file is in charge.
153+
Map<String, String> conf = Collections.singletonMap(
154+
HiveConnectorProperties.CONF_ENABLE_CREATE_BUCKET_TABLE, "true");
155+
ConnectorBucketSpec bucket = new ConnectorBucketSpec(
156+
Collections.singletonList("id"), 8, "doris_default");
157+
158+
metadataWithConf(client, conf, Collections.emptyMap())
159+
.createTable(session(), request().bucketSpec(bucket).build());
160+
161+
Assertions.assertEquals(Collections.singletonList("id"), client.lastCreateTable.getBucketCols());
162+
Assertions.assertEquals(8, client.lastCreateTable.getNumBuckets());
163+
}
164+
165+
@Test
166+
public void theConfTemplateIsNamedAfterTheProvider() {
167+
// WHY: the engine reads <name>.conf, and this connector's name is "hms" while its plugin
168+
// directory is "hive". A template under any other name deploys a file nothing ever opens --
169+
// silently, with every setting in it ignored. Renaming getType() must break here.
170+
String expected = new HiveConnectorProvider().name() + ".conf.template";
171+
Assertions.assertNotNull(
172+
HiveConnectorMetadataDdlTest.class.getClassLoader().getResource(expected),
173+
"the plugin must ship " + expected + " on its classpath");
174+
}
175+
115176
// ==================== createTable: transactional rejection ====================
116177

117178
@Test
@@ -368,6 +429,12 @@ private static HiveConnectorMetadata metadata(RecordingHmsClient client,
368429
return new HiveConnectorMetadata(client, catalogProps, new FakeConnectorContext(env));
369430
}
370431

432+
private static HiveConnectorMetadata metadataWithConf(RecordingHmsClient client,
433+
Map<String, String> conf, Map<String, String> env) {
434+
return new HiveConnectorMetadata(client, Collections.emptyMap(),
435+
new FakeConnectorContext("test_catalog", 0L, env, conf));
436+
}
437+
371438
/**
372439
* Columns are nullable: hive rejects a {@code NOT NULL} column up front (validateColumns runs before every
373440
* other createTable check), so a NOT NULL fixture column would short-circuit every test in this class before

0 commit comments

Comments
 (0)