Repository navigation
Commit 82646c3
[feat](connector) give each connector plugin its own conf file (apache#66347)
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
Giving one connector a new deployment-level setting currently costs two
edits in
the engine: an `@ConfField` in fe-common's `Config`, and a line in
fe-core's
`DefaultConnectorContext.buildEnvironment()` that forwards it. A
connector plugin
cannot read `Config` itself — it loads child-first, so its own bundled
copy
shadows the engine's and every field reads back as a code default — so
the engine
has to carry each key by name. The result is that `fe-core` knows the
config key
names of connectors it is otherwise entirely agnostic about, and that
list only
grows.
**This PR gives a connector plugin a configuration file of its own.**
The engine
reads `<pluginDir>/<name>.conf` (where `<name>` is
`ConnectorProvider.name()`),
parses it generically, and serves it back through the new
`ConnectorContext.getConnectorConfig()`. No key name of any connector
reaches
`fe-core`, and **a new connector needs no engine change at all** to add
a
deployment-level setting.
Connectors read a setting through one entry point:
```java
ConnectorConf.get(context, "drivers_dir", "jdbc_drivers_dir", null)
// ^ key in <name>.conf ^ the fe.conf key it used to live under
```
Resolution is **plugin conf → fe.conf → default**, with blank treated as
"not
set" at each step (an operator who writes `key=` has not configured it,
and
reading it as a set empty string would let one stray line mask the
fe.conf value
actually in effect).
Six settings across five connectors are moved onto the new channel.
**Every
`@ConfField` is kept** and stays the fallback, so an existing deployment
upgrades
with nothing to edit and behaves exactly as before:
| connector (`name()`) | conf file | key | fe.conf fallback |
| --- | --- | --- | --- |
| `trino-connector` | `trino-connector.conf` | `plugin_dir` |
`trino_connector_plugin_dir` |
| `hms` (hive) | `hms.conf` | `default_file_format` |
`hive_default_file_format` |
| `hms` (hive) | `hms.conf` | `enable_create_bucket_table` |
`enable_create_hive_bucket_table` |
| `jdbc` | `jdbc.conf` | `drivers_dir` | `jdbc_drivers_dir` |
| `jdbc` | `jdbc.conf` | `force_sqlserver_encrypt_false` |
`force_sqlserver_jdbc_encrypt_false` |
| `iceberg` / `paimon` | `iceberg.conf` / `paimon.conf` | `drivers_dir`
| `jdbc_drivers_dir` |
| `iceberg` / `paimon` | `iceberg.conf` / `paimon.conf` |
`metastore_client_timeout_second` |
`hive_metastore_client_timeout_second` |
Two of these are shared by several connectors at the fe.conf end (one
`jdbc_drivers_dir` serves jdbc, iceberg and paimon). A per-plugin file
cannot
express that, so a deployment that moves them sets the value in each
plugin's
conf; leaving them commented out keeps the single shared fe.conf value,
which is
what every existing deployment gets. Both templates say so.
`doris_home` and `doris_version` stay in `getEnvironment()` — they are
not
connector settings. `jdbc_driver_secure_path` is dropped from it: no
connector
ever read it (the JDBC allow-list is enforced in fe-core by
`JdbcResource`, which
reads `Config` directly), so it was a dead key that read like a
connector
setting. The `Config` field itself is unchanged.
**Packaging.** A plugin ships `src/main/resources/<name>.conf.template`;
`build.sh`
seeds the live `<name>.conf` from it with `cp -n`, globbing
`*.conf.template` with
no connector named, so a new connector needs no `build.sh` change
either. The live
`.conf` is deliberately **not** in the plugin zip, so the ordinary
upgrade —
unzipping a newer plugin build over the deployed directory — refreshes
the jars
and the template but never the administrator's file.
Note the conf file is named after `ConnectorProvider.name()`, **not**
after the
plugin directory: `plugins/connector/hive/` holds `hms.conf` and
`plugins/connector/trino/` holds `trino-connector.conf`. The directory
name is the
deployer's choice and cannot be what the engine keys on. Each connector
carries a
test asserting its shipped template name still tracks `name()`.
`connector.plugin.api.version` stays at **1.0**. The added
`ConnectorContext.getConnectorConfig()` is a `default` method, nothing
outside
fe-connector-spi implements `ConnectorContext`, and the only decorators
of it
extend the parent-first `ForwardingConnectorContext` — loaded from the
FE's own
classpath, so they inherit the new forward without being rebuilt. A
connector
plugin built before this change loads and behaves exactly as it did; it
simply
never reads the new map.
### Release note
Connector plugins can now carry their own deployment-level configuration
file,
`<DORIS_HOME>/plugins/connector/<dir>/<name>.conf`, seeded from a
template shipped
with each plugin. Settings there take precedence over the corresponding
`fe.conf`
keys, which keep working unchanged — no action is required when
upgrading. The
file must be maintained on every FE node and takes effect after an FE
restart.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
**Unit tests** — new: `ConnectorConfFileTest`, `ConnectorConfTest`
(spi),
`ConnectorPluginConfTest` (fe-core, drives `loadPlugins` +
`createConnector` on real
plugin directories), `IcebergConnectorConfTest`,
`PaimonConnectorConfTest`, plus
per-connector cases and a template-name guard in `TrinoBootstrapTest`,
`HiveConnectorMetadataDdlTest`, `JdbcUrlNormalizerTest`. Existing
`ForwardingConnectorContextTest` covers the new forward by reflection.
Each was mutation-checked (precedence reversed, forward removed, surface
baseline
reverted, conf not keyed per provider, conf never attached, template
renamed) and
confirmed to fail.
**Manual test** — `sh build.sh --fe`, then verified in
`output/fe/plugins/connector/`:
- `hive/hms.conf`, `trino/trino-connector.conf`, `iceberg/iceberg.conf`,
`jdbc/jdbc.conf`, `paimon/paimon.conf` are created and byte-identical to
their
templates; `es/`, `hudi/`, `maxcompute/` ship no template and get no
file, with
no error.
- Every seeded file has **0 active settings** (all commented out), so a
fresh
deployment behaves exactly as before.
- Upgrade safety: hand-edited `hms.conf`, replayed the deploy step
(`unzip -o` +
the `cp -n` loop) from the real plugin zip — the edit survives and only
the
template is refreshed.
- Behavior changed:
- [x] Yes.
Two, both narrow:
1. A deployment-level setting now resolves from the plugin's
`<name>.conf` before
`fe.conf`. For a deployment that does not edit the seeded
(all-commented) file,
nothing changes.
2. A blank `hive_default_file_format` in fe.conf now falls through to
`orc` instead
of reaching the metastore create as an empty format string. An empty
file format
was never a working configuration.
- Does this need documentation?
- [x] Yes. <!-- doris-website PR not filed yet -->
Each shipped `.conf.template` documents its own keys inline, and
`fe/fe-connector/README.md` plus `fe-connector-api/package-info.java`
(Rule 7) are
updated for connector authors. A doris-website page for operators is
still to be
written.
---------
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>1 parent 8b0cafd commit 82646c3
54 files changed
Lines changed: 2040 additions & 134 deletions
File tree
- fe
- fe-connector
- fe-connector-api/src/main/java/org/apache/doris/connector/api
- fe-connector-hive/src
- main
- assembly
- java/org/apache/doris/connector/hive
- resources
- test/java/org/apache/doris/connector/hive
- fe-connector-iceberg/src
- main
- assembly
- java/org/apache/doris/connector/iceberg
- resources
- test/java/org/apache/doris/connector/iceberg
- fe-connector-jdbc/src
- main
- assembly
- java/org/apache/doris/connector/jdbc
- resources
- test/java/org/apache/doris/connector/jdbc
- fe-connector-metastore-paimon/src/test/java/org/apache/doris/connector/metastore/paimon/jdbc
- fe-connector-metastore-spi/src/main/java/org/apache/doris/connector/metastore/spi
- fe-connector-paimon/src
- main
- assembly
- java/org/apache/doris/connector/paimon
- resources
- test/java/org/apache/doris/connector/paimon
- fe-connector-spi/src
- main/java/org/apache/doris/connector/spi
- test
- java/org/apache/doris/connector/spi
- resources
- fe-connector-trino/src
- main
- assembly
- java/org/apache/doris/connector/trino
- resources
- test/java/org/apache/doris/connector/trino
- fe-core/src
- main/java/org/apache/doris/connector
- test/java/org/apache/doris
- connectorconf/testplugins
- connector
- fe-filesystem/fe-filesystem-obs/src
- main/java/org/apache/doris/filesystem/obs
- test/java/org/apache/doris/filesystem/obs
- regression-test/suites/external_table_p0/hive/ddl
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
62 | 72 | | |
63 | 73 | | |
64 | 74 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1082 | 1082 | | |
1083 | 1083 | | |
1084 | 1084 | | |
| 1085 | + | |
| 1086 | + | |
| 1087 | + | |
| 1088 | + | |
| 1089 | + | |
| 1090 | + | |
| 1091 | + | |
| 1092 | + | |
| 1093 | + | |
1085 | 1094 | | |
1086 | | - | |
| 1095 | + | |
1087 | 1096 | | |
1088 | 1097 | | |
1089 | 1098 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
211 | 211 | | |
212 | 212 | | |
213 | 213 | | |
214 | | - | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
215 | 231 | | |
216 | 232 | | |
217 | 233 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
160 | 160 | | |
161 | 161 | | |
162 | 162 | | |
163 | | - | |
164 | | - | |
| 163 | + | |
| 164 | + | |
165 | 165 | | |
166 | 166 | | |
167 | 167 | | |
| |||
174 | 174 | | |
175 | 175 | | |
176 | 176 | | |
177 | | - | |
178 | | - | |
179 | | - | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
180 | 193 | | |
181 | 194 | | |
182 | 195 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
43 | 54 | | |
44 | 55 | | |
45 | 56 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
67 | 67 | | |
68 | 68 | | |
69 | 69 | | |
| 70 | + | |
70 | 71 | | |
71 | 72 | | |
72 | 73 | | |
| |||
1618 | 1619 | | |
1619 | 1620 | | |
1620 | 1621 | | |
1621 | | - | |
| 1622 | + | |
| 1623 | + | |
1622 | 1624 | | |
1623 | 1625 | | |
1624 | 1626 | | |
| |||
1673 | 1675 | | |
1674 | 1676 | | |
1675 | 1677 | | |
1676 | | - | |
| 1678 | + | |
| 1679 | + | |
1677 | 1680 | | |
1678 | 1681 | | |
1679 | | - | |
1680 | | - | |
| 1682 | + | |
| 1683 | + | |
| 1684 | + | |
| 1685 | + | |
1681 | 1686 | | |
1682 | 1687 | | |
1683 | 1688 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
73 | 73 | | |
74 | 74 | | |
75 | 75 | | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
76 | 83 | | |
77 | 84 | | |
78 | 85 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
28 | | - | |
29 | | - | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
30 | 31 | | |
31 | 32 | | |
32 | 33 | | |
| |||
40 | 41 | | |
41 | 42 | | |
42 | 43 | | |
| 44 | + | |
43 | 45 | | |
44 | 46 | | |
45 | 47 | | |
| |||
50 | 52 | | |
51 | 53 | | |
52 | 54 | | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
53 | 60 | | |
54 | 61 | | |
55 | 62 | | |
| 63 | + | |
56 | 64 | | |
57 | 65 | | |
58 | 66 | | |
| |||
69 | 77 | | |
70 | 78 | | |
71 | 79 | | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
72 | 85 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
112 | 112 | | |
113 | 113 | | |
114 | 114 | | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
115 | 176 | | |
116 | 177 | | |
117 | 178 | | |
| |||
368 | 429 | | |
369 | 430 | | |
370 | 431 | | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
371 | 438 | | |
372 | 439 | | |
373 | 440 | | |
| |||
0 commit comments