Skip to content

[feature](plugin) Add information_schema.plugins for kernel plugin metadata - #65644

Merged
CalvinKirs merged 11 commits into
apache:masterfrom
CalvinKirs:plugin-meta
Jul 27, 2026
Merged

CalvinKirs merged 11 commits into
apache:masterfrom
CalvinKirs:plugin-meta

Conversation

@CalvinKirs

@CalvinKirs CalvinKirs commented Jul 15, 2026 •

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue #65479

Kernel plugin families (filesystem, connector, authentication, lineage) currently have no unified, queryable inventory: to know which plugins are loaded on an FE, one has to read logs. This PR adds a unified information_schema.plugins system table listing all successfully loaded kernel plugins of the currently connected FE.

Columns: PLUGIN_NAME, PLUGIN_TYPE, PLUGIN_VERSION, SOURCE, DESCRIPTION.

Design points:

  • (plugin_type, plugin_name) is the primary key, enforced at load time: the first registration wins and is never silently overridden, so a directory jar can never displace a built-in plugin.
  • name() / description() are snapshotted once at load time; querying the table never executes plugin code. A broken self-report fails the load.
  • Plugin names are validated at load (trimmed, non-blank, <=64 chars, [A-Za-z0-9._-]); an invalid name fails the load for external plugins.
  • PLUGIN_VERSION comes from the jar MANIFEST Implementation-Version (NULL when unavailable), never from plugin interface methods.
  • Only successfully loaded plugins appear; load failures go to logs. No status/fail_reason columns.
  • Querying requires global ADMIN privilege; non-admin users see no rows.
  • View semantics are per-FE: the BE scanner RPCs the currently connected FE (same pattern as catalog_meta_cache_statistics), so per-FE jar deployment can be audited connection by connection.

Implementation:

  • fe-extension-spi: add default String description() to PluginFactory (empty by default, zero migration cost).
  • fe-extension-loader: new process-wide PluginRegistry (load-time snapshots only) and PluginNames validation; DirectoryPluginRuntimeManager snapshots name/description and reads Implementation-Version from the jar containing the factory class.
  • The four family managers register BUILTIN/EXTERNAL rows; filesystem and connector managers now also reject external providers whose name conflicts with an already-registered one.
  • FE: SchemaTable definition + MetadataGenerator data source; thrift TSchemaTableName.PLUGINS. Reuses the existing MySQL-compat TSchemaTableType.SCH_PLUGINS.
  • BE: new schema_plugins_scanner wired into schema_scanner.

Example:

mysql> SELECT * FROM information_schema.plugins;
+-------------+----------------+----------------+---------+---------------------------+
| PLUGIN_NAME | PLUGIN_TYPE    | PLUGIN_VERSION | SOURCE  | DESCRIPTION               |
+-------------+----------------+----------------+---------+---------------------------+
| hdfs        | FILESYSTEM     | NULL           | BUILTIN |                           |
| jdbc        | CONNECTOR      | NULL           | BUILTIN |                           |
| oidc        | AUTHENTICATION | 4.0.0          | EXTERNAL| OIDC authentication       |
+-------------+----------------+----------------+---------+---------------------------+

Release note

Add information_schema.plugins system table listing loaded kernel plugins (filesystem/connector/authentication/lineage) of the currently connected FE; requires ADMIN privilege.

Check List (For Author)

  • Test

    • Unit test: PluginRegistryTest, DirectoryPluginRuntimeManagerMetadataTest
    • Regression test: regression-test/suites/query_p0/schema_table/test_plugins_schema.groovy
  • Behavior changed:

    • Yes, new information_schema.plugins table (additive only).
  • Does this need documentation?

    • Yes, information_schema.plugins should be documented as a new system table.

…tadata

Add a unified information_schema.plugins system table that lists all
loaded kernel plugins (filesystem, connector, authentication, lineage)
of the currently connected FE.

Columns: PLUGIN_NAME, PLUGIN_TYPE, PLUGIN_VERSION, SOURCE, DESCRIPTION.

Design points:
- (plugin_type, plugin_name) is the primary key, enforced at load time:
  the first registration wins and is never silently overridden, so
  directory jars cannot displace built-in plugins.
- name()/description() are snapshotted once at load time; querying the
  table never executes plugin code. A broken self-report fails the load.
- Plugin names are validated at load (trimmed, non-blank, <=64 chars,
  [A-Za-z0-9._-]); invalid names fail the load for external plugins.
- PLUGIN_VERSION comes from the jar MANIFEST Implementation-Version
  (null when unavailable), never from plugin interface methods.
- Only successfully loaded plugins appear; failures go to logs.
- Querying requires global ADMIN privilege; non-admin users see no rows.
- View semantics are per-FE: the BE scanner RPCs the currently
  connected FE (same pattern as catalog_meta_cache_statistics).

Implementation:
- fe-extension-spi: add default description() to PluginFactory.
- fe-extension-loader: new PluginRegistry (process-wide, load-time
  snapshots) and PluginNames validation; DirectoryPluginRuntimeManager
  snapshots name/description and reads Implementation-Version.
- Four family managers register BUILTIN/EXTERNAL rows; filesystem and
  connector managers now also reject external providers whose name
  conflicts with an already-registered one.
- FE: SchemaTable definition + MetadataGenerator data source; thrift
  TSchemaTableName.PLUGINS. Reuses existing TSchemaTableType.SCH_PLUGINS.
- BE: new schema_plugins_scanner wired into schema_scanner.
- Tests: loader unit tests and regression suite test_plugins_schema.
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@CalvinKirs

Copy link
Copy Markdown
Member Author

run buildall

@CalvinKirs

CalvinKirs commented Jul 15, 2026 •

Copy link
Copy Markdown
Member Author

One open design question worth discussing: should querying information_schema.plugins trigger loading of not-yet-loaded plugins?

Today only authentication is lazy (external auth plugins load on first use via ensurePluginFactoryLoaded), but lazy loading could well become the norm for more plugin families down the road. If that happens, this table would only show whatever happens to have been loaded so far, and someone might reasonably ask for the query to force-load everything so the inventory is always complete.

@CalvinKirs

Copy link
Copy Markdown
Member Author

/review

@CalvinKirs

Copy link
Copy Markdown
Member Author

run buildall

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review context preparation failed before Codex ran; inspect the 'Prepare authoritative PR context and required AGENTS guides' step.
Workflow run: https://github.com/apache/doris/actions/runs/29403481123

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 22.95% (14/61) 🎉
Increment coverage report
Complete coverage report

@CalvinKirs

Copy link
Copy Markdown
Member Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for eight correctness, compatibility, lifecycle, and test-contract issues documented inline.

Review checkpoints:

  • Goal and tests: the intended result is an ADMIN-only, five-column inventory of successfully loaded plugins on the FE to which the session is connected. The happy-path unit/regression coverage does not exercise the advertised SQL NULL, family-to-registry identity, conflict cleanup, or split root/lib version paths. The regression also bypasses Doris's required generated-result convention and requires a BUILTIN row that the standard packaged cluster need not contain because filesystem/connector implementations are deployed as EXTERNAL directory plugins.
  • Scope and compatibility: the default description() method is binary-compatible, but tightening every external plugin name to a new ASCII/64-character grammar is not additive and can reject previously loadable deployments.
  • Runtime consistency and lifecycle: the concurrent containers and sorted registry snapshot are individually safe, with no new lock or hot-path concern. Publication across family runtime state and the singleton registry is not one decision, however; duplicate/invalid names can make inventory disagree with execution, built-in metadata failures can leave partial state, and newly rejected connector/filesystem handles remain retained with open classloaders.
  • FE/BE contract: schema order, requested-column handling, current-user propagation, ADMIN filtering, connected-FE routing, scanner batching, and the supported BE-before-FE rolling-upgrade direction were traced and found consistent. Missing versions are not encoded/materialized as SQL NULL.
  • Persistence, configuration, and data writes: this patch adds no persistence/replay, transaction, data-write, or dynamic-configuration path.
  • Security model: ADMIN filtering is consistent with the stated contract. Internal FE/BE RPC and plugin deployment are trusted boundaries under the repository threat model; I found no separate in-scope security vulnerability.
  • Tooling and observability: a literal NUL makes the new production registry source binary to Git and normal search/review tooling even though compilation succeeds.

Validation was static as required by the review prompt; I did not run local builds or tests. At the reviewed head, COMPILE, FE UT, CheckStyle, Cloud UT, External Regression, NonConcurrent Regression, and vault_p0 are green. Several external regression/coverage/performance/BE jobs are red with unavailable TeamCity details; the macOS BE UT failure is an environment JDK 25-versus-17 mismatch rather than evidence for these findings.

Automation convergence status: the three-round limit was reached when the final packaging/test candidate was found. That candidate was independently verified and merged into the test comment, and every suspicious point now has a concrete disposition, but the exact final comment set could not receive another all-agent unchanged reconfirmation within the allowed rounds; this review is therefore marked incomplete under the review-run contract.

- PluginRegistry: replace NUL-byte key separator with a space (file was
  treated as binary by git); safe because plugin names are charset-restricted
- DirectoryPluginRuntimeManager: add discard() so rejected duplicate handles
  release their classloaders; read Implementation-Version from the factory
  class's own code-source jar first (covers root-descriptor + lib-impl layouts)
- Managers (fs/connector/auth/lineage): snapshot self-reported metadata before
  publishing each built-in so one throwing plugin is skipped instead of
  aborting startup; discard rejected duplicate handles
- AuthenticationPluginManager: built-in registration is now first-wins
  (putIfAbsent), matching the registry's precedence
- BE scanner: surface unset string cells as SQL NULL instead of empty string
- Regression test: don't require BUILTIN rows (packaged clusters deploy
  fs/connector providers as directory plugins); assert PLUGIN_VERSION is
  NULL or non-empty
@CalvinKirs

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29600 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bf33dd70fe048b38573514fcb8e0cdacd90f4568, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17713	4161	4011	4011
q2	2024	318	201	201
q3	10316	1427	842	842
q4	4694	469	343	343
q5	7501	879	586	586
q6	188	173	140	140
q7	757	848	615	615
q8	9389	1587	1536	1536
q9	5707	4368	4358	4358
q10	6736	1725	1456	1456
q11	501	360	324	324
q12	716	566	456	456
q13	18093	3449	2769	2769
q14	261	261	246	246
q15	q16	782	787	712	712
q17	1035	967	1019	967
q18	6899	5820	5529	5529
q19	1175	1300	1121	1121
q20	790	702	588	588
q21	5596	2580	2498	2498
q22	424	357	302	302
Total cold run time: 101297 ms
Total hot run time: 29600 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4441	4364	4378	4364
q2	296	315	221	221
q3	4584	4944	4418	4418
q4	2056	2146	1523	1523
q5	4401	4300	4310	4300
q6	230	174	128	128
q7	1758	1867	1861	1861
q8	2794	2275	2203	2203
q9	7977	8210	7844	7844
q10	4666	4630	4271	4271
q11	569	421	373	373
q12	744	774	563	563
q13	3367	3707	2971	2971
q14	317	302	287	287
q15	q16	720	719	641	641
q17	1367	1320	1486	1320
q18	7786	7233	7243	7233
q19	1216	1082	1135	1082
q20	2208	2188	1932	1932
q21	5257	4617	4390	4390
q22	517	449	398	398
Total cold run time: 57271 ms
Total hot run time: 52323 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 177956 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bf33dd70fe048b38573514fcb8e0cdacd90f4568, data reload: false

query5	4331	638	480	480
query6	477	242	241	241
query7	4847	617	340	340
query8	324	183	175	175
query9	8757	4078	4094	4078
query10	452	349	313	313
query11	5887	2338	2122	2122
query12	156	108	110	108
query13	1322	630	441	441
query14	6236	5237	4911	4911
query14_1	4299	4276	4279	4276
query15	220	209	189	189
query16	1051	502	489	489
query17	1128	741	596	596
query18	2738	490	351	351
query19	211	196	157	157
query20	112	108	126	108
query21	235	164	142	142
query22	13563	13609	13364	13364
query23	17450	16424	16057	16057
query23_1	16146	16186	16209	16186
query24	7583	1793	1303	1303
query24_1	1321	1296	1300	1296
query25	579	469	398	398
query26	1354	359	218	218
query27	2552	587	387	387
query28	4456	1997	2012	1997
query29	1085	657	495	495
query30	343	262	231	231
query31	1117	1103	990	990
query32	111	65	63	63
query33	539	325	253	253
query34	1178	1139	654	654
query35	760	782	673	673
query36	1236	1200	1078	1078
query37	160	107	95	95
query38	1871	1729	1663	1663
query39	859	885	853	853
query39_1	837	835	840	835
query40	244	163	139	139
query41	66	64	63	63
query42	93	94	99	94
query43	323	338	290	290
query44	1411	772	761	761
query45	198	177	173	173
query46	1077	1186	724	724
query47	2127	2132	1936	1936
query48	405	426	289	289
query49	577	417	297	297
query50	1067	424	338	338
query51	10522	10548	10436	10436
query52	90	89	79	79
query53	264	282	199	199
query54	278	238	242	238
query55	72	70	65	65
query56	321	303	308	303
query57	1327	1295	1149	1149
query58	288	270	278	270
query59	1559	1688	1469	1469
query60	313	276	264	264
query61	156	156	158	156
query62	550	499	436	436
query63	242	204	197	197
query64	2787	1069	871	871
query65	4729	4653	4692	4653
query66	1777	505	383	383
query67	29218	29272	29009	29009
query68	3156	1618	1024	1024
query69	415	310	273	273
query70	1080	986	982	982
query71	364	334	321	321
query72	3078	2662	2381	2381
query73	843	778	430	430
query74	5046	4906	4688	4688
query75	2511	2496	2140	2140
query76	2340	1161	804	804
query77	357	372	286	286
query78	11733	11779	11383	11383
query79	1384	1209	785	785
query80	1294	547	475	475
query81	508	327	285	285
query82	582	160	117	117
query83	383	332	303	303
query84	283	151	126	126
query85	966	608	524	524
query86	464	302	294	294
query87	1834	1819	1747	1747
query88	3692	2792	2770	2770
query89	449	377	329	329
query90	1951	207	207	207
query91	209	190	163	163
query92	62	58	55	55
query93	1570	1583	1032	1032
query94	724	348	306	306
query95	789	595	491	491
query96	1091	788	339	339
query97	2610	2618	2516	2516
query98	209	205	200	200
query99	1093	1108	987	987
Total cold run time: 263422 ms
Total hot run time: 177956 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.16 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bf33dd70fe048b38573514fcb8e0cdacd90f4568, data reload: false

query1	0.01	0.00	0.00
query2	0.10	0.04	0.04
query3	0.29	0.14	0.13
query4	1.61	0.13	0.14
query5	0.24	0.21	0.22
query6	1.23	1.06	1.07
query7	0.04	0.02	0.00
query8	0.05	0.03	0.04
query9	0.39	0.31	0.31
query10	0.55	0.54	0.59
query11	0.18	0.14	0.13
query12	0.18	0.14	0.15
query13	0.48	0.47	0.47
query14	1.03	1.00	1.01
query15	0.61	0.60	0.59
query16	0.31	0.32	0.32
query17	1.12	1.10	1.14
query18	0.22	0.21	0.22
query19	1.97	1.99	1.94
query20	0.02	0.01	0.01
query21	15.43	0.20	0.13
query22	4.95	0.05	0.05
query23	16.14	0.31	0.13
query24	2.94	0.43	0.35
query25	0.11	0.05	0.04
query26	0.76	0.21	0.16
query27	0.04	0.04	0.04
query28	3.57	0.96	0.56
query29	12.53	4.12	3.30
query30	0.28	0.15	0.15
query31	2.77	0.58	0.31
query32	3.23	0.60	0.48
query33	3.09	3.32	3.24
query34	15.64	4.24	3.54
query35	3.54	3.52	3.51
query36	0.57	0.44	0.43
query37	0.09	0.07	0.07
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.16	0.16
query41	0.09	0.03	0.04
query42	0.04	0.02	0.03
query43	0.05	0.04	0.03
Total cold run time: 96.76 s
Total hot run time: 25.16 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 17.57% (39/222) 🎉
Increment coverage report
Complete coverage report

@CalvinKirs

Copy link
Copy Markdown
Member Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for three additional plugin-inventory issues documented inline. I did not duplicate the eight existing threads. The current head fixes the textual registry source, SQL NULL transport, defining-jar lookup, rejected-handle cleanup, and the packaged BUILTIN assumption; the existing name-contract, canonical-publication, callback, version-layout, and result-test threads remain applicable.

Review checkpoints:

  • Goal and behavior: this adds an ADMIN-only, five-column, per-connected-FE factory inventory. FE/BE schema order, projection, nullable transport, batching, privilege gating, and connected-FE routing are consistent, but one public-SPI compatibility path and two metadata/publication paths remain incorrect.
  • Scope and compatibility: the schema table is additive and the patch is focused. Adding the public description() default is not migration-safe for every precompiled factory. During a mixed new-FE/old-BE rollout, an old BE can still select its pre-feature dummy scanner and return an empty result until a supporting BE executes the table.
  • Lifecycle and concurrency: the concurrent maps, copy-on-write snapshots, lifecycle lock, and accepted-handle discard path show no new race or deadlock. Filesystem sensitive-key discovery still mutates runtime and inventory state in inconsistent orders on callback failure.
  • FE/BE and persistence: column identity/order, SQL NULL insertion, current-FE RPC behavior, and ADMIN-empty behavior are correct. The change does not add persisted state, configuration migration, or data writes.
  • Performance and observability: metadata is snapshotted at load time and the registry sorts a small inventory; no material hot-path concern found. Failure paths are logged, but the uncaught linkage error bypasses that isolation.
  • Tests and validation: changed tests do not cover an old-SPI default-method collision, a package-section manifest version, or filesystem sensitive-key callback failure. Review validation was static only as required by the review contract; no local build or test command was run. Hosted compile/FE UT/checkstyle checks are green; macOS BE UT failed before tests because the runner used JDK 25 while Doris requires JDK 17, and several TeamCity failures could not be inspected because their details endpoint requires authorization.

… grant

- FileSystemPluginManager: snapshot self-reported sensitive keys before
  mutating any store on both builtin and external paths, so a throwing
  provider is rejected atomically (no phantom inventory row, no active
  provider without a row)
- DirectoryPluginRuntimeManager: catch LinkageError alongside
  RuntimeException when reading factory name/description, keeping a
  broken-classpath jar's failure isolated to that plugin; honor jar-spec
  package manifest sections when resolving Implementation-Version
- test_plugins_schema: grant SELECT_PRIV on regression_test (and cloud
  cluster usage) to the fresh non-admin user, since the JDBC URL carries
  a default database and the connection is refused without it
@CalvinKirs

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.62% (1908/2458)
Line Coverage 64.48% (34152/52966)
Region Coverage 64.91% (17574/27076)
Branch Coverage 54.06% (9419/17424)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 20.88% (19/91) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 84.21% (80/95) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.05% (30590/40758)
Line Coverage 59.23% (337542/569869)
Region Coverage 56.03% (283560/506084)
Branch Coverage 57.28% (125864/219716)

@gavinchou

Copy link
Copy Markdown
Contributor

gensrc reviewed, LGTM

@github-actions github-actions Bot added approved Indicates a PR has been approved by one committer. labels Jul 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

1 similar comment
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@CalvinKirs
CalvinKirs merged commit 0c35a6e into apache:master Jul 27, 2026
34 of 35 checks passed
morningman added a commit that referenced this pull request Jul 27, 2026
62 commits replayed; the only conflicts were two import-block unions in
MetadataGenerator.java (upstream #65644's PluginRegistry import landing between
our datasource.plugin.PluginDriven* imports). Zero capability migration this
round: #65987's JDBC hardening already targets fe-connector-jdbc upstream and
its JdbcResource half is shared, #65644's plugin inventory was verified safe for
all 8 of our connectors, and the other three upstream commits are BE-only.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
morningman added a commit to morningman/doris that referenced this pull request Jul 27, 2026
…ion that carries semantics

Two textual conflicts, both in ConnectorPluginManager, both from upstream apache#65644
adding a plugin-inventory registration into the exact loop bodies this branch had
already given an admission gate. Resolution is the union, with one deliberate
narrowing: registerDiscovered stays outside upstream's catch(RuntimeException),
because its fail-loud IllegalStateException reports a build error and would
otherwise be silently downgraded to warn-and-skip.

Also records what the four verification layers actually covered, and the two
things they did not: those loop bodies have no test on either side, and the
merged loadPlugins now has one reject path that discards the plugin classloader
and one that does not.
morningman added a commit to morningman/doris that referenced this pull request Jul 27, 2026
… reject path

Merging this branch's admission gate into the loadPlugins loop that upstream
apache#65644 had just restructured left the loop with two reject paths and only one
of them releasing the plugin's classloader: the name-conflict path discarded,
the registerDiscovered path did not. FileSystemPluginManager, the sibling this
loop mirrors, discards on both of its reject paths.

Safe because a false return from registerDiscovered has claimed nothing — the
claimedTypes add is short-circuited while a problem is pending, claimedEngineNames
is untouched and the provider never reaches the providers list — so no part of the
manager still references that classloader. No extra log line: registerDiscovered
already reports why it refused.

fe-core targeted 26/26, checkstyle 0. The loop body itself remains untested on
both sides; see the handoff for why a test there was judged not worth its cost.
0AyanamiRei pushed a commit to 0AyanamiRei/doris that referenced this pull request Jul 27, 2026
…tadata (apache#65644)

### What problem does this PR solve?

Issue apache#65479

Kernel plugin families (filesystem, connector, authentication, lineage)
currently have no unified, queryable inventory: to know which plugins
are loaded on an FE, one has to read logs. This PR adds a unified
`information_schema.plugins` system table listing all successfully
loaded kernel plugins of the currently connected FE.

Columns: `PLUGIN_NAME`, `PLUGIN_TYPE`, `PLUGIN_VERSION`, `SOURCE`,
`DESCRIPTION`.

Design points:
- `(plugin_type, plugin_name)` is the primary key, enforced at load
time: the first registration wins and is never silently overridden, so a
directory jar can never displace a built-in plugin.
- `name()` / `description()` are snapshotted once at load time; querying
the table never executes plugin code. A broken self-report fails the
load.
- Plugin names are validated at load (trimmed, non-blank, <=64 chars,
`[A-Za-z0-9._-]`); an invalid name fails the load for external plugins.
- `PLUGIN_VERSION` comes from the jar MANIFEST `Implementation-Version`
(NULL when unavailable), never from plugin interface methods.
- Only successfully loaded plugins appear; load failures go to logs. No
status/fail_reason columns.
- Querying requires global ADMIN privilege; non-admin users see no rows.
- View semantics are per-FE: the BE scanner RPCs the currently connected
FE (same pattern as `catalog_meta_cache_statistics`), so per-FE jar
deployment can be audited connection by connection.

Implementation:
- `fe-extension-spi`: add `default String description()` to
`PluginFactory` (empty by default, zero migration cost).
- `fe-extension-loader`: new process-wide `PluginRegistry` (load-time
snapshots only) and `PluginNames` validation;
`DirectoryPluginRuntimeManager` snapshots name/description and reads
`Implementation-Version` from the jar containing the factory class.
- The four family managers register BUILTIN/EXTERNAL rows; filesystem
and connector managers now also reject external providers whose name
conflicts with an already-registered one.
- FE: `SchemaTable` definition + `MetadataGenerator` data source; thrift
`TSchemaTableName.PLUGINS`. Reuses the existing MySQL-compat
`TSchemaTableType.SCH_PLUGINS`.
- BE: new `schema_plugins_scanner` wired into `schema_scanner`.

Example:

```
mysql> SELECT * FROM information_schema.plugins;
+-------------+----------------+----------------+---------+---------------------------+
| PLUGIN_NAME | PLUGIN_TYPE    | PLUGIN_VERSION | SOURCE  | DESCRIPTION               |
+-------------+----------------+----------------+---------+---------------------------+
| hdfs        | FILESYSTEM     | NULL           | BUILTIN |                           |
| jdbc        | CONNECTOR      | NULL           | BUILTIN |                           |
| oidc        | AUTHENTICATION | 4.0.0          | EXTERNAL| OIDC authentication       |
+-------------+----------------+----------------+---------+---------------------------+
```

### Release note

Add `information_schema.plugins` system table listing loaded kernel
plugins (filesystem/connector/authentication/lineage) of the currently
connected FE; requires ADMIN privilege.

### Check List (For Author)

- Test
- [x] Unit test: `PluginRegistryTest`,
`DirectoryPluginRuntimeManagerMetadataTest`
- [x] Regression test:
`regression-test/suites/query_p0/schema_table/test_plugins_schema.groovy`

- Behavior changed:
    - [x] Yes, new `information_schema.plugins` table (additive only).

- Does this need documentation?
- [x] Yes, `information_schema.plugins` should be documented as a new
system table.
morningman added a commit that referenced this pull request Jul 27, 2026
62 commits replayed; the only conflicts were two import-block unions in
MetadataGenerator.java (upstream #65644's PluginRegistry import landing between
our datasource.plugin.PluginDriven* imports). Zero capability migration this
round: #65987's JDBC hardening already targets fe-connector-jdbc upstream and
its JdbcResource half is shared, #65644's plugin inventory was verified safe for
all 8 of our connectors, and the other three upstream commits are BE-only.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
morningman added a commit to morningman/doris that referenced this pull request Jul 27, 2026
…ion that carries semantics

Two textual conflicts, both in ConnectorPluginManager, both from upstream apache#65644
adding a plugin-inventory registration into the exact loop bodies this branch had
already given an admission gate. Resolution is the union, with one deliberate
narrowing: registerDiscovered stays outside upstream's catch(RuntimeException),
because its fail-loud IllegalStateException reports a build error and would
otherwise be silently downgraded to warn-and-skip.

Also records what the four verification layers actually covered, and the two
things they did not: those loop bodies have no test on either side, and the
merged loadPlugins now has one reject path that discards the plugin classloader
and one that does not.
morningman added a commit to morningman/doris that referenced this pull request Jul 27, 2026
… reject path

Merging this branch's admission gate into the loadPlugins loop that upstream
apache#65644 had just restructured left the loop with two reject paths and only one
of them releasing the plugin's classloader: the name-conflict path discarded,
the registerDiscovered path did not. FileSystemPluginManager, the sibling this
loop mirrors, discards on both of its reject paths.

Safe because a false return from registerDiscovered has claimed nothing — the
claimedTypes add is short-circuited while a problem is pending, claimedEngineNames
is untouched and the provider never reaches the providers list — so no part of the
manager still references that classloader. No extra log line: registerDiscovered
already reports why it refused.

fe-core targeted 26/26, checkstyle 0. The loop body itself remains untested on
both sides; see the handoff for why a test there was judged not worth its cost.
morningman added a commit that referenced this pull request Jul 27, 2026
62 commits replayed; the only conflicts were two import-block unions in
MetadataGenerator.java (upstream #65644's PluginRegistry import landing between
our datasource.plugin.PluginDriven* imports). Zero capability migration this
round: #65987's JDBC hardening already targets fe-connector-jdbc upstream and
its JdbcResource half is shared, #65644's plugin inventory was verified safe for
all 8 of our connectors, and the other three upstream commits are BE-only.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
morningman added a commit that referenced this pull request Jul 28, 2026
62 commits replayed; the only conflicts were two import-block unions in
MetadataGenerator.java (upstream #65644's PluginRegistry import landing between
our datasource.plugin.PluginDriven* imports). Zero capability migration this
round: #65987's JDBC hardening already targets fe-connector-jdbc upstream and
its JdbcResource half is shared, #65644's plugin inventory was verified safe for
all 8 of our connectors, and the other three upstream commits are BE-only.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
morningman added a commit that referenced this pull request Jul 28, 2026
62 commits replayed; the only conflicts were two import-block unions in
MetadataGenerator.java (upstream #65644's PluginRegistry import landing between
our datasource.plugin.PluginDriven* imports). Zero capability migration this
round: #65987's JDBC hardening already targets fe-connector-jdbc upstream and
its JdbcResource half is shared, #65644's plugin inventory was verified safe for
all 8 of our connectors, and the other three upstream commits are BE-only.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
morningman added a commit that referenced this pull request Jul 29, 2026
62 commits replayed; the only conflicts were two import-block unions in
MetadataGenerator.java (upstream #65644's PluginRegistry import landing between
our datasource.plugin.PluginDriven* imports). Zero capability migration this
round: #65987's JDBC hardening already targets fe-connector-jdbc upstream and
its JdbcResource half is shared, #65644's plugin inventory was verified safe for
all 8 of our connectors, and the other three upstream commits are BE-only.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
morningman added a commit that referenced this pull request Jul 29, 2026
62 commits replayed; the only conflicts were two import-block unions in
MetadataGenerator.java (upstream #65644's PluginRegistry import landing between
our datasource.plugin.PluginDriven* imports). Zero capability migration this
round: #65987's JDBC hardening already targets fe-connector-jdbc upstream and
its JdbcResource half is shared, #65644's plugin inventory was verified safe for
all 8 of our connectors, and the other three upstream commits are BE-only.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 17, 2026
…tadata (apache#65644)

### What problem does this PR solve?

Issue apache#65479

Kernel plugin families (filesystem, connector, authentication, lineage)
currently have no unified, queryable inventory: to know which plugins
are loaded on an FE, one has to read logs. This PR adds a unified
`information_schema.plugins` system table listing all successfully
loaded kernel plugins of the currently connected FE.

Columns: `PLUGIN_NAME`, `PLUGIN_TYPE`, `PLUGIN_VERSION`, `SOURCE`,
`DESCRIPTION`.

Design points:
- `(plugin_type, plugin_name)` is the primary key, enforced at load
time: the first registration wins and is never silently overridden, so a
directory jar can never displace a built-in plugin.
- `name()` / `description()` are snapshotted once at load time; querying
the table never executes plugin code. A broken self-report fails the
load.
- Plugin names are validated at load (trimmed, non-blank, <=64 chars,
`[A-Za-z0-9._-]`); an invalid name fails the load for external plugins.
- `PLUGIN_VERSION` comes from the jar MANIFEST `Implementation-Version`
(NULL when unavailable), never from plugin interface methods.
- Only successfully loaded plugins appear; load failures go to logs. No
status/fail_reason columns.
- Querying requires global ADMIN privilege; non-admin users see no rows.
- View semantics are per-FE: the BE scanner RPCs the currently connected
FE (same pattern as `catalog_meta_cache_statistics`), so per-FE jar
deployment can be audited connection by connection.

Implementation:
- `fe-extension-spi`: add `default String description()` to
`PluginFactory` (empty by default, zero migration cost).
- `fe-extension-loader`: new process-wide `PluginRegistry` (load-time
snapshots only) and `PluginNames` validation;
`DirectoryPluginRuntimeManager` snapshots name/description and reads
`Implementation-Version` from the jar containing the factory class.
- The four family managers register BUILTIN/EXTERNAL rows; filesystem
and connector managers now also reject external providers whose name
conflicts with an already-registered one.
- FE: `SchemaTable` definition + `MetadataGenerator` data source; thrift
`TSchemaTableName.PLUGINS`. Reuses the existing MySQL-compat
`TSchemaTableType.SCH_PLUGINS`.
- BE: new `schema_plugins_scanner` wired into `schema_scanner`.

Example:

```
mysql> SELECT * FROM information_schema.plugins;
+-------------+----------------+----------------+---------+---------------------------+
| PLUGIN_NAME | PLUGIN_TYPE    | PLUGIN_VERSION | SOURCE  | DESCRIPTION               |
+-------------+----------------+----------------+---------+---------------------------+
| hdfs        | FILESYSTEM     | NULL           | BUILTIN |                           |
| jdbc        | CONNECTOR      | NULL           | BUILTIN |                           |
| oidc        | AUTHENTICATION | 4.0.0          | EXTERNAL| OIDC authentication       |
+-------------+----------------+----------------+---------+---------------------------+
```

### Release note

Add `information_schema.plugins` system table listing loaded kernel
plugins (filesystem/connector/authentication/lineage) of the currently
connected FE; requires ADMIN privilege.

### Check List (For Author)

- Test
- [x] Unit test: `PluginRegistryTest`,
`DirectoryPluginRuntimeManagerMetadataTest`
- [x] Regression test:
`regression-test/suites/query_p0/schema_table/test_plugins_schema.groovy`

- Behavior changed:
    - [x] Yes, new `information_schema.plugins` table (additive only).

- Does this need documentation?
- [x] Yes, `information_schema.plugins` should be documented as a new
system table.
yiguolei pushed a commit that referenced this pull request Sep 20, 2026
…el plugin metadata (#68176)

#65644

---------

Signed-off-by: Aurélien Pupier <[email protected]>
Co-authored-by: Aurélien Pupier <[email protected]>
@CalvinKirs
CalvinKirs deleted the plugin-meta branch September 21, 2026 03:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/4.2.0-merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants