Skip to content

Commit 0c35a6e

Browse files
authored
[feature](plugin) Add information_schema.plugins for kernel plugin metadata (#65644)
### What problem does this PR solve? Issue #65479 Kernel plugin families (filesystem, connector, authentication, lineage) currently have no unified, queryable inventory: to know which plugins are loaded on an FE, one has to read logs. This PR adds a unified `information_schema.plugins` system table listing all successfully loaded kernel plugins of the currently connected FE. Columns: `PLUGIN_NAME`, `PLUGIN_TYPE`, `PLUGIN_VERSION`, `SOURCE`, `DESCRIPTION`. Design points: - `(plugin_type, plugin_name)` is the primary key, enforced at load time: the first registration wins and is never silently overridden, so a directory jar can never displace a built-in plugin. - `name()` / `description()` are snapshotted once at load time; querying the table never executes plugin code. A broken self-report fails the load. - Plugin names are validated at load (trimmed, non-blank, <=64 chars, `[A-Za-z0-9._-]`); an invalid name fails the load for external plugins. - `PLUGIN_VERSION` comes from the jar MANIFEST `Implementation-Version` (NULL when unavailable), never from plugin interface methods. - Only successfully loaded plugins appear; load failures go to logs. No status/fail_reason columns. - Querying requires global ADMIN privilege; non-admin users see no rows. - View semantics are per-FE: the BE scanner RPCs the currently connected FE (same pattern as `catalog_meta_cache_statistics`), so per-FE jar deployment can be audited connection by connection. Implementation: - `fe-extension-spi`: add `default String description()` to `PluginFactory` (empty by default, zero migration cost). - `fe-extension-loader`: new process-wide `PluginRegistry` (load-time snapshots only) and `PluginNames` validation; `DirectoryPluginRuntimeManager` snapshots name/description and reads `Implementation-Version` from the jar containing the factory class. - The four family managers register BUILTIN/EXTERNAL rows; filesystem and connector managers now also reject external providers whose name conflicts with an already-registered one. - FE: `SchemaTable` definition + `MetadataGenerator` data source; thrift `TSchemaTableName.PLUGINS`. Reuses the existing MySQL-compat `TSchemaTableType.SCH_PLUGINS`. - BE: new `schema_plugins_scanner` wired into `schema_scanner`. Example: ``` mysql> SELECT * FROM information_schema.plugins; +-------------+----------------+----------------+---------+---------------------------+ | PLUGIN_NAME | PLUGIN_TYPE | PLUGIN_VERSION | SOURCE | DESCRIPTION | +-------------+----------------+----------------+---------+---------------------------+ | hdfs | FILESYSTEM | NULL | BUILTIN | | | jdbc | CONNECTOR | NULL | BUILTIN | | | oidc | AUTHENTICATION | 4.0.0 | EXTERNAL| OIDC authentication | +-------------+----------------+----------------+---------+---------------------------+ ``` ### Release note Add `information_schema.plugins` system table listing loaded kernel plugins (filesystem/connector/authentication/lineage) of the currently connected FE; requires ADMIN privilege. ### Check List (For Author) - Test - [x] Unit test: `PluginRegistryTest`, `DirectoryPluginRuntimeManagerMetadataTest` - [x] Regression test: `regression-test/suites/query_p0/schema_table/test_plugins_schema.groovy` - Behavior changed: - [x] Yes, new `information_schema.plugins` table (additive only). - Does this need documentation? - [x] Yes, `information_schema.plugins` should be documented as a new system table.
1 parent cb56b23 commit 0c35a6e

26 files changed

Lines changed: 1473 additions & 26 deletions

File tree

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#include "information_schema/schema_extensions_scanner.h"
19+
20+
#include <utility>
21+
22+
#include "core/assert_cast.h"
23+
#include "core/block/block.h"
24+
#include "core/column/column_nullable.h"
25+
#include "core/data_type/data_type_factory.hpp"
26+
#include "core/string_ref.h"
27+
#include "runtime/exec_env.h"
28+
#include "runtime/query_context.h"
29+
#include "runtime/runtime_state.h"
30+
#include "util/client_cache.h"
31+
#include "util/thrift_rpc_helper.h"
32+
33+
namespace doris {
34+
35+
std::vector<SchemaScanner::ColumnDesc> SchemaExtensionsScanner::_s_tbls_columns = {
36+
{"EXTENSION_NAME", TYPE_STRING, sizeof(StringRef), true},
37+
{"EXTENSION_TYPE", TYPE_STRING, sizeof(StringRef), true},
38+
{"EXTENSION_VERSION", TYPE_STRING, sizeof(StringRef), true},
39+
{"SOURCE", TYPE_STRING, sizeof(StringRef), true},
40+
{"DESCRIPTION", TYPE_STRING, sizeof(StringRef), true},
41+
};
42+
43+
SchemaExtensionsScanner::SchemaExtensionsScanner()
44+
: SchemaScanner(_s_tbls_columns, TSchemaTableType::SCH_EXTENSIONS) {}
45+
46+
Status SchemaExtensionsScanner::start(RuntimeState* state) {
47+
if (!_is_init) {
48+
return Status::InternalError("used before initialized.");
49+
}
50+
_block_rows_limit = state->batch_size();
51+
_rpc_timeout_ms = state->execution_timeout() * 1000;
52+
// Extensions are per-FE local state: ask the FE this session is connected to.
53+
auto* query_ctx = state->get_query_ctx();
54+
if (query_ctx == nullptr) {
55+
return Status::InternalError("query context is null");
56+
}
57+
_fe_addr = query_ctx->current_connect_fe;
58+
return Status::OK();
59+
}
60+
61+
Status SchemaExtensionsScanner::_get_extensions_block_from_fe() {
62+
TSchemaTableRequestParams schema_table_request_params;
63+
for (int i = 0; i < _s_tbls_columns.size(); i++) {
64+
schema_table_request_params.__isset.columns_name = true;
65+
schema_table_request_params.columns_name.emplace_back(_s_tbls_columns[i].name);
66+
}
67+
schema_table_request_params.__set_current_user_ident(*_param->common_param->current_user_ident);
68+
69+
TFetchSchemaTableDataRequest request;
70+
request.__set_schema_table_name(TSchemaTableName::EXTENSIONS);
71+
request.__set_schema_table_params(schema_table_request_params);
72+
73+
TFetchSchemaTableDataResult result;
74+
RETURN_IF_ERROR(ThriftRpcHelper::rpc<FrontendServiceClient>(
75+
_fe_addr.hostname, _fe_addr.port,
76+
[&request, &result](FrontendServiceConnection& client) {
77+
client->fetchSchemaTableData(result, request);
78+
},
79+
_rpc_timeout_ms));
80+
81+
Status status(Status::create(result.status));
82+
if (!status.ok()) {
83+
LOG(WARNING) << "fetch extensions from FE(" << _fe_addr.hostname
84+
<< ") failed, errmsg=" << status;
85+
return status;
86+
}
87+
88+
std::vector<TRow> result_data = std::move(result.data_batch);
89+
if (!result_data.empty()) {
90+
auto col_size = result_data[0].column_value.size();
91+
if (col_size != _s_tbls_columns.size()) {
92+
return Status::InternalError<false>("extensions schema is not match for FE and BE");
93+
}
94+
}
95+
96+
_extensions_block = Block::create_unique();
97+
for (int i = 0; i < _s_tbls_columns.size(); ++i) {
98+
auto data_type =
99+
DataTypeFactory::instance().create_data_type(_s_tbls_columns[i].type, true);
100+
_extensions_block->insert(ColumnWithTypeAndName(data_type->create_column(), data_type,
101+
_s_tbls_columns[i].name));
102+
}
103+
_extensions_block->reserve(_block_rows_limit);
104+
105+
for (int i = 0; i < result_data.size(); i++) {
106+
const TRow& row = result_data[i];
107+
for (int j = 0; j < _s_tbls_columns.size(); j++) {
108+
const TCell& cell = row.column_value[j];
109+
// An unset string cell means SQL NULL (e.g. unknown EXTENSION_VERSION);
110+
// insert_block_column would materialize it as an empty string.
111+
if (!cell.__isset.stringVal) {
112+
MutableColumnPtr mutable_col_ptr =
113+
IColumn::mutate(_extensions_block->get_by_position(j).column);
114+
// All _s_tbls_columns are declared nullable, so the column is
115+
// always a ColumnNullable wrapping the string column.
116+
assert_cast<ColumnNullable*>(mutable_col_ptr.get())->insert_data(nullptr, 0);
117+
_extensions_block->replace_by_position(j, std::move(mutable_col_ptr));
118+
continue;
119+
}
120+
RETURN_IF_ERROR(
121+
insert_block_column(cell, j, _extensions_block.get(), _s_tbls_columns[j].type));
122+
}
123+
}
124+
return Status::OK();
125+
}
126+
127+
Status SchemaExtensionsScanner::get_next_block_internal(Block* block, bool* eos) {
128+
if (!_is_init) {
129+
return Status::InternalError("Used before initialized.");
130+
}
131+
132+
if (nullptr == block || nullptr == eos) {
133+
return Status::InternalError("input pointer is nullptr.");
134+
}
135+
136+
if (_extensions_block == nullptr) {
137+
RETURN_IF_ERROR(_get_extensions_block_from_fe());
138+
_total_rows = static_cast<int>(_extensions_block->rows());
139+
}
140+
141+
if (_row_idx == _total_rows) {
142+
*eos = true;
143+
return Status::OK();
144+
}
145+
146+
int current_batch_rows = std::min(_block_rows_limit, _total_rows - _row_idx);
147+
ScopedMutableBlock scoped_mblock(block);
148+
auto& mblock = scoped_mblock.mutable_block();
149+
RETURN_IF_ERROR(mblock.add_rows(_extensions_block.get(), _row_idx, current_batch_rows));
150+
_row_idx += current_batch_rows;
151+
152+
*eos = _row_idx == _total_rows;
153+
return Status::OK();
154+
}
155+
156+
} // namespace doris
Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#pragma once
19+
20+
#include <vector>
21+
22+
#include "common/status.h"
23+
#include "information_schema/schema_scanner.h"
24+
25+
namespace doris {
26+
class RuntimeState;
27+
class Block;
28+
29+
// information_schema.extensions: kernel extension inventory of the currently
30+
// connected FE (extensions are per-FE local state, so the RPC targets the
31+
// connected FE instead of the master).
32+
class SchemaExtensionsScanner : public SchemaScanner {
33+
ENABLE_FACTORY_CREATOR(SchemaExtensionsScanner);
34+
35+
public:
36+
SchemaExtensionsScanner();
37+
~SchemaExtensionsScanner() override = default;
38+
39+
Status start(RuntimeState* state) override;
40+
Status get_next_block_internal(Block* block, bool* eos) override;
41+
42+
static std::vector<SchemaScanner::ColumnDesc> _s_tbls_columns;
43+
44+
private:
45+
Status _get_extensions_block_from_fe();
46+
47+
TNetworkAddress _fe_addr;
48+
49+
int _block_rows_limit = 4096;
50+
int _row_idx = 0;
51+
int _total_rows = 0;
52+
int _rpc_timeout_ms = 3000;
53+
std::unique_ptr<Block> _extensions_block = nullptr;
54+
};
55+
56+
} // namespace doris

‎be/src/information_schema/schema_scanner.cpp‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@
6060
#include "information_schema/schema_database_properties_scanner.h"
6161
#include "information_schema/schema_dummy_scanner.h"
6262
#include "information_schema/schema_encryption_keys_scanner.h"
63+
#include "information_schema/schema_extensions_scanner.h"
6364
#include "information_schema/schema_file_cache_info_scanner.h"
6465
#include "information_schema/schema_file_cache_statistics.h"
6566
#include "information_schema/schema_files_scanner.h"
@@ -286,6 +287,8 @@ std::unique_ptr<SchemaScanner> SchemaScanner::create(TSchemaTableType::type type
286287
return SchemaFileCacheInfoScanner::create_unique();
287288
case TSchemaTableType::SCH_AUTHENTICATION_INTEGRATIONS:
288289
return SchemaAuthenticationIntegrationsScanner::create_unique();
290+
case TSchemaTableType::SCH_EXTENSIONS:
291+
return SchemaExtensionsScanner::create_unique();
289292
case TSchemaTableType::SCH_ROLE_MAPPINGS:
290293
return SchemaRoleMappingsScanner::create_unique();
291294
case TSchemaTableType::SCH_TABLE_STREAMS:
Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#include "information_schema/schema_extensions_scanner.h"
19+
20+
#include <gen_cpp/Descriptors_types.h>
21+
#include <gtest/gtest.h>
22+
23+
#include "common/object_pool.h"
24+
#include "common/status.h"
25+
#include "core/block/block.h"
26+
#include "core/data_type/define_primitive_type.h"
27+
#include "information_schema/schema_scanner.h"
28+
#include "runtime/runtime_state.h"
29+
#include "testutil/mock/mock_runtime_state.h"
30+
31+
namespace doris {
32+
33+
// The RPC data path (_get_extensions_block_from_fe) requires a live FE and is
34+
// covered by the FE-side regression test; ThriftRpcHelper has no result-injection
35+
// hook, so these unit tests exercise the FE-independent surface only: the factory
36+
// registration, the column schema, and the pre-/post-init guard branches of
37+
// start() and get_next_block_internal().
38+
class SchemaExtensionsScannerTest : public testing::Test {
39+
void SetUp() override {}
40+
void TearDown() override {}
41+
};
42+
43+
TEST_F(SchemaExtensionsScannerTest, test_create_extensions_scanner) {
44+
auto scanner = SchemaScanner::create(TSchemaTableType::SCH_EXTENSIONS);
45+
ASSERT_NE(nullptr, scanner);
46+
EXPECT_EQ(TSchemaTableType::SCH_EXTENSIONS, scanner->type());
47+
48+
const auto& columns = scanner->get_column_desc();
49+
ASSERT_EQ(5, columns.size());
50+
EXPECT_STREQ("EXTENSION_NAME", columns[0].name);
51+
EXPECT_STREQ("EXTENSION_TYPE", columns[1].name);
52+
EXPECT_STREQ("EXTENSION_VERSION", columns[2].name);
53+
EXPECT_STREQ("SOURCE", columns[3].name);
54+
EXPECT_STREQ("DESCRIPTION", columns[4].name);
55+
56+
// All extension columns are nullable strings (e.g. unknown EXTENSION_VERSION
57+
// is surfaced as SQL NULL rather than an empty string).
58+
for (const auto& column : columns) {
59+
EXPECT_EQ(TYPE_STRING, column.type);
60+
EXPECT_TRUE(column.is_null);
61+
}
62+
}
63+
64+
TEST_F(SchemaExtensionsScannerTest, test_start_before_init) {
65+
SchemaExtensionsScanner scanner;
66+
// start() must reject use before init() regardless of the RuntimeState.
67+
auto st = scanner.start(nullptr);
68+
EXPECT_FALSE(st.ok());
69+
}
70+
71+
TEST_F(SchemaExtensionsScannerTest, test_get_next_block_before_init) {
72+
SchemaExtensionsScanner scanner;
73+
auto block = Block::create_unique();
74+
bool eos = false;
75+
auto st = scanner.get_next_block_internal(block.get(), &eos);
76+
EXPECT_FALSE(st.ok());
77+
EXPECT_FALSE(eos);
78+
}
79+
80+
// After init(), start() still fails when the session has no query context, since
81+
// extensions are fetched from the FE recorded on the query context.
82+
TEST_F(SchemaExtensionsScannerTest, test_start_without_query_context) {
83+
// A default RuntimeState has a null query context.
84+
RuntimeState state;
85+
SchemaScannerParam param;
86+
ObjectPool pool;
87+
88+
SchemaExtensionsScanner scanner;
89+
ASSERT_TRUE(scanner.init(&state, &param, &pool).ok());
90+
91+
auto st = scanner.start(&state);
92+
EXPECT_FALSE(st.ok());
93+
}
94+
95+
// With a query context present, start() succeeds and stashes the per-session
96+
// state (batch size, rpc timeout, connected FE) used by the later RPC.
97+
TEST_F(SchemaExtensionsScannerTest, test_start_with_query_context) {
98+
MockRuntimeState state;
99+
SchemaScannerParam param;
100+
ObjectPool pool;
101+
102+
SchemaExtensionsScanner scanner;
103+
ASSERT_TRUE(scanner.init(&state, &param, &pool).ok());
104+
105+
auto st = scanner.start(&state);
106+
EXPECT_TRUE(st.ok());
107+
}
108+
109+
// After init(), get_next_block_internal() rejects null output pointers before
110+
// attempting any FE RPC.
111+
TEST_F(SchemaExtensionsScannerTest, test_get_next_block_null_output) {
112+
RuntimeState state;
113+
SchemaScannerParam param;
114+
ObjectPool pool;
115+
116+
SchemaExtensionsScanner scanner;
117+
ASSERT_TRUE(scanner.init(&state, &param, &pool).ok());
118+
119+
bool eos = false;
120+
EXPECT_FALSE(scanner.get_next_block_internal(nullptr, &eos).ok());
121+
122+
auto block = Block::create_unique();
123+
EXPECT_FALSE(scanner.get_next_block_internal(block.get(), nullptr).ok());
124+
}
125+
126+
} // namespace doris

0 commit comments

Comments
 (0)