Skip to content

Commit 1086409

Browse files
zy-kkkpull[bot]
authored andcommitted
[Doc](AuditLoader) improvement auditLoader doc (#23758)
1 parent a6e2104 commit 1086409

2 files changed

Lines changed: 101 additions & 37 deletions

File tree

‎docs/en/docs/ecosystem/audit-plugin.md‎

Lines changed: 52 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -5,21 +5,21 @@
55
}
66
---
77

8-
<!--
8+
<!--
99
Licensed to the Apache Software Foundation (ASF) under one
10-
or more contributor license agreements. See the NOTICE file
10+
or more contributor license agreements. See the NOTICE file
1111
distributed with this work for additional information
12-
regarding copyright ownership. The ASF licenses this file
12+
regarding copyright ownership. The ASF licenses this file
1313
to you under the Apache License, Version 2.0 (the
1414
"License"); you may not use this file except in compliance
15-
with the License. You may obtain a copy of the License at
15+
with the License. You may obtain a copy of the License at
1616
17-
  http://www.apache.org/licenses/LICENSE-2.0
17+
http://www.apache.org/licenses/LICENSE-2.0
1818
1919
Unless required by applicable law or agreed to in writing,
2020
software distributed under the License is distributed on an
2121
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
22-
KIND, either express or implied. See the License for the
22+
KIND, either express or implied. See the License for the
2323
specific language governing permissions and limitations
2424
under the License.
2525
-->
@@ -34,30 +34,47 @@ This plugin can periodically import the FE audit log into the specified Doris cl
3434

3535
### FE Configuration
3636

37-
The audit log plug-in framework is disabled by default in Doris. You need to add `plugin_enable = true `to the FE configuration file to enable plugin framework.
37+
The audit log plug-in framework is enabled by default in Doris and is controlled by the FE configuration `plugin_enable`
3838

3939
### AuditLoader Configuration
4040

41-
The configuration of the auditloader plugin is located in `$ {DORIS}/fe_plugins/auditloader/src/main/assembly/`.
41+
1. Download the Audit Loader plugin
4242

43-
Open `plugin.conf` for configuration. See the comments of the configuration items.
43+
The Audit Loader plug-in is provided by default in the Doris distribution. After downloading the Doris installation package through [DOWNLOAD](https://doris.apache.org/download), decompress it and enter the directory, you can find the auditloader.zip file in the extensionsaudit_loader subdirectory.
4444

45-
<version since="1.2.0"></version>
46-
Audit log plugin supports importing slow query logs into a separate slow table since version 1.2, `doris_slow_log_tbl__`, which is closed by default. In the plugin configuration file, add `enable_slow_log = true` to enable the function. And you could modify 'qe_slow_log_ms' item in FE configuration file to change slow query threshold.
45+
2. Unzip the installation package
4746

48-
### Compile
47+
```shell
48+
unzip auditloader.zip
49+
```
4950

50-
After executing `sh build_plugin.sh` in the Doris code directory, you will get the `auditloader.zip` file in the `fe_plugins/output` directory.
51+
Unzip and generate the following files:
5152

52-
### Deployment
53+
* auditloader.jar: plug-in code package.
54+
* plugin.properties: plugin properties file.
55+
* plugin.conf: plugin configuration file.
5356

54-
You can place this file on an http download server or copy(or unzip) it to the specified directory of all FEs. Here we use the latter.
57+
3. Modify plugin.conf
5558

56-
### Installation
59+
The following configurations are available for modification:
5760

58-
After deployment is complete, and before installing the plugin, you need to create the audit log database and tables previously specified in `plugin.conf`. If `enable_slow_log` is set true, the slow table `doris_slow_log_tbl__` needs to be created, with the same schema as `doris_audit_log_tbl__`. The database and table creation statement is as follows:
61+
* frontend_host_port: FE node IP address and HTTP port in the format <fe_ip>:<fe_http_port>. The default value is 127.0.0.1:8030.
62+
* database: Audit log database name.
63+
* audit_log_table: Audit log table name.
64+
* slow_log_table: Slow query log table name.
65+
* enable_slow_log: Whether to enable the slow query log import function. The default value is false.
66+
* user: Cluster username. The user must have INSERT permission on the corresponding table.
67+
* password: Cluster user password.
5968

60-
```
69+
### 创建库表
70+
71+
In Doris, you need to create the library and table of the audit log. The table structure is as follows:
72+
73+
If you need to enable the slow query log import function, you need to create an additional slow table `doris_slow_log_tbl__`, whose table structure is consistent with `doris_audit_log_tbl__`.
74+
75+
Among them, the `dynamic_partition` attribute selects the number of days for audit log retention according to your own needs.
76+
77+
```sql
6178
create database doris_audit_db__;
6279
6380
create table doris_audit_db__.doris_audit_log_tbl__
@@ -81,7 +98,7 @@ create table doris_audit_db__.doris_audit_log_tbl__
8198
sql_hash varchar(48) comment "Hash value for this query",
8299
sql_digest varchar(48) comment "Sql digest for this query",
83100
peak_memory_bytes bigint comment "Peak memory bytes used on all backends of this query",
84-
stmt string comment "The original statement, trimed if longer than 2G "
101+
stmt string comment "The original statement, trimed if longer than 2G"
85102
) engine=OLAP
86103
duplicate key(query_id, `time`, client_ip)
87104
partition by range(`time`) ()
@@ -117,7 +134,7 @@ create table doris_audit_db__.doris_slow_log_tbl__
117134
sql_hash varchar(48) comment "Hash value for this query",
118135
sql_digest varchar(48) comment "Sql digest for this query",
119136
peak_memory_bytes bigint comment "Peak memory bytes used on all backends of this query",
120-
stmt string comment "The original statement, trimed if longer than 2G"
137+
stmt string comment "The original statement, trimed if longer than 2G "
121138
) engine=OLAP
122139
duplicate key(query_id, `time`, client_ip)
123140
partition by range(`time`) ()
@@ -137,8 +154,20 @@ properties(
137154
>
138155
> In the above table structure: stmt string, this can only be used in 0.15 and later versions, in previous versions, the field type used varchar
139156

140-
The `dynamic_partition` attribute selects the number of days to keep the audit log based on your needs.
157+
### Deployment
158+
159+
You can place the packaged auditloader.zip on an http server, or copy `auditloader.zip` to the same specified directory in all FEs.
160+
161+
### Installation
162+
163+
通过以下语句安装 Audit Loader 插件:
164+
165+
```sql
166+
INSTALL PLUGIN FROM [source] [PROPERTIES ("key"="value", ...)]
167+
```
168+
169+
Detailed command reference: [INSTALL-PLUGIN.md](../sql-manual/sql-reference/Database-Administration-Statements/INSTALL-PLUGIN)
141170

142-
After that, connect to Doris and use the `INSTALL PLUGIN` command to complete the installation. After successful installation, you can see the installed plug-ins through `SHOW PLUGINS`, and the status is `INSTALLED`.
171+
After successful installation, you can see the installed plug-ins through `SHOW PLUGINS`, and the status is `INSTALLED`.
143172

144-
Upon completion, the plug-in will continuously import audit date into this table at specified intervals.
173+
After completion, the plugin will continuously insert audit logs into this table at specified intervals.

‎docs/zh-CN/docs/ecosystem/audit-plugin.md‎

Lines changed: 49 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -34,30 +34,53 @@ Doris 的审计日志插件是在 FE 的插件框架基础上开发的。是一
3434

3535
### FE 配置
3636

37-
审计日志插件框架在 Doris 中是默认关闭的,需要在FE的配置文件中,增加`plugin_enable = true`启用plugin框架
37+
审计日志插件框架在 Doris 中是默认开启的的,由 FE 的配置 `plugin_enable` 控制
3838

3939
### AuditLoader 配置
4040

41-
auditloader plugin的配置位于`${DORIS}/fe_plugins/auditloader/src/main/assembly/`.
41+
1. 下载 Audit Loader 插件
4242

43-
打开 `plugin.conf` 进行配置。配置项说明参见注释。
43+
Audit Loader 插件在 Doris 的发行版中默认提供,通过 [DOWNLOAD](https://doris.apache.org/zh-CN/download) 下载 Doris 安装包解压并进入目录后即可在 extensions/audit_loader 子目录下找到 auditloader.zip 文件。
4444

45-
<version since="1.2.0"></version>
46-
从 1.2 版本开始,审计日志插件支持将慢查询日志导入到单独的慢表 `doris_slow_log_tbl__` 中,Doris 中默认关闭,在审计日志的配置文件中,增加 `enable_slow_log = true`,开启该功能。并且可以在 FE 配置文件中修改 `qe_slow_log_ms` 项来修改慢查询阈值。
45+
2. 解压安装包
4746

48-
### 编译
47+
```shell
48+
unzip auditloader.zip
49+
```
4950

50-
在 Doris 代码目录下执行 `sh build_plugin.sh` 后,会在 `fe_plugins/output` 目录下得到 `auditloader.zip` 文件。
51+
解压生成以下文件:
5152

52-
### 部署
53+
* auditloader.jar:插件代码包。
54+
* plugin.properties:插件属性文件。
55+
* plugin.conf:插件配置文件。
5356

54-
您可以将这个文件放置在一个 http 服务器上,或者拷贝`auditloader.zip`(或者解压`auditloader.zip`)到所有 FE 的指定目录下。这里我们使用后者。
57+
3. 修改 plugin.conf
5558

56-
### 安装
59+
以下配置可供修改:
5760

58-
部署完成后,安装插件前,需要创建之前在 `plugin.conf` 中指定的审计日志数据库和表。若开启了慢查询日志导入功能,需要创建慢表 `doris_slow_log_tbl__`,其表结构与 `doris_audit_log_tbl__` 一致。其中建库与建表语句如下:
61+
* frontend_host_port:FE 节点 IP 地址和 HTTP 端口,格式为 <fe_ip>:<fe_http_port>。 默认值为 127.0.0.1:8030。
62+
* database:审计日志库名。
63+
* audit_log_table:审计日志表名。
64+
* slow_log_table:慢查询日志表名。
65+
* enable_slow_log:是否开启慢查询日志导入功能。默认值为 false。
66+
* user:集群用户名。该用户必须具有对应表的 INSERT 权限。
67+
* password:集群用户密码。
5968

60-
```
69+
4. 重新打包 Audit Loader 插件
70+
71+
```shell
72+
zip -r -q -m auditloader.zip auditloader.jar plugin.properties plugin.conf
73+
```
74+
75+
### 创建库表
76+
77+
在 Doris 中,需要创建审计日志的库和表,表结构如下:
78+
79+
若需开启慢查询日志导入功能,还需要额外创建慢表 `doris_slow_log_tbl__`,其表结构与 `doris_audit_log_tbl__` 一致。
80+
81+
其中 `dynamic_partition` 属性根据自己的需要,选择审计日志保留的天数。
82+
83+
```sql
6184
create database doris_audit_db__;
6285
6386
create table doris_audit_db__.doris_audit_log_tbl__
@@ -137,8 +160,20 @@ properties(
137160
>
138161
> 上面表结构中:stmt string ,这个只能在0.15及之后版本中使用,之前版本,字段类型使用varchar
139162

140-
其中 `dynamic_partition` 属性根据自己的需要,选择审计日志保留的天数。
163+
### 部署
164+
165+
您可以将 打包好的 auditloader.zip 放置在一个 http 服务器上,或者拷贝`auditloader.zip` 到所有 FE 的相同指定目录下。
166+
167+
### 安装
168+
169+
通过以下语句安装 Audit Loader 插件:
170+
171+
```sql
172+
INSTALL PLUGIN FROM [source] [PROPERTIES ("key"="value", ...)]
173+
```
174+
175+
详细命令参考:[INSTALL-PLUGIN.md](../sql-manual/sql-reference/Database-Administration-Statements/INSTALL-PLUGIN)
141176

142-
之后,连接到 Doris 后使用 `INSTALL PLUGIN` 命令完成安装。安装成功后,可以通过 `SHOW PLUGINS` 看到已经安装的插件,并且状态为 `INSTALLED`。
177+
安装成功后,可以通过 `SHOW PLUGINS` 看到已经安装的插件,并且状态为 `INSTALLED`。
143178

144179
完成后,插件会不断的以指定的时间间隔将审计日志插入到这个表中。

0 commit comments

Comments
 (0)