Skip to content

Commit c94cb29

Browse files
committed
[fix](nereids) HBO: address persistence code review findings
### What problem does this PR solve? Problem Summary: v4pro 审阅 commit 3962b55(pinned 落盘 internal db)后 0 Blocker / 0 Major / 7 Minor / 6 Nit,结论可合入;本轮修订全部 Minor 与 可落地 Nit: - P-1:加载与 SET/DELETE 并发互斥 —— 内存阶段(load 批量写入、SET put、 DELETE invalidate)统一走 pinnedLoadLock;加载改用 `asMap().putIfAbsent` 合并,杜绝加载覆盖并发 SET 新值/复活已删条目。 - P-2:加载失败不再每个 group 重复 DDL+SELECT —— loadAll 失败返回 null (原返回空表语义无法区分),manager 侧 30s backoff(lastLoadFailedMs)后 才重试。 - P-3:多 FE 语义文档化 —— 每个 FE 进程只加载一次快照、不刷新其它 FE 的 SET(类 javadoc + Config javadoc)。 - P-4:转义复用 StatisticsUtil.escapeSQL(同时双写 ' 与 \\,internal ctx 默认 sql_mode 无 NO_BACKSLASH_ESCAPES 时反斜杠不再被解析吞掉)。 - P-5:struct_info 截断改按 UTF-8 字节并在字符边界截断(varchar 上限按 字节计,原 UTF-16 char 截断对多字节文本可能超限被拒)。 - P-6:replication_num 从硬编码 1 改为 Math.max(1, Config.min_replication_num_per_tablet),与 InternalSchemaInitializer 约定一致,min_replication>1 集群不再静默建表失败。 - P-7:配置热切换语义文档化(默认 false 期 SET 不落盘、DELETE 不清行; 重新开启后加载以 putIfAbsent 合并、内存新值优先)+ putIfAbsent 落实。 - N-1/N-2/N-3:删除本地 escape(null)死代码、改用 InternalCatalog.INTERNAL_CATALOG_NAME、catch Throwable 收窄为 Exception。 - N-4:回归 finally 先还原 config 再 DELETE,清理失败不再泄漏热配置。 - 记录在案:N-5(.out 尾随空白为生成器固有,惯例接受)、N-6(空串/NULL 以内存 '' 为规范,读侧 NULL->'' 已统一;DB 列保持 NULL 兼容)。 验证:build.sh --fe EXIT=0(checkstyle 0 违规);70k ASCII 与 30k 多字节 STRUCT 截断实测(65533 字节、21844 个完整 3 字节字符、无截断字符);含 引号/反斜杠 STRUCT 落盘往返无损;重启后惰性加载恢复 explain 翻转 (loaded 1 entries);hbo_structinfo_inject_test 与 hbo_persist_internal_db_test 均 Test 1 suites, failed 0。 ### Release note None(实验特性内部修订,默认关闭) ### Check List (For Author) - Test: FE 构建通过;回归两套通过;手工验证转义/截断/重启恢复 - Behavior changed: No - Does this need documentation: No
1 parent 3962b55 commit c94cb29

4 files changed

Lines changed: 122 additions & 51 deletions

File tree

‎fe/fe-common/src/main/java/org/apache/doris/common/Config.java‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2333,7 +2333,12 @@ public class Config extends ConfigBase {
23332333
* Persist manually injected (pinned) hbo statistics into the internal database table
23342334
* __internal_schema.hbo_statistics, so that they survive FE restarts. In-memory pinned
23352335
* statistics stay authoritative; changes are written through synchronously (best effort)
2336-
* and the table is loaded lazily on first use.
2336+
* and the table is loaded lazily on first use. Persistence applies to SET/DELETE issued
2337+
* while this config is true: entries SET while it is false are not persisted, and entries
2338+
* DELETEd while it is false keep their stored rows, which are reloaded after the config is
2339+
* re-enabled (or the FE restarts); on re-enable, existing in-memory entries win over the
2340+
* stored snapshot. Each FE loads the table once per process and does not refresh entries
2341+
* SET by other FEs during its lifetime.
23372342
*/
23382343
@ConfField(mutable = true, description = "The default setting is false. When true, HBO SET/DELETE STATISTICS "
23392344
+ "entries are persisted into __internal_schema.hbo_statistics and reloaded after FE "

‎fe/fe-core/src/main/java/org/apache/doris/nereids/stats/HboPlanStatisticsManager.java‎

Lines changed: 50 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
import org.apache.logging.log4j.Logger;
2727

2828
import java.util.Collections;
29+
import java.util.List;
2930
import java.util.Map;
3031
import java.util.Optional;
3132

@@ -36,18 +37,34 @@
3637
* - pinned hbo statistics: manually injected statistics via the {@code HBO SET STATISTICS}
3738
* statement, keyed by the hbo fingerprint; pinned entries take precedence over learned ones
3839
* and are never overwritten or evicted by the automatic profile-based collection.
40+
*
41+
* <p>Pinned statistics are per-FE in memory; when
42+
* {@code Config.hbo_persist_pinned_to_internal_db} is enabled they are additionally written
43+
* through to {@code __internal_schema.hbo_statistics} and each FE loads the table into memory
44+
* once, lazily on first use. The in-memory cache stays authoritative: loaded entries are merged
45+
* with putIfAbsent, so entries injected while persistence was off (or concurrently) win over the
46+
* stored snapshot. A FE never refreshes entries SET by another FE during its lifetime; the
47+
* snapshot is only taken at load time.
3948
*/
4049
public class HboPlanStatisticsManager {
4150
private static final Logger LOG = LogManager.getLogger(HboPlanStatisticsManager.class);
4251

52+
private static final long LOAD_RETRY_INTERVAL_MS = 30_000L;
53+
4354
private HboPlanStatisticsProvider hboPlanStatisticsProvider;
4455
private HboPlanInfoProvider hboPlanInfoProvider;
4556
private final Cache<String, PinnedHboStatistics> pinnedPlanStatistics = Caffeine.newBuilder()
4657
.maximumSize(Math.max(Config.hbo_pinned_stats_cache_num, 0))
4758
.build();
4859
// whether pinned statistics have been loaded from the internal table (only when persistence
49-
// is enabled); single-threaded planner/command access makes this flag safe enough
60+
// is enabled); planner/command threads access it concurrently
5061
private volatile boolean hboPinnedLoaded = false;
62+
// serializes the memory phases of the lazy load, SET and DELETE against each other; the
63+
// best-effort persistence SQL of SET/DELETE runs outside this lock
64+
private final Object pinnedLoadLock = new Object();
65+
// backoff timestamp after a failed load attempt, so a not-ready internal schema does not
66+
// trigger a DDL+SELECT retry for every group of every query (guarded by pinnedLoadLock)
67+
private long lastLoadFailedMs = 0L;
5168

5269
public HboPlanStatisticsManager() {
5370
hboPlanStatisticsProvider = new MemoryHboPlanStatisticsProvider();
@@ -80,9 +97,11 @@ public void putPinnedPlanStatistics(String fingerprint, long rows, String nodeTy
8097
// LRU bounded by Config.hbo_pinned_stats_cache_num; pinned entries are otherwise never
8198
// expired automatically and are only removed by HBO DELETE STATISTICS or eviction
8299
long createTimeMs = System.currentTimeMillis();
83-
pinnedPlanStatistics.put(fingerprint,
84-
new PinnedHboStatistics(fingerprint, rows, nodeType, structCanonical, createTimeMs));
85-
if (Config.hbo_persist_pinned_to_internal_db && FeConstants.enableInternalSchemaDb) {
100+
synchronized (pinnedLoadLock) {
101+
pinnedPlanStatistics.put(fingerprint,
102+
new PinnedHboStatistics(fingerprint, rows, nodeType, structCanonical, createTimeMs));
103+
}
104+
if (persistenceEnabled()) {
86105
HboStatisticsStore.persist(fingerprint, rows, nodeType, structCanonical, createTimeMs);
87106
}
88107
}
@@ -97,8 +116,10 @@ public Optional<PinnedHboStatistics> getPinnedPlanStatistics(String fingerprint)
97116
* untouched here; callers may invalidate them explicitly.
98117
*/
99118
public void removePinnedPlanStatistics(String fingerprint) {
100-
pinnedPlanStatistics.invalidate(fingerprint);
101-
if (Config.hbo_persist_pinned_to_internal_db && FeConstants.enableInternalSchemaDb) {
119+
synchronized (pinnedLoadLock) {
120+
pinnedPlanStatistics.invalidate(fingerprint);
121+
}
122+
if (persistenceEnabled()) {
102123
HboStatisticsStore.delete(fingerprint);
103124
}
104125
}
@@ -109,24 +130,36 @@ public Map<String, PinnedHboStatistics> getAllPinnedPlanStatistics() {
109130
}
110131

111132
private void ensurePinnedLoaded() {
112-
if (hboPinnedLoaded || !Config.hbo_persist_pinned_to_internal_db
113-
|| !FeConstants.enableInternalSchemaDb) {
133+
if (hboPinnedLoaded || !persistenceEnabled()) {
114134
return;
115135
}
116-
try {
117-
for (PinnedHboStatistics pinned : HboStatisticsStore.loadAll()) {
118-
pinnedPlanStatistics.put(pinned.getFingerprint(), pinned);
136+
synchronized (pinnedLoadLock) {
137+
if (hboPinnedLoaded || !persistenceEnabled()) {
138+
return;
139+
}
140+
long now = System.currentTimeMillis();
141+
if (now - lastLoadFailedMs < LOAD_RETRY_INTERVAL_MS) {
142+
return;
143+
}
144+
List<PinnedHboStatistics> loaded = HboStatisticsStore.loadAll();
145+
if (loaded == null) {
146+
lastLoadFailedMs = now;
147+
return;
148+
}
149+
for (PinnedHboStatistics pinned : loaded) {
150+
// entries injected while persistence was off, or concurrently with the load,
151+
// must win over the stored snapshot: the in-memory cache stays authoritative
152+
pinnedPlanStatistics.asMap().putIfAbsent(pinned.getFingerprint(), pinned);
119153
}
120154
hboPinnedLoaded = true;
121-
LOG.info("loaded {} hbo pinned statistics entries from internal table",
122-
pinnedPlanStatistics.asMap().size());
123-
} catch (Throwable t) {
124-
// table may not be ready yet (e.g. internal schema initialization in progress);
125-
// retry on the next access
126-
LOG.warn("failed to load hbo pinned statistics from internal table", t);
155+
LOG.info("loaded {} hbo pinned statistics entries from internal table", loaded.size());
127156
}
128157
}
129158

159+
private static boolean persistenceEnabled() {
160+
return Config.hbo_persist_pinned_to_internal_db && FeConstants.enableInternalSchemaDb;
161+
}
162+
130163
/**
131164
* A manually injected hbo statistics entry.
132165
*/

‎fe/fe-core/src/main/java/org/apache/doris/nereids/stats/HboStatisticsStore.java‎

Lines changed: 64 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -17,13 +17,16 @@
1717

1818
package org.apache.doris.nereids.stats;
1919

20+
import org.apache.doris.common.Config;
2021
import org.apache.doris.common.FeConstants;
22+
import org.apache.doris.datasource.InternalCatalog;
2123
import org.apache.doris.statistics.ResultRow;
2224
import org.apache.doris.statistics.util.StatisticsUtil;
2325

2426
import org.apache.logging.log4j.LogManager;
2527
import org.apache.logging.log4j.Logger;
2628

29+
import java.nio.charset.StandardCharsets;
2730
import java.util.ArrayList;
2831
import java.util.List;
2932

@@ -42,46 +45,35 @@ public class HboStatisticsStore {
4245

4346
private static final String INTERNAL_DB = FeConstants.INTERNAL_DB_NAME;
4447
private static final String TABLE = "hbo_statistics";
45-
private static final String FULL_QUALIFIED = "internal." + INTERNAL_DB + "." + TABLE;
46-
private static final int STRUCT_MAX_LEN = 65533;
47-
48-
private static final String DDL = "CREATE TABLE IF NOT EXISTS `internal`.`" + INTERNAL_DB + "`.`" + TABLE
49-
+ "` (\n"
50-
+ " `fingerprint` varchar(64) NOT NULL COMMENT \"\",\n"
51-
+ " `row_count` bigint NOT NULL COMMENT \"\",\n"
52-
+ " `node_type` varchar(1024) NULL COMMENT \"\",\n"
53-
+ " `struct_info` varchar(" + STRUCT_MAX_LEN + ") NULL COMMENT \"\",\n"
54-
+ " `create_time_ms` bigint NOT NULL COMMENT \"\"\n"
55-
+ ") ENGINE = olap\n"
56-
+ "UNIQUE KEY(`fingerprint`)\n"
57-
+ "COMMENT \"Doris internal hbo pinned statistics table, DO NOT MODIFY IT\"\n"
58-
+ "DISTRIBUTED BY HASH(`fingerprint`)\n"
59-
+ "BUCKETS 1\n"
60-
+ "PROPERTIES (\"replication_num\" = \"1\")";
48+
private static final String FULL_QUALIFIED =
49+
InternalCatalog.INTERNAL_CATALOG_NAME + "." + INTERNAL_DB + "." + TABLE;
50+
/** max length (bytes) of the struct_info varchar column */
51+
private static final int STRUCT_MAX_BYTES = 65533;
6152

6253
private HboStatisticsStore() {
6354
}
6455

6556
/** Create the table if it does not exist. */
6657
public static void ensureTable() throws Exception {
67-
StatisticsUtil.execUpdate(DDL);
58+
StatisticsUtil.execUpdate(createDdl());
6859
}
6960

70-
/** Upsert one pinned entry (UNIQUE KEY fingerprint replaces on conflict). */
61+
/**
62+
* Upsert one pinned entry (UNIQUE KEY fingerprint replaces on conflict).
63+
*/
7164
public static void persist(String fingerprint, long rows, String nodeType, String structCanonical,
7265
long createTimeMs) {
7366
try {
7467
ensureTable();
75-
String struct = structCanonical == null ? "" : structCanonical;
76-
if (struct.length() > STRUCT_MAX_LEN) {
77-
struct = struct.substring(0, STRUCT_MAX_LEN);
78-
}
68+
String nodeTypeVal = nodeType == null ? "" : nodeType;
69+
String struct = truncateUtf8(structCanonical == null ? "" : structCanonical, STRUCT_MAX_BYTES);
7970
String sql = "INSERT INTO " + FULL_QUALIFIED
8071
+ " (`fingerprint`, `row_count`, `node_type`, `struct_info`, `create_time_ms`) VALUES ('"
81-
+ escape(fingerprint) + "', " + rows + ", '" + escape(nodeType) + "', '"
82-
+ escape(struct) + "', " + createTimeMs + ")";
72+
+ StatisticsUtil.escapeSQL(fingerprint) + "', " + rows + ", '"
73+
+ StatisticsUtil.escapeSQL(nodeTypeVal) + "', '" + StatisticsUtil.escapeSQL(struct)
74+
+ "', " + createTimeMs + ")";
8375
StatisticsUtil.execUpdate(sql);
84-
} catch (Throwable t) {
76+
} catch (Exception t) {
8577
LOG.warn("failed to persist hbo pinned statistics for fingerprint {}", fingerprint, t);
8678
}
8779
}
@@ -90,14 +82,20 @@ public static void persist(String fingerprint, long rows, String nodeType, Strin
9082
public static void delete(String fingerprint) {
9183
try {
9284
ensureTable();
93-
String sql = "DELETE FROM " + FULL_QUALIFIED + " WHERE `fingerprint` = '" + escape(fingerprint) + "'";
85+
String sql = "DELETE FROM " + FULL_QUALIFIED + " WHERE `fingerprint` = '"
86+
+ StatisticsUtil.escapeSQL(fingerprint) + "'";
9487
StatisticsUtil.execUpdate(sql);
95-
} catch (Throwable t) {
88+
} catch (Exception t) {
9689
LOG.warn("failed to delete hbo pinned statistics for fingerprint {}", fingerprint, t);
9790
}
9891
}
9992

100-
/** Load all pinned entries into memory. Returns empty list when the table is not present. */
93+
/**
94+
* Load all pinned entries from the internal table.
95+
*
96+
* @return the loaded entries, or {@code null} when the load failed (e.g. the internal schema
97+
* is not ready yet); the caller decides when to retry
98+
*/
10199
public static List<HboPlanStatisticsManager.PinnedHboStatistics> loadAll() {
102100
List<HboPlanStatisticsManager.PinnedHboStatistics> result = new ArrayList<>();
103101
try {
@@ -117,13 +115,47 @@ public static List<HboPlanStatisticsManager.PinnedHboStatistics> loadAll() {
117115
LOG.warn("skip malformed hbo pinned statistics row {}", row, e);
118116
}
119117
}
120-
} catch (Throwable t) {
118+
return result;
119+
} catch (Exception t) {
121120
LOG.warn("failed to load hbo pinned statistics from internal table", t);
121+
return null;
122122
}
123-
return result;
124123
}
125124

126-
private static String escape(String value) {
127-
return value == null ? "" : value.replace("'", "''");
125+
private static String createDdl() {
126+
// follow the internal-table convention (see InternalSchemaInitializer) so that CREATE
127+
// never fails on clusters whose min_replication_num_per_tablet exceeds 1
128+
int replication = Math.max(1, Config.min_replication_num_per_tablet);
129+
return "CREATE TABLE IF NOT EXISTS `" + InternalCatalog.INTERNAL_CATALOG_NAME + "`.`" + INTERNAL_DB
130+
+ "`.`" + TABLE + "` (\n"
131+
+ " `fingerprint` varchar(64) NOT NULL COMMENT \"\",\n"
132+
+ " `row_count` bigint NOT NULL COMMENT \"\",\n"
133+
+ " `node_type` varchar(1024) NULL COMMENT \"\",\n"
134+
+ " `struct_info` varchar(" + STRUCT_MAX_BYTES + ") NULL COMMENT \"\",\n"
135+
+ " `create_time_ms` bigint NOT NULL COMMENT \"\"\n"
136+
+ ") ENGINE = olap\n"
137+
+ "UNIQUE KEY(`fingerprint`)\n"
138+
+ "COMMENT \"Doris internal hbo pinned statistics table, DO NOT MODIFY IT\"\n"
139+
+ "DISTRIBUTED BY HASH(`fingerprint`)\n"
140+
+ "BUCKETS 1\n"
141+
+ "PROPERTIES (\"replication_num\" = \"" + replication + "\")";
142+
}
143+
144+
/**
145+
* Truncate to at most {@code maxBytes} UTF-8 bytes without splitting a multi-byte character
146+
* (the varchar column limit is byte-based).
147+
*/
148+
private static String truncateUtf8(String value, int maxBytes) {
149+
byte[] bytes = value.getBytes(StandardCharsets.UTF_8);
150+
if (bytes.length <= maxBytes) {
151+
return value;
152+
}
153+
// a cut inside a multi-byte sequence points at a continuation byte; back off to the
154+
// preceding lead byte so that only complete characters are kept
155+
int end = maxBytes;
156+
while (end > 0 && (bytes[end] & 0xC0) == 0x80) {
157+
end--;
158+
}
159+
return new String(bytes, 0, end, StandardCharsets.UTF_8);
128160
}
129161
}

‎regression-test/suites/query_p0/hbo/hbo_persist_internal_db_test.groovy‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,8 @@ suite("hbo_persist_internal_db_test", "nonConcurrent") {
3636
qt_delete_cleared """ SELECT fingerprint, row_count, node_type, struct_info FROM ${tableName}
3737
WHERE fingerprint = '${fingerprint}'; """
3838
} finally {
39-
sql """ HBO DELETE STATISTICS '${fingerprint}'; """
39+
// restore the config first so a failing cleanup can not leak the hot config to later suites
4040
sql """ ADMIN SET FRONTEND CONFIG ("hbo_persist_pinned_to_internal_db" = "${prevPersist}"); """
41+
sql """ HBO DELETE STATISTICS '${fingerprint}'; """
4142
}
4243
}

0 commit comments

Comments
 (0)