Skip to content

Commit 1005da0

Browse files
committed
[fix](nereids) HBO: tombstone re-delete on load success + javadoc polish
### What problem does this PR solve? Problem Summary: v4pro 复审 round3(commit f908d4d)给出 0 Blocker / 0 Major / 0 Minor / 3 Nit;本轮修订: - M-1-R(墓碑键跨重启复活):未就绪窗口内 DB 删除失败的 DELETE 由墓碑在 进程内封死复活,但首次成功 load(证明 DB 可达)只是清空墓碑、未对墓碑 键补发 DB 删除,库行残留导致 FE 重启后仍复活。修复:load 成功后对墓碑 键逐个 best-effort 重发幂等 HboStatisticsStore.delete(失败仍仅 warn, 行保留并已在 javadoc 注明),再清空墓碑。 - N-3:Config javadoc 对"pending load 期间 honored 的删除"补 "issued while this config is true" 限定,与 OFF 期删除行为区分清楚; manager 类 javadoc 同步墓碑语义(待加载删除被墓碑化、load 成功后重删)。 - N-4:墓碑字段/类 javadoc 点明跨 FE 弱一致模型角落 —— 本 FE 墓碑可能压 制他 FE 在加载窗口内重建的同键行,重启后随一次性加载恢复。 验证:build.sh --fe EXIT=0(checkstyle 0 违规);SET/重启加载/翻转/DELETE 清行手工链路通过;hbo_structinfo_inject_test 与 hbo_persist_internal_db_test 均 Test 1 suites, failed 0。 ### Release note None(实验特性内部修订,默认关闭) ### Check List (For Author) - Test: FE 构建通过;回归两套通过;手工验证持久化链路 - Behavior changed: No - Does this need documentation: No
1 parent f908d4d commit 1005da0

2 files changed

Lines changed: 25 additions & 8 deletions

File tree

‎fe/fe-common/src/main/java/org/apache/doris/common/Config.java‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2336,11 +2336,13 @@ public class Config extends ConfigBase {
23362336
* and the table is loaded lazily on first use. Persistence applies to SET/DELETE issued
23372337
* while this config is true: entries SET while it is false are not persisted, and entries
23382338
* DELETEd while it is false keep their stored rows. A FE loads the table into memory at most
2339-
* once per process (on first use while this config is true), so such rows reappear only when
2340-
* that one-time load is still pending (e.g. right after FE start with the config on). On
2341-
* load, existing in-memory entries win over the stored snapshot and deletions that happened
2342-
* while the load was pending are honored; a FE never refreshes entries SET by other FEs
2343-
* during its lifetime.
2339+
* once per process (on first use while this config is true), so rows deleted while it is
2340+
* false reappear only for a FE whose one-time load is still pending (e.g. right after FE
2341+
* start with the config on). On that load, existing in-memory entries win over the stored
2342+
* snapshot, deletions issued while this config is true during the pending load are honored
2343+
* (their stored rows are best-effort removed again once the table is reachable), and rows a
2344+
* DELETE could not remove reappear after a FE restart; a FE never refreshes entries SET by
2345+
* other FEs during its lifetime.
23442346
*/
23452347
@ConfField(mutable = true, description = "The default setting is false. When true, HBO SET/DELETE STATISTICS "
23462348
+ "entries are persisted into __internal_schema.hbo_statistics and reloaded after FE "

‎fe/fe-core/src/main/java/org/apache/doris/nereids/stats/HboPlanStatisticsManager.java‎

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -45,8 +45,11 @@
4545
* through to {@code __internal_schema.hbo_statistics} and each FE loads the table into memory
4646
* once, lazily on first use. The in-memory cache stays authoritative: loaded entries are merged
4747
* with putIfAbsent, so entries injected while persistence was off (or concurrently) win over the
48-
* stored snapshot. A FE never refreshes entries SET by another FE during its lifetime; the
49-
* snapshot is only taken at load time.
48+
* stored snapshot, and a DELETE whose DB removal failed while the load was still pending is
49+
* tombstoned so the pending load cannot resurrect it (the removal is retried once the load
50+
* succeeds). A FE never refreshes entries SET by another FE during its lifetime; in particular a
51+
* tombstone may hide a row that another FE re-created during this FE's load window until this FE
52+
* restarts and reloads. The snapshot is only taken at load time.
5053
*/
5154
public class HboPlanStatisticsManager {
5255
private static final Logger LOG = LogManager.getLogger(HboPlanStatisticsManager.class);
@@ -70,7 +73,10 @@ public class HboPlanStatisticsManager {
7073
private long lastLoadFailedMs = 0L;
7174
// fingerprints whose DELETE could not be removed from the internal table while a lazy load
7275
// is still pending (hboPinnedLoaded false); the pending or retrying load must not resurrect
73-
// them into memory (guarded by pinnedLoadLock, cleared after a successful load)
76+
// them into memory. Cleared after a successful load, at which point the suppressed rows are
77+
// best-effort removed from the table again. Under the per-FE weak consistency model a
78+
// tombstone may also suppress a row that another FE re-created during the load window; it
79+
// reappears after this FE restarts and reloads. (guarded by pinnedLoadLock)
7480
private final Set<String> pendingLoadTombstones = new HashSet<>();
7581

7682
public HboPlanStatisticsManager() {
@@ -171,6 +177,15 @@ private void ensurePinnedLoaded() {
171177
}
172178
}
173179
hboPinnedLoaded = true;
180+
if (!pendingLoadTombstones.isEmpty()) {
181+
// the load succeeded, so the internal table is reachable: retry the failed DB
182+
// removals (idempotent) so their rows do not reappear after a FE restart; a row
183+
// re-created by another FE during the tombstone window is removed here too,
184+
// which is acceptable under the per-FE weak consistency model (see class javadoc)
185+
for (String fingerprint : pendingLoadTombstones) {
186+
HboStatisticsStore.delete(fingerprint);
187+
}
188+
}
174189
pendingLoadTombstones.clear();
175190
LOG.info("loaded {} hbo pinned statistics entries from internal table", loaded.size());
176191
}

0 commit comments

Comments
 (0)