Skip to content

[fix](temp-table) not clean temp table temporary until fix mem leak (#59535) - #93

Merged
wenzhenghu merged 4 commits into
HYDCP:branch-3.1from
Ryan19929:pick-59535-branch-3.1
Sep 2, 2026
Merged

wenzhenghu merged 4 commits into
HYDCP:branch-3.1from
Ryan19929:pick-59535-branch-3.1

Conversation

@Ryan19929

Copy link
Copy Markdown

What problem does this PR solve?

Related PR: apache#59535 (cherry-picked from master commit d172147)

Problem Summary:

branch-3.1 上 FESessionMgr 的 session 跟踪机制存在内存泄漏:

  1. Env.registerSessionInfo() 生效,且注册面远超真实客户端连接——Master 处理每条 follower 转发的语句时(FrontendServiceImpl.forward)都会经 ConnectContext 三参构造器用一个随机 UUID 注册进 aliveSessionSet 后再覆盖 sessionId,导致每条转发语句永久泄漏一个 UUID;数十处内部任务代码 new ConnectContext() 同样只注册不注销。实测集群 aliveSessionSet 已积压约 1450 万条目。
  2. Master 每 5s 向所有 FE 全量拉取 session 列表(Thrift getAliveSessions),反序列化出数百万元素的 Object[];随集合膨胀,单轮分配从 389MB 涨到 843MB(JFR P99 达 7.3GB),巨型数组晋升老年代后不回收,Master FE 老年代被持续撑大,最终有 OOM 风险。
  3. 泄漏的死 session 每轮仍被上报为"活着",sessionReportTimeMap 时间戳被不断刷新,孤儿临时表永不超时,TemporaryTableMgr 的自动回收已实质失效。

修复方式(与上游 master/4.0 一致):

  • 注释 Env.registerSessionInfo() 的 aliveSessionSet.add(sessionId);
  • 注释 Env.refreshSession() 的 sessionReportTimeMap.put(...);
  • 注释 TemporaryTableMgr.runAfterCatalogReady() 的孤儿临时表清理逻辑(必须同步停用:tracking 关闭后 map 恒空,保留清理会误删所有临时表)。

另追加一个 test 提交:skip test_temp_table_with_restart(该 case 断言的正是被停用的孤儿清理能力,参照上游 apache#64525 的做法)。test_temp_table、test_temp_table_with_conn_timeout 不受影响,保持运行。

Release note

临时表的孤儿自动清理(session 所在 FE 异常退出后的临时表回收)被停用,与上游 apache#59535 一致;正常断开连接时的临时表删除不受影响。

Check List (For Author)

morrySnow and others added 2 commits September 1, 2026 11:33
…pache#59535)

### What problem does this PR solve?

Related PR: apache#40680

Problem Summary:

This pull request temporarily disables session tracking and automatic
cleanup for temporary tables due to a memory leak issue involving
`Env#sessionReportTimeMap` and `Env#aliveSessionSet`. The affected logic
is commented out with TODO notes, and related imports are cleaned up.

Session management changes:

* Disabled the code that adds sessions to `aliveSessionSet` in
`Env.registerSessionInfo`, preventing new session tracking until the
memory leak is fixed.
* Disabled the code that updates session report times in
`Env.refreshSession`, pausing session activity tracking.

Temporary table cleanup changes:

* Commented out the logic in `TemporaryTableMgr.runAfterCatalogReady`
that deletes temporary tables when their creating session is gone,
halting automatic cleanup.

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [x] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [x] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

(cherry picked from commit d172147)
… backport

### What problem does this PR solve?

Related PR: apache#59535

Problem Summary: The backport of apache#59535 disables orphan temporary table
auto-cleanup (TemporaryTableMgr) to stop the aliveSessionSet memory leak.
test_temp_table_with_restart asserts exactly that auto-cleanup behavior and
would always fail now. Skip it, mirroring upstream's practice in apache#64525.

### Release note

None

### Check List (For Author)

- Test: No need to test (test-only change, skipping a case)
- Behavior changed: No
- Does this need documentation: No

@xuchenhao xuchenhao left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for one mixed-version correctness regression that is outside the PR's accepted scope.

The PR description explicitly accepts disabling orphan temporary-table cleanup after all relevant FEs run the new behavior. It does not accept deleting a table owned by a still-live session during a rolling upgrade. There is no deployment/traffic constraint in the description, and the session-report RPC carries only cluster/token (no capability/version handshake), so an old-master/new-follower window is possible and unguarded.

Part 1.3 review conclusions:

  1. Goal / proof: The steady-state all-new-version path does stop aliveSessionSet / sessionReportTimeMap growth and matches upstream d172147f55e in the two production files. However, the added test only skips the orphan-cleanup case; no mixed-version test proves upgrade safety.
  2. Scope: The patch is small and focused, but the transition semantics are not contained to the described orphan-cleanup behavior.
  3. Concurrency: No new local lock/deadlock issue. The correctness failure is cross-FE daemon concurrency: an old master's 5s FESessionMgr polling and 30s TemporaryTableMgr cycle observe a new follower's empty report.
  4. Lifecycle: New follower sessions remain live, but their IDs are never registered/reported; the old master therefore ages out their temporary tables. This lifecycle mismatch is the blocker.
  5. Configuration: No new config. Existing defaults make the issue concrete: alive_session_update_interval_second=5, loss_conn_fe_temp_table_keep_second=60, and TemporaryTableMgr uses the 30s default daemon interval.
  6. Compatibility: No storage/RPC schema change, but rolling-upgrade behavioral compatibility is broken in the old-master/new-follower direction. New-master/old-follower is safe because cleanup is disabled on the master.
  7. Parallel paths: Master-local/all-new paths were checked; only the mixed direction above retains the destructive old cleanup while suppressing the new follower's liveness report.
  8. Conditions: Old cleanup's containsKey / timestamp checks have no FE-version or tracking-enabled guard.
  9. Tests: Existing normal-session tests remain, but the restart test is skipped and there is no mixed-version coverage.
  10. Observability: The old master logs the drop, but logging cannot prevent or recover the active temporary-table deletion.
  11. Transaction / persistence: The erroneous cleanup calls dropTableWithoutCheck, and that metadata deletion is then propagated; this is not a transient read-only inconsistency.
  12. Data writes: Active temporary-table data can be removed while its client session is still using it.
  13. FE/BE variables: None added; not applicable.
  14. Performance: The intended all-new steady-state allocation reduction is valid; no additional hot-path regression found.
  15. Other: Apart from the inline P1, I found no other P0–P3 issues.

Verification performed: reviewed the complete base-to-head diff, final merge result, relevant ConnectContext / forwarding / FESessionMgr / cleanup call chains, and the upstream merged commit; git diff --check passed. I did not run FE build/UT/regression locally because this isolated worktree lacks .worktree_initialized, the referenced setup hook, and thirdparty/installed/bin/protoc. The PR's FE compile claim is author-provided and was not independently reproduced; no CI checks were exposed for the current head.


public void registerSessionInfo(String sessionId) {
this.aliveSessionSet.add(sessionId);
// TODO: do nothing now until we fix memory link on Env#sessionReportTimeMap and Env#aliveSessionSet

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve live sessions across mixed-version upgrades

This no-op is safe only after the master also runs the new code. In an old-master/new-follower window, a client can connect to the upgraded follower and create a temporary table. The old master handles the forwarded CREATE and its old registerTempTableAndSession() -> refreshSession() records that real session ID. But this follower now returns an empty list from getAliveSessions() (the RPC has no version/capability flag), so the old master's 5s FESessionMgr polls never refresh the timestamp. Its still-active TemporaryTableMgr runs every 30s and, after the default 60s keep time, calls dropTableWithoutCheck() on a table whose client session is still alive. The PR description accepts retaining orphan tables, not deleting active ones, and specifies no rollout/traffic constraint that avoids this state. Please make disabling cleanup version-aware, preserve leak-free reporting for real client sessions, or otherwise guard the mixed-version path before this backport is safe.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at b05ef22f775dd0bb730e005a0c23723e6fd6adb7: this mixed-version issue is now fixed in code. getAllAliveSessionIds() derives the report from ConnectScheduler.getConnectionMap(), which combines the live MySQL and Arrow Flight SQL pools. Both protocols obtain their connection ID from the same scheduler, register before queries can run, and unregister on close; the stored ConnectContext retains the same session ID sent in forwarded requests. Therefore an old master now receives and refreshes the active session ID, while internal/proxy contexts remain unregistered and the original leaking aliveSessionSet bookkeeping stays disabled. Withdrawing this blocker; I am leaving the thread unresolved as requested and posting an approval on the new head.

…rolling upgrade

### What problem does this PR solve?

Related PR: apache#59535

Problem Summary: apache#59535 disabled the leaking session-tracking bookkeeping
(Env#aliveSessionSet / Env#sessionReportTimeMap) by commenting out
registerSessionInfo/refreshSession, so aliveSessionSet always stays empty.
During a rolling upgrade, an old-version master FE still queries follower
FEs for their alive session ids (getAllAliveSessionIds RPC) and drops
temporary tables whose owning session is not reported alive after
loss_conn_fe_temp_table_keep_second (default 60s). A new-version follower
would report an empty list, so the old master would wrongly delete temp
tables of sessions that are actually alive on the new follower.

Fix: derive getAllAliveSessionIds() from the live connection registry
(ExecuteEnv -> ConnectScheduler -> getConnectionMap, which covers both the
MySQL protocol pool and the Arrow Flight SQL pool) instead of the
always-empty aliveSessionSet. The registry always reflects real client
connections and needs no register/unregister bookkeeping, so alive-session
reporting stays accurate for an old-version master while the leak fix of
apache#59535 remains intact (registerSessionInfo/refreshSession/TemporaryTableMgr
stay commented out).

### Release note

None

### Check List (For Author)

- Test: Unit Test (EnvTest#testGetAllAliveSessionIds), ./build.sh --fe
- Behavior changed: No
- Does this need documentation: No

@xuchenhao xuchenhao left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM on b05ef22f775dd0bb730e005a0c23723e6fd6adb7. The previous mixed-version P1 is fixed by code rather than by a documentation-only assertion.

Scope note: the current top-level PR description still explicitly accepts only disabling orphan temporary-table cleanup; it does not make active-table deletion during rolling upgrade accepted or out-of-scope. The previous P1 therefore remained in scope. The new commit resolves it by reporting real live connections while leaving the leaking bookkeeping and new-master cleanup disabled.

Part 1.3 review conclusions:

  1. Goal / proof: The original all-new-version memory fix remains intact, and old-master/new-follower compatibility is restored. EnvTest#testGetAllAliveSessionIds covers registered session inclusion and removal; there is no full mixed-version cluster test.
  2. Scope: The follow-up is focused: one production method plus one targeted FE unit test.
  3. Concurrency: Both underlying connection maps are concurrent. MySQL and Arrow Flight share ConnectScheduler.nextConnectionId, so combining their maps does not overwrite live entries. Registration occurs before query processing and unregistration on close; a concurrent snapshot can at most transiently lag a connection transition and introduces no lock/deadlock issue.
  4. Lifecycle: Real client contexts live in the scheduler registry for exactly their connection/session lifetime. Forwarded proxy and internal contexts are excluded, preventing the original random-UUID leak.
  5. Configuration: No config is added or changed. Existing 5s reporting, 60s keep time, and 30s cleanup cadence are now safe in the mixed-version direction because the active session continues to be reported.
  6. Compatibility: No RPC/storage format changes are required. The existing getAliveSessions response now preserves its old semantic contract for an old master, so rolling upgrade is compatible in both old-master/new-follower and new-master/old-follower directions.
  7. Parallel paths: MySQL and Arrow Flight SQL pools are both included. Master-local, follower RPC, normal disconnect, forwarded proxy, and internal-context paths were checked.
  8. Conditions: No new special condition is introduced; the old master's existing checkAndRefreshSession timestamp condition now receives the correct session ID.
  9. Tests: The added unit test verifies the new registry-derived result and removal behavior. The skipped restart test still corresponds to the PR's explicitly accepted orphan-cleanup disablement. Mixed-version end-to-end coverage is absent but not blocking for this focused compatibility bridge.
  10. Observability: Existing FE session/temporary-table logs remain sufficient; no new failure state or metric is required.
  11. Transaction / persistence: No EditLog format or replay path changes. The fix prevents an old master from reaching the erroneous persistent drop path for a live session.
  12. Data writes: The new method is read-only over the connection registry and does not alter transaction or table-write semantics.
  13. FE/BE variables: None added; not applicable.
  14. Performance: Reporting is bounded by configured real connections instead of the unbounded leaked set. The combined-map/list snapshot is proportional to live clients and runs on the existing reporting path.
  15. Other: No new P0-P3 findings.

Verification performed personally: re-read the current full PR body and all commits/files/reviews/comments/checks, reviewed the old-to-new diff and the updated full base-to-merge diff, inspected final files plus MySQL/Arrow registration and cleanup call chains, and ran git diff --check successfully. I did not run FE build/UT/regression locally because this isolated worktree has no .worktree_initialized, no referenced setup hook, and no thirdparty/installed/bin/protoc. The new commit message reports EnvTest#testGetAllAliveSessionIds and ./build.sh --fe; those are author-provided claims, not independently reproduced here. The Clang Formatter and License Check failures occur at actions/checkout because the fork PR is blocked in pull_request_target, before project code runs; Check Large File passed.

@wenzhenghu wenzhenghu left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

当前 head 存在一个需要在合入稳定版前处理的 P1:通过 follower 创建临时表时,实际表名只登记在 master 的代理 ConnectContext;原始 follower ConnectContext 没有该记录。停用 TemporaryTableMgr 后,正常断开、连接超时或正常 kill 都可能永久遗留临时表,形成无界的元数据、tablet 和存储资源泄漏。详细调用链和修复建议见内联评论及补充复核评论。按要求未运行 FE UT,本结论基于静态调用链检查。

…n disconnect

### What problem does this PR solve?

Related PR: apache#59535

Problem Summary: When a client connects to a follower FE and creates a
temporary table, the CREATE statement is forwarded to master FE and executed
there with a proxy ConnectContext (isProxy=true). Env.registerTempTableAndSession()
therefore records the temp table's inner name only in the master-side proxy
session, and the original client session's dbToTempTableNamesMap on the
follower stays empty. When the follower session later disconnects, times out
or is killed, ConnectContext.deleteTempTable() iterates an empty map and drops
nothing. apache#59535 disabled the TemporaryTableMgr background cleanup that
previously collected such orphan temp tables, so temp tables created via a
follower leak permanently (unbounded metadata/tablet/storage leak).

Fix: after a statement is successfully forwarded to master FE
(StmtExecutor.forwardToMaster), if it is a CREATE TEMPORARY TABLE (nereids
CreateTableCommand or legacy CreateTableStmt), register the deterministic
inner name (sessionId + sign + tableName, same as master side since the
session id is forwarded) into the follower session's dbToTempTableNamesMap;
if it is a forwarded DROP TABLE, remove the entry symmetrically so a later
disconnect does not re-drop it. Also fix the previously dead non-master
branch of ConnectContext.deleteTempTable() to qualify the forwarded DROP
with the db name recorded in the map, instead of relying on the session's
current database at disconnect time.

### Release note

Fix the issue that temporary tables created on a follower FE are not cleaned
up after the session disconnects.

### Check List (For Author)

- Test: Unit Test (existing org.apache.doris.catalog.EnvTest passed);
    added docker regression test
    temp_table_p0/test_temp_table_follower_disconnect (NOT run locally,
    docker regression environment unavailable)
- Behavior changed: Yes - temporary tables created through a follower FE
    connection are now dropped when the session exits, as originally intended
- Does this need documentation: No
@Ryan19929

Copy link
Copy Markdown
Author

关于 follower 临时表清理 P1 的补充说明:上游现状

针对 @wenzhenghu 提出的 P1(follower 创建的临时表在正常断开后永久遗留),我们核实了上游 apache/doris 的 master 分支,确认该问题在上游同样存在且未被修复:

  1. TemporaryTableMgr.runAfterCatalogReady() 自 [fix](temp-table) not clean temp table temporary until fix mem leak apache/doris#59535(d172147f55e)起整段注释,此后 TemporaryTableMgr.java / FESessionMgr.java 再无任何提交;
  2. Env.registerTempTableAndSession() 在 master 上原样保留——临时表名仍只登记到执行时刻的 thread-local 上下文,转发场景下即 master 侧的一次性代理 ConnectContext(isProxy=true,deleteTempTable() 直接 return),follower 原始客户端上下文始终拿不到表名;
  3. addTempTableToDB 在 master 全库仍只有 registerTempTableAndSession 一个调用点,没有任何回写同步机制;
  4. 上游对临时表特性的态度是 "temp table feature is no longer maintained"(见 branch-4.0: [fix](test) fix branch-4.0 P0 regression failures (build #198126) apache/doris#64525 中 skip test_temp_table 的注释),该缺口处于事实性停维护状态。

即上游 master/4.0 的现状是:follower 创建的临时表在正常断开后同样永久遗留,且滚动升级期间 old-master/new-follower 窗口也存在误删活跃临时表的风险。

本次提交的 5d845095668(转发 CREATE 成功后把内部表名登记回 follower 原始 ConnectContext + DROP 对称移除 + 修复转发 DROP 缺 db 限定符)和此前的 b05ef22f775(getAllAliveSessionIds 改从实时连接注册表派生)使得本 PR 在这两个场景上超出上游现状:follower 直连会话的正常断开/超时/kill 均能直接清理临时表,混合版本升级双向安全。

验证情况(如实说明):./build.sh --fe 编译通过(含 checkstyle);EnvTest(含新增 testGetAllAliveSessionIds)通过;新增 docker 回归 test_temp_table_follower_disconnect.groovy(2 FE,follower 连接下分别验证正常关闭与 wait_timeout 超时后的临时表清理)因本地无 docker 回归环境未实际执行,建议合入前在 CI 跑通。

@wenzhenghu wenzhenghu left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

基于最新 head 5d84509 重新检查:上次 P1 的普通 CREATE TEMPORARY TABLE 路径已得到修复,但其他受支持的临时表创建方式和转发部分成功边界尚未覆盖;新增 timeout 回归也不能独立证明服务端超时路径。详细意见见内联评论。按此前约束,本轮仅做静态检查,未运行 FE UT。

@Ryan19929

Copy link
Copy Markdown
Author

关于临时表特性的上游态度与本 PR 范围的说明

我们评估了上游对临时表特性的定位,结论同步如下,作为本 PR 范围取舍的依据:

上游态度:实验特性、不再维护、不建议生产使用

  1. 官方文档(4.x Temporary Table)首页标注:"The temporary table is an experimental feature. It is not recommended for use in a production environment.";且 3.x 文档中该特性页面不存在(branch-3.1: [feature](schema) support temporary table like mysql #40680 apache/doris#52347 backport 进 3.1 时未伴随文档发布)。
  2. 维护状态:branch-4.0: [fix](test) fix branch-4.0 P0 regression failures (build #198126) apache/doris#64525(4.0)/ branch-4.1: [fix](test) skip test_temp_table (temp table feature no longer maintained, backport of #64525) apache/doris#64607(4.1)skip test_temp_table 的注释原话为 "temp table feature is no longer maintained";[fix](temp-table) not clean temp table temporary until fix mem leak apache/doris#59535 停用清理机制后留有 "until we fix mem leak" 的 TODO,但至今无后续修复。
  3. 无禁用开关:3.1 的 Config/SessionVariable 中不存在禁用临时表语法的配置项。

我们的业务决策与对本 PR 范围的影响

我们已在业务行为层面禁止使用临时表。在此前提下重新评估本 PR 四个提交的必要性:

  • e515d6f6b03([fix](temp-table) not clean temp table temporary until fix mem leak apache/doris#59535 cherry-pick):必需,与临时表是否使用无关。 aliveSessionSet 泄漏的触发源是每条转发语句和内部任务上下文(proxy/internal ConnectContext 注册),不依赖临时表语法;我们生产集群约 1450 万条积压即由此产生。这是本 PR 的核心修复。
  • 27044897b87(skip restart 用例):保留,CI 卫生。
  • b05ef22f775 / 5d845095668(加固提交):针对的是"临时表被实际使用"的场景。由于上游无硬开关、业务禁令属于软约束,我们保留这两个提交作为防御性加固,防止误用后产生无清理路径的临时表。

对未决 review 意见的处理口径

  • P1(CREATE TEMPORARY TABLE ... LIKE ... / legacy CTAS 经 follower 创建时仍不登记):结合上游"特性不再维护"的定性、我们的业务禁用前提、以及 master 同样存在该缺口的事实,本次选择作为已知限制明确披露、不做代码覆盖。如后续放开临时表使用,再补齐这些路径的登记。
  • 两条 P2(master 成功但 replay 超时导致漏登记;timeout 用例未独立证明超时路径):均已确认成立,修复方案明确(finally 中按 getStatusCode()==0 登记;用例中先断言原连接已失效),当前暂缓合入,将在后续提交中处理。

@wenzhenghu
wenzhenghu dismissed their stale review September 2, 2026 02:52

Review scope updated: temporary-table behavior is explicitly deferred to a separate PR. The prior blocking review is no longer applicable.

@wenzhenghu wenzhenghu left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

基于最新 head 5d8450956681fb089572f8d5c17d7c2545fe906d 重新审查,结论:APPROVE,未发现临时表功能之外的 P0–P3 问题。

本轮范围已按当前决策调整:临时表功能语义、创建路径覆盖、会话退出清理以及相关回归测试充分性不再作为本 PR 的阻塞依据,将在其他 PR 单独处置。

关键检查结论:

  1. 目标与实现:Env.registerSessionInfo() 不再向 aliveSessionSet 写入,直接阻断普通客户端、follower 转发代理和内部任务创建 ConnectContext 时随机 session UUID 的无界积累。该核心修改与上游 apache#59535 一致,且不依赖生产是否实际使用临时表语法。
  2. 资源与并发:移除写入后集合保持为空;既有 unregisterSessionInfo() 对空并发集合执行 remove 不产生正确性或并发风险。没有新增锁、线程或资源所有权关系。
  3. 兼容性:没有 EditLog、image、RPC/Thrift、FE-BE 协议、配置或持久化结构变化;滚动升级期间新旧 FE 的非临时表查询、连接和会话协议不受影响。
  4. 性能:核心效果是消除随 ConnectContext 构造次数无界增长的 session 集合及其周期性复制成本;未发现新的非临时表热路径开销。
  5. 错误处理与可观测性:本修改不改变客户端查询错误语义,也没有新增需要处理的异常边界。
  6. 测试与静态验证:精确 merge-base 6b84416279527cab64114d098e3fdeddf891c391 到当前 head 的最小 diff 为 7 个文件,git diff --check 通过。本轮未运行 FE UT。当前 Clang Formatter/License Check 失败发生在 fork PR 的 pull_request_target checkout 阶段,项目代码未被检出执行,因此不是代码检查失败。

历史上由临时表语义产生的我方阻塞 review 已撤销,相关评论已删除;本 review 取代此前结论。

@wenzhenghu
wenzhenghu merged commit 28387f7 into HYDCP:branch-3.1 Sep 2, 2026
3 of 6 checks passed
wenzhenghu pushed a commit that referenced this pull request Oct 8, 2026
…8689)

### What problem does this PR solve?

Related PR: apache#68687

Pin lance-c to merged upstream main commit
`cd63420bfbe27f6f0a1edcc873b9191af7d52852`, including the
distance-bounded search (#90), independent batch nearest search (#92),
and boolean scalar-segment predicates (#93) changes. Update the archive
checksum and refresh the existing Foyer patch against that base with an
unchanged implementation payload.

Foyer PR
[lance-format/lance-c#73](lance-format/lance-c#73)
is still open. Its patch must remain temporarily because Doris already
calls its cache APIs; it can be removed after upstream merge.

This PR contains the dependency pin/Foyer refresh and its build
integration: `build.sh`, `thirdparty/build-thirdparty.sh`,
`thirdparty/lance-install.sh`, and the installation regression harness.
The build checks a source/archive/patch fingerprint before reusing
Lance, invalidates it during publication, and rejects stale external
build output. Before removing an existing installation, it verifies
required rebuild inputs and rejects external definitions that differ
from the checkout. The third-party CI script job runs the installation
harness under the existing workflow path filters. FE predicate changes
and SQL fixtures/tests from apache#68687 are not included.

### Validation

- Verified the downloaded archive checksum and exact patch application.
- Third-party extraction, cached-source reuse/refresh, and invalid-patch
rejection tests passed on master.
- The updated dependency plus the unchanged Foyer patch passed 75 Rust
unit tests, 376 C API tests, and all 3 native C/C++/static OSS consumer
tests.
- The same dependency passed 27 FE-generated Substrait integration
scenarios as part of apache#68687 validation.
- The installation harness passed on master and branch-4.1 using
independent external source definitions: legacy/matching installs,
missing helper/vars/patch/downloader/builder, mismatched and
synchronized pin/patch updates, incomplete artifacts, stale builder
output, interrupted publication, and retry. Rejected preflight cases
verify that the entire installation and builder invocation count remain
unchanged. The missing-helper regression failed before the fix. Shell
syntax, workflow YAML, and CI path-filter checks passed.
- Full Doris compilation and BE UT for this revision are pending CI.

### Release note

Update the Lance C dependency to merged upstream search and
scalar-predicate improvements.

### Check List (For Author)

- Test: Dependency unit and integration tests described above.
- Behavior changed: Yes; incorporates upstream lance-c fixes and
additive APIs.
- Does this need documentation: No new Doris user interface in this
dependency-only update.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants