Repository navigation
[fix](temp-table) not clean temp table temporary until fix mem leak (#59535) - #93
Conversation
…pache#59535) ### What problem does this PR solve? Related PR: apache#40680 Problem Summary: This pull request temporarily disables session tracking and automatic cleanup for temporary tables due to a memory leak issue involving `Env#sessionReportTimeMap` and `Env#aliveSessionSet`. The affected logic is commented out with TODO notes, and related imports are cleaned up. Session management changes: * Disabled the code that adds sessions to `aliveSessionSet` in `Env.registerSessionInfo`, preventing new session tracking until the memory leak is fixed. * Disabled the code that updates session report times in `Env.refreshSession`, pausing session activity tracking. Temporary table cleanup changes: * Commented out the logic in `TemporaryTableMgr.runAfterCatalogReady` that deletes temporary tables when their creating session is gone, halting automatic cleanup. ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [x] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [x] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> (cherry picked from commit d172147)
… backport ### What problem does this PR solve? Related PR: apache#59535 Problem Summary: The backport of apache#59535 disables orphan temporary table auto-cleanup (TemporaryTableMgr) to stop the aliveSessionSet memory leak. test_temp_table_with_restart asserts exactly that auto-cleanup behavior and would always fail now. Skip it, mirroring upstream's practice in apache#64525. ### Release note None ### Check List (For Author) - Test: No need to test (test-only change, skipping a case) - Behavior changed: No - Does this need documentation: No
xuchenhao
left a comment
There was a problem hiding this comment.
Requesting changes for one mixed-version correctness regression that is outside the PR's accepted scope.
The PR description explicitly accepts disabling orphan temporary-table cleanup after all relevant FEs run the new behavior. It does not accept deleting a table owned by a still-live session during a rolling upgrade. There is no deployment/traffic constraint in the description, and the session-report RPC carries only cluster/token (no capability/version handshake), so an old-master/new-follower window is possible and unguarded.
Part 1.3 review conclusions:
- Goal / proof: The steady-state all-new-version path does stop
aliveSessionSet/sessionReportTimeMapgrowth and matches upstreamd172147f55ein the two production files. However, the added test only skips the orphan-cleanup case; no mixed-version test proves upgrade safety. - Scope: The patch is small and focused, but the transition semantics are not contained to the described orphan-cleanup behavior.
- Concurrency: No new local lock/deadlock issue. The correctness failure is cross-FE daemon concurrency: an old master's 5s
FESessionMgrpolling and 30sTemporaryTableMgrcycle observe a new follower's empty report. - Lifecycle: New follower sessions remain live, but their IDs are never registered/reported; the old master therefore ages out their temporary tables. This lifecycle mismatch is the blocker.
- Configuration: No new config. Existing defaults make the issue concrete:
alive_session_update_interval_second=5,loss_conn_fe_temp_table_keep_second=60, andTemporaryTableMgruses the 30s default daemon interval. - Compatibility: No storage/RPC schema change, but rolling-upgrade behavioral compatibility is broken in the old-master/new-follower direction. New-master/old-follower is safe because cleanup is disabled on the master.
- Parallel paths: Master-local/all-new paths were checked; only the mixed direction above retains the destructive old cleanup while suppressing the new follower's liveness report.
- Conditions: Old cleanup's
containsKey/ timestamp checks have no FE-version or tracking-enabled guard. - Tests: Existing normal-session tests remain, but the restart test is skipped and there is no mixed-version coverage.
- Observability: The old master logs the drop, but logging cannot prevent or recover the active temporary-table deletion.
- Transaction / persistence: The erroneous cleanup calls
dropTableWithoutCheck, and that metadata deletion is then propagated; this is not a transient read-only inconsistency. - Data writes: Active temporary-table data can be removed while its client session is still using it.
- FE/BE variables: None added; not applicable.
- Performance: The intended all-new steady-state allocation reduction is valid; no additional hot-path regression found.
- Other: Apart from the inline P1, I found no other P0–P3 issues.
Verification performed: reviewed the complete base-to-head diff, final merge result, relevant ConnectContext / forwarding / FESessionMgr / cleanup call chains, and the upstream merged commit; git diff --check passed. I did not run FE build/UT/regression locally because this isolated worktree lacks .worktree_initialized, the referenced setup hook, and thirdparty/installed/bin/protoc. The PR's FE compile claim is author-provided and was not independently reproduced; no CI checks were exposed for the current head.
|
|
||
| public void registerSessionInfo(String sessionId) { | ||
| this.aliveSessionSet.add(sessionId); | ||
| // TODO: do nothing now until we fix memory link on Env#sessionReportTimeMap and Env#aliveSessionSet |
There was a problem hiding this comment.
[P1] Preserve live sessions across mixed-version upgrades
This no-op is safe only after the master also runs the new code. In an old-master/new-follower window, a client can connect to the upgraded follower and create a temporary table. The old master handles the forwarded CREATE and its old registerTempTableAndSession() -> refreshSession() records that real session ID. But this follower now returns an empty list from getAliveSessions() (the RPC has no version/capability flag), so the old master's 5s FESessionMgr polls never refresh the timestamp. Its still-active TemporaryTableMgr runs every 30s and, after the default 60s keep time, calls dropTableWithoutCheck() on a table whose client session is still alive. The PR description accepts retaining orphan tables, not deleting active ones, and specifies no rollout/traffic constraint that avoids this state. Please make disabling cleanup version-aware, preserve leak-free reporting for real client sessions, or otherwise guard the mixed-version path before this backport is safe.
There was a problem hiding this comment.
Re-reviewed at b05ef22f775dd0bb730e005a0c23723e6fd6adb7: this mixed-version issue is now fixed in code. getAllAliveSessionIds() derives the report from ConnectScheduler.getConnectionMap(), which combines the live MySQL and Arrow Flight SQL pools. Both protocols obtain their connection ID from the same scheduler, register before queries can run, and unregister on close; the stored ConnectContext retains the same session ID sent in forwarded requests. Therefore an old master now receives and refreshes the active session ID, while internal/proxy contexts remain unregistered and the original leaking aliveSessionSet bookkeeping stays disabled. Withdrawing this blocker; I am leaving the thread unresolved as requested and posting an approval on the new head.
…rolling upgrade ### What problem does this PR solve? Related PR: apache#59535 Problem Summary: apache#59535 disabled the leaking session-tracking bookkeeping (Env#aliveSessionSet / Env#sessionReportTimeMap) by commenting out registerSessionInfo/refreshSession, so aliveSessionSet always stays empty. During a rolling upgrade, an old-version master FE still queries follower FEs for their alive session ids (getAllAliveSessionIds RPC) and drops temporary tables whose owning session is not reported alive after loss_conn_fe_temp_table_keep_second (default 60s). A new-version follower would report an empty list, so the old master would wrongly delete temp tables of sessions that are actually alive on the new follower. Fix: derive getAllAliveSessionIds() from the live connection registry (ExecuteEnv -> ConnectScheduler -> getConnectionMap, which covers both the MySQL protocol pool and the Arrow Flight SQL pool) instead of the always-empty aliveSessionSet. The registry always reflects real client connections and needs no register/unregister bookkeeping, so alive-session reporting stays accurate for an old-version master while the leak fix of apache#59535 remains intact (registerSessionInfo/refreshSession/TemporaryTableMgr stay commented out). ### Release note None ### Check List (For Author) - Test: Unit Test (EnvTest#testGetAllAliveSessionIds), ./build.sh --fe - Behavior changed: No - Does this need documentation: No
xuchenhao
left a comment
There was a problem hiding this comment.
LGTM on b05ef22f775dd0bb730e005a0c23723e6fd6adb7. The previous mixed-version P1 is fixed by code rather than by a documentation-only assertion.
Scope note: the current top-level PR description still explicitly accepts only disabling orphan temporary-table cleanup; it does not make active-table deletion during rolling upgrade accepted or out-of-scope. The previous P1 therefore remained in scope. The new commit resolves it by reporting real live connections while leaving the leaking bookkeeping and new-master cleanup disabled.
Part 1.3 review conclusions:
- Goal / proof: The original all-new-version memory fix remains intact, and old-master/new-follower compatibility is restored.
EnvTest#testGetAllAliveSessionIdscovers registered session inclusion and removal; there is no full mixed-version cluster test. - Scope: The follow-up is focused: one production method plus one targeted FE unit test.
- Concurrency: Both underlying connection maps are concurrent. MySQL and Arrow Flight share
ConnectScheduler.nextConnectionId, so combining their maps does not overwrite live entries. Registration occurs before query processing and unregistration on close; a concurrent snapshot can at most transiently lag a connection transition and introduces no lock/deadlock issue. - Lifecycle: Real client contexts live in the scheduler registry for exactly their connection/session lifetime. Forwarded proxy and internal contexts are excluded, preventing the original random-UUID leak.
- Configuration: No config is added or changed. Existing 5s reporting, 60s keep time, and 30s cleanup cadence are now safe in the mixed-version direction because the active session continues to be reported.
- Compatibility: No RPC/storage format changes are required. The existing
getAliveSessionsresponse now preserves its old semantic contract for an old master, so rolling upgrade is compatible in both old-master/new-follower and new-master/old-follower directions. - Parallel paths: MySQL and Arrow Flight SQL pools are both included. Master-local, follower RPC, normal disconnect, forwarded proxy, and internal-context paths were checked.
- Conditions: No new special condition is introduced; the old master's existing
checkAndRefreshSessiontimestamp condition now receives the correct session ID. - Tests: The added unit test verifies the new registry-derived result and removal behavior. The skipped restart test still corresponds to the PR's explicitly accepted orphan-cleanup disablement. Mixed-version end-to-end coverage is absent but not blocking for this focused compatibility bridge.
- Observability: Existing FE session/temporary-table logs remain sufficient; no new failure state or metric is required.
- Transaction / persistence: No EditLog format or replay path changes. The fix prevents an old master from reaching the erroneous persistent drop path for a live session.
- Data writes: The new method is read-only over the connection registry and does not alter transaction or table-write semantics.
- FE/BE variables: None added; not applicable.
- Performance: Reporting is bounded by configured real connections instead of the unbounded leaked set. The combined-map/list snapshot is proportional to live clients and runs on the existing reporting path.
- Other: No new P0-P3 findings.
Verification performed personally: re-read the current full PR body and all commits/files/reviews/comments/checks, reviewed the old-to-new diff and the updated full base-to-merge diff, inspected final files plus MySQL/Arrow registration and cleanup call chains, and ran git diff --check successfully. I did not run FE build/UT/regression locally because this isolated worktree has no .worktree_initialized, no referenced setup hook, and no thirdparty/installed/bin/protoc. The new commit message reports EnvTest#testGetAllAliveSessionIds and ./build.sh --fe; those are author-provided claims, not independently reproduced here. The Clang Formatter and License Check failures occur at actions/checkout because the fork PR is blocked in pull_request_target, before project code runs; Check Large File passed.
wenzhenghu
left a comment
There was a problem hiding this comment.
当前 head 存在一个需要在合入稳定版前处理的 P1:通过 follower 创建临时表时,实际表名只登记在 master 的代理 ConnectContext;原始 follower ConnectContext 没有该记录。停用 TemporaryTableMgr 后,正常断开、连接超时或正常 kill 都可能永久遗留临时表,形成无界的元数据、tablet 和存储资源泄漏。详细调用链和修复建议见内联评论及补充复核评论。按要求未运行 FE UT,本结论基于静态调用链检查。
…n disconnect ### What problem does this PR solve? Related PR: apache#59535 Problem Summary: When a client connects to a follower FE and creates a temporary table, the CREATE statement is forwarded to master FE and executed there with a proxy ConnectContext (isProxy=true). Env.registerTempTableAndSession() therefore records the temp table's inner name only in the master-side proxy session, and the original client session's dbToTempTableNamesMap on the follower stays empty. When the follower session later disconnects, times out or is killed, ConnectContext.deleteTempTable() iterates an empty map and drops nothing. apache#59535 disabled the TemporaryTableMgr background cleanup that previously collected such orphan temp tables, so temp tables created via a follower leak permanently (unbounded metadata/tablet/storage leak). Fix: after a statement is successfully forwarded to master FE (StmtExecutor.forwardToMaster), if it is a CREATE TEMPORARY TABLE (nereids CreateTableCommand or legacy CreateTableStmt), register the deterministic inner name (sessionId + sign + tableName, same as master side since the session id is forwarded) into the follower session's dbToTempTableNamesMap; if it is a forwarded DROP TABLE, remove the entry symmetrically so a later disconnect does not re-drop it. Also fix the previously dead non-master branch of ConnectContext.deleteTempTable() to qualify the forwarded DROP with the db name recorded in the map, instead of relying on the session's current database at disconnect time. ### Release note Fix the issue that temporary tables created on a follower FE are not cleaned up after the session disconnects. ### Check List (For Author) - Test: Unit Test (existing org.apache.doris.catalog.EnvTest passed); added docker regression test temp_table_p0/test_temp_table_follower_disconnect (NOT run locally, docker regression environment unavailable) - Behavior changed: Yes - temporary tables created through a follower FE connection are now dropped when the session exits, as originally intended - Does this need documentation: No
关于 follower 临时表清理 P1 的补充说明:上游现状针对 @wenzhenghu 提出的 P1(follower 创建的临时表在正常断开后永久遗留),我们核实了上游 apache/doris 的 master 分支,确认该问题在上游同样存在且未被修复:
即上游 master/4.0 的现状是:follower 创建的临时表在正常断开后同样永久遗留,且滚动升级期间 old-master/new-follower 窗口也存在误删活跃临时表的风险。 本次提交的 验证情况(如实说明): |
wenzhenghu
left a comment
There was a problem hiding this comment.
基于最新 head 5d84509 重新检查:上次 P1 的普通 CREATE TEMPORARY TABLE 路径已得到修复,但其他受支持的临时表创建方式和转发部分成功边界尚未覆盖;新增 timeout 回归也不能独立证明服务端超时路径。详细意见见内联评论。按此前约束,本轮仅做静态检查,未运行 FE UT。
关于临时表特性的上游态度与本 PR 范围的说明我们评估了上游对临时表特性的定位,结论同步如下,作为本 PR 范围取舍的依据: 上游态度:实验特性、不再维护、不建议生产使用
我们的业务决策与对本 PR 范围的影响我们已在业务行为层面禁止使用临时表。在此前提下重新评估本 PR 四个提交的必要性:
对未决 review 意见的处理口径
|
Review scope updated: temporary-table behavior is explicitly deferred to a separate PR. The prior blocking review is no longer applicable.
wenzhenghu
left a comment
There was a problem hiding this comment.
基于最新 head 5d8450956681fb089572f8d5c17d7c2545fe906d 重新审查,结论:APPROVE,未发现临时表功能之外的 P0–P3 问题。
本轮范围已按当前决策调整:临时表功能语义、创建路径覆盖、会话退出清理以及相关回归测试充分性不再作为本 PR 的阻塞依据,将在其他 PR 单独处置。
关键检查结论:
- 目标与实现:
Env.registerSessionInfo()不再向aliveSessionSet写入,直接阻断普通客户端、follower 转发代理和内部任务创建ConnectContext时随机 session UUID 的无界积累。该核心修改与上游 apache#59535 一致,且不依赖生产是否实际使用临时表语法。 - 资源与并发:移除写入后集合保持为空;既有
unregisterSessionInfo()对空并发集合执行 remove 不产生正确性或并发风险。没有新增锁、线程或资源所有权关系。 - 兼容性:没有 EditLog、image、RPC/Thrift、FE-BE 协议、配置或持久化结构变化;滚动升级期间新旧 FE 的非临时表查询、连接和会话协议不受影响。
- 性能:核心效果是消除随
ConnectContext构造次数无界增长的 session 集合及其周期性复制成本;未发现新的非临时表热路径开销。 - 错误处理与可观测性:本修改不改变客户端查询错误语义,也没有新增需要处理的异常边界。
- 测试与静态验证:精确 merge-base
6b84416279527cab64114d098e3fdeddf891c391到当前 head 的最小 diff 为 7 个文件,git diff --check通过。本轮未运行 FE UT。当前 Clang Formatter/License Check 失败发生在 fork PR 的pull_request_targetcheckout 阶段,项目代码未被检出执行,因此不是代码检查失败。
历史上由临时表语义产生的我方阻塞 review 已撤销,相关评论已删除;本 review 取代此前结论。
…8689) ### What problem does this PR solve? Related PR: apache#68687 Pin lance-c to merged upstream main commit `cd63420bfbe27f6f0a1edcc873b9191af7d52852`, including the distance-bounded search (#90), independent batch nearest search (#92), and boolean scalar-segment predicates (#93) changes. Update the archive checksum and refresh the existing Foyer patch against that base with an unchanged implementation payload. Foyer PR [lance-format/lance-c#73](lance-format/lance-c#73) is still open. Its patch must remain temporarily because Doris already calls its cache APIs; it can be removed after upstream merge. This PR contains the dependency pin/Foyer refresh and its build integration: `build.sh`, `thirdparty/build-thirdparty.sh`, `thirdparty/lance-install.sh`, and the installation regression harness. The build checks a source/archive/patch fingerprint before reusing Lance, invalidates it during publication, and rejects stale external build output. Before removing an existing installation, it verifies required rebuild inputs and rejects external definitions that differ from the checkout. The third-party CI script job runs the installation harness under the existing workflow path filters. FE predicate changes and SQL fixtures/tests from apache#68687 are not included. ### Validation - Verified the downloaded archive checksum and exact patch application. - Third-party extraction, cached-source reuse/refresh, and invalid-patch rejection tests passed on master. - The updated dependency plus the unchanged Foyer patch passed 75 Rust unit tests, 376 C API tests, and all 3 native C/C++/static OSS consumer tests. - The same dependency passed 27 FE-generated Substrait integration scenarios as part of apache#68687 validation. - The installation harness passed on master and branch-4.1 using independent external source definitions: legacy/matching installs, missing helper/vars/patch/downloader/builder, mismatched and synchronized pin/patch updates, incomplete artifacts, stale builder output, interrupted publication, and retry. Rejected preflight cases verify that the entire installation and builder invocation count remain unchanged. The missing-helper regression failed before the fix. Shell syntax, workflow YAML, and CI path-filter checks passed. - Full Doris compilation and BE UT for this revision are pending CI. ### Release note Update the Lance C dependency to merged upstream search and scalar-predicate improvements. ### Check List (For Author) - Test: Dependency unit and integration tests described above. - Behavior changed: Yes; incorporates upstream lance-c fixes and additive APIs. - Does this need documentation: No new Doris user interface in this dependency-only update.
What problem does this PR solve?
Related PR: apache#59535 (cherry-picked from master commit d172147)
Problem Summary:
branch-3.1 上
FESessionMgr的 session 跟踪机制存在内存泄漏:Env.registerSessionInfo()生效,且注册面远超真实客户端连接——Master 处理每条 follower 转发的语句时(FrontendServiceImpl.forward)都会经ConnectContext三参构造器用一个随机 UUID 注册进aliveSessionSet后再覆盖 sessionId,导致每条转发语句永久泄漏一个 UUID;数十处内部任务代码new ConnectContext()同样只注册不注销。实测集群aliveSessionSet已积压约 1450 万条目。getAliveSessions),反序列化出数百万元素的Object[];随集合膨胀,单轮分配从 389MB 涨到 843MB(JFR P99 达 7.3GB),巨型数组晋升老年代后不回收,Master FE 老年代被持续撑大,最终有 OOM 风险。sessionReportTimeMap时间戳被不断刷新,孤儿临时表永不超时,TemporaryTableMgr的自动回收已实质失效。修复方式(与上游 master/4.0 一致):
Env.registerSessionInfo()的aliveSessionSet.add(sessionId);Env.refreshSession()的sessionReportTimeMap.put(...);TemporaryTableMgr.runAfterCatalogReady()的孤儿临时表清理逻辑(必须同步停用:tracking 关闭后 map 恒空,保留清理会误删所有临时表)。另追加一个 test 提交:skip
test_temp_table_with_restart(该 case 断言的正是被停用的孤儿清理能力,参照上游 apache#64525 的做法)。test_temp_table、test_temp_table_with_conn_timeout不受影响,保持运行。Release note
临时表的孤儿自动清理(session 所在 FE 异常退出后的临时表回收)被停用,与上游 apache#59535 一致;正常断开连接时的临时表删除不受影响。
Check List (For Author)
./build.sh --fe)test_temp_table_with_restartskipped (asserts the disabled behavior)aliveSessionSetstays empty and Old Gen baseline stabilizes