Skip to content

Commit 7398c4f

Browse files
committed
[refactor](fe) Avoid unnecessary ConnectContext creation
### Background Synchronous helpers can use their caller's execution context, while parameter-only helpers do not need a session. Independent SQL and background execution still need explicit session ownership. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Refactor Problem Summary: Several helpers created a ConnectContext only to obtain defaults, copy a request or wrap an already owned load task. Remove eight redundant construction sites, reduce view metadata to one context per request and keep statement-only isolation local. ### Details - **Change map:** 18 production Java files, +168/-232; 11 test files, +465/-71. No generated files or standalone documentation. - **Moves vs. logic changes:** Pass the current executor and compute group directly; pass the task context through the load coordinator chain. Remove redundant cloud/HTTP contexts and duplicate Flight/MV TLS management. Keep private contexts for independent identity, settings and SQL execution. - **Behavior change:** Group Commit block rules and audit use the current SQL, user and planner. HTTP Group Commit uses request settings; auto-start without a context or explicit cluster returns no selection. View collection closes statement resources per view without replacing the borrowed statement. - **Compatibility:** Internal Java call sites migrate together. No protocol, journal, stored format or configuration changes. - **Performance impact:** Direct construction sites fall from 46 to 38. A metadata request creates at most one context only when a view needs parsing. No new lifecycle state or executor. - **Rollback:** Revert this commit; no metadata migration. ### Testing Combined final results: 316 tests passed across 28 FE classes. The initial run passed 27 classes and found two exception-format assertion errors in LoadActionTest. After correcting those test assertions, the repository run-fe-ut.sh script passed all 26 LoadActionTest cases and completed with BUILD SUCCESS. FE Checkstyle and staged diff checks passed. Cluster regression tests were not run. ### Notes for Reviewers Private execution/identity contexts remain owned by their scope. Synchronous borrowing does not transfer cleanup responsibility. The broader audit found pre-existing MysqlLoad reader, InsertTask cancellation and queryId lookup races; these are documented in the draft PR and not modified here. ### Release note Group Commit planning uses the active statement's user and SQL for block-rule checks and its planner for audit information. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: Yes; ConnectContext's English class comment explains when to create or borrow a context. - Pre-submission status: Not ready: Draft, awaiting human review; broader audit findings are documented and cluster regression is unrun. ### AI Disclosure - Tool: OpenAI Codex (GPT-6). - Generated: Implementation, tests, class comments and PR description. - Human review: Pending.
1 parent 60cecaa commit 7398c4f

29 files changed

Lines changed: 633 additions & 303 deletions

‎fe/fe-core/src/main/java/org/apache/doris/arrowflight/FlightSqlConnectProcessor.java‎

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -54,17 +54,16 @@
5454
import java.util.concurrent.TimeoutException;
5555

5656
/**
57-
* Process one flgiht sql connection.
58-
* <p>
59-
* Must use try-with-resources.
57+
* Processes statements under the enclosing Flight command's execution ownership and thread binding.
58+
* Use try-with-resources to release statement resources; the protocol entry retains the session and
59+
* restores the caller's thread binding after the complete command exits.
6060
*/
6161
public class FlightSqlConnectProcessor extends ConnectProcessor implements AutoCloseable {
6262
private static final Logger LOG = LogManager.getLogger(FlightSqlConnectProcessor.class);
6363
private Schema arrowSchema;
6464

6565
public FlightSqlConnectProcessor(ConnectContext context) {
6666
super(context);
67-
context.setThreadLocalInfo();
6867
}
6968

7069
public Schema getArrowSchema() {
@@ -217,6 +216,5 @@ public void close() throws Exception {
217216
executor.finalizeQuery();
218217
}
219218
ctx.clear();
220-
ConnectContext.remove();
221219
}
222220
}

‎fe/fe-core/src/main/java/org/apache/doris/catalog/EnvFactory.java‎

Lines changed: 20 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -151,45 +151,34 @@ public Coordinator createCoordinator(ConnectContext context, Planner planner,
151151
return new Coordinator(context, (NereidsPlanner) planner, statsErrorEstimator, jobId);
152152
}
153153

154-
// Used for broker load task/export task/update coordinator
155-
public Coordinator createCoordinator(Long jobId, TUniqueId queryId, DescriptorTable descTable,
154+
// The broker load task owns the context until its coordinator finishes.
155+
public Coordinator createCoordinator(ConnectContext connectContext, Long jobId,
156+
TUniqueId queryId, DescriptorTable descTable,
156157
List<PlanFragment> fragments, List<ScanNode> scanNodes,
157158
String timezone, boolean loadZeroTolerance, boolean enableProfile) {
158159
if (queryId == null) {
159160
UUID taskId = UUID.randomUUID();
160161
queryId = new TUniqueId(taskId.getMostSignificantBits(), taskId.getLeastSignificantBits());
161162
}
162-
ConnectContext connectContext = ConnectContext.get();
163-
boolean ownsContext = connectContext == null;
164-
if (ownsContext) {
165-
// Keep this fallback unbound: the load coordinator captures the caller's thread-local context.
166-
connectContext = new ConnectContext();
163+
if (connectContext.getLoadId() == null) {
164+
connectContext.setLoadId(queryId);
167165
}
168-
try {
169-
if (connectContext.getLoadId() == null) {
170-
connectContext.setLoadId(queryId);
171-
}
172-
if (connectContext.getEnv() == null) {
173-
connectContext.setEnv(Env.getCurrentEnv());
174-
}
175-
StatementContext statementContext = connectContext.getStatementContext();
176-
if (statementContext == null) {
177-
statementContext = new StatementContext(connectContext, new OriginStatement("", 0));
178-
connectContext.setStatementContext(statementContext);
179-
}
180-
DistributePlanner distributePlanner = new DistributePlanner(
181-
statementContext, fragments, false, true, true);
182-
FragmentIdMapping<DistributedPlan> distributedPlans = distributePlanner.plan();
183-
184-
return new Coordinator(
185-
jobId, queryId, descTable, fragments, distributedPlans.valueList(),
186-
scanNodes, timezone, loadZeroTolerance, enableProfile
187-
);
188-
} finally {
189-
if (ownsContext) {
190-
connectContext.requestClose(ConnectContext.CloseMode.AFTER_RESPONSE);
191-
}
166+
if (connectContext.getEnv() == null) {
167+
connectContext.setEnv(Env.getCurrentEnv());
192168
}
169+
StatementContext statementContext = connectContext.getStatementContext();
170+
if (statementContext == null) {
171+
statementContext = new StatementContext(connectContext, new OriginStatement("", 0));
172+
connectContext.setStatementContext(statementContext);
173+
}
174+
DistributePlanner distributePlanner = new DistributePlanner(
175+
statementContext, fragments, false, true, true);
176+
FragmentIdMapping<DistributedPlan> distributedPlans = distributePlanner.plan();
177+
178+
return new Coordinator(
179+
connectContext, jobId, queryId, descTable, fragments, distributedPlans.valueList(),
180+
scanNodes, timezone, loadZeroTolerance, enableProfile
181+
);
193182
}
194183

195184
public GroupCommitPlanner createGroupCommitPlanner(Database db, OlapTable table, List<String> targetColumnNames,

‎fe/fe-core/src/main/java/org/apache/doris/cloud/CloudWarmUpJob.java‎

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -32,8 +32,6 @@
3232
import org.apache.doris.common.util.TimeUtils;
3333
import org.apache.doris.metric.MetricRepo;
3434
import org.apache.doris.persist.gson.GsonUtils;
35-
import org.apache.doris.qe.AutoCloseConnectContext;
36-
import org.apache.doris.qe.ConnectContext;
3735
import org.apache.doris.system.Backend;
3836
import org.apache.doris.thrift.BackendService.Client;
3937
import org.apache.doris.thrift.TDownloadType;
@@ -731,11 +729,7 @@ public synchronized void run() {
731729
cancel("Timeout", false);
732730
return;
733731
}
734-
try (AutoCloseConnectContext scope = Config.isCloudMode()
735-
? new AutoCloseConnectContext(new ConnectContext()) : null) {
736-
if (scope != null) {
737-
scope.connectContext.setCloudCluster(dstClusterName);
738-
}
732+
try {
739733
switch (jobState) {
740734
case PENDING:
741735
runPendingJob();

‎fe/fe-core/src/main/java/org/apache/doris/cloud/load/CloudBrokerLoadJob.java‎

Lines changed: 14 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@
4545
import org.apache.doris.qe.Coordinator;
4646
import org.apache.doris.qe.OriginStatement;
4747
import org.apache.doris.qe.QeProcessorImpl;
48+
import org.apache.doris.qe.SessionVariable;
4849
import org.apache.doris.qe.StmtExecutor;
4950
import org.apache.doris.resource.computegroup.ComputeGroupMgr;
5051
import org.apache.doris.system.Backend;
@@ -186,24 +187,19 @@ protected void executeFinish() {
186187

187188
@Override
188189
protected void afterLoadingTaskCommitTransaction(List<Table> tableList) {
189-
try (AutoCloseConnectContext scope = ConnectContext.get() == null
190-
? new AutoCloseConnectContext(new ConnectContext()) : null) {
191-
ConnectContext ctx = ConnectContext.get();
192-
193-
if (ctx.getSessionVariable().enableMultiClusterSyncLoad()) {
194-
// get the backends of each cluster expect the load cluster
195-
CloudSystemInfoService infoService = (CloudSystemInfoService) Env.getCurrentSystemInfo();
196-
List<List<Backend>> backendsList = infoService.getCloudClusterIds()
197-
.stream()
198-
.filter(id -> !id.equals(cloudClusterId))
199-
.map(id -> infoService.getBackendsByClusterId(id))
200-
.collect(Collectors.toList());
201-
// for each all load table, get its tablets
202-
tableList.forEach(table -> {
203-
List<Long> allTabletIds = ((OlapTable) table).getAllTabletIds();
204-
StmtExecutor.syncLoadForTablets(backendsList, allTabletIds);
205-
});
206-
}
190+
if (SessionVariable.cloudEnableMultiClusterSyncLoad) {
191+
// get the backends of each cluster except the load cluster
192+
CloudSystemInfoService infoService = (CloudSystemInfoService) Env.getCurrentSystemInfo();
193+
List<List<Backend>> backendsList = infoService.getCloudClusterIds()
194+
.stream()
195+
.filter(id -> !id.equals(cloudClusterId))
196+
.map(id -> infoService.getBackendsByClusterId(id))
197+
.collect(Collectors.toList());
198+
// for each load table, get its tablets
199+
tableList.forEach(table -> {
200+
List<Long> allTabletIds = ((OlapTable) table).getAllTabletIds();
201+
StmtExecutor.syncLoadForTablets(backendsList, allTabletIds);
202+
});
207203
}
208204
}
209205

‎fe/fe-core/src/main/java/org/apache/doris/cloud/system/CloudSystemInfoService.java‎

Lines changed: 18 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1690,33 +1690,27 @@ public String getClusterNameAutoStart(final String clusterName) {
16901690
return clusterName;
16911691
}
16921692

1693-
ConnectContext previousContext = ConnectContext.get();
1694-
ConnectContext context = previousContext;
1693+
ConnectContext context = ConnectContext.get();
16951694
if (context == null) {
1696-
LOG.warn("auto start cant get context so new it");
1697-
context = new ConnectContext();
1695+
// Policy selection needs the caller's identity. Background callers must name the cluster.
1696+
LOG.warn("cannot choose a compute group without a context or an explicit cluster name");
1697+
return null;
16981698
}
1699-
try {
1700-
ConnectContext.CloudClusterResult cloudClusterTypeAndName = context.getCloudClusterByPolicy();
1701-
if (cloudClusterTypeAndName == null) {
1702-
LOG.warn("get cluster from ctx err");
1703-
return null;
1704-
}
1705-
if (cloudClusterTypeAndName.comment
1706-
== ConnectContext.CloudClusterResult.Comment.DEFAULT_CLUSTER_SET_BUT_NOT_EXIST) {
1707-
LOG.warn("get default cluster from ctx err");
1708-
return null;
1709-
}
1710-
1711-
Preconditions.checkState(!Strings.isNullOrEmpty(cloudClusterTypeAndName.clusterName),
1712-
"get cluster name empty");
1713-
LOG.info("get cluster to resume {}", cloudClusterTypeAndName);
1714-
return cloudClusterTypeAndName.clusterName;
1715-
} finally {
1716-
if (previousContext == null) {
1717-
context.requestClose(ConnectContext.CloseMode.AFTER_RESPONSE);
1718-
}
1699+
ConnectContext.CloudClusterResult cloudClusterTypeAndName = context.getCloudClusterByPolicy();
1700+
if (cloudClusterTypeAndName == null) {
1701+
LOG.warn("get cluster from ctx err");
1702+
return null;
17191703
}
1704+
if (cloudClusterTypeAndName.comment
1705+
== ConnectContext.CloudClusterResult.Comment.DEFAULT_CLUSTER_SET_BUT_NOT_EXIST) {
1706+
LOG.warn("get default cluster from ctx err");
1707+
return null;
1708+
}
1709+
1710+
Preconditions.checkState(!Strings.isNullOrEmpty(cloudClusterTypeAndName.clusterName),
1711+
"get cluster name empty");
1712+
LOG.info("get cluster to resume {}", cloudClusterTypeAndName);
1713+
return cloudClusterTypeAndName.clusterName;
17201714
}
17211715

17221716
public String waitForAutoStart(String clusterName) throws DdlException {

‎fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FederationBackendPolicy.java‎

Lines changed: 29 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020

2121
package org.apache.doris.datasource.scan;
2222

23+
import org.apache.doris.catalog.Env;
2324
import org.apache.doris.common.AnalysisException;
2425
import org.apache.doris.common.Config;
2526
import org.apache.doris.common.IndexedPriorityQueue;
@@ -29,6 +30,7 @@
2930
import org.apache.doris.common.util.ConsistentHash;
3031
import org.apache.doris.datasource.split.SplitWeight;
3132
import org.apache.doris.qe.ConnectContext;
33+
import org.apache.doris.resource.Tag;
3234
import org.apache.doris.resource.computegroup.ComputeGroup;
3335
import org.apache.doris.spi.Split;
3436
import org.apache.doris.system.Backend;
@@ -189,47 +191,40 @@ public void init(List<String> preLocations) throws UserException {
189191

190192
public void init(BeSelectionPolicy policy) throws UserException {
191193
ConnectContext ctx = ConnectContext.get();
192-
boolean ownsContext = ctx == null;
193-
if (ownsContext) {
194+
ComputeGroup computeGroup;
195+
if (ctx == null) {
194196
if (Config.isCloudMode()) {
195197
throw new AnalysisException("ConnectContext is null");
196-
} else {
197-
ctx = new ConnectContext();
198198
}
199-
199+
computeGroup = Env.getCurrentEnv().getComputeGroupMgr().getComputeGroupByName(Tag.VALUE_DEFAULT_TAG);
200+
} else {
201+
computeGroup = ctx.getComputeGroup();
202+
}
203+
if (Config.isNotCloudMode() && computeGroup.equals(ComputeGroup.INVALID_COMPUTE_GROUP)) {
204+
throw new LoadException(ComputeGroup.INVALID_COMPUTE_GROUP_ERR_MSG);
200205
}
201-
try {
202-
ComputeGroup computeGroup = ctx.getComputeGroup();
203-
if (Config.isNotCloudMode() && computeGroup.equals(ComputeGroup.INVALID_COMPUTE_GROUP)) {
204-
throw new LoadException(ComputeGroup.INVALID_COMPUTE_GROUP_ERR_MSG);
205-
}
206206

207-
backends.addAll(policy.getCandidateBackends(computeGroup.getBackendList()));
208-
if (backends.isEmpty()) {
209-
if (Config.isCloudMode()) {
210-
throw new UserException("No available backends, "
211-
+ "in cloud maybe this cluster has been dropped, please `use @otherClusterName` switch it");
212-
} else {
213-
throw new UserException("No available backends for compute group: " + computeGroup.toString());
214-
}
215-
}
216-
assignedWeightPerBackend.clear();
217-
assignedWeightCounts.clear();
218-
backendMap.clear();
219-
backendMap.putAll(backends.stream().collect(Collectors.groupingBy(Backend::getHost)));
220-
for (Backend backend : backends) {
221-
setAssignedWeight(backend, 0L);
207+
backends.addAll(policy.getCandidateBackends(computeGroup.getBackendList()));
208+
if (backends.isEmpty()) {
209+
if (Config.isCloudMode()) {
210+
throw new UserException("No available backends, "
211+
+ "in cloud maybe this cluster has been dropped, please `use @otherClusterName` switch it");
212+
} else {
213+
throw new UserException("No available backends for compute group: " + computeGroup.toString());
222214
}
215+
}
216+
assignedWeightPerBackend.clear();
217+
assignedWeightCounts.clear();
218+
backendMap.clear();
219+
backendMap.putAll(backends.stream().collect(Collectors.groupingBy(Backend::getHost)));
220+
for (Backend backend : backends) {
221+
setAssignedWeight(backend, 0L);
222+
}
223223

224-
try {
225-
consistentHash = consistentHashCache.get(new HashCacheKey(backends));
226-
} catch (ExecutionException e) {
227-
throw new UserException("failed to get consistent hash", e);
228-
}
229-
} finally {
230-
if (ownsContext) {
231-
ctx.requestClose(ConnectContext.CloseMode.AFTER_RESPONSE);
232-
}
224+
try {
225+
consistentHash = consistentHashCache.get(new HashCacheKey(backends));
226+
} catch (ExecutionException e) {
227+
throw new UserException("failed to get consistent hash", e);
233228
}
234229
}
235230

‎fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/LoadAction.java‎

Lines changed: 8 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,6 @@
3838
import org.apache.doris.load.StreamLoadHandler;
3939
import org.apache.doris.mysql.privilege.PrivPredicate;
4040
import org.apache.doris.planner.GroupCommitPlanner;
41-
import org.apache.doris.qe.AutoCloseConnectContext;
4241
import org.apache.doris.qe.ConnectContext;
4342
import org.apache.doris.qe.SessionVariable;
4443
import org.apache.doris.resource.BackendSelectionManager;
@@ -438,7 +437,7 @@ private TNetworkAddress selectLocalRedirectBackend(boolean groupCommit, HttpServ
438437
if (groupCommit) {
439438
// Use pre-selected backend if provided to avoid duplicate calls
440439
backend = preSelectedBackend != null ? preSelectedBackend
441-
: selectBackendForGroupCommit("", request, tableId);
440+
: selectBackendForGroupCommit("", tableId);
442441
} else {
443442
backend = chooseRedirectBackendWithSelection(ctx, backendIds);
444443
}
@@ -466,7 +465,7 @@ private TNetworkAddress selectCloudRedirectBackend(String clusterName, HttpServl
466465
if (groupCommit) {
467466
// Use pre-selected backend if provided to avoid duplicate calls
468467
backend = preSelectedBackend != null ? preSelectedBackend
469-
: selectBackendForGroupCommit(clusterName, req, tableId);
468+
: selectBackendForGroupCommit(clusterName, tableId);
470469
} else {
471470
backend = StreamLoadHandler.selectBackend(clusterName);
472471
}
@@ -755,17 +754,12 @@ private enum DrainDecision {
755754
DRAIN
756755
}
757756

758-
private Backend selectBackendForGroupCommit(String clusterName, HttpServletRequest req, long tableId)
757+
private Backend selectBackendForGroupCommit(String clusterName, long tableId)
759758
throws LoadException {
760-
ConnectContext currentCtx = ConnectContext.get();
761-
try (AutoCloseConnectContext scope = new AutoCloseConnectContext(new ConnectContext())) {
762-
ConnectContext ctx = scope.connectContext;
763-
ctx.setEnv(Env.getCurrentEnv());
764-
ctx.setRemoteIP(req.getRemoteAddr());
765-
// We set this variable to fulfill required field 'user' in
766-
// TMasterOpRequest(FrontendService.thrift)
767-
ctx.setCurrentUserIdentity(UserIdentity.ADMIN);
768-
inheritLoadSelectionContext(currentCtx, ctx);
759+
ConnectContext ctx = ConnectContext.get();
760+
try {
761+
// Authentication owns this request's context. Backend selection borrows its identity and
762+
// selection settings, and applies the compute group selected by this request's headers.
769763
if (Config.isCloudMode()) {
770764
ctx.setCloudCluster(clusterName);
771765
}
@@ -775,23 +769,6 @@ private Backend selectBackendForGroupCommit(String clusterName, HttpServletReque
775769
}
776770
}
777771

778-
private static void inheritLoadSelectionContext(ConnectContext source, ConnectContext target) {
779-
if (source == null) {
780-
return;
781-
}
782-
UserIdentity sourceIdentity = source.getCurrentUserIdentity();
783-
if (sourceIdentity != null) {
784-
target.setCurrentUserIdentity(sourceIdentity);
785-
}
786-
target.getSessionVariable().enableLoadBackendSelection =
787-
source.getSessionVariable().isEnableLoadBackendSelection();
788-
target.getSessionVariable().preferredBackendSelectionKey =
789-
source.getSessionVariable().getPreferredBackendSelectionKey();
790-
target.getSessionVariable().backendSelectionMode =
791-
source.getSessionVariable().getBackendSelectionMode();
792-
target.setConnectingFeLocalResourceGroup(source.getConnectingFeLocalResourceGroup());
793-
}
794-
795772
/*
796773
* Create redirect URL for stream load forward mode.
797774
*
@@ -880,7 +857,7 @@ private TNetworkAddress handleStreamLoadRedirect(HttpServletRequest request, boo
880857
}
881858

882859
// Get target backend for group commit
883-
Backend targetBackend = selectBackendForGroupCommit(cloudClusterName, request, tableId);
860+
Backend targetBackend = selectBackendForGroupCommit(cloudClusterName, tableId);
884861
if (targetBackend == null) {
885862
throw new LoadException("Failed to select target backend for group commit forwarding.");
886863
}

‎fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadLoadingTask.java‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -179,8 +179,8 @@ protected void executeOnce() throws Exception {
179179
BackendSelectionManager.restoreLoadSelection(executionContext, loadBackendSelectionHint);
180180
final boolean enableProfile = this.jobProfile != null;
181181
// New one query id,
182-
Coordinator curCoordinator = EnvFactory.getInstance().createCoordinator(callback.getCallbackId(),
183-
loadId, planner.getDescTable(),
182+
Coordinator curCoordinator = EnvFactory.getInstance().createCoordinator(executionContext,
183+
callback.getCallbackId(), loadId, planner.getDescTable(),
184184
planner.getFragments(), planner.getScanNodes(), planner.getTimezone(), loadZeroTolerance,
185185
enableProfile);
186186
if (enableProfile) {

0 commit comments

Comments
 (0)