Skip to content

Commit 695c88b

Browse files
authored
[fix](binlog) Refresh incremental partition versions before pruning (#68138)
### What problem does this PR solve? Issue Number: None Related PR: #67181 Problem Summary: A cloud incremental read may wait for its target transaction to become visible before query planning continues. During that wait, the FE `CloudPartition` cache can still contain `PARTITION_INIT_VERSION` for a partition even though the transaction has already become visible in MetaService. `PruneEmptyPartition` previously relied only on that cached version. It could therefore classify the partition as empty and remove it before scan-node planning. The later version refresh in `ScanNode.setVisibleVersionForOlapScanNodes` could observe the new version, but it was already too late because the partition had been pruned from the scan. This PR changes empty-partition pruning for direct cloud incremental reads as follows: - A partition that is already known to be non-empty from the FE cache is retained directly. Partition versions are monotonic, so a cached version greater than `PARTITION_INIT_VERSION` remains sufficient for this pruning decision. - Only cached-empty or unknown candidate partitions are refreshed from MetaService. This catches transactions that became visible after the FE cache entry was populated while avoiding unnecessary RPC work for known non-empty partitions. - The refreshed and cached results are combined in the original candidate-partition order. - Normal reads and incremental reads backed by an `OlapTableWrapper` with fixed visible versions continue to use their existing snapshot-aware path. The refresh in `ScanNode.setVisibleVersionForOlapScanNodes` is intentionally retained. `PruneEmptyPartition` only needs an empty-or-non-empty decision, while scan execution still needs the exact snapshot versions of all final selected partitions. ### Release note Fix cloud incremental reads that could miss newly visible data because a stale FE partition-version cache caused premature empty-partition pruning. ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - `PruneEmptyPartitionTest` - `OlapScanNodeTest` - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [x] Yes. Direct cloud incremental reads refresh cached-empty or unknown partition versions from MetaService before empty-partition pruning. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
1 parent 22044ef commit 695c88b

4 files changed

Lines changed: 184 additions & 10 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/catalog/RowBinlogTableWrapper.java‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,14 +17,21 @@
1717

1818
package org.apache.doris.catalog;
1919

20+
import org.apache.doris.catalog.stream.StreamReadMode;
21+
import org.apache.doris.cloud.catalog.CloudPartition;
22+
import org.apache.doris.common.Config;
2023
import org.apache.doris.common.Pair;
2124

2225
import com.google.common.base.Preconditions;
2326
import com.google.common.collect.Maps;
2427

28+
import java.util.Collection;
2529
import java.util.Collections;
30+
import java.util.List;
2631
import java.util.Map;
2732
import java.util.Objects;
33+
import java.util.Optional;
34+
import java.util.stream.Collectors;
2835

2936
/**
3037
* A lightweight wrapper base for read binlog<Row> of table
@@ -79,6 +86,22 @@ public KeysType getKeysType() {
7986
return KeysType.DUP_KEYS;
8087
}
8188

89+
@Override
90+
public List<Long> selectNonEmptyPartitionIds(Collection<Long> partitionIds,
91+
Optional<StreamReadMode> streamReadMode) {
92+
if (Config.isCloudMode() && !hasFixedVisibleVersions()) {
93+
// A row-binlog scan can start immediately after its target transaction becomes visible. Refresh
94+
// cached-empty or unknown partitions so an older cache entry cannot prune newly visible binlog data.
95+
List<CloudPartition> partitions = partitionIds.stream()
96+
.map(this::getPartition)
97+
.filter(Objects::nonNull)
98+
.map(partition -> (CloudPartition) partition)
99+
.collect(Collectors.toList());
100+
return CloudPartition.selectNonEmptyPartitionIdsFromMs(partitions);
101+
}
102+
return super.selectNonEmptyPartitionIds(partitionIds, streamReadMode);
103+
}
104+
82105
@Override
83106
public boolean equals(Object obj) {
84107
if (!super.equals(obj)) {

‎fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudPartition.java‎

Lines changed: 22 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -213,25 +213,37 @@ private long getVisibleVersionFromMs(boolean waitForPendingTxns) {
213213

214214
// Select the non-empty partitions and return the ids.
215215
public static List<Long> selectNonEmptyPartitionIds(List<CloudPartition> partitions) {
216-
List<Long> nonEmptyPartitionIds = partitions.stream()
217-
.filter(CloudPartition::hasDataCached)
218-
.map(CloudPartition::getId)
219-
.collect(Collectors.toList());
220-
if (nonEmptyPartitionIds.size() == partitions.size()) {
216+
return selectNonEmptyPartitionIds(partitions, false);
217+
}
218+
219+
// Select non-empty partitions while bypassing the version cache for cached-empty or unknown partitions.
220+
public static List<Long> selectNonEmptyPartitionIdsFromMs(List<CloudPartition> partitions) {
221+
return selectNonEmptyPartitionIds(partitions, true);
222+
}
223+
224+
private static List<Long> selectNonEmptyPartitionIds(List<CloudPartition> partitions, boolean forceRefresh) {
225+
List<Long> nonEmptyPartitionIds = new ArrayList<>(partitions.size());
226+
List<CloudPartition> unknowns = new ArrayList<>(partitions.size());
227+
for (CloudPartition partition : partitions) {
228+
if (partition.hasDataCached()) {
229+
nonEmptyPartitionIds.add(partition.getId());
230+
} else {
231+
unknowns.add(partition);
232+
}
233+
}
234+
if (unknowns.isEmpty()) {
221235
return nonEmptyPartitionIds;
222236
}
223237

224-
List<CloudPartition> unknowns = partitions.stream()
225-
.filter(p -> !p.hasDataCached())
226-
.collect(Collectors.toList());
227-
228238
SummaryProfile profile = getSummaryProfile();
229239
if (profile != null) {
230240
profile.incGetPartitionVersionByHasDataCount();
231241
}
232242

233243
try {
234-
List<Long> versions = CloudPartition.getSnapshotVisibleVersion(unknowns);
244+
List<Long> versions = forceRefresh
245+
? CloudPartition.getSnapshotVisibleVersionFromMs(unknowns, false)
246+
: CloudPartition.getSnapshotVisibleVersion(unknowns);
235247

236248
int size = versions.size();
237249
for (int i = 0; i < size; i++) {

‎fe/fe-core/src/test/java/org/apache/doris/catalog/OlapTableWrapperTest.java‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,13 +18,21 @@
1818
package org.apache.doris.catalog;
1919

2020
import org.apache.doris.binlog.BinlogTestUtils;
21+
import org.apache.doris.cloud.catalog.CloudPartition;
22+
import org.apache.doris.common.Config;
2123
import org.apache.doris.thrift.TStorageType;
2224

25+
import com.google.common.collect.ImmutableList;
26+
import com.google.common.collect.ImmutableMap;
2327
import com.google.common.collect.Lists;
2428
import org.junit.jupiter.api.Assertions;
2529
import org.junit.jupiter.api.Test;
30+
import org.mockito.MockedStatic;
31+
import org.mockito.Mockito;
2632

33+
import java.util.Collections;
2734
import java.util.List;
35+
import java.util.Optional;
2836
import java.util.concurrent.TimeUnit;
2937

3038
public class OlapTableWrapperTest {
@@ -117,4 +125,63 @@ public void testRowBinlogTableWrapper() {
117125
Assertions.assertEquals(table.getSchemaByIndexId(table.getBaseIndexId()), wrapper.getSchemaByIndexId(table.getBaseIndexId()));
118126
Assertions.assertEquals(table.getIndexSchemaVersion(table.getBaseIndexId()), wrapper.getIndexSchemaVersion(table.getBaseIndexId()));
119127
}
128+
129+
@Test
130+
public void testCloudRowBinlogWrapperRefreshesCachedEmptyPartitions() {
131+
long stalePartitionId = 100L;
132+
long cachedNonEmptyPartitionId = 101L;
133+
List<Long> partitionIds = ImmutableList.of(stalePartitionId, cachedNonEmptyPartitionId);
134+
135+
CloudPartition stalePartition = Mockito.mock(CloudPartition.class);
136+
Mockito.when(stalePartition.getId()).thenReturn(stalePartitionId);
137+
// Simulate another query refreshing the shared cache after the first cached-state check.
138+
Mockito.when(stalePartition.hasDataCached()).thenReturn(false, true);
139+
CloudPartition cachedNonEmptyPartition = Mockito.mock(CloudPartition.class);
140+
Mockito.when(cachedNonEmptyPartition.getId()).thenReturn(cachedNonEmptyPartitionId);
141+
Mockito.when(cachedNonEmptyPartition.hasDataCached()).thenReturn(true);
142+
143+
OlapTable table = Mockito.spy(newTestTable(BinlogTestUtils.newTestRowBinlogConfig(true, true)));
144+
Mockito.doReturn(stalePartition).when(table).getPartition(stalePartitionId);
145+
Mockito.doReturn(cachedNonEmptyPartition).when(table).getPartition(cachedNonEmptyPartitionId);
146+
RowBinlogTableWrapper wrapper = new RowBinlogTableWrapper(table);
147+
148+
try (MockedStatic<Config> mockedConfig = Mockito.mockStatic(Config.class);
149+
MockedStatic<CloudPartition> mockedPartition = Mockito.mockStatic(
150+
CloudPartition.class, Mockito.CALLS_REAL_METHODS)) {
151+
mockedConfig.when(Config::isCloudMode).thenReturn(true);
152+
mockedPartition.when(() -> CloudPartition.getSnapshotVisibleVersionFromMs(
153+
ImmutableList.of(stalePartition), false)).thenReturn(ImmutableList.of(2L));
154+
mockedPartition.clearInvocations();
155+
156+
Assertions.assertEquals(ImmutableList.of(cachedNonEmptyPartitionId, stalePartitionId),
157+
wrapper.selectNonEmptyPartitionIds(partitionIds, Optional.empty()));
158+
mockedPartition.verify(() -> CloudPartition.getSnapshotVisibleVersionFromMs(
159+
ImmutableList.of(stalePartition), false));
160+
Mockito.verify(stalePartition).hasDataCached();
161+
Mockito.verify(cachedNonEmptyPartition).hasDataCached();
162+
}
163+
}
164+
165+
@Test
166+
public void testCloudRowBinlogWrapperWithFixedVisibleVersionsUsesOriginTable() {
167+
long emptyPartitionId = 100L;
168+
long nonEmptyPartitionId = 101L;
169+
List<Long> partitionIds = ImmutableList.of(emptyPartitionId, nonEmptyPartitionId);
170+
171+
OlapTable table = Mockito.spy(newTestTable(BinlogTestUtils.newTestRowBinlogConfig(true, true)));
172+
Mockito.doReturn(ImmutableList.of(nonEmptyPartitionId)).when(table)
173+
.selectNonEmptyPartitionIds(partitionIds, Optional.empty());
174+
RowBinlogTableWrapper wrapper = new RowBinlogTableWrapper(table, Collections.emptyMap(),
175+
ImmutableMap.of(emptyPartitionId, Partition.PARTITION_INIT_VERSION, nonEmptyPartitionId, 2L));
176+
177+
try (MockedStatic<Config> mockedConfig = Mockito.mockStatic(Config.class);
178+
MockedStatic<CloudPartition> mockedPartition = Mockito.mockStatic(CloudPartition.class)) {
179+
mockedConfig.when(Config::isCloudMode).thenReturn(true);
180+
181+
Assertions.assertEquals(ImmutableList.of(nonEmptyPartitionId),
182+
wrapper.selectNonEmptyPartitionIds(partitionIds, Optional.empty()));
183+
Mockito.verify(table).selectNonEmptyPartitionIds(partitionIds, Optional.empty());
184+
mockedPartition.verifyNoInteractions();
185+
}
186+
}
120187
}
Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.nereids.rules.rewrite;
19+
20+
import org.apache.doris.catalog.OlapTable;
21+
import org.apache.doris.cloud.catalog.CloudPartition;
22+
import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
23+
import org.apache.doris.nereids.util.MemoPatternMatchSupported;
24+
import org.apache.doris.nereids.util.MemoTestUtils;
25+
import org.apache.doris.nereids.util.PlanChecker;
26+
import org.apache.doris.nereids.util.PlanConstructor;
27+
import org.apache.doris.qe.ConnectContext;
28+
29+
import com.google.common.collect.ImmutableList;
30+
import org.junit.jupiter.api.Assertions;
31+
import org.junit.jupiter.api.Test;
32+
import org.mockito.MockedStatic;
33+
import org.mockito.Mockito;
34+
35+
import java.util.List;
36+
import java.util.Optional;
37+
38+
class PruneEmptyPartitionTest implements MemoPatternMatchSupported {
39+
40+
@Test
41+
void testNormalReadUsesCachedPartitionVersions() {
42+
long emptyPartitionId = 100L;
43+
long nonEmptyPartitionId = 101L;
44+
List<Long> partitionIds = ImmutableList.of(emptyPartitionId, nonEmptyPartitionId);
45+
46+
CloudPartition emptyPartition = Mockito.mock(CloudPartition.class);
47+
Mockito.when(emptyPartition.getId()).thenReturn(emptyPartitionId);
48+
CloudPartition nonEmptyPartition = Mockito.mock(CloudPartition.class);
49+
Mockito.when(nonEmptyPartition.getId()).thenReturn(nonEmptyPartitionId);
50+
51+
OlapTable table = Mockito.spy(PlanConstructor.newOlapTable(10L, "normal_tbl", 0));
52+
Mockito.doReturn(partitionIds).when(table).getPartitionIds();
53+
Mockito.doReturn(emptyPartition).when(table).getPartition(emptyPartitionId);
54+
Mockito.doReturn(nonEmptyPartition).when(table).getPartition(nonEmptyPartitionId);
55+
Mockito.doReturn(ImmutableList.of(nonEmptyPartitionId)).when(table)
56+
.selectNonEmptyPartitionIds(Mockito.anyCollection(), Mockito.any());
57+
58+
LogicalOlapScan scan = new LogicalOlapScan(
59+
PlanConstructor.getNextRelationId(), table, ImmutableList.of("normal_tbl"));
60+
ConnectContext connectContext = MemoTestUtils.createConnectContext();
61+
62+
try (MockedStatic<CloudPartition> mockedPartition = Mockito.mockStatic(CloudPartition.class)) {
63+
LogicalOlapScan rewritten = (LogicalOlapScan) PlanChecker.from(connectContext, scan)
64+
.applyTopDown(new PruneEmptyPartition())
65+
.getPlan();
66+
67+
Assertions.assertEquals(ImmutableList.of(nonEmptyPartitionId), rewritten.getSelectedPartitionIds());
68+
Mockito.verify(table).selectNonEmptyPartitionIds(partitionIds, Optional.empty());
69+
mockedPartition.verifyNoInteractions();
70+
}
71+
}
72+
}

0 commit comments

Comments
 (0)