Repository navigation
Commit 695c88b
authored
[fix](binlog) Refresh incremental partition versions before pruning (#68138)
### What problem does this PR solve?
Issue Number: None
Related PR: #67181
Problem Summary:
A cloud incremental read may wait for its target transaction to become
visible before query planning continues. During that wait, the FE
`CloudPartition` cache can still contain `PARTITION_INIT_VERSION` for a
partition even though the transaction has already become visible in
MetaService.
`PruneEmptyPartition` previously relied only on that cached version. It
could therefore classify the partition as empty and remove it before
scan-node planning. The later version refresh in
`ScanNode.setVisibleVersionForOlapScanNodes` could observe the new
version, but it was already too late because the partition had been
pruned from the scan.
This PR changes empty-partition pruning for direct cloud incremental
reads as follows:
- A partition that is already known to be non-empty from the FE cache is
retained directly. Partition versions are monotonic, so a cached version
greater than `PARTITION_INIT_VERSION` remains sufficient for this
pruning decision.
- Only cached-empty or unknown candidate partitions are refreshed from
MetaService. This catches transactions that became visible after the FE
cache entry was populated while avoiding unnecessary RPC work for known
non-empty partitions.
- The refreshed and cached results are combined in the original
candidate-partition order.
- Normal reads and incremental reads backed by an `OlapTableWrapper`
with fixed visible versions continue to use their existing
snapshot-aware path.
The refresh in `ScanNode.setVisibleVersionForOlapScanNodes` is
intentionally retained. `PruneEmptyPartition` only needs an
empty-or-non-empty decision, while scan execution still needs the exact
snapshot versions of all final selected partitions.
### Release note
Fix cloud incremental reads that could miss newly visible data because a
stale FE partition-version cache caused premature empty-partition
pruning.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- `PruneEmptyPartitionTest`
- `OlapScanNodeTest`
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes. Direct cloud incremental reads refresh cached-empty or
unknown partition versions from MetaService before empty-partition
pruning.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label1 parent 22044ef commit 695c88b
4 files changed
Lines changed: 184 additions & 10 deletions
File tree
- fe/fe-core/src
- main/java/org/apache/doris
- catalog
- cloud/catalog
- test/java/org/apache/doris
- catalog
- nereids/rules/rewrite
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
20 | 23 | | |
21 | 24 | | |
22 | 25 | | |
23 | 26 | | |
24 | 27 | | |
| 28 | + | |
25 | 29 | | |
| 30 | + | |
26 | 31 | | |
27 | 32 | | |
| 33 | + | |
| 34 | + | |
28 | 35 | | |
29 | 36 | | |
30 | 37 | | |
| |||
79 | 86 | | |
80 | 87 | | |
81 | 88 | | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
82 | 105 | | |
83 | 106 | | |
84 | 107 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
213 | 213 | | |
214 | 214 | | |
215 | 215 | | |
216 | | - | |
217 | | - | |
218 | | - | |
219 | | - | |
220 | | - | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
221 | 235 | | |
222 | 236 | | |
223 | 237 | | |
224 | | - | |
225 | | - | |
226 | | - | |
227 | | - | |
228 | 238 | | |
229 | 239 | | |
230 | 240 | | |
231 | 241 | | |
232 | 242 | | |
233 | 243 | | |
234 | | - | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
235 | 247 | | |
236 | 248 | | |
237 | 249 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
| 22 | + | |
21 | 23 | | |
22 | 24 | | |
| 25 | + | |
| 26 | + | |
23 | 27 | | |
24 | 28 | | |
25 | 29 | | |
| 30 | + | |
| 31 | + | |
26 | 32 | | |
| 33 | + | |
27 | 34 | | |
| 35 | + | |
28 | 36 | | |
29 | 37 | | |
30 | 38 | | |
| |||
117 | 125 | | |
118 | 126 | | |
119 | 127 | | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
120 | 187 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
0 commit comments