Repository navigation
Commit 7bd89a0
authored
[fix](ivm) Stop the incremental delta from reading partitions the MV dropped (apache#67814)
`partition_sync_limit` keeps only a recent slice of each base table's
partitions, so an MV can be missing a partition its base table still
has. The incremental delta read that base partition anyway, through the
delta scan and through the join-opposite snapshot. A change to a
non-partitioned dimension therefore produced delta rows for dates the MV
has no partition for, and the insert failed with `no partition for this
tuple`.
Re-syncing cannot bring an expired partition back, so the retry loop in
`executeIvmAttempt` could never recover: the task failed after
exhausting its attempts, and because the write is atomic, the partitions
the MV does keep were left unrepaired as well. A strict `REFRESH ...
INCREMENTAL`, and a scheduled refresh of an MV declared without
`FALLBACK`, fail outright.
### What changed
The incremental delta now reads only the base partitions the MV's
partition definition keeps.
- `MTMVPartitionUtil.generateRelatedBasePartitionIds` returns that
partition set per base table partitioned by the MV's partition column,
and an empty value when `partition_sync_limit` is not set, which is the
only property that can leave the MV without a base partition.
- `IvmIncrRefreshManager` passes it to `IvmRewriteContext.incremental`,
and `IvmDeltaRewriter` applies it as an upper bound on the partitions
each base table may be read from, intersected with
`ivm_partition_window_limit` when that is set too.
- The bound is the partition set the MV is aligned to rather than the
partitions it already has, so a base partition whose MV partition has
not been added yet stays readable: the refresh still reports the missing
partition and recovers it by syncing, which is what keeps a newly added
base partition working.
- A base table absent from the set keeps its full read, since the MV
partition column does not come from it and limiting it would change join
results without narrowing the target MV partitions. A set covering every
partition of its table leaves the plan untouched, so an MV that mirrors
all of its base partitions is unaffected.
- Only olap tables are reachable: the delta reads them through their
stream and restricts a scan by partition id, which a connector table has
no equivalent of.
The COMPLETE paths are unchanged.
The PR also removes the `nonConcurrent` group from seven `mtmv_p0/ivm`
suites that use neither debug points nor global variables or config, so
they run in the parallel pool again.
Trace issue: apache#654181 parent 4b5ab04 commit 7bd89a0
20 files changed
Lines changed: 670 additions & 20 deletions
File tree
- fe/fe-core/src
- main/java/org/apache/doris/mtmv
- ivm
- test/java/org/apache/doris/mtmv
- ivm
- regression-test
- data/mtmv_p0/ivm
- suites/mtmv_p0/ivm
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
237 | 237 | | |
238 | 238 | | |
239 | 239 | | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
240 | 297 | | |
241 | 298 | | |
242 | 299 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
218 | 218 | | |
219 | 219 | | |
220 | 220 | | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
221 | 234 | | |
222 | 235 | | |
223 | 236 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| 29 | + | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
| |||
37 | 38 | | |
38 | 39 | | |
39 | 40 | | |
| 41 | + | |
40 | 42 | | |
41 | 43 | | |
42 | 44 | | |
| |||
66 | 68 | | |
67 | 69 | | |
68 | 70 | | |
69 | | - | |
| 71 | + | |
| 72 | + | |
70 | 73 | | |
71 | 74 | | |
72 | 75 | | |
| |||
153 | 156 | | |
154 | 157 | | |
155 | 158 | | |
156 | | - | |
| 159 | + | |
| 160 | + | |
157 | 161 | | |
158 | 162 | | |
159 | 163 | | |
| |||
181 | 185 | | |
182 | 186 | | |
183 | 187 | | |
| 188 | + | |
184 | 189 | | |
185 | 190 | | |
186 | 191 | | |
187 | 192 | | |
188 | 193 | | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
189 | 249 | | |
190 | 250 | | |
191 | 251 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| 25 | + | |
| 26 | + | |
25 | 27 | | |
26 | 28 | | |
27 | 29 | | |
| |||
39 | 41 | | |
40 | 42 | | |
41 | 43 | | |
| 44 | + | |
42 | 45 | | |
| 46 | + | |
43 | 47 | | |
44 | 48 | | |
| 49 | + | |
45 | 50 | | |
46 | 51 | | |
47 | 52 | | |
| |||
92 | 97 | | |
93 | 98 | | |
94 | 99 | | |
95 | | - | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
96 | 110 | | |
97 | 111 | | |
98 | 112 | | |
| |||
0 commit comments