Repository navigation
Commit bbaaecf
[fix](fe) Repair partition-level IVM baselines before refresh (apache#28)
Base-table DROP or TRUNCATE PARTITION can remove data without row-level binlog records. If the materialized view continues an incremental refresh from the old baseline, its result can be incorrect. This change records affected MV partitions in the existing IVM metadata and repairs them before any incremental attempt. Partitioned MVs repair only affected partitions; non-partitioned MVs fall back to COMPLETE refresh. ADD PARTITION remains eligible for normal binlog incremental refresh.
Root cause: The refresh path previously treated a base partition metadata removal as a global broken state or allowed incremental refresh to continue without rebuilding the affected baseline.
Change summary:
| Area | Change |
|------|--------|
| IVM metadata | Track partition-level broken state with the existing `IvmInfo.binlogBrokenPartitions` field. |
| Refresh flow | Resolve broken MV partitions during partition synchronization, repair them before IVM, and preserve broken state when repair fails. |
| Refresh scope | Use partition refresh for affected partitioned MVs and COMPLETE for non-partitioned MVs or ambiguous mappings. |
| Tests | Add FE unit coverage for successful and failed repairs, multi-base-partition mappings, and regression coverage for ADD, DROP, TRUNCATE, fallback, and recovery. |
```mermaid
graph TD
A[Refresh MV] --> B[syncPartitionsIfNeeded]
B --> C[Resolve broken MV partitions]
C --> D[Repair affected baseline]
D --> E{Repair complete?}
E -->|yes| F[IVM incremental refresh]
E -->|no| G[Keep broken state and fail or use COMPLETE]1 parent 01649db commit bbaaecf
37 files changed
Lines changed: 1317 additions & 406 deletions
File tree
- fe/fe-core/src
- main/java/org/apache/doris
- alter
- catalog
- datasource
- job/extensions/mtmv
- mtmv
- ivm
- persist
- test/java/org/apache/doris
- mtmv
- ivm
- nereids/trees/plans
- regression-test
- data
- mtmv_ivm_restart_p0
- mtmv_p0
- ivm
- suites
- mtmv_ivm_restart_p0
- mtmv_p0
- ivm
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
366 | 366 | | |
367 | 367 | | |
368 | 368 | | |
369 | | - | |
370 | | - | |
371 | | - | |
372 | 369 | | |
373 | 370 | | |
374 | 371 | | |
| |||
464 | 461 | | |
465 | 462 | | |
466 | 463 | | |
467 | | - | |
468 | | - | |
469 | | - | |
470 | | - | |
471 | | - | |
472 | | - | |
473 | | - | |
474 | | - | |
475 | | - | |
476 | | - | |
477 | | - | |
478 | | - | |
479 | | - | |
480 | | - | |
481 | | - | |
482 | | - | |
483 | | - | |
484 | | - | |
485 | | - | |
486 | | - | |
487 | | - | |
488 | | - | |
489 | | - | |
490 | | - | |
491 | | - | |
492 | | - | |
493 | | - | |
494 | | - | |
495 | | - | |
496 | | - | |
497 | | - | |
498 | | - | |
499 | 464 | | |
500 | 465 | | |
501 | 466 | | |
| |||
719 | 684 | | |
720 | 685 | | |
721 | 686 | | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
722 | 693 | | |
723 | 694 | | |
724 | 695 | | |
| |||
1312 | 1283 | | |
1313 | 1284 | | |
1314 | 1285 | | |
1315 | | - | |
1316 | 1286 | | |
1317 | 1287 | | |
1318 | 1288 | | |
| |||
1333 | 1303 | | |
1334 | 1304 | | |
1335 | 1305 | | |
1336 | | - | |
1337 | | - | |
| 1306 | + | |
| 1307 | + | |
1338 | 1308 | | |
1339 | | - | |
1340 | | - | |
1341 | | - | |
| 1309 | + | |
| 1310 | + | |
| 1311 | + | |
1342 | 1312 | | |
1343 | 1313 | | |
1344 | 1314 | | |
1345 | 1315 | | |
1346 | 1316 | | |
1347 | | - | |
| 1317 | + | |
1348 | 1318 | | |
| 1319 | + | |
1349 | 1320 | | |
1350 | 1321 | | |
1351 | 1322 | | |
| |||
1354 | 1325 | | |
1355 | 1326 | | |
1356 | 1327 | | |
1357 | | - | |
1358 | | - | |
| 1328 | + | |
1359 | 1329 | | |
1360 | 1330 | | |
1361 | 1331 | | |
| |||
1372 | 1342 | | |
1373 | 1343 | | |
1374 | 1344 | | |
1375 | | - | |
1376 | 1345 | | |
1377 | 1346 | | |
1378 | 1347 | | |
| |||
1381 | 1350 | | |
1382 | 1351 | | |
1383 | 1352 | | |
1384 | | - | |
1385 | 1353 | | |
1386 | 1354 | | |
1387 | 1355 | | |
1388 | 1356 | | |
1389 | 1357 | | |
1390 | 1358 | | |
1391 | | - | |
1392 | | - | |
1393 | | - | |
1394 | 1359 | | |
1395 | 1360 | | |
1396 | 1361 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| 53 | + | |
53 | 54 | | |
54 | 55 | | |
55 | 56 | | |
| |||
1066 | 1067 | | |
1067 | 1068 | | |
1068 | 1069 | | |
1069 | | - | |
1070 | | - | |
1071 | | - | |
| 1070 | + | |
| 1071 | + | |
| 1072 | + | |
| 1073 | + | |
1072 | 1074 | | |
1073 | 1075 | | |
1074 | 1076 | | |
| |||
1129 | 1131 | | |
1130 | 1132 | | |
1131 | 1133 | | |
1132 | | - | |
1133 | | - | |
1134 | | - | |
1135 | 1134 | | |
1136 | 1135 | | |
1137 | 1136 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
162 | 162 | | |
163 | 163 | | |
164 | 164 | | |
165 | | - | |
166 | 165 | | |
167 | 166 | | |
168 | 167 | | |
| |||
7015 | 7014 | | |
7016 | 7015 | | |
7017 | 7016 | | |
7018 | | - | |
7019 | | - | |
7020 | | - | |
| 7017 | + | |
| 7018 | + | |
| 7019 | + | |
| 7020 | + | |
| 7021 | + | |
| 7022 | + | |
| 7023 | + | |
| 7024 | + | |
| 7025 | + | |
| 7026 | + | |
| 7027 | + | |
| 7028 | + | |
7021 | 7029 | | |
7022 | 7030 | | |
7023 | 7031 | | |
| |||
7069 | 7077 | | |
7070 | 7078 | | |
7071 | 7079 | | |
7072 | | - | |
7073 | | - | |
7074 | 7080 | | |
7075 | 7081 | | |
7076 | 7082 | | |
| |||
7612 | 7618 | | |
7613 | 7619 | | |
7614 | 7620 | | |
7615 | | - | |
7616 | | - | |
7617 | | - | |
7618 | | - | |
7619 | | - | |
7620 | | - | |
7621 | 7621 | | |
7622 | 7622 | | |
7623 | 7623 | | |
| |||
0 commit comments