Repository navigation
Commit a560c74
[feature](ivm) Support incremental view maintenance (IVM) for materialized tables (MTMV) (#62606)
### What problem does this PR solve?
Implement Incremental View Maintenance (IVM) for materialized tables
(MTMV) in Apache Doris. When a base table of an MTMV changes, only the
affected rows are derived from the base-table deltas (row binlog / Table
Stream) and transactionally applied to the materialized table, instead
of recomputing the whole result with a full refresh. This enables
maintaining near-real-time aggregation and join results at a fraction of
the cost of full refresh.
Supported operators: projection, filter, aggregate
(COUNT/SUM/AVG/MIN/MAX, bitmap aggregates, expression arguments, bare
GROUP BY, grouping sets), joins (INNER/CROSS/OUTER JOIN, nested join and
self join), UNION ALL, subquery aliases, and OneRowRelation.
Supported refresh: `REFRESH MATERIALIZED VIEW ...
INCREMENTAL/PARTITIONS`, refresh explain, dry-run, and automatic
full-refresh fallback when incremental maintenance is not valid (e.g.
binlog broken).
Lifecycle: stream creation/lifecycle integrated with MTMV, chained IVM
MTMV support.
This PR is the MTMV incremental-maintenance layer of the end-to-end
incremental computation stack tracked by #65418.
Issue Number: close #66719
Related issue: #65418 (tracking issue), #57921 (superseded umbrella
issue)
---------
Co-authored-by: seawinde <[email protected]>1 parent 099d5bd commit a560c74
347 files changed
Lines changed: 48763 additions & 920 deletions
File tree
- fe
- fe-catalog/src/main/java/org/apache/doris/catalog
- fe-core/src
- main/java/org/apache/doris
- alter
- catalog
- stream
- common/util
- datasource
- job
- extensions/mtmv
- task
- mtmv
- ivm
- agg
- nereids
- glue/translator
- jobs/executor
- parser
- rules
- analysis
- exploration/mv
- trees
- expressions/functions/scalar
- plans
- commands
- info
- insert
- logical
- visitor
- persist
- qe
- tablefunction
- transaction
- test/java/org/apache/doris
- catalog
- job/extensions/mtmv
- mtmv
- ivm
- agg
- nereids
- glue/translator
- parser
- rules
- analysis
- rewrite
- sqltest
- trees/plans
- algebra
- commands
- info
- insert
- logical
- qe
- transaction
- fe-sql-parser/src/main/antlr4/org/apache/doris/nereids
- regression-test
- data
- mtmv_ivm_restart_p0
- mtmv_p0
- ivm
- plugins
- suites
- mtmv_ivm_restart_p0
- mtmv_p0
- ivm
- table_stream_p0
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
| 54 | + | |
54 | 55 | | |
55 | 56 | | |
56 | 57 | | |
57 | 58 | | |
58 | 59 | | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
59 | 75 | | |
60 | 76 | | |
61 | 77 | | |
| |||
285 | 301 | | |
286 | 302 | | |
287 | 303 | | |
| 304 | + | |
288 | 305 | | |
289 | 306 | | |
290 | 307 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| 38 | + | |
38 | 39 | | |
39 | 40 | | |
40 | 41 | | |
| |||
49 | 50 | | |
50 | 51 | | |
51 | 52 | | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
52 | 58 | | |
53 | 59 | | |
54 | 60 | | |
| 61 | + | |
55 | 62 | | |
56 | 63 | | |
57 | 64 | | |
| |||
359 | 366 | | |
360 | 367 | | |
361 | 368 | | |
362 | | - | |
363 | 369 | | |
364 | 370 | | |
365 | 371 | | |
| |||
678 | 684 | | |
679 | 685 | | |
680 | 686 | | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
681 | 693 | | |
682 | 694 | | |
683 | 695 | | |
| |||
743 | 755 | | |
744 | 756 | | |
745 | 757 | | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
746 | 762 | | |
747 | 763 | | |
748 | 764 | | |
| |||
1267 | 1283 | | |
1268 | 1284 | | |
1269 | 1285 | | |
1270 | | - | |
1271 | 1286 | | |
1272 | 1287 | | |
1273 | 1288 | | |
| |||
1279 | 1294 | | |
1280 | 1295 | | |
1281 | 1296 | | |
1282 | | - | |
1283 | | - | |
| 1297 | + | |
| 1298 | + | |
| 1299 | + | |
| 1300 | + | |
| 1301 | + | |
| 1302 | + | |
| 1303 | + | |
| 1304 | + | |
| 1305 | + | |
| 1306 | + | |
| 1307 | + | |
1284 | 1308 | | |
1285 | | - | |
1286 | | - | |
1287 | | - | |
| 1309 | + | |
| 1310 | + | |
| 1311 | + | |
1288 | 1312 | | |
1289 | 1313 | | |
1290 | 1314 | | |
1291 | 1315 | | |
1292 | 1316 | | |
| 1317 | + | |
| 1318 | + | |
| 1319 | + | |
| 1320 | + | |
1293 | 1321 | | |
1294 | 1322 | | |
1295 | 1323 | | |
1296 | 1324 | | |
1297 | 1325 | | |
1298 | 1326 | | |
1299 | 1327 | | |
1300 | | - | |
1301 | | - | |
| 1328 | + | |
1302 | 1329 | | |
1303 | 1330 | | |
1304 | 1331 | | |
1305 | 1332 | | |
1306 | 1333 | | |
1307 | 1334 | | |
1308 | 1335 | | |
| 1336 | + | |
| 1337 | + | |
| 1338 | + | |
| 1339 | + | |
| 1340 | + | |
| 1341 | + | |
| 1342 | + | |
| 1343 | + | |
| 1344 | + | |
| 1345 | + | |
| 1346 | + | |
| 1347 | + | |
| 1348 | + | |
| 1349 | + | |
| 1350 | + | |
| 1351 | + | |
| 1352 | + | |
| 1353 | + | |
| 1354 | + | |
| 1355 | + | |
| 1356 | + | |
| 1357 | + | |
| 1358 | + | |
| 1359 | + | |
| 1360 | + | |
| 1361 | + | |
| 1362 | + | |
| 1363 | + | |
| 1364 | + | |
| 1365 | + | |
| 1366 | + | |
| 1367 | + | |
| 1368 | + | |
| 1369 | + | |
| 1370 | + | |
| 1371 | + | |
| 1372 | + | |
| 1373 | + | |
| 1374 | + | |
| 1375 | + | |
| 1376 | + | |
| 1377 | + | |
| 1378 | + | |
| 1379 | + | |
| 1380 | + | |
| 1381 | + | |
| 1382 | + | |
| 1383 | + | |
| 1384 | + | |
| 1385 | + | |
| 1386 | + | |
| 1387 | + | |
| 1388 | + | |
| 1389 | + | |
| 1390 | + | |
| 1391 | + | |
| 1392 | + | |
| 1393 | + | |
| 1394 | + | |
| 1395 | + | |
1309 | 1396 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| 32 | + | |
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
| |||
50 | 51 | | |
51 | 52 | | |
52 | 53 | | |
| 54 | + | |
53 | 55 | | |
54 | 56 | | |
55 | 57 | | |
| |||
1075 | 1077 | | |
1076 | 1078 | | |
1077 | 1079 | | |
| 1080 | + | |
| 1081 | + | |
| 1082 | + | |
| 1083 | + | |
1078 | 1084 | | |
1079 | 1085 | | |
1080 | 1086 | | |
| |||
0 commit comments