Repository navigation
Commit 57a1b3d
committed
[fix](ivm) Match the changed base partition by incarnation and read the sync limit twice
Two ways the selection still picked the wrong partitions, both by reading less than what decides it.
RECOVER PARTITION reports the recycled partition under the name it had, and a partition added after
the drop can be live under that name again by then, with a different range. The membership check
matched names only, so it accepted that replacement: the barrier went to the MV partitions of the
replacement's range, and the recovered range -- whose rows no row binlog can repair -- was left
without one, so the MV partition that partition sync then adds for it stays empty.
The partition_sync_limit decides whether the mapping may be trusted, but was read only after the
mapping was built. MV properties are mutated in between (ALTER MATERIALIZED VIEW ... SET is not
generation guarded), so a limit cleared while the mapping was built read as no limit at all, and a
change outside the window was answered with "no MV partition reads it": no barrier, no version bump.
Key changes:
- Require each changed base partition to be the live partition the caller supplied, by id, not merely a partition carrying the same name; a missing or mismatched incarnation rebuilds the whole MV, which is what the missing-name case already did. Only RECOVER can supply an incarnation that is not live: DROP, TRUNCATE and strict REPLACE all pass the id of the partition they are changing
- Read the limit on both sides of the mapping construction, so a limit in effect either before or after it makes the mapping untrustworthy; a limit that is gone by the time the guard runs still describes the mapping it produced
- End the recent partition of both sync-window tests two years out: the cutoff is now() truncated to the year and a partition is kept while its upper bound is after it, so a year rollover between building the DDL and marking the change used to put its upper bound exactly on the cutoff, drop it from the mapping, and let the test pass through the answer it exists to rule out
Unit Test:
- IvmBaselineRebuildTest.testRecoveredPartitionWhoseNameWasReusedRebuildsTheWholeMv: DROP p202001, add a different-range p202001, then RECOVER ... AS p202003. Without the id check it fails with expected: <true> but was: <false> -- no barrier at all
- IvmBaselineRebuildTest.testTheSyncLimitIsReadOnBothSidesOfTheMapping pins both reads; with a single read it fails with "Wanted but not invoked ... exactly 1 interaction". The interleaving itself is not staged: the mapping is built with no injection point between the reads
- Regression: mtmv_p0/ivm test_ivm_baseline_marker_scope, test_ivm_partition_baseline_rebuild, test_ivm_partition_baseline_rebuild_dup_keys, test_ivm_drop_referenced_column_baseline_rebuild, test_ivm_partition_drop_live_delta, test_ivm_replace_table_reconcile, test_ivm_replace_stream_cleanup and test_ivm_drop_column_fallback_reason green on a local cluster built from this head; IvmBaselineRebuildTest 36/36; checkstyle clean1 parent 892b20c commit 57a1b3d
2 files changed
Lines changed: 111 additions & 16 deletions
File tree
- fe/fe-core/src
- main/java/org/apache/doris/catalog
- test/java/org/apache/doris/mtmv/ivm
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
803 | 803 | | |
804 | 804 | | |
805 | 805 | | |
806 | | - | |
807 | | - | |
808 | | - | |
809 | | - | |
810 | | - | |
811 | | - | |
812 | | - | |
813 | | - | |
814 | | - | |
815 | | - | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
816 | 823 | | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
817 | 834 | | |
818 | 835 | | |
| 836 | + | |
| 837 | + | |
819 | 838 | | |
820 | 839 | | |
821 | 840 | | |
| |||
856 | 875 | | |
857 | 876 | | |
858 | 877 | | |
859 | | - | |
860 | | - | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
861 | 882 | | |
862 | 883 | | |
863 | 884 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| 36 | + | |
36 | 37 | | |
| 38 | + | |
37 | 39 | | |
38 | 40 | | |
39 | 41 | | |
| |||
47 | 49 | | |
48 | 50 | | |
49 | 51 | | |
| 52 | + | |
| 53 | + | |
50 | 54 | | |
51 | 55 | | |
52 | 56 | | |
| |||
178 | 182 | | |
179 | 183 | | |
180 | 184 | | |
181 | | - | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
182 | 191 | | |
183 | 192 | | |
184 | 193 | | |
| |||
189 | 198 | | |
190 | 199 | | |
191 | 200 | | |
192 | | - | |
| 201 | + | |
193 | 202 | | |
194 | 203 | | |
195 | 204 | | |
| |||
223 | 232 | | |
224 | 233 | | |
225 | 234 | | |
226 | | - | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
227 | 241 | | |
228 | 242 | | |
229 | 243 | | |
| |||
234 | 248 | | |
235 | 249 | | |
236 | 250 | | |
237 | | - | |
| 251 | + | |
238 | 252 | | |
239 | 253 | | |
240 | 254 | | |
| |||
256 | 270 | | |
257 | 271 | | |
258 | 272 | | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
259 | 307 | | |
260 | 308 | | |
261 | 309 | | |
| |||
447 | 495 | | |
448 | 496 | | |
449 | 497 | | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
450 | 524 | | |
451 | 525 | | |
452 | 526 | | |
| |||
0 commit comments