You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f8ed33f
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](ivm) Refresh the surviving partitions after an IVM baseline rebuild (apache#67802)
### What problem does this PR solve?
Dropping a base-table partition marks the IVM baseline as broken,
because those rows disappear through metadata rather than through row
binlog entries. Partition sync then removes the very MV partitions that
barrier named, so `handlePendingIvmBaselineRebuild` intersected the
barrier with the current MV partitions and always got an empty set. The
refresh reported SUCCESS with refresh mode `NOT_REFRESH` and cleared the
barrier without refreshing anything, leaving the delta that had
accumulated on the surviving partitions unapplied: the task claimed the
MV was up to date while the MV was missing rows.
Reproduction: drop a base partition, insert a row into a surviving
partition, then run `REFRESH MATERIALIZED VIEW ... INCREMENTAL
FALLBACK`. The task succeeds, the expired MV partition is dropped, and
the new row never reaches the MV.
### What changed
- `MTMVTask.handlePendingIvmBaselineRebuild` is a pre-step now instead
of a terminal branch. It rebuilds only the barrier partitions that still
exist, releases the barrier, and then lets the normal attempt list run,
so the surviving partitions catch up in the same task. Partitions the
barrier named that partition sync already dropped need no rebuild: the
partition and its IVM offsets are both gone, which is what satisfies
those entries.
- The attempt list is rewritten in place. A lone `COMPLETE` attempt
rebuilds everything anyway and needs no pre-rebuild; a complete baseline
rebuild rewrites the list to `COMPLETE` instead of executing it inline.
- `MTMV.releaseIvmBaselineRebuild` releases the barrier under a
`schemaChangeVersion` compare-and-clear and journals the new state
immediately, like the other `ivmInfo` mutations. Without the release the
IVM attempt that follows would be rejected by `validateIvmRefreshStart`;
without the compare-and-clear a concurrent base-table change would lose
the barrier entry it had just recorded.
- `MTMVRelationManager.markIvmBaselineRebuild` takes an explicit
all-partitions-changed flag instead of inferring it from an empty
partition map.
The COMPLETE paths are unchanged. They end the task, so the existing
`addTaskResult` cleanup releases the barrier once the refresh succeeds.
A strict `REFRESH ... INCREMENTAL`, and a scheduled refresh of an MV
declared without `FALLBACK`, still fail while the barrier is pending,
because that check runs before partition sync. Relaxing it is left to a
follow-up PR.
### Test
-
`regression-test/suites/mtmv_p0/ivm/test_ivm_partition_drop_live_delta.groovy`
(new): partitions added and dropped by hand with literal dates and no
dynamic-partition scheduler, so the case is deterministic. Asserts the
MV matches the base table right after `INCREMENTAL FALLBACK`. The
scenario was reproduced on the pre-fix build, where the fallback task
reported SUCCESS while the MV was missing the row written to the
surviving partition.
-
`regression-test/suites/mtmv_p0/ivm/test_ivm_partition_baseline_rebuild_dup_keys.groovy`
(new): covers the `TRUNCATE` path, where the affected MV partition
survives and the pre-step really has something to rebuild. Runs on a
duplicate-key MV with repeated identical rows, so a delta applied twice
would show up as extra rows instead of hiding behind a unique key.
- `test_ivm_partition_baseline_rebuild`,
`test_ivm_partition_sync_retry`, `test_ivm_partition_unique_key`,
`test_ivm_partition_window_limit`, `test_ivm_partition_window_remove`,
`test_ivm_one_row_relation_partitioned`,
`test_ivm_strict_failure_partition_atomicity`
Trace issue: apache#65418
0 commit comments