[fix](ivm) Read a base partition with no stream offset from the base table in an IVM refresh by morningman · Pull Request #69778 · apache/doris

morningman · 2026-10-28T14:06:07Z

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

In short. An IVM materialized view silently misses the rows of a base partition its internal stream has no offset for (a partition added after the MV was built, or one that was still empty then) when those rows are older than the base table's changelog.ttl_seconds at refresh time. The refresh succeeds, the rows are never read, and a base-table query of the MV's shape that transparent rewrite answers from the MV returns the same wrong result. This PR makes an IVM read take such a partition from the base table, as an append of all its rows, instead of from its changelog.

Background. An IVM MV reads each base table through an internal table stream. The stream keeps an offset per partition and records one only when it consumes the partition with data in it. It is created with show_initial_rows, so a partition that has data at creation is a historical partition, read once from the base table; a partition that is empty then gets nothing; and every consumption, the reset of a COMPLETE refresh included, records offsets only for the partitions it read with data. Two kinds of partitions therefore have no offset and none of their rows in the MV: a partition added after the stream was created, and a partition that was empty when the stream was created and at every consumption since (a range partition created ahead of its data is the usual one).

The problem, and what it cost.

NormalizeOlapTableStreamScan.makeTableStreamScan reads a partition with no offset like any other incremental partition: from its changelog, with no start. applyChangelogTtl turns the missing start into the first TSO the changelog retains and does not reject it.
Once the partition's rows are older than the TTL when the MV is refreshed, the read skips them. The refresh succeeds and records the partition's offset past them, so no later refresh reads them either: the MV stays short of those rows for good.
This holds for an MV over a unique-key table and for an append-only detail MV over a duplicate-key table. An MV partitioned like its base table rebuilds the MV partition of an added base partition, but its partition that was empty when the MV was built is caught up incrementally and loses its rows the same way.

How this PR fixes it.

In an IVM statement, makeTableStreamScan moves the incremental partitions that have no consumption baseline (OlapTableStreamWrapper.filterConsumedPartitionIds: no offset, or the sentinel offset of a partition empty at stream creation) to the branch that reads historical partitions from the base table: every row as an append, with the commit TSO as the sequence and the row LSN as the row identity. For an MV that holds none of the partition's rows that is the partition's whole change, whatever the TTL. The offset the refresh records afterwards is the same as before.
The base read and the end TSO the refresh commits come from the same snapshot, as for a historical partition: in shared-nothing mode both are taken under the table read lock that planning holds, and in cloud mode the base read is pinned to the visible version of the stream read state.
A user stream is not changed: reading the changelog it retains, from the first retained TSO, is what it promises. The choice is made by the statement being an IVM rewrite, not by the stream.

Results (local shared-nothing cluster; rows written to an added partition and to the partition that was empty, older than a one-second TTL when the MV is refreshed):

MV	Before	After
non-partitioned, over a unique-key table	SUCCESS, the rows of both partitions missing for good	SUCCESS, the MV matches the base table
append-only detail MV, over a duplicate-key table	SUCCESS, the rows of both partitions missing for good	SUCCESS, the MV matches the base table
partitioned like its unique-key base table	SUCCESS, the added partition rebuilt, the rows of the one that was empty missing	SUCCESS, the MV matches the base table
the next refresh, after more changes, with the TTL back to a day	incremental, the rows still missing	incremental, the new changes read from the changelog

Classes, and how they call each other.

IvmIncrRefreshMTMV → IvmDeltaRewriteState.createDeltaScan builds a LogicalOlapTableStreamScan over an OlapTableStreamWrapper.
NormalizeOlapTableStreamScan.makeTableStreamScan → filterHistoryPartitionIds / filterIncrementalPartitionIds, and in an IVM statement filterConsumedPartitionIds → historical and unconsumed partitions: makeOlapScanOnBaseTable; consumed partitions: makeIncrementalScanFromChangelog → applyChangelogTtl.
StreamConsumptionInfoExtractor records the new offsets from the wrapper's output map, as before.

Not changed: the TTL check, the snapshot and reset reads (the snapshot read already leaves the partitions without a baseline out and reports that it did), user streams, and anything persisted.

Release note

Fix an IVM materialized view missing the rows of a base partition that was added after the MV was created, or was empty then, when those rows were older than the table's changelog.ttl_seconds at refresh time.

Check List (For Author)

Test: Regression test / Unit Test
- Regression test: mtmv_p0/ivm/test_ivm_partition_without_offset (new, nonConcurrent): a non-partitioned MV over a unique-key table, an append-only detail MV over a duplicate-key table and an MV partitioned like its unique-key base table, each refreshed after rows written to an added partition and to the partition that was empty became older than the TTL, then again after more changes with the TTL back to a day, and compared with the base table with MV rewrite off. Local shared-nothing cluster: first on a build without this change, where all three refreshes succeeded without those rows; with the change the suite passes, and passes again against the generated .out. On the same build the whole mtmv_p0/ivm directory (113 suites), mtmv_ivm_restart_p0 and changelog_p0/test_changelog_before_column_name pass. After the rebase onto the latest master (two commits that touch neither this code nor IVM), the new suite and the other test_ivm_partition_* suites pass again. Not run on a cloud cluster.
- Unit Test: ExplainTableStreamPlanTest (two new cases: a user stream reads a partition without a consumption baseline from the changelog; the same read in an IVM statement takes it from the base table and still reads the partition with an offset from the changelog), run together with IvmIncrRefreshMTMVTest, IvmFullRefreshMTMVTest, InsertIntoTableCommandTableStreamTest, UpdateMvByPartitionCommandTest and CollectRelationChangelogTtlTest: 84 tests pass (before the rebase).
Behavior changed: Yes. An IVM refresh reads a partition with no stream offset from the base table instead of from its changelog; for such a partition EXPLAIN REFRESH ... INCREMENTAL shows a scan of the base table.
Does this need documentation: No

…table in an IVM refresh ### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: An IVM materialized view reads each base table through an internal table stream, and the stream records an offset for a partition only when it consumes the partition with data in it. Two kinds of partitions therefore have no offset, and none of their rows are in the MV: a partition added after the stream was created, and a partition that was still empty when the stream was created and whenever it was consumed. A COMPLETE refresh records offsets only for the partitions that have data, so a range partition created ahead of its data stays without one. `NormalizeOlapTableStreamScan.makeTableStreamScan` reads such a partition from its changelog with no start, and `applyChangelogTtl` turns a missing start into the first TSO the changelog retains. Once the partition's rows are older than `changelog.ttl_seconds` when the MV is refreshed, they are skipped: the refresh succeeds, the offset is recorded past them, and no later refresh reads them. A query on the base table with the MV's shape is answered from the MV by transparent rewrite and returns the same wrong result. An MV partitioned like its base table rebuilds the MV partition of an added base partition, but its partition that was empty when the MV was built is caught up incrementally, so it loses those rows too. How to reproduce: a range-partitioned table with the changelog enabled and a TTL of a day, with rows in one partition only; an IVM MV on it, refreshed; then add a partition, write rows to it and to the partition that was empty, set `changelog.ttl_seconds` to 1, wait two seconds and run `REFRESH MATERIALIZED VIEW mv INCREMENTAL`. | MV | Before | After | | --- | --- | --- | | non-partitioned, over a unique-key table | SUCCESS, the rows of both partitions missing for good | SUCCESS, the MV matches the base table | | append-only detail MV, over a duplicate-key table | SUCCESS, the rows of both partitions missing for good | SUCCESS, the MV matches the base table | | partitioned like its unique-key base table | SUCCESS, the added partition rebuilt, the rows of the one that was empty missing | SUCCESS, the MV matches the base table | The fix makes an IVM read take an incremental partition with no consumption baseline (no offset, or the sentinel offset of a partition empty at stream creation) from the base table, the way it already takes a historical partition of a stream created with `show_initial_rows`: every row as an append, with the commit TSO as the sequence and the row LSN as the row identity. For an MV that holds none of the partition's rows that is the partition's whole change, whatever the TTL, and the offset the refresh records is the same as before. In shared-nothing mode the base read and the end TSO it commits are both taken under the table read lock of planning, as for a historical partition; in cloud mode the base read is pinned to the visible version of the stream read state. The choice is made by the statement being an IVM rewrite: a user stream keeps reading the changelog it retains, which is what it promises. ### Release note Fix an IVM materialized view missing the rows of a base partition that was added after the MV was created, or was empty then, when those rows were older than the table's `changelog.ttl_seconds` at refresh time. ### Check List (For Author) - Test: Regression test / Unit Test - Regression test: `mtmv_p0/ivm/test_ivm_partition_without_offset` (new, nonConcurrent): a non-partitioned MV over a unique-key table, an append-only detail MV over a duplicate-key table and an MV partitioned like its unique-key base table, each refreshed after rows written to an added partition and to the partition that was empty became older than the TTL, then again after more changes with the TTL back to a day, and compared with the base table with MV rewrite off. Run on a local shared-nothing cluster, first on a build without this change, where all three refreshes succeeded without those rows; with the change the suite passes, and passes again against the generated `.out`. On the same build the whole `mtmv_p0/ivm` directory (113 suites), `mtmv_ivm_restart_p0` and `changelog_p0/test_changelog_before_column_name` pass. Not run on a cloud cluster. - Unit Test: `ExplainTableStreamPlanTest` (two new cases: a user stream reads a partition without a consumption baseline from the changelog, the same read in an IVM statement takes it from the base table and still reads the partition with an offset from the changelog). Run together with `IvmIncrRefreshMTMVTest`, `IvmFullRefreshMTMVTest`, `InsertIntoTableCommandTableStreamTest`, `UpdateMvByPartitionCommandTest` and `CollectRelationChangelogTtlTest`: 84 tests pass. - Behavior changed: Yes. An IVM refresh reads a partition with no stream offset from the base table instead of from its changelog; for such a partition `EXPLAIN REFRESH ... INCREMENTAL` shows a scan of the base table. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>

hello-stephen · 2026-10-28T14:06:15Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

morningman · 2026-10-28T14:10:38Z

/review

morningman · 2026-10-28T14:10:47Z

run buildall

github-actions

Reviewed all four changed files at head 7549bc2bb2184cee197e66ee0c905c3d50f272e9 through two rounds, including independent production, test, and baseline/snapshot investigations. One P2 finding concerns the new unit-test fixture's commit timestamps. No production correctness defect or P0/P1 blocker was found. There were no existing inline findings to carry forward and no additional user-provided focus points.

Critical checkpoint conclusions:

Goal/data correctness: The IVM-only classification supplies the complete current partition image where the pre-refresh image is empty, preventing TTL truncation. The added FE and regression cases target this behavior.
Scope/reuse: The localized change reuses the existing historical scan, projections, delete-sign filter and union helpers.
Concurrency/locking: Shared-nothing planning holds the base read lock through offset/version capture and scan finalization. Cloud scans use the visible version paired with the read-state end TSO. No new lock, thread, lock-order change or locked IO is introduced.
Lifecycle: Added collections are statement-local. Initial, consumed, reset, partial-rebuild and failed-refresh baseline transitions were traced; no new lifetime or initialization issue was found.
Configuration: No configuration item is added.
Compatibility: No public symbol, persisted metadata or storage format changes.
Thrift/protobuf: No definitions or field/enum IDs change.
Parallel paths: Both unique-key and append-only detail IVM models receive the fix. User streams retain their behavior; RESET/SNAPSHOT bypass this branch. Shared-nothing and cloud paths were inspected.
Conditional logic: The IVM gate and no-baseline premise are documented and agree with snapshot exclusion of these partitions.
Test coverage: Added tests cover user versus IVM routing, unique/detail/partitioned MVs, added and initially empty partitions past TTL, and subsequent offset consumption. New NULL, join, aggregate and cloud-concurrency cases are absent; those interactions were examined statically.
Expected results: All 17 query/output sections, business rows, ordering and task-mode expectations match their producers. The inline finding identifies a timing-dependent fixture failure. Output-generation provenance and test execution were not verified here.
Observability: Existing EXPLAIN scans and refresh task/fallback status expose the changed route adequately.
Transactions/persistence: The original analyzed wrapper still supplies offset updates through existing transaction attachments; commit/replay and cloud atomic rowset/offset handling remain in use.
Writes/crashes: The rule changes read routing. Existing offset/data commit boundaries and durable rebuild requirements cover retries and interrupted rebuilds; no new write/failure boundary was found.
FE/BE propagation: No transmitted variable or protocol path is added.
Performance/memory: Classification is linear in selected partitions using set membership. A complete base scan is required to recover rows outside changelog retention; no new BE ownership or allocation issue applies.
Other issues: Final whole-PR sweep found no additional valuable issue; error/precondition behavior remains unchanged.

The review converged in round 2: all three reviewers returned NO_NEW_VALUABLE_FINDINGS in the latest round, every candidate and initial risk has a recorded disposition, and no concrete question remains unresolved. Validation was static only: no builds, tests, benchmarks or runtime reproduction were performed, and the fixture failure frequency was not measured.

Review converged after 2 round(s).

morningman · 2026-10-28T14:28:36Z

When does the problem this PR fixes show up? Three examples

Rows go missing only when three things hold at the same time:

The partition has no stream offset. An IVM MV's internal stream records an offset for a partition only when it consumes the partition with data in it, so two kinds of partitions have none:
- a partition added after the MV was built (dynamic or auto partitioning, ALTER TABLE ... ADD PARTITION);
- a partition that already existed but was still empty when the MV was built (future partitions pre-created by dynamic_partition.end, a LIST partition that is not in use yet).
Its rows wait longer than changelog.ttl_seconds before the first refresh that reads the partition (the TTL is one day when the property is not set). Either the refresh interval is longer than the TTL, or refreshes stopped for a while: the job was paused, refreshes kept failing, or nobody refreshed a manually refreshed MV.
Nothing else in that refresh trips the expiry check. If a partition that does have an offset also has new data and its offset has expired, the whole incremental read is rejected. On a unique-key table that fails the refresh today, and with [fix](ivm) Fall back to COMPLETE when an IVM stream offset has expired #69772 a refresh that may fall back runs COMPLETE, which happens to repair the MV. Either way it is loud. A table partitioned by time and written only in its latest partitions meets this condition naturally, because its old partitions do not change.

Example 1: daily partitions, refreshed once a week

CREATE TABLE orders (
    order_id BIGINT NOT NULL,
    dt       DATE   NOT NULL,
    amount   DECIMAL(12, 2)
)
UNIQUE KEY(order_id, dt)
PARTITION BY RANGE(dt) ()
DISTRIBUTED BY HASH(order_id) BUCKETS 8
PROPERTIES (
    "enable_unique_key_merge_on_write" = "true",
    "changelog.enable" = "true",
    "changelog.need_historical_value" = "true",
    "dynamic_partition.enable" = "true",
    "dynamic_partition.time_unit" = "DAY",
    "dynamic_partition.start" = "-30",
    "dynamic_partition.end" = "3",
    "dynamic_partition.prefix" = "p"
);  -- changelog.ttl_seconds is not set: one day

CREATE MATERIALIZED VIEW sales_by_day
BUILD IMMEDIATE REFRESH INCREMENTAL ON SCHEDULE EVERY 7 DAY STARTS '2026-11-02 02:00:00'
DISTRIBUTED BY HASH(dt) BUCKETS 1
AS SELECT dt, COUNT(*) AS cnt, SUM(amount) AS total FROM orders GROUP BY dt;

Orders are loaded in batches during the day (09:00 to 18:00) into that day's partition, and old partitions are not modified:

Time	What happens
Mon 11-02 02:00, refresh	Last week's partitions are consumed and have offsets. `p20261102` is still empty, so it has no offset
11-02 to 11-08, daytime	Each day's orders are loaded into that day's partition: `p20261102` to `p20261108` all have data, and none of them has an offset
Mon 11-09 02:00, refresh	Only these seven partitions have new data, and none of them has an offset to expire, so nothing is rejected. All seven are read from the first TSO the changelog retains, which is 11-08 02:00
Result	The orders of 11-02 to 11-07 are skipped and only 11-08 is read. The refresh is SUCCESS, with an empty `IvmFallbackReason`

After that refresh the seven partitions have offsets at their latest TSO, so the six missing days are never read again; only a manual COMPLETE refresh brings them back. The same happens every week. SELECT dt, SUM(amount) FROM orders GROUP BY dt is rewritten to the MV and misses the same six days.

Example 2: refreshes interrupted for a few days

Same table, with the MV refreshed every night at 02:00. Normally every partition is first read less than a day after its rows were written, so nothing is lost. Now the refresh fails three nights in a row for an unrelated reason, or the job is paused during an upgrade. When it succeeds on the fourth night, the partitions of those days were never consumed: they have no offsets, they are read from "now minus one day", and only the last day reaches the MV. What the user sees is a refresh that recovered.

Example 3: a partition that existed, empty, when the MV was built; a partitioned (PCT) MV is affected too

With dynamic_partition.end = 3, the partitions of the next three days exist, empty, when the MV is created. Take an MV partitioned like its base table:

CREATE MATERIALIZED VIEW sales_pmv
BUILD IMMEDIATE REFRESH INCREMENTAL ON SCHEDULE EVERY 7 DAY
PARTITION BY (dt)
DISTRIBUTED BY HASH(dt) BUCKETS 1
AS SELECT dt, order_id, amount FROM orders;

The three pre-created partitions already have their MV partitions, so when their data arrives they are caught up incrementally instead of rebuilt. Under the timing of example 1 or 2 they lose rows.
A partition created after the MV is a new base partition: the PCT MV rebuilds its MV partition, so that one is not affected.
A non-partitioned MV is affected by both kinds.

A LIST partition behaves the same way: PARTITION p_eu VALUES IN ('EU') stays empty until the region launches, and if its first import lands more than a TTL before the next refresh, that import never reaches the MV.

When it does not happen

Refreshes always succeed on time, and the interval is shorter than the TTL: every row is read within the TTL.
The partition already has an offset. An expired offset stops the read of a unique-key table: a failure today, and COMPLETE with [fix](ivm) Fall back to COMPLETE when an IVM stream offset has expired #69772 when the refresh may fall back. (A duplicate-key table clips an expired offset today; that is a separate problem with its own fix.)
In a PCT MV, a base partition added after the MV was built: its MV partition is rebuilt.
User streams: reading the changelog from the first retained TSO is their semantics, and this PR keeps it.

The new regression test test_ivm_partition_without_offset compresses these timelines into a one-second TTL. On a build without this PR, all three MVs in it end up without those rows.

hello-stephen · 2026-10-28T16:19:17Z

TPC-H: Total hot run time: 28513 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7549bc2bb2184cee197e66ee0c905c3d50f272e9, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17607	5122	5364	5122
q2	2144	292	251	251
q3	10247	1383	803	803
q4	4678	462	342	342
q5	7427	786	526	526
q6	182	190	154	154
q7	745	811	589	589
q8	9250	1334	1063	1063
q9	5468	4168	4178	4168
q10	6826	1282	1047	1047
q11	429	246	226	226
q12	639	404	283	283
q13	18058	3521	3054	3054
q14	272	272	249	249
q15	q16	732	722	692	692
q17	1222	765	576	576
q18	6972	6180	6124	6124
q19	1122	950	609	609
q20	375	324	221	221
q21	5166	2164	2260	2164
q22	343	292	250	250
Total cold run time: 99904 ms
Total hot run time: 28513 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5586	5598	5585	5585
q2	674	558	556	556
q3	5225	5325	4984	4984
q4	1987	2129	1562	1562
q5	5234	5150	5565	5150
q6	246	201	150	150
q7	2363	2024	1900	1900
q8	3072	2643	2642	2642
q9	8074	7952	7928	7928
q10	3857	3645	3229	3229
q11	637	463	412	412
q12	723	679	516	516
q13	3486	3701	3310	3310
q14	320	307	294	294
q15	q16	702	700	636	636
q17	8379	6980	6896	6896
q18	13017	12245	12798	12245
q19	905	777	793	777
q20	2145	2149	1900	1900
q21	5497	4545	4647	4545
q22	484	452	424	424
Total cold run time: 72613 ms
Total hot run time: 65641 ms

morningman · 2026-10-28T16:19:30Z

Steps to reproduce

One unique-key table, one non-partitioned IVM MV, one BE. The FE needs enable_feature_changelog = true and enable_table_stream = true in fe.conf (the regression pipelines set both).

CREATE DATABASE IF NOT EXISTS ivm_repro;
USE ivm_repro;
-- Keep the queries on t answered by t itself; step 6 turns this back on.
SET enable_materialized_view_rewrite = false;

DROP MATERIALIZED VIEW IF EXISTS mv;
DROP TABLE IF EXISTS t;
CREATE TABLE t (
    k  BIGINT NOT NULL,
    dt DATE   NOT NULL,
    v  BIGINT
)
UNIQUE KEY(k, dt)
PARTITION BY RANGE(dt) (
    PARTITION p202601 VALUES [('2026-01-01'), ('2026-02-01')),
    PARTITION p202602 VALUES [('2026-02-01'), ('2026-03-01'))
)
DISTRIBUTED BY HASH(k) BUCKETS 1
PROPERTIES (
    "replication_num" = "1",
    "enable_unique_key_merge_on_write" = "true",
    "changelog.enable" = "true",
    "changelog.need_historical_value" = "true",
    "changelog.ttl_seconds" = "86400"
);
INSERT INTO t VALUES (1, '2026-01-10', 10);   -- p202602 stays empty for now

-- 1. Build the MV. Its first refresh is a COMPLETE one: p202601 gets a stream offset, the empty p202602 gets none.
CREATE MATERIALIZED VIEW mv
BUILD IMMEDIATE REFRESH INCREMENTAL ON MANUAL
DISTRIBUTED BY HASH(k) BUCKETS 1
PROPERTIES ("replication_num" = "1")
AS SELECT k, dt, v FROM t;
-- repeat until Status is SUCCESS
SELECT Status, RefreshMode FROM tasks('type'='mv')
WHERE MvDatabaseName = 'ivm_repro' AND MvName = 'mv' ORDER BY CreateTime DESC LIMIT 1;

-- 2. Rows in the partition that was empty and in a partition added after the MV: neither partition has an offset.
ALTER TABLE t ADD PARTITION p202603 VALUES [('2026-03-01'), ('2026-04-01'));
INSERT INTO t VALUES (2, '2026-02-10', 20), (3, '2026-03-10', 30);

-- 3. Let those rows grow older than the TTL before the MV reads them. This stands in for a refresh that comes
--    more than a day after the write with the default TTL of one day.
ALTER TABLE t SET ("changelog.ttl_seconds" = "1");
SELECT sleep(2);

-- 4. A strict incremental refresh; repeat the query until the task has finished.
REFRESH MATERIALIZED VIEW mv INCREMENTAL;
SELECT Status, RefreshMode, IvmFallbackReason, ErrorMsg FROM tasks('type'='mv')
WHERE MvDatabaseName = 'ivm_repro' AND MvName = 'mv' ORDER BY CreateTime DESC LIMIT 1;

-- 5. Compare the MV with the table.
SELECT * FROM mv ORDER BY k;
SELECT * FROM t ORDER BY k;

-- 6. With transparent rewrite on, the same query on t is answered from mv.
SET enable_materialized_view_rewrite = true;
SELECT k, dt, v FROM t ORDER BY k;
EXPLAIN SHAPE PLAN SELECT k, dt, v FROM t ORDER BY k;

What it printed on a local shared-nothing cluster:

Step	Without this PR (master `86d62ae00c9`, with #69785 on top, which does not touch this path)	With this PR (`7549bc2bb21`)
1. build task	`SUCCESS`, `COMPLETE`	`SUCCESS`, `COMPLETE`
4. refresh task	`SUCCESS`; `RefreshMode` and `IvmFallbackReason` are `\N`, `ErrorMsg` is empty	the same
5. `SELECT * FROM mv`	one row: `1, 2026-01-10, 10`	three rows: `1, 2026-01-10, 10` / `2, 2026-02-10, 20` / `3, 2026-03-10, 30`
5. `SELECT * FROM t`	the three rows	the three rows
6. the same query with rewrite on	one row; the plan is a `PhysicalOlapScan[mv]`	three rows; the plan is a `PhysicalOlapScan[mv]`

So without the fix the refresh reports success, the MV is short of both rows for good (their partitions now have offsets past them), and the query on t returns the MV's answer. Nothing in the task, its error or its fallback reason says that anything went wrong.

Variants, all in test_ivm_partition_without_offset: a duplicate-key table with an append-only detail MV loses the same two rows; an MV partitioned by dt rebuilds the MV partition of p202603 but loses the row of p202602, whose MV partition existed already.

hello-stephen · 2026-10-28T16:29:05Z

TPC-DS: Total hot run time: 150727 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7549bc2bb2184cee197e66ee0c905c3d50f272e9, data reload: false

query5	4308	599	466	466
query6	426	210	192	192
query7	4793	466	225	225
query8	315	167	161	161
query9	8784	3253	3249	3249
query10	480	305	255	255
query11	5952	3848	3573	3573
query12	151	90	88	88
query13	1258	440	325	325
query14	6100	3931	3656	3656
query14_1	3423	3423	3426	3423
query15	215	197	176	176
query16	936	424	397	397
query17	838	650	510	510
query18	2407	408	302	302
query19	198	156	130	130
query20	83	83	79	79
query21	206	125	105	105
query22	12986	12976	12709	12709
query23	13360	12822	12393	12393
query23_1	12485	12543	12562	12543
query24	7334	816	469	469
query24_1	476	481	497	481
query25	536	424	362	362
query26	1249	255	140	140
query27	2769	452	272	272
query28	4622	2060	2071	2060
query29	1566	581	440	440
query30	300	220	184	184
query31	1022	932	828	828
query32	148	92	94	92
query33	523	297	237	237
query34	893	865	525	525
query35	737	776	641	641
query36	824	822	734	734
query37	129	100	85	85
query38	1775	1737	1678	1678
query39	792	786	744	744
query39_1	727	737	746	737
query40	221	115	99	99
query41	71	73	68	68
query42	88	85	81	81
query43	343	362	315	315
query44	1326	710	712	710
query45	180	181	168	168
query46	838	967	569	569
query47	3060	3019	2864	2864
query48	299	308	232	232
query49	574	412	331	331
query50	743	295	224	224
query51	10410	10510	10172	10172
query52	80	82	77	77
query53	190	208	156	156
query54	262	205	190	190
query55	74	71	73	71
query56	244	240	220	220
query57	1650	1634	1556	1556
query58	298	276	262	262
query59	2134	2144	1946	1946
query60	290	256	241	241
query61	174	168	160	160
query62	393	332	288	288
query63	192	161	161	161
query64	2875	945	789	789
query65	3335	3285	3291	3285
query66	1792	425	308	308
query67	20244	20157	20007	20007
query68	2947	1009	625	625
query69	382	290	252	252
query70	861	769	791	769
query71	299	235	208	208
query72	2670	2561	1914	1914
query73	527	526	294	294
query74	4609	4501	4370	4370
query75	2324	2314	1896	1896
query76	2321	1018	594	594
query77	335	367	256	256
query78	9964	9721	9132	9132
query79	1023	854	544	544
query80	1252	412	340	340
query81	592	310	270	270
query82	599	135	114	114
query83	300	194	184	184
query84	312	124	97	97
query85	863	422	372	372
query86	399	237	226	226
query87	1977	1951	1804	1804
query88	3871	2791	2781	2781
query89	341	290	237	237
query90	1922	168	164	164
query91	166	152	124	124
query92	100	91	88	88
query93	1041	1000	572	572
query94	649	284	261	261
query95	583	388	324	324
query96	691	494	240	240
query97	2421	2448	2310	2310
query98	162	154	150	150
query99	774	768	643	643
Total cold run time: 235402 ms
Total hot run time: 150727 ms

hello-stephen · 2026-10-28T16:33:58Z

ClickBench: Total hot run time: 24.62 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7549bc2bb2184cee197e66ee0c905c3d50f272e9, data reload: false

query1	0.00	0.00	0.01
query2	0.10	0.05	0.05
query3	0.26	0.09	0.08
query4	1.61	0.10	0.10
query5	0.23	0.25	0.24
query6	1.18	0.70	0.68
query7	0.04	0.00	0.00
query8	0.05	0.04	0.03
query9	0.39	0.33	0.34
query10	0.58	0.58	0.56
query11	0.19	0.14	0.14
query12	0.18	0.15	0.15
query13	0.49	0.50	0.49
query14	1.03	1.03	1.01
query15	0.64	0.60	0.61
query16	0.34	0.37	0.35
query17	1.19	1.15	1.12
query18	0.19	0.18	0.18
query19	2.12	2.02	1.97
query20	0.02	0.01	0.02
query21	15.50	0.19	0.13
query22	4.96	0.05	0.05
query23	16.26	0.32	0.13
query24	2.72	0.70	0.71
query25	0.12	0.06	0.04
query26	1.64	0.21	0.16
query27	0.04	0.04	0.03
query28	3.59	0.74	0.35
query29	12.48	4.14	3.35
query30	0.28	0.12	0.11
query31	2.78	0.56	0.33
query32	3.22	0.60	0.51
query33	3.30	3.38	3.12
query34	15.22	4.00	3.39
query35	3.33	3.36	3.34
query36	0.59	0.47	0.44
query37	0.09	0.07	0.06
query38	0.05	0.03	0.03
query39	0.04	0.03	0.03
query40	0.17	0.15	0.14
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 97.38 s
Total hot run time: 24.62 s

hello-stephen · 2026-10-28T16:46:10Z

FE UT Coverage Report

Increment line coverage 100.00% (10/10) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-10-28T18:14:30Z

FE Regression Coverage Report

Increment line coverage 47.62% (10/21) 🎉
Increment coverage report
Complete coverage report

yujun777

LGTM

yujun777 · 2026-10-29T02:04:58Z

        List<Long> incrementalPartitionIds =
                streamWrapper.filterIncrementalPartitionIds(scan.getSelectedPartitionIds());
+        if (cascadesContext.getStatementContext().isIvmMTMVRewrite()) {
+            // A partition with no consumption baseline -- added after the stream was created, or still empty when


这个感觉不是IVM 专有的，IVM 只是用Stream 功能，如果有BUG，这里对非IVM也是适用的。 read from stream 本来就应该那些没有offset的新分区

…cate-key table (#69785) ### What problem does this PR solve? Issue Number: None Related PR: #69772 Problem Summary: **In short.** An IVM materialized view over a duplicate-key base table silently misses rows once the table's `changelog.ttl_seconds` has expired its stream offset: the refresh succeeds, the rows written between the offset and the cutoff are never read, and a base-table query of the MV's shape that transparent rewrite answers from the MV returns the same wrong result. This PR makes every read of an IVM refresh reject an expired offset, whatever its scan type, so the refresh fails with `CHANGELOG_BROKEN` or, when it may fall back, rebuilds the MV with COMPLETE (#69772). It also carries the two follow-ups yujun777 suggested on #69772. **Background.** An IVM MV reads each base table through an internal table stream. A duplicate-key table is read `APPEND_ONLY`: `OlapTableStreamWrapper.getStreamScanType()` returns `APPEND_ONLY` for `DUP_KEYS`, whatever type the stream declares. `NormalizeOlapTableStreamScan.applyChangelogTtl` decides what an expired start means: a `MIN_DELTA` read is rejected (#69772 turns that into `CHANGELOG_BROKEN` and COMPLETE), any other read is clipped to the first TSO the changelog retains. The clipping is right for a user stream, which reads the changelog it retains. **The problem, and what it cost.** - For an IVM refresh the clipping loses data without a trace. The rows written between the offset and the cutoff are never read, the refresh succeeds, the offset moves past them, and no later incremental refresh brings them back. - It hits every IVM MV over a duplicate-key table, the aggregate ones (unique-key MV) and the append-only detail ones alike, whenever the MV is not refreshed within the TTL of a write. **How this PR fixes it.** 1. `applyChangelogTtl` rejects an expired start for every read of an IVM statement (the statement carries an `IvmRewriteContext`), whatever the scan type. The condition is the statement, not the type the stream declares: a user stream on a duplicate-key table also declares `MIN_DELTA` by default, and it keeps its clipping. An IVM statement never reads a user stream: an MV query that scans a stream is rejected when the MV is created. The snapshot read of a duplicate-key table is bounded by the offset on the base table itself and reads no changelog, so it is unchanged. 2. The follow-ups from the review of #69772, in their own commit and without a behavior change: - the rebuild an incremental attempt runs before its delta now turns a failure into a fallback when its reason's only repair is COMPLETE (`IvmFailureReason.requiresCompleteRefresh()`), and passes that reason through, instead of naming `CHANGELOG_BROKEN`; - the `CHANGELOG_BROKEN` branch of `handleIvmFallbackResult`, which returned the same result as the `requiresCompleteRefresh()` branch after it, is removed; why only COMPLETE repairs an expired changelog now sits on `IvmFailureReason.CHANGELOG_BROKEN`. **Results** (local shared-nothing cluster; a duplicate-key table, a row written after the MV's offset, then `changelog.ttl_seconds` set to 1 and two seconds of waiting): | Refresh after the offset expired | Before | After | | --- | --- | --- | | `REFRESH ... INCREMENTAL` (strict) | SUCCESS, the row missing from the MV | FAILED, `IvmFallbackReason = CHANGELOG_BROKEN`, the MV keeps its rows | | `REFRESH ... AUTO` | SUCCESS with nothing left to read; the row stays missing | SUCCESS through COMPLETE, `IvmFallbackReason = CHANGELOG_BROKEN`, the MV matches the base table | | the next refresh, with the TTL back to a day | incremental; the row missing for good | incremental | | a user stream on the same table | reads from the first retained TSO | the same | Both an aggregate MV and an append-only detail MV over the table behave as above. **Classes, and how they call each other.** - `IvmIncrRefreshMTMV` → `IvmDeltaRewriteState.createDeltaScan` → `LogicalOlapTableStreamScan` → `NormalizeOlapTableStreamScan.makeTableStreamScan` → `makeIncrementalScanFromChangelog` → `applyChangelogTtl` → `ChangelogUtils.effectiveStartTso`, which now rejects for an IVM statement. - The rejection reaches `IvmIncrRefreshManager.doRefresh` → `IvmFailureClassifier.classifyExecutionFailure` → `CHANGELOG_BROKEN` → `MTMVTask.handleIvmFallbackResult` → strict: failure with the reason; otherwise `FALLBACK_TO_COMPLETE`. - The rebuild before the delta: `MTMVTask.executeIvmAttempt` → `executePartitionBasedRefresh` → on a failure whose reason `requiresCompleteRefresh()`, `handleIvmFallbackResult` with that reason. Not changed: user streams, direct incremental queries (`BindRelation.applyChangelogTtl`), the snapshot and reset reads, and anything persisted. A partition with no stream offset is a separate problem, fixed by #69778.

…inTableStreamPlanTest (#69809) ### What problem does this PR solve? Issue Number: None Related PR: #69778, #69785 Problem Summary: **In short:** the FE build on master fails, because `ExplainTableStreamPlanTest` now declares the same private method twice. This PR deletes the second copy; nothing else changes. #69778 and #69785 each added the same private helper `normalizeStreamRead(String, long, Optional<IvmRewriteContext>)` to `ExplainTableStreamPlanTest`, at different places in the file. Each PR compiled on its own, and they were merged 41 seconds apart without a rebase onto each other, so master has both copies, and javac rejects the file: ``` ExplainTableStreamPlanTest.java:843: error: method normalizeStreamRead(String,long,Optional<IvmRewriteContext>) is already defined in class ExplainTableStreamPlanTest ``` `build.sh --fe` passes `-DskipTests`, which skips running the tests but still compiles them, so this fails the whole FE build on master and on every PR based on it, not only the FE unit tests. The fix removes the copy #69785 added after its own tests. The two copies are identical, so the tests of both PRs keep calling the same code.

morningman requested review from 924060929, englefly, morrySnow and starocean999 as code owners October 28, 2026 14:06

morningman added the incremental-computation label Oct 28, 2026

github-actions Bot reviewed Oct 28, 2026

View reviewed changes

Comment thread fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/ExplainTableStreamPlanTest.java Resolved

morningman mentioned this pull request Oct 28, 2026

[fix](ivm) Reject an expired stream offset for an IVM read of a duplicate-key table #69785

Merged

yujun777 approved these changes Oct 29, 2026

View reviewed changes

morrySnow approved these changes Oct 29, 2026

View reviewed changes

morningman merged commit bc49fd1 into apache:master Oct 29, 2026
40 checks passed

morningman mentioned this pull request Oct 29, 2026

[fix](fe) Remove the duplicated normalizeStreamRead helper from ExplainTableStreamPlanTest #69809

Merged

3 of 16 tasks

morningman mentioned this pull request Oct 31, 2026

[Tracking] Changelog and IVM monitoring: metrics and status tables for table streams and IVM materialized views #69907

Open

6 of 13 tasks

morningman added the incremental-computation-picked label Nov 4, 2026

Conversation

morningman commented Oct 28, 2026

What problem does this PR solve?

Release note

Check List (For Author)

Uh oh!

hello-stephen commented Oct 28, 2026

Uh oh!

morningman commented Oct 28, 2026

Uh oh!

morningman commented Oct 28, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

morningman commented Oct 28, 2026

When does the problem this PR fixes show up? Three examples

Example 1: daily partitions, refreshed once a week

Example 2: refreshes interrupted for a few days

Example 3: a partition that existed, empty, when the MV was built; a partitioned (PCT) MV is affected too

When it does not happen

Uh oh!

hello-stephen commented Oct 28, 2026

Uh oh!

morningman commented Oct 28, 2026

Steps to reproduce

Uh oh!

hello-stephen commented Oct 28, 2026

Uh oh!

hello-stephen commented Oct 28, 2026

Uh oh!

hello-stephen commented Oct 28, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Oct 28, 2026

FE Regression Coverage Report

Uh oh!

yujun777 left a comment

Choose a reason for hiding this comment

Uh oh!

yujun777 Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants