Repository navigation
[Bug] (ivm) An expression over an aggregate result passes CREATE ... REFRESH INCREMENTAL, but every incremental refresh fails with PLAN_REWRITE_FAILED #68742
Description
Search before asking
- I had searched in the issues and found no similar issues.
Version
master @ 7ac0eb2 (2026-10-04). The IVM code used by the reproduction below is identical to this commit.
What's Wrong?
An IVM materialized view whose select list wraps an aggregate result in an expression, for example SUM(v) * 100, is accepted by CREATE MATERIALIZED VIEW ... REFRESH INCREMENTAL, and its first COMPLETE refresh succeeds. Every later incremental refresh fails:
IVM incremental refresh failed for mv=mv_expr_over_agg, reason=PLAN_REWRITE_FAILED, detail=IVM failed to find slot: sum(v)
A strict REFRESH ... INCREMENTAL task fails, and REFRESH ... INCREMENTAL FALLBACK falls back to COMPLETE on every run (IvmFallbackReason = PLAN_REWRITE_FAILED), so the MV is never maintained incrementally.
What You Expected?
Either the definition is maintained incrementally, or CREATE MATERIALIZED VIEW ... REFRESH INCREMENTAL rejects it at creation time, as it already does for other unsupported shapes (for example HAVING, which fails with IVM aggregate must be the top-level operator). Today it creates an MV whose incremental refresh can never succeed.
How to Reproduce?
FE configured with enable_feature_binlog = true and enable_table_stream = true.
CREATE TABLE t (
k INT,
g VARCHAR(16),
v INT
)
UNIQUE KEY(k)
DISTRIBUTED BY HASH(k) BUCKETS 1
PROPERTIES (
"replication_num" = "1",
"enable_unique_key_merge_on_write" = "true",
"binlog.enable" = "true",
"binlog.format" = "ROW",
"binlog.need_historical_value" = "true"
);
INSERT INTO t VALUES (1, 'a', 10), (2, 'a', 20), (3, 'b', 30);
CREATE MATERIALIZED VIEW mv_expr_over_agg
BUILD DEFERRED REFRESH INCREMENTAL ON MANUAL
DISTRIBUTED BY RANDOM BUCKETS 1
PROPERTIES ("replication_num" = "1")
AS
SELECT g, SUM(v) * 100 AS s100
FROM t
GROUP BY g;
REFRESH MATERIALIZED VIEW mv_expr_over_agg COMPLETE; -- SUCCESS
INSERT INTO t VALUES (4, 'b', 40);
REFRESH MATERIALIZED VIEW mv_expr_over_agg INCREMENTAL; -- FAILED, PLAN_REWRITE_FAILED
SELECT Status, RefreshMode, IvmFallbackReason, ErrorMsg
FROM tasks('type'='mv')
WHERE MvName = 'mv_expr_over_agg'
ORDER BY CreateTime DESC LIMIT 1;+--------+-------------+---------------------+----------------------------------------------------------------------------------------------------------------------------+
| Status | RefreshMode | IvmFallbackReason | ErrorMsg |
+--------+-------------+---------------------+----------------------------------------------------------------------------------------------------------------------------+
| FAILED | \N | PLAN_REWRITE_FAILED | IVM incremental refresh failed for mv=mv_expr_over_agg, reason=PLAN_REWRITE_FAILED, detail=IVM failed to find slot: sum(v) |
+--------+-------------+---------------------+----------------------------------------------------------------------------------------------------------------------------+
REFRESH MATERIALIZED VIEW mv_expr_over_agg INCREMENTAL FALLBACK then succeeds with RefreshMode = COMPLETE and IvmFallbackReason = PLAN_REWRITE_FAILED, and keeps doing so on every later refresh.
Anything Else?
Same steps (baseline COMPLETE, insert one row, strict INCREMENTAL) with other select lists:
| Select list | Incremental refresh |
|---|---|
SUM(v) * 100, CAST(SUM(v) AS DOUBLE), SUM(v) / COUNT(*) |
fails: IVM failed to find slot: sum(v) |
COUNT(v) + 1 |
fails: IVM failed to find slot: count(v) |
MIN(v) * 2, MAX(v) + 1 |
fails: IVM failed to find slot: min(v) / max(v) |
ARRAY_SIZE(COLLECT_LIST(v)) |
fails: IVM failed to find slot: collect_list(v) |
COUNT(*) * 2, AVG(v) * 2, ROUND(AVG(v), 2), BITMAP_UNION_COUNT(TO_BITMAP(v)) + 0 |
OK |
SUM(v) AS s, SUM(v) * 100 AS s100 (the bare aggregate is selected as well) |
OK |
SUM(v * 100) (the expression is inside the aggregate) |
OK |
UPPER(g) over a GROUP BY g key |
OK |
The failing functions are the ones whose visible MV column holds the aggregate state itself (SUM, COUNT(expr), MIN, MAX, COLLECT_LIST). COUNT(*), AVG and BITMAP_UNION_COUNT are maintained through hidden state columns and are not affected. The apply step appears to resolve the MV column by the aggregate's generated name (sum(v); the message comes from findSlotByName), and that column does not exist once the aggregate is wrapped in an expression.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
Doris-Breakwater commentedon Oct 5, 2026
Breakwater-GitHub-Analysis-Slot: slot_a493a51b0fe9
Triage: This is a reproducible IVM create/refresh contract gap on the stated
7ac0eb2fdb4revision. The supplied SQL and task output are sufficient for initial triage; I have not run a Doris cluster reproduction.Code evidence: Normalization permits a project above an aggregate and records each aggregate's own output slot as an
IvmAggTarget(normalization project, target construction). During incremental apply,IvmAggSumProcessorreads the old MV value using that target slot's name (SUM apply); the lookup searches the physical MV scan output and throws the reported error if the slot is absent (slot lookup). In this definition, the target issum(v)while the stored visible column is the outer expressions100, so the state the SUM apply step expects is unavailable. The issue's workingCOUNT(*)andAVGcases fit the code paths:COUNT(*)uses group count, andAVGreads hidden SUM/COUNT state (COUNT apply, AVG apply). This explains the failure mechanism for the reported case; the other listed expressions still need regression coverage.Recommended next step: Add create-time validation that every aggregate state the incremental apply processors read is actually persisted in the MV schema. Reject unsupported outer expressions with a clear analysis error unless the required aggregate state is also stored. If supporting these expressions, persist the missing aggregate states and recompute the outer expression from their updated values; simply treating
s100as the oldsum(v)would produce incorrect results. CoverSUM(v) * 100,COUNT(v) + 1,MIN/MAX,COLLECT_LIST, and the workingCOUNT(*)/AVG/bare-aggregate cases in create and strict incremental refresh tests. ACOMPLETErefresh (orINCREMENTAL FALLBACK, which the report shows choosesCOMPLETE) is the current operational workaround.Missing information: None needed to locate the reported planning failure. A maintainer reproducing it should capture
SHOW CREATE MATERIALIZED VIEW, the physical MV column list, andEXPLAIN REFRESH MATERIALIZED VIEW ... INCREMENTALalongside the task error to confirm the schema/plan boundary on the target build.[docs] Add verified IVM query examples, update IVM baseline rebuild, …
[docs](ivm) Document incremental refresh for expressions over aggrega…