Repository navigation
[feature](ivm) Track MIN/MAX extreme counts for incremental refresh - #69286
Merged
Merged
Conversation
Add aggregate functions that return how many input rows equal the group's minimum or maximum value. The state keeps the current extreme and its multiplicity, so a single pass produces the count. NULL inputs are ignored and empty or all-NULL groups return 0. The functions accept the same argument types as MIN/MAX. Key changes: - Add AggregateFunctionsCountAtExtreme with min/max state wrappers. - Register count_at_min and count_at_max in the aggregate function factory. - Add value-equality helpers to the single-value state types. - Add BE unit coverage for duplicates, empty input, strings, and merges. - Include runtime/exec_env.h in function_dict_get_many.cpp; the new source changed the unity batch and exposed the missing include. Unit Test: - Added be/test/exprs/aggregate/agg_count_at_min_max_test.cpp. The local unit-test target could not link because the toolchain already provides glibc-compatibility symbols; the main BE build succeeded.
Expose COUNT_AT_MIN and COUNT_AT_MAX as aggregate functions returning BIGINT: the number of input rows equal to the group's minimum or maximum value. NULL inputs are ignored, empty or all-NULL input returns 0, and DISTINCT is rejected because it carries no useful meaning for this function. Key changes: - Add CountAtMin and CountAtMax expression classes with a BIGINT return type and an (extreme, count) intermediate state. - Register both functions in BuiltinAggregateFunctions. - Add default AggregateFunctionVisitor methods. Unit Test: - Verified with SQL on the local cluster, including duplicate extremes, an all-NULL input, and DISTINCT rejection. - FE build and checkstyle pass.
Store how many rows equal the current MIN/MAX in a hidden MV column and use it during incremental refresh. The delta plan now emits the extreme count for both the insert and delete sides. Apply keeps the old extreme while its surviving count is positive; deleting a duplicate extreme therefore stays incremental. Deleting the last row at the extreme still trips the existing boundary guard and falls back to a complete refresh. Key changes: - Add MIN_COUNT and MAX_COUNT hidden state keys and delta slots. - Emit COUNT_AT_MIN/COUNT_AT_MAX over the insert- and delete-only arguments. - Replace the deleted-extreme guard with the surviving-count calculation. - Keep the apply predicates non-NULL when an insert or delete side is empty. - Update IVM tests for the new hidden columns and delta outputs. Unit Test: - IvmAggMinProcessorTest and IvmAggMaxProcessorTest pass. - IvmAggDeltaHandlerTest and IvmNormalizeUniqKeysMTMVTest pass.
Cover deleting a duplicate minimum or maximum, deleting the last row at the extreme, and inserting a smaller value. The duplicate and smaller-insert cases must stay incremental; deleting the last extreme must keep the complete-refresh fallback. Key changes: - Add test_ivm_minmax_count.groovy and its generated .out file. Unit Test: - run-regression-test.sh -d mtmv_p0/ivm -s test_ivm_minmax_count
yujun777
requested review from
924060929,
englefly,
morrySnow and
starocean999
as code owners
October 19, 2026 12:07
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Request changes: one P1 data-correctness defect can leave an IVM MIN/MAX stale after a later refresh, and two P2 defects make the new tests fail or inspect the wrong task. All 23 authoritative changed paths were reviewed at c5d12fa. There were no existing review threads or additional user focus points.
The reviewed execution flow is normalization into visible extrema plus hidden counts, signed stream/join delta aggregation, a join to persisted MV state, guarded state application, and the existing transactional MV refresh. BE partial aggregation carries both the extreme and its count through serialization and merge.
Critical-checkpoint conclusions:
- Goal and evidence: Duplicate-extreme deletion is intended to remain incremental. The new persisted count enables it, but canceled join contributions in an old-NULL group corrupt that count and can later retain a deleted extreme (P1). New FE/BE/regression cases exercise ordinary shapes; evidence here is static.
- Scope and reuse: The change is focused and shares MIN/MAX logic and existing SingleValueData ownership/comparison implementations. The dictionary change supplies an existing ExecEnv use with a concrete implementation-file include.
- Concurrency and locks: No new production threads, shared mutable state, locks or lock ordering. Aggregate states are per group and FE rewrite contexts are statement-local. The regression task poll introduces a real asynchronous lifecycle race (P2).
- Lifecycle and static initialization: State create/reset/serialize/deserialize/merge/destroy and nullable-wrapper lifecycles were checked, including empty partials, strings and complex columns. Existing call_once registration remains ordered before combinator registration; no new cross-translation-unit initialization dependency was found.
- Configuration: None added or changed; dynamic configuration is inapplicable.
- Compatibility: Hidden IVM schema gains MIN_COUNT/MAX_COUNT; the IVM guide explicitly exempts these pre-January-2027 layouts from migration compatibility. Existing ordinary MIN/MAX serialized layouts remain unchanged.
- Thrift/protobuf: No definitions, ids, enum values or required fields changed.
- Parallel paths: MIN and MAX have symmetric affected logic; nullable/nonnullable, scalar/grouped, repeated/shared aggregate state, initial/complete/incremental refresh and shared cloud execution paths were examined.
- Conditional checks: Last-extreme loss retains the existing assertion/fallback mechanism for sound counts. The old-NULL initializer violates that premise in the reported two-refresh scenario. Missing delta slots still fail explicitly.
- Test coverage: BE cases cover integer/string/empty/direct merge and FE cases check normalized schema and delta/apply expression shapes. The end-to-end case covers duplicate deletes, last-extreme fallback and better inserts. Direct nullable/serialized/long-string/complex fixtures and a two-refresh canceled-join case are missing.
- Expected results: All new regression output rows are arithmetically consistent with the DML; the BE merge expectation incorrectly requires 5 instead of 2 (P2). Output-generation provenance cannot be proved by static inspection.
- Observability: Existing MIN_MAX_BOUNDARY_HIT fallback classification and task metadata remain available. No separate new production observability obligation was identified.
- Transactions and persistence: New state is carried in normalized MV columns through existing refresh persistence. No edit-log, failover, stream-offset or transaction protocol change was introduced.
- Writes, atomicity and crashes: Refresh continues through the existing transaction/upsert path. Version visibility and delete-bitmap handling are unchanged; the P1 concerns the values committed by that path. No additional crash/resource-release defect was found statically.
- FE/BE contract: Function names, BIGINT nonnullable results, zero-on-empty behavior and string partial-state lowering agree. Constants are materialized before concrete nullable access; both null wrappers skip NULL rows and retain initialized zero counts. No new transmitted variable is introduced.
- Performance: Added counts require the intended aggregate work and per-group state. No independently substantiated additional hot-path, locking or allocation regression was found.
- Other issues: The regression poll can accept the preceding terminal task during the current task's running-to-history gap; an existing task-identity-aware wait should be reused (P2).
No unresolved review candidate or uncovered changed path remains. This was a static review: no builds, tests, source edits or runtime validation were performed under the review contract. The absent test scenarios above remain validation limits.
Review converged after 2 round(s).
| Expression newCountRaw = new CaseWhen( | ||
| ImmutableList.of( | ||
| new WhenClause(new EqualTo(newTotalCount, zero), zero), | ||
| new WhenClause(new IsNull(oldExtreme), deltaInsertCount), |
Contributor
There was a problem hiding this comment.
[P1] Subtract matching deletes when initializing the extreme count
When oldExtreme is NULL, persisting deltaInsertCount ignores canceled delete contributions. A reachable plan is Aggregate(MIN(L.v)) -> InnerJoin(L.k = R.k): initially L has (k=0,v=NULL) and R has keys 0,1,2,3. Before refresh, insert L rows (1,1),(2,1),(3,2) and delete R key 1. The join rewrite emits +{1,1,2} from leftDelta JOIN rightPre and -{1} from leftPost JOIN rightDelta. The resulting MIN is correctly 1, but this branch stores MIN_COUNT=2 instead of 1. On the next refresh, deleting L key 2 leaves only non-NULL value 2; survivingOldCount=2-1 stays positive, so the guard silently retains MIN=1 instead of falling back. This later wrong result is introduced by the new count state; the previous guard would reject that boundary deletion. Initialize the count from net contributions at the selected extreme and preserve fallback when its survival cannot be proved. MAX has the symmetric failure.
Contributor
Author
There was a problem hiding this comment.
Fixed in 87a8d55. When the old extreme is NULL, the count now comes from the net insert/delete contributions at the insert-side extreme: subtract deltaDeleteCount when the delete-side extreme equals it, use deltaInsertCount when the delete extreme is farther from the aggregate extreme, and keep the complete-refresh fallback when a delete is closer or cancels every inserted row. I also added the two-refresh canceled-join regression test_ivm_minmax_count_join, which reproduced the stale MIN=1 behavior before the fix.
|
|
||
| ColumnInt64 result; | ||
| function->insert_result_into(lhs, result); | ||
| EXPECT_EQ(5, result.get_element(0)); |
Contributor
There was a problem hiding this comment.
[P2] Expect two minima in the merge fixture
This fixture selects count_at_min over partial inputs {5,1,5} and {5,1,9}. Each partial has minimum 1 with count 1, so merging equal extrema correctly returns 2. EXPECT_EQ(5, ...) therefore fails deterministically with the implementation in this PR. Expect 2, or change the fixture to contain five occurrences of the global minimum.
Contributor
Author
There was a problem hiding this comment.
Fixed in 87a8d55. The merge fixture now expects 2: each partial has minimum 1 with count 1, so merging equal extrema returns 2.
| """) | ||
| return !taskResult.isEmpty() | ||
| && taskResult[0].Status.toString() != 'PENDING' | ||
| && taskResult[0].Status.toString() != 'RUNNING' |
Contributor
There was a problem hiding this comment.
[P2] Wait for the submitted refresh task before checking fallback
This predicate accepts any latest terminal row without checking its TaskId. During MTMVTask.onSuccess, the current task is removed from the running list before after() publishes it into MV history, so tasks() can temporarily return the preceding completed refresh. At the last-min deletion, this helper can then return the previous INCREMENTAL task and fail the expected COMPLETE assertion; after the smaller insert, it can return the previous fallback task and fail assertNoFallback. Suite.pollMTMVTaskTerminal already handles this exact gap. Reuse that wait logic and inspect the same completed TaskId, or capture the previous TaskId and wait for a distinct terminal task.
Contributor
Author
There was a problem hiding this comment.
Fixed in 87a8d55. Both MIN/MAX count suites now remember the previous TaskId and only accept a terminal row whose TaskId differs, so the running-list-to-history gap can no longer return the preceding refresh. The new join suite uses the same wait.
When the old extreme is NULL, the insert-side extreme became the new extreme but its stored count ignored deletes emitted for the same value by the same join refresh. The inflated MIN_COUNT/MAX_COUNT could later keep a deleted extreme instead of falling back. Initialize the count from the net insert/delete contributions at the selected extreme, and keep the complete refresh fallback when the surviving count cannot be proved. Key changes: - Compute the old-NULL count as insert count minus matching delete count; fall back when a delete is closer to the extreme or cancels every inserted row. - Add the MIN/MAX beyond-insert comparison and cover the net-count expression in the FE processor tests. - Fix the BE merge fixture to expect the two merged minima. - Make both MIN/MAX count regression suites wait for a task id distinct from the previous refresh. Unit Test: - IvmAggMinProcessorTest, IvmAggMaxProcessorTest, IvmAggDeltaHandlerTest, IvmNormalizeUniqKeysMTMVTest - regression-test: mtmv_p0/ivm test_ivm_minmax_count, test_ivm_minmax_count_join
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
There was a problem hiding this comment.
No new actionable finding in the static review of head 87a8d552ab57c552322af4ce4a77ca290d6829ac. All 26 changed paths were covered by the main review, two complete-scope reviewers, and a separate join-cancellation risk reviewer. All three reviewers returned NO_NEW_VALUABLE_FINDINGS; no candidate or risk remains unresolved.
The three existing findings are fixed at this head:
- Old-NULL extreme-count cancellation: matching deletes now reduce the inserted extreme count; unprovable or exhausted extrema retain complete-refresh fallback. The original two-refresh example now persists count 1, then falls back when that last instance is deleted. MAX has the symmetric correction.
- BE merge fixture: the expected count is now 2, matching the two input minima.
- Refresh-task history race: both suites require a distinct terminal TaskId before checking refresh mode/reason.
Critical checkpoint conclusions:
- Goal and supporting tests: hidden extreme counts allow duplicate-boundary deletions to remain incremental and preserve last-boundary fallback. Added BE/FE unit cases and regression scenarios support those paths by inspection.
- Scope and reuse: the change is focused and reuses SingleValueData, nullable adapters, shared extremal processing and existing normalization hooks. The dictionary change supplies an existing ExecEnv dependency.
- Concurrency and locks: no new thread, shared mutable processor state or lock-order edge. Aggregate states follow evaluator ownership; existing MV job locking serializes refreshes. Task-history synchronization is corrected.
- Lifecycle and initialization: add/reset/merge/serialization/destruction and fixed/string/column ownership are consistent. No new cross-translation-unit static initialization dependency was found.
- Configuration: no new configuration item.
- Compatibility: existing public aggregate formats are unchanged. New function names require supporting BE binaries; prerelease IVM hidden-layout migration is explicitly exempted by the IVM guide.
- Thrift/protobuf: no definition or field ID changes.
- Parallel paths: MIN/MAX, scalar/grouped states, nullable adapters, distributed merge, analytic and aggregate-state adapters were checked. No missing counterpart was identified.
- Special conditions: NULL-only/empty groups and signed join cancellation are handled conservatively. The non-NULL old-boundary guard prevents deletes from canceling a selected better inserted extreme; otherwise the exact surviving old count must remain positive.
- Test coverage: added cases cover duplicate MIN/MAX deletion, empty/string/merge inputs, hidden-state/delta structure, better inserts, last-boundary fallback and the two-refresh canceled-join case. Tests were inspected, not executed.
- Expected results: changed regression output rows and the BE merge expectation agree with the input operations. Output-generation provenance cannot be established from file inspection.
- Observability: the existing boundary error prefix, MIN_MAX_BOUNDARY_HIT classification, task fallback fields and MV/task/query logging remain usable.
- Transactions, persistence and failover: counts persist as ordinary MV columns alongside visible values; existing transaction/stream-offset and metadata persistence paths remain in use. No new independent write phase or EditLog field is introduced.
- Writes, atomicity and crashes: execution/precommit errors prevent commit and enter ordinary abort handling before permitted fallback. No new partial-count write or resource-leak path was identified statically; crash behavior was not exercised.
- FE/BE handoff: registrations, lowercase names, nonnullable BIGINT results and string-serialized intermediate state agree; no new protocol variable is added.
- Performance: the change adds bounded per-row counter/comparison work and per-group state, preserving shared COUNT deduplication. No additional source scan or confirmed asymptotic regression was found; no benchmark was run.
- Other checks: optimizer slot/state retention, empty-input handling, constant materialization, memory ownership and BE source/test inclusion were checked without an additional defect.
User focus: no additional focus points were supplied. The prior review concerns received explicit revalidation above.
Validation limits: this review performed no builds, tests or source modifications. Added runtime cases are not exhaustive for MAX join cancellation, outer joins, long-string/complex serialization and reset; those mechanisms were reviewed statically. No changed path or concrete candidate remains uncovered.
Review converged after 1 round(s).
Contributor
TPC-H: Total hot run time: 28306 ms |
Contributor
TPC-DS: Total hot run time: 150451 ms |
Contributor
ClickBench: Total hot run time: 24.09 s |
Contributor
TPC-H: Total hot run time: 28165 ms |
Contributor
TPC-DS: Total hot run time: 150262 ms |
Contributor
ClickBench: Total hot run time: 24.32 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
3 similar comments
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
Add a query regression suite for the public COUNT_AT_MIN/COUNT_AT_MAX aggregates so the SQL-level registration, type handling and NULL semantics are covered in addition to the BE aggregate unit test. Key changes: - Cover grouped BIGINT, STRING, DECIMAL and DATETIME arguments, duplicate extremes, NULL rows and an all-NULL group. - Cover scalar aggregation, empty input returning zero, and DISTINCT rejection for both functions. Unit Test: - regression-test: query_p0/aggregate count_at_min_max
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
TPC-H: Total hot run time: 28259 ms |
Contributor
TPC-DS: Total hot run time: 150979 ms |
Contributor
ClickBench: Total hot run time: 24.13 s |
The foreach combinator passes result_is_nullable=true for its nested aggregate. count_at_min/count_at_max are non-nullable: an all-NULL array position must return 0, but the nullable-result wrapper turned it into NULL. Create the nested aggregate with the non-nullable unary creator so the count state's zero result survives the wrapper. Key changes: - Use create_unary_arguments_return_not_nullable in every count_at_min/max type branch. - Cover count_at_min_foreach and count_at_max_foreach with an all-NULL array position under both the default V2 and old nullable adapters. Unit Test: - regression-test: query_p0/aggregate count_at_min_max
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
|
Codex automated review failed and did not complete. Error: Review context preparation failed before Codex ran; inspect the 'Prepare authoritative PR context and required AGENTS guides' step. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
TPC-H: Total hot run time: 28433 ms |
Contributor
TPC-DS: Total hot run time: 150970 ms |
Contributor
There was a problem hiding this comment.
Reviewed all 30 changed paths at 008d08b586b9e981e19d07e8e5ca01d63f244045. No new actionable defect was substantiated by this static review. The review converged in 1 round: both complete-scope reviewers and the separate risk reviewer returned NO_NEW_VALUABLE_FINDINGS, with every risk and candidate disposed and all changed paths covered.
The existing old-NULL joined-delta P1, merge fixture, refresh task wait, and foreach zero-result issues are fixed for their reported scenarios on this head. The P1 case now stores the net count at the inserted extreme and falls back when its remaining count reaches zero. No existing P0/P1 finding was independently confirmed to remain applicable, and none was reposted. No additional user focus was supplied.
Required checkpoint conclusions:
| Checkpoint | Conclusion |
|---|---|
| Goal and supporting coverage | Hidden MIN_COUNT/MAX_COUNT preserves duplicate extremes on incremental deletion; last-extreme deletion retains complete fallback. New grouped and joined regression cases express both behaviors. |
| Scope and reuse | Changes reuse the shared extremal processor, SingleValueData, aggregate factory and existing normalize/apply infrastructure. The dictionary change adds its directly used include. |
| Concurrency and locks | New aggregate states are execution-local; processor metadata is immutable. No new thread, lock acquisition, or lock-order edge was introduced. Factory registration retains call_once. |
| Lifecycle and initialization | Value/count create, reset, merge, serialization and destruction were traced, including owned strings and shared immutable one-row columns. No new cross-TU static initialization dependency was found. |
| Configuration | No new configuration was added. Both existing nullable-adapter modes were checked. |
| Compatibility | Existing aggregate serialization remains unchanged. New hidden IVM layout follows the applicable AGENTS waiver for pre-January-2027 IVM compatibility. New functions use generic aggregate transport. |
| Thrift/protobuf fields | No gensrc/thrift or gensrc/proto change; the cross-branch field-ID gate does not apply. |
| Parallel paths | MIN/MAX, scalar/grouped, full/incremental, old/V2 nullable, const materialization, foreach/state and direct windows were inspected. Nonnullable window results preserve zero for empty frames. |
| Conditional guards | Equal delete contributions reduce counts. Accepted branches prove the selected extreme survives; unprovable cases retain the existing assertion-driven fallback. |
| Test coverage | BE core fixtures, FE plan/state assertions and SQL regression suites were reviewed. Focused runtime cases for reset/serialization, long strings, complex arrays, NaNs and windows remain coverage limitations. |
| Changed expected results | All five changed result files agree with their SQL data or normalized hidden-column schema by static inspection, including the corrected merge count. |
| Observability | Existing boundary assertion text maps to MIN_MAX_BOUNDARY_HIT and task fallback diagnostics. No new distributed stage requiring separate observability was introduced. |
| Persistence and failover | Count state uses existing MV columns and INSERT persistence. Snapshot/epoch publication remains success-only; no new EditLog payload or recovery protocol was added. |
| Writes and atomicity | Count and visible extreme use the same established MV sink/transaction. Guard failure follows existing execution failure and fallback handling. |
| FE/BE propagation | Lowercase names, argument types/decimal coercion, nonnullable BIGINT results and generic String intermediate transport agree. No separate new wire variable was added. |
| Performance | Adds two count aggregates and one persisted count per extremal target while retaining bounded per-state ownership and the existing source scan. No additional material performance defect was substantiated statically. |
| Further missed-case sweep | Rechecked signed join cancellations, repeated refreshes, shared/dropped state, NULL/new/deleted groups, NaN equality and empty/sliding windows; no unresolved static correctness candidate remains. |
Validation and limits: static analysis only; no Doris build, tests, formatting checks, cluster execution or benchmark was run, as required by the review contract. Expected-output generation provenance was not independently established. A separate finite mathematical model, inspected and reproduced by the main reviewer, matched both extreme and count for all 414,230 accepted MIN/MAX states over values NULL/0/1/2 and multiplicities 0..2; this supports the branch analysis and is not Doris runtime validation. All assigned static scope is covered; the runtime and test-coverage limits above remain.
Review converged after 1 round(s).
Contributor
ClickBench: Total hot run time: 24.36 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
morrySnow
approved these changes
Oct 20, 2026
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
|
PR approved by anyone and no changes requested. |
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
morningman
added a commit
to morningman/doris
that referenced
this pull request
Oct 20, 2026
… count suites ### What problem does this PR solve? Issue Number: None Related PR: apache#69286 Problem Summary: apache#69286 added test_ivm_minmax_count and test_ivm_minmax_count_join after this PR was opened. Their base tables still enable row binlog the old way: "binlog.enable" = "true", "binlog.format" = "ROW", "binlog.need_historical_value" = "true" With this PR, CREATE TABLE rejects those keys ("Property binlog.format is no longer supported, use "changelog.enable" = "true" to enable changelog, or remove it for CCR binlog"), so both suites would fail before their first refresh. Switch them to "changelog.enable" and "changelog.need_historical_value", as this PR does for the other IVM suites. ### Release note None ### Check List (For Author) - Test: No need to test (regression suites only) - The two suites were not run locally. The change is the same property rewrite this PR applies to the other suites under mtmv_p0/ivm. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
5 of 16 tasks
morningman
added a commit
to morningman/doris
that referenced
this pull request
Oct 20, 2026
… count suites ### What problem does this PR solve? Issue Number: None Related PR: apache#69286 Problem Summary: apache#69286 added test_ivm_minmax_count and test_ivm_minmax_count_join after this PR was opened. Their base tables still enable row binlog the old way: "binlog.enable" = "true", "binlog.format" = "ROW", "binlog.need_historical_value" = "true" With this PR, CREATE TABLE rejects those keys ("Property binlog.format is no longer supported, use "changelog.enable" = "true" to enable changelog, or remove it for CCR binlog"), so both suites would fail before their first refresh. Switch them to "changelog.enable" and "changelog.need_historical_value", as this PR does for the other IVM suites. ### Release note None ### Check List (For Author) - Test: No need to test (regression suites only) - The two suites were not run locally. The change is the same property rewrite this PR applies to the other suites under mtmv_p0/ivm. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
hello-stephen
pushed a commit
that referenced
this pull request
Oct 30, 2026
…tions (#69456) ### What problem does this PR solve? Issue Number: N/A (tracked in #65418) Related PR: #69286 Problem Summary: Adds the generic building blocks that incremental MIN/MAX maintenance builds on; the IVM integration itself is a separate PR. - `min_topk(x, k)` / `max_topk(x, k)`: an aggregate returning `ARRAY<STRUCT<value T, count BIGINT>>` — the k smallest (largest) distinct non-NULL values with their exact row counts, ordered by value. The array carries no truncation flag: a caller that needs one infers it from the length, treating a list of exactly k entries as possibly truncated. With `k = 0` the array holds at most the single `(extreme, count)` pair. `k` is a constant `BIGINT` with `0 <= k <= 65536`. - `array_sorted_merge(merge_fn, arr1, arr2, is_asc)`: a generic higher-order function that merges two arrays already sorted by the elements' natural order. `merge_fn` receives the two current candidates and returns the combined element, or NULL when the two elements must be kept separately. `is_asc` states the sort direction, so one function covers ascending and descending inputs, and calls can be chained to merge more than two arrays. The aggregates keep a sorted `(value, count)` state with checked int64 counting, so partial aggregation and merge stay exact (`topK(A ∪ B) ⊆ topK(A) ∪ topK(B)`). FLOAT/DOUBLE order NaN as greater than every number so the distinct list keeps a strict order. Returning the array alone keeps the aggregate in the same shape as the other array aggregates and leaves the truncation flag to the caller that actually persists a bound: incremental MIN/MAX maintenance still stores one, but as its own state column rather than inside the aggregate result. `array_sorted_merge` keeps its non-lambda `is_asc` argument through the higher-order binding: the existing pipeline passed only the lambda arguments to the bound function, so the binding contract gained a way to return extra inputs. ### Release note Add the `min_topk`/`max_topk` aggregate functions and the `array_sorted_merge` scalar function.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Incremental refresh of a MIN/MAX materialized view currently falls back to a
complete refresh whenever a deleted row could be the current extreme. Deleting
one of several rows that share the minimum value is a common case, yet the MV
does not remember how many rows hold that extreme, so the refresh cannot tell
that the extreme survives.
This PR adds
COUNT_AT_MIN/COUNT_AT_MAXand stores that count as a hiddenIVM state column. The delta plan now emits the count at the insert-side and
delete-side extreme. Apply keeps the old extreme while its surviving count is
positive; only deleting the last row at the extreme still falls back to a
complete refresh.
Trace issue: #65418
What changed?
COUNT_AT_MIN(expr)/COUNT_AT_MAX(expr)aggregate functions.They return the number of input rows equal to the group's minimum/maximum,
ignore NULLs, return 0 for empty or all-NULL input, and reject DISTINCT.
They are equivalent to Oracle's
COUNT(*) KEEP (DENSE_RANK FIRST/LAST ORDER BY expr).MIN_COUNT/MAX_COUNTIVM hidden state keys and hidden columns namedin the existing
__DORIS_IVM_AGG_{ordinal}_{STATE}_COL__format.count state.
Testing
IvmAggMinProcessorTest,IvmAggMaxProcessorTest,IvmAggDeltaHandlerTest,IvmNormalizeUniqKeysMTMVTest.test_ivm_minmax_count, plus existingtest_ivm_minmax_runtime_fallback,test_ivm_agg_1,test_ivm_agg_2, andtest_ivm_agg_shared_columns.rejection.
pass.
target could not link in this environment because the toolchain already
provides glibc-compatibility symbols; the main BE build succeeded.