Skip to content

[core] Read single-run sections of aggregation tables without sort merge - #1370

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JeonDaehong:perf-agg-single-run
Nov 11, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
JeonDaehong:perf-agg-single-run

Conversation

@JeonDaehong

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: none, follows #1164

#1164 reads a single-run section without the sort merge for deduplicate and first-row. Aggregation can take the same path: when a key has one version, the merge function already returns that row as is (singleton_reducer_result) and omits a lone retract, which is what the unmerged path does per batch. The exception is changelog-producer = lookup, where aggregation reduces singletons too, so it keeps merging.

Brief change log

  • The single-run check also accepts aggregation unless changelog-producer = lookup.
  • reduces_singletons(options) holds that rule, and the three merge functions that had it inline use it.

PK table, 2M keys in 4 files with disjoint keys, sum on one column, full read (release, in-memory FileIO): 187–194 → 89–90 ms, the same as deduplicate. Measured in one binary with the merge path switched back by an env var, 3 alternating rounds. With overlapping rounds (the merge still runs) and for deduplicate, times are unchanged.

Tests

  • aggregation_reads_disjoint_files_without_merge_and_overlaps_still_merge: disjoint files read as written, an overlapping file still sums.
  • single_run_read_merges_repeated_keys_under_aggregation: a pre-fix(core): align evolution stats and aggregation flush with Java #839 file that repeats a key still sums.
  • single_run_read_drops_retracts_hidden_by_legacy_metadata gets an aggregation case.

The new tests also pass on main, so the output matches the merge path. Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, integration, datafusion, python, go).

API and Format

None.

Documentation

None.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit ed80e07 into apache:main Nov 11, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants