You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit afb1601
Browse filesBrowse the repository at this point in the historyBrowse files
[Improvement](scan) update limit pushdown to tablet reader (apache#61713)
This pull request implements a general limit pushdown optimization for
storage-layer scans on DUP_KEYS and UNIQUE_KEYS (with Merge-on-Write,
MOW) tables. The optimization ensures that LIMIT clauses are enforced
after filtering, preventing the scan from returning fewer rows than
requested when filters are present. The change is carefully scoped to
avoid interfering with existing optimizations (like TopN or storage
merge paths) and includes comprehensive regression tests to validate
correctness across various scenarios.
**General Limit Pushdown Implementation:**
* Added a `general_read_limit` parameter to the storage layer
(`TabletReader`, `RowsetReaderContext`, and `VCollectIterator`) to
enforce a post-filter row limit in non-merge scan paths for DUP_KEYS and
UNIQUE_KEYS with MOW. The iterator now tracks the number of rows
returned and stops reading once the limit is reached, ensuring correct
LIMIT semantics even in the presence of filters.
[[1]](diffhunk://#diff-4d3f2b2a64e1dac3bc076994d3cd9a6c569f7a8c6ad5bc69425188ffccb4266bR494-R503)
[[2]](diffhunk://#diff-4d3f2b2a64e1dac3bc076994d3cd9a6c569f7a8c6ad5bc69425188ffccb4266bR518-R531)
[[3]](diffhunk://#diff-88c1391d53c67a5ab8475c103bb743a2a80cc282e0c752e8fc52b7aaeb2e0e35L95-R102)
[[4]](diffhunk://#diff-88c1391d53c67a5ab8475c103bb743a2a80cc282e0c752e8fc52b7aaeb2e0e35R258-R294)
[[5]](diffhunk://#diff-b7a74d17e270719c904e8c31fb1308cb82ec0b89459a245dce9cb5505021ac63R353-R358)
[[6]](diffhunk://#diff-0025ea2b49c400923787cd77990575f94ba5d925efc95cb16336fbaec92ebfb1R106-R109)
[[7]](diffhunk://#diff-979b99e23ca7a4bb73f5beaa00270494778545296929a3de64b2e87dc8299cd6R202-R203)
[[8]](diffhunk://#diff-48c14ad6bacc0a0ed497ad0cde34756af85aa98dcb7dc909609a288a503be0c9R216-R220)
**Testing and Validation:**
* Added a new regression test suite
(`test_general_limit_pushdown.groovy`) covering DUP_KEYS, UNIQUE_KEYS
(MOW), AGG_KEYS, and MOR tables. The tests verify correct row counts and
output for LIMIT queries with and without filters, as well as negative
cases where the optimization must not apply.
* Added corresponding expected output results for the new tests.
0 commit comments