Skip to content

perf(sql): avoid expensive row counting in EXPLAIN query with LIMIT#6540

Merged
bluestreak01 merged 93 commits intomasterfrom
mt_limit-improve
Dec 22, 2025
Merged

perf(sql): avoid expensive row counting in EXPLAIN query with LIMIT#6540
bluestreak01 merged 93 commits intomasterfrom
mt_limit-improve

Conversation

@mtopolnik
Copy link
Copy Markdown
Contributor

@mtopolnik mtopolnik commented Dec 16, 2025

In the current code, LimitRecordCursorFactory.toPlan() calls calculateSize() to resolve the effective LIMIT bounds. This executes the complete query. This PR lets toPlan() work with the information that's already available and never make the expensive call.

The PR also resolves other related issues with LimitRecordCursorFactory:

  • avoids calculateSize() in hasNext() when LIMIT bounds are non-negative (no need to count from the back)
  • avoids calculateSize() in size(), propagating -1 if the size isn't cheaply available

On the other hand, the new code calls baseCursor.size() eagerly in of(). This provides the invariant: "base row count missing indicates the expensive count is unavoidable".

The PR clears up the semantics of LIMIT, which are currently in contradiction with the specification in the docs.

Cleaned-up LIMIT semantics

The main complication in the semantics stems from LIMIT args being allowed to be both positive and negative, and even a mix of both. Here are the new rules (m and n are positive numbers, so -m is a negative number):

  • LIMIT n = LIMIT n, 0 = LIMIT 0, n = LIMIT n, = LIMIT , n: take first n rows
  • LIMIT m, n: take rows from the zero-based range [m, n) (left-inclusive, right-exclusive). If m > n, implicitly swap them.
  • LIMIT -n= LIMIT -n, 0 = LIMIT -n,: take last n rows
  • LIMIT -m, -n: take rows from the range [-m, -n), where -1 denotes the last row, -2 next-to-last, and so on. If m < n, implicitly swap them.
  • LIMIT m, -n: take rows from the range [m, -n). In other words, skip the first m rows, and drop the last n rows. These arguments will not be swapped.

Additional bugfixes

Since it touches LimitRecordCursorFactory.calculateSize() andskipRows(), this PR surfaced a number of bugs in these methods in other cursor classes. It fixes them.

skipRows() -> skipBaseRows()
skippedRows -> baseRowsToSkip
rowsToSkip -> pendingBaseRowsToSkip
It no longer resolves actual cursor size when LIMIT bounds aren't
negative.
@glasstiger
Copy link
Copy Markdown
Contributor

[PR Coverage check]

😍 pass : 199 / 201 (99.00%)

file detail

path covered line new line coverage
🔵 io/questdb/griffin/engine/LimitRecordCursorFactory.java 139 141 98.58%
🔵 io/questdb/griffin/engine/functions/date/GenerateSeriesTimestampRecordCursorFactory.java 4 4 100.00%
🔵 io/questdb/griffin/engine/table/AsyncFilteredRecordCursor.java 1 1 100.00%
🔵 io/questdb/griffin/engine/groupby/vect/GroupByNotKeyedVectorRecordCursorFactory.java 8 8 100.00%
🔵 io/questdb/griffin/SqlCodeGenerator.java 2 2 100.00%
🔵 io/questdb/griffin/engine/table/PageFrameRecordCursorImpl.java 16 16 100.00%
🔵 io/questdb/griffin/engine/table/AsyncTopKRecordCursor.java 10 10 100.00%
🔵 io/questdb/griffin/engine/functions/date/GenerateSeriesTimestampStringRecordCursorFactory.java 4 4 100.00%
🔵 io/questdb/griffin/engine/functions/table/ReadParquetPageFrameCursor.java 1 1 100.00%
🔵 io/questdb/griffin/engine/functions/date/GenerateSeriesLongRecordCursorFactory.java 4 4 100.00%
🔵 io/questdb/cairo/RecordArray.java 1 1 100.00%
🔵 io/questdb/griffin/engine/groupby/GroupByNotKeyedRecordCursorFactory.java 9 9 100.00%

@bluestreak01 bluestreak01 merged commit f4183e7 into master Dec 22, 2025
43 checks passed
@bluestreak01 bluestreak01 deleted the mt_limit-improve branch December 22, 2025 13:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants