Skip to content

feat(sql): add dynamic window support in WINDOW JOIN#6859

Merged
bluestreak01 merged 22 commits intomasterfrom
puzpuzpuz_dynamic_window_join
Mar 17, 2026
Merged

feat(sql): add dynamic window support in WINDOW JOIN#6859
bluestreak01 merged 22 commits intomasterfrom
puzpuzpuz_dynamic_window_join

Conversation

@puzpuzpuz
Copy link
Copy Markdown
Contributor

@puzpuzpuz puzpuzpuz commented Mar 9, 2026

WINDOW JOIN now accepts column references and expressions as RANGE BETWEEN boundaries, in addition to static constants. This allows each left-hand-side (LHS) row to define its own window size based on its data.

Example:

SELECT t.ts, sum(d.val) AS agg
FROM fx_trades t
WINDOW JOIN market_data d
RANGE BETWEEN t.lookback minutes PRECEDING AND t.lookahead minutes FOLLOWING;

Either or both of the lo/hi boundaries can be dynamic. The other can remain a static constant.

Boundary expressions must evaluate to an integer and must only reference LHS table columns. Negative values are clamped to zero (equivalent to CURRENT ROW). NULL values produce NULL aggregates — the row is skipped.

Each boundary, whether static or dynamic, can optionally include a time unit suffix (seconds, minutes, hours, etc.). When a time unit is present, the value is scaled to the LHS table's timestamp resolution at runtime (microseconds or nanoseconds). When the time unit is omitted, the raw integer value is interpreted in the LHS table's native timestamp resolution.

Dynamic windows disable the fast (symbol-keyed) and vectorized execution paths. Queries with an ON key equality clause fall back to the general path with a join filter instead.

Also, makes parallel HORIZON and WINDOW JOIN queries more responsive to query cancellation.

Technical details

  • SqlOptimiser.validateWindowJoins uses tryEvalNonNegativeLongConstant to distinguish static vs dynamic bounds. Column references and non-constant expressions fail to compile against EmptyRecordMetadata and are classified as dynamic. resolveWindowJoinBoundColumns walks the expression tree to strip LHS table prefixes and reject right-hand-side (RHS) table column references.
  • SqlCodeGenerator compiles dynamic bound functions against LHS metadata, computes sign/timeUnit parameters, and creates per-worker function copies for thread-unsafe expressions via compileWorkerFunctionsConditionally.
  • AsyncWindowJoinRecordCursorFactory implements 6 new reducer methods for dynamic windows using a two-pass algorithm: the first pass computes overall RHS bounds across all LHS rows in a page frame, the second pass does per-row binary search within the pre-scanned data. computeEffectiveBound evaluates the dynamic function per LHS row, handles NULL and negative clamping.
  • WindowJoinRecordCursorFactory (single-threaded path) integrates computeEffectiveBound for per-row bound computation.
  • WindowJoinTimeFrameHelper supports non-monotonic access patterns from dynamic windows via bookmark-based navigation.
  • Each async WINDOW JOIN and HORIZON JOIN reduce method receives a circuitBreaker parameter but only uses it for slot acquisition, never inside the per-row processing loops. When a single page frame contains many rows, the worker thread runs uninterruptibly until the entire frame is processed, making the query unresponsive to cancellation. The fix adds circuitBreaker.statefulThrowExceptionIfTripped() at the top of every master-row iteration loop in all four async factories, so that cancellation, timeout, and broken connection signals are detected promptly during heavy frame processing.

Test plan

  • Deterministic tests for dynamic lo, dynamic hi, both dynamic, mixed static+dynamic, expression-based bounds, NULL bounds, negative bounds, non-monotonic bounds, bound column not in SELECT, error cases (non-integer bound, RHS column reference), fast path fallback
  • All deterministic tests cover both async and single-threaded (via LIMIT) paths, with and without INCLUDE PREVAILING
  • Parallel fuzz tests with thread-unsafe (varchar-to-long cast) bound expressions to exercise per-worker function copies
  • Randomized fuzz test independently randomizes dynamic lo and hi bounds across various combinations of filters, symbol equality, join filters, and prevailing
  • Unit tests for WindowJoinTimeFrameHelper bookmark logic

@puzpuzpuz puzpuzpuz self-assigned this Mar 9, 2026
@puzpuzpuz puzpuzpuz added Enhancement Enhance existing functionality SQL Issues or changes relating to SQL execution labels Mar 9, 2026
@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Mar 9, 2026

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 68da28dd-9665-4683-a8de-6af3549ac0e5

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch puzpuzpuz_dynamic_window_join
📝 Coding Plan
  • Generate coding plan for human review comments

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@puzpuzpuz puzpuzpuz marked this pull request as ready for review March 11, 2026 12:34
@puzpuzpuz puzpuzpuz changed the title feat(sql): dynamic window support in WINDOW JOIN feat(sql): add dynamic window support in WINDOW JOIN Mar 16, 2026
@puzpuzpuz puzpuzpuz added the New feature Feature requests label Mar 16, 2026
Store per-worker window function lists in local variables before
passing them to the AsyncWindowJoinRecordCursorFactory constructor.
This prevents a resource leak if a later constructor argument
compilation throws, since the catch block can now free them.

Also add setDynamicLo(true)/setDynamicHi(true) to
MutableModelsTest.testWindowJoinContextClear() so the test will
catch a regression if clear() stops resetting these fields.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
bluestreak01
bluestreak01 previously approved these changes Mar 16, 2026
If functionParser.parseFunction() throws mid-loop, previously created
Function objects in the local workerFunctions list were leaked because
the list was never returned to the caller. The catch block in
generateJoins could not free them since the assignment to
perWorkerWindowLoFuncs/perWorkerWindowHiFuncs never completed.

Wrap the loop in a try-catch that frees the partial list on failure.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@glasstiger
Copy link
Copy Markdown
Contributor

[PR Coverage check]

😍 pass : 1055 / 1441 (73.21%)

file detail

path covered line new line coverage
🔵 io/questdb/griffin/engine/join/AsyncWindowJoinRecordCursorFactory.java 687 1045 65.74%
🔵 io/questdb/griffin/engine/join/WindowJoinRecordCursorFactory.java 28 31 90.32%
🔵 io/questdb/griffin/engine/join/AsyncWindowJoinFastRecordCursorFactory.java 63 70 90.00%
🔵 io/questdb/griffin/SqlOptimiser.java 128 140 91.43%
🔵 io/questdb/griffin/SqlCodeGenerator.java 55 58 94.83%
🔵 io/questdb/griffin/engine/join/WindowJoinTimeFrameHelper.java 25 26 96.15%
🔵 io/questdb/griffin/engine/join/AsyncWindowJoinAtom.java 57 59 96.61%
🔵 io/questdb/griffin/SqlParser.java 2 2 100.00%
🔵 io/questdb/griffin/model/WindowJoinContext.java 8 8 100.00%
🔵 io/questdb/griffin/engine/table/AsyncHorizonJoinRecordCursorFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/table/AsyncHorizonJoinNotKeyedRecordCursorFactory.java 1 1 100.00%

@bluestreak01 bluestreak01 merged commit 68dab9c into master Mar 17, 2026
51 checks passed
@bluestreak01 bluestreak01 deleted the puzpuzpuz_dynamic_window_join branch March 17, 2026 01:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Enhancement Enhance existing functionality New feature Feature requests SQL Issues or changes relating to SQL execution

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants