You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit c6c5afd
Browse filesBrowse the repository at this point in the historyBrowse files
feat: BufferExec waits for hash join dynamic filters before buffering
Adds ProducerKind to DynamicFilterPhysicalExpr to distinguish between
hash join (safe to wait), TopK, and Aggregate (would deadlock) producers.
BufferExec now walks its child subtree at execution time, collects any
HashJoin dynamic filters, and waits for them to complete before the
background task begins polling the input stream. This allows scans below
BufferExec to benefit from dynamic filters rather than being eagerly
read before the filter is populated.
A new session config option `hash_join_buffering_dynamic_filter_wait_ms`
controls the wait behavior (0 = disabled, usize::MAX = wait indefinitely,
any other value = timeout in ms). Defaults to usize::MAX.
Closesapache#20778
/// Produced by a hash join build side. Safe to wait on — fully populated before probe starts.
44
+
HashJoin,
45
+
/// Produced by a TopK operator. Not safe to wait on — populated incrementally.
46
+
TopK,
47
+
/// Produced by an aggregate operator. Not safe to wait on — may never complete.
48
+
Aggregate,
49
+
}
50
+
31
51
/// State of a dynamic filter, tracking both updates and completion.
32
52
#[derive(Debug,Clone,Copy,PartialEq,Eq)]
33
53
enumFilterState{
@@ -55,7 +75,7 @@ impl FilterState {
55
75
/// For more background, please also see the [Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries blog]
56
76
///
57
77
/// [Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries blog]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters
58
-
#[derive(Debug)]
78
+
#[derive(Debug,Clone)]
59
79
pubstructDynamicFilterPhysicalExpr{
60
80
/// The original children of this PhysicalExpr, if any.
61
81
/// This is necessary because the dynamic filter may be initialized with a placeholder (e.g. `lit(true)`)
0 commit comments