Skip to content

Commit 3f9c68b

Browse files
unikdahalPerseus
authored andcommitted
feat(reader): support execution-time predicates in ArrowReader
Execution engines learn restrictions while a query runs, such as a completed join build's key range or an improving TopK or MIN/MAX bound. Add RuntimePredicateSnapshot and RuntimePredicateProvider so engines can publish them as Iceberg predicates, and ArrowReaderBuilder::with_runtime_predicate_provider to apply them. When a data-file task starts, the reader checks the provider's generation. Only when it changed does the reader take a snapshot, push NOT down to the leaves and bind the predicate. Refreshes are serialized, so a generation is snapshotted and bound once, and the cache never moves to an older generation. The bound predicate is shared, without copying, by later tasks of the scan with the same generation and schema. The runtime predicate is planned separately from the planned and equality-delete predicates, so the shared bound tree is never copied. They are then combined into one Arrow row filter, which decodes shared columns once, and their row-group, bloom-filter and page-index selections are intersected, row groups in linear time. A bloom filter is read at most once per row group and column, even when both predicates use it. Runtime predicates are advisory. Provider, binding and planning failures, and page-index failures, drop only the runtime part for that file; the planned and delete filtering are unchanged. A predicate on a column missing from the file (its initial default or null is invisible to physical filters) or stored with a type that needs promotion (a literal cast before promotion could round or overflow) is ignored for that file. Positional deletes keep their original file positions. The provider contract documents that data skipped under a published predicate must never be required later, that a later None only stops runtime pruning for tasks that start afterwards, that generations increase monotonically and are never reused, and that provider, snapshot, binding, planning and page-pruning failures are advisory. The reader accepts the provider; passing one through TableScan is a follow-up. (cherry picked from commit 821fc65)
1 parent 5b82ab9 commit 3f9c68b

7 files changed

Lines changed: 2254 additions & 121 deletions

File tree

‎crates/iceberg/public-api.txt‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,7 @@ pub fn iceberg::arrow::ArrowReaderBuilder::with_range_coalesce_bytes(self, range
7979
pub fn iceberg::arrow::ArrowReaderBuilder::with_range_fetch_concurrency(self, range_fetch_concurrency: usize) -> Self
8080
pub fn iceberg::arrow::ArrowReaderBuilder::with_row_group_filtering_enabled(self, row_group_filtering_enabled: bool) -> Self
8181
pub fn iceberg::arrow::ArrowReaderBuilder::with_row_selection_enabled(self, row_selection_enabled: bool) -> Self
82+
pub fn iceberg::arrow::ArrowReaderBuilder::with_runtime_predicate_provider(self, runtime_predicate_provider: alloc::sync::Arc<dyn iceberg::arrow::RuntimePredicateProvider>) -> Self
8283
pub struct iceberg::arrow::PartitionValueCalculator
8384
impl iceberg::arrow::PartitionValueCalculator
8485
pub fn iceberg::arrow::PartitionValueCalculator::calculate(&self, batch: &arrow_array::record_batch::RecordBatch) -> iceberg::Result<arrow_array::array::ArrayRef>
@@ -93,6 +94,13 @@ pub fn iceberg::arrow::RecordBatchPartitionSplitter::split(&self, batch: &arrow_
9394
pub fn iceberg::arrow::RecordBatchPartitionSplitter::try_new(iceberg_schema: iceberg::spec::SchemaRef, partition_spec: iceberg::spec::PartitionSpecRef, calculator: core::option::Option<iceberg::arrow::PartitionValueCalculator>) -> iceberg::Result<Self>
9495
pub fn iceberg::arrow::RecordBatchPartitionSplitter::try_new_with_computed_values(iceberg_schema: iceberg::spec::SchemaRef, partition_spec: iceberg::spec::PartitionSpecRef) -> iceberg::Result<Self>
9596
pub fn iceberg::arrow::RecordBatchPartitionSplitter::try_new_with_precomputed_values(iceberg_schema: iceberg::spec::SchemaRef, partition_spec: iceberg::spec::PartitionSpecRef) -> iceberg::Result<Self>
97+
pub struct iceberg::arrow::RuntimePredicateSnapshot
98+
impl iceberg::arrow::RuntimePredicateSnapshot
99+
pub fn iceberg::arrow::RuntimePredicateSnapshot::generation(&self) -> u64
100+
pub fn iceberg::arrow::RuntimePredicateSnapshot::into_predicate(self) -> core::option::Option<iceberg::expr::Predicate>
101+
pub fn iceberg::arrow::RuntimePredicateSnapshot::new(predicate: core::option::Option<iceberg::expr::Predicate>, generation: u64) -> Self
102+
impl core::fmt::Debug for iceberg::arrow::RuntimePredicateSnapshot
103+
pub fn iceberg::arrow::RuntimePredicateSnapshot::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result
96104
pub struct iceberg::arrow::ScanMetrics
97105
impl iceberg::scan::ScanMetrics
98106
pub fn iceberg::scan::ScanMetrics::bytes_read(&self) -> u64
@@ -124,6 +132,9 @@ pub fn iceberg::arrow::ArrowSchemaVisitor::primitive(&mut self, p: &arrow_schema
124132
pub fn iceberg::arrow::ArrowSchemaVisitor::schema(&mut self, schema: &arrow_schema::schema::Schema, values: alloc::vec::Vec<Self::T>) -> iceberg::Result<Self::U>
125133
pub fn iceberg::arrow::ArrowSchemaVisitor::struct(&mut self, fields: &arrow_schema::fields::Fields, results: alloc::vec::Vec<Self::T>) -> iceberg::Result<Self::T>
126134
pub fn iceberg::arrow::ArrowSchemaVisitor::variant(&mut self, field: &arrow_schema::field::FieldRef) -> iceberg::Result<Self::T> where Self: core::marker::Sized
135+
pub trait iceberg::arrow::RuntimePredicateProvider: core::marker::Send + core::marker::Sync
136+
pub fn iceberg::arrow::RuntimePredicateProvider::generation(&self) -> u64
137+
pub fn iceberg::arrow::RuntimePredicateProvider::snapshot(&self) -> iceberg::Result<iceberg::arrow::RuntimePredicateSnapshot>
127138
pub fn iceberg::arrow::arrow_primitive_to_literal(primitive_array: &arrow_array::array::ArrayRef, ty: &iceberg::spec::Type) -> iceberg::Result<alloc::vec::Vec<core::option::Option<iceberg::spec::Literal>>>
128139
pub fn iceberg::arrow::arrow_schema_to_schema(schema: &arrow_schema::schema::Schema) -> iceberg::Result<iceberg::spec::Schema>
129140
pub fn iceberg::arrow::arrow_schema_to_schema_auto_assign_ids(schema: &arrow_schema::schema::Schema) -> iceberg::Result<iceberg::spec::Schema>

‎crates/iceberg/src/arrow/reader/mod.rs‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@
1717

1818
//! Parquet file data reader
1919
20+
use std::sync::Arc;
21+
2022
use crate::arrow::caching_delete_file_loader::CachingDeleteFileLoader;
2123
use crate::io::FileIO;
2224
use crate::runtime::Runtime;
@@ -42,13 +44,17 @@ mod predicate_visitor;
4244
mod projection;
4345
mod row_filter;
4446
mod row_lineage;
47+
mod runtime_predicate;
48+
#[cfg(test)]
49+
mod runtime_predicate_tests;
4550
pub use file_reader::ArrowFileReader;
4651
pub(crate) use options::ParquetReadOptions;
4752
use predicate_visitor::{CollectFieldIdVisitor, PredicateConverter};
4853
use projection::{
4954
add_fallback_field_ids_to_arrow_schema, apply_name_mapping_to_arrow_schema,
5055
find_leaf_by_field_id,
5156
};
57+
pub use runtime_predicate::{RuntimePredicateProvider, RuntimePredicateSnapshot};
5258

5359
/// Builder to create ArrowReader
5460
pub struct ArrowReaderBuilder {
@@ -58,6 +64,7 @@ pub struct ArrowReaderBuilder {
5864
row_group_filtering_enabled: bool,
5965
row_selection_enabled: bool,
6066
bloom_filter_enabled: bool,
67+
runtime_predicate_provider: Option<Arc<dyn RuntimePredicateProvider>>,
6168
parquet_read_options: ParquetReadOptions,
6269
runtime: Runtime,
6370
}
@@ -74,6 +81,7 @@ impl ArrowReaderBuilder {
7481
row_group_filtering_enabled: true,
7582
row_selection_enabled: false,
7683
bloom_filter_enabled: false,
84+
runtime_predicate_provider: None,
7785
parquet_read_options: ParquetReadOptions::builder().build(),
7886
runtime,
7987
}
@@ -119,6 +127,21 @@ impl ArrowReaderBuilder {
119127
self
120128
}
121129

130+
/// Supplies execution-time predicates, sampled when each data-file task
131+
/// starts and combined with the task's filters using AND. See
132+
/// [`RuntimePredicateProvider`].
133+
///
134+
/// Only the reader accepts a provider. `TableScan::to_arrow` cannot pass
135+
/// one, so wiring it through `TableScan` and execution engines is a
136+
/// follow-up.
137+
pub fn with_runtime_predicate_provider(
138+
mut self,
139+
runtime_predicate_provider: Arc<dyn RuntimePredicateProvider>,
140+
) -> Self {
141+
self.runtime_predicate_provider = Some(runtime_predicate_provider);
142+
self
143+
}
144+
122145
/// Provide a hint as to the number of bytes to prefetch for parsing the Parquet metadata
123146
///
124147
/// This hint can help reduce the number of fetch requests. For more details see the
@@ -159,6 +182,7 @@ impl ArrowReaderBuilder {
159182
row_group_filtering_enabled: self.row_group_filtering_enabled,
160183
row_selection_enabled: self.row_selection_enabled,
161184
bloom_filter_enabled: self.bloom_filter_enabled,
185+
runtime_predicate_provider: self.runtime_predicate_provider,
162186
parquet_read_options: self.parquet_read_options,
163187
}
164188
}
@@ -177,5 +201,6 @@ pub struct ArrowReader {
177201
row_group_filtering_enabled: bool,
178202
row_selection_enabled: bool,
179203
bloom_filter_enabled: bool,
204+
runtime_predicate_provider: Option<Arc<dyn RuntimePredicateProvider>>,
180205
parquet_read_options: ParquetReadOptions,
181206
}

0 commit comments

Comments
 (0)