Skip to content

Commit 8653851

Browse files
authored
feat(spark): Add SessionStateBuilderSpark to datafusion-spark (#19865)
## Which issue does this PR close? - Closes #19843 ## Rationale for this change Currently, combining DataFusion's default features with Spark features is awkward because: 1. Expression planners must be registered **before** calling `with_default_features().build()` to take precedence 2. UDFs must be registered **after** the state is built (if using `register_all`) This requires splitting the setup into multiple phases, which is verbose and error-prone. ## What changes are included in this PR? - Added `SessionStateBuilderSpark` extension trait in `datafusion-spark` that provides `with_spark_features()` method to register both the Spark expression planner (with correct precedence) and all Spark UDFs in one call - Added `core` feature flag to `datafusion-spark` with `datafusion` as an optional dependency (this avoids having `datafusion-core` depend on `datafusion-spark`) - Updated `datafusion-spark` crate documentation with usage example - Simplified test context setup in `datafusion-sqllogictest` to use the new extension trait ## Are these changes tested? Yes, there is a unit test in `datafusion-spark/src/session_state.rs` plus the existing Spark SQLLogicTest suite validates that all Spark functions work correctly. The test context in datafusion-sqllogictest now uses the `SessionStateBuilderSpark` extension trait, serving as both a usage example and integration test. ## Are there any user-facing changes? Yes, this adds a new public API: `SessionStateBuilderSpark` extension trait (behind the `core` feature flag in `datafusion-spark`).
1 parent 073174b commit 8653851

6 files changed

Lines changed: 147 additions & 12 deletions

File tree

‎Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎datafusion/spark/Cargo.toml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,10 @@ edition = { workspace = true }
2929
[package.metadata.docs.rs]
3030
all-features = true
3131

32+
[features]
33+
default = []
34+
core = ["datafusion"]
35+
3236
# Note: add additional linter rules in lib.rs.
3337
# Rust does not support workspace + new linter rules in subcrates yet
3438
# https://github.com/rust-lang/cargo/issues/13157
@@ -43,6 +47,8 @@ arrow = { workspace = true }
4347
bigdecimal = { workspace = true }
4448
chrono = { workspace = true }
4549
crc32fast = "1.4"
50+
# Optional dependency for SessionStateBuilderSpark extension trait
51+
datafusion = { workspace = true, optional = true, default-features = false }
4652
datafusion-catalog = { workspace = true }
4753
datafusion-common = { workspace = true }
4854
datafusion-execution = { workspace = true }
@@ -59,6 +65,8 @@ url = { workspace = true }
5965
[dev-dependencies]
6066
arrow = { workspace = true, features = ["test_utils"] }
6167
criterion = { workspace = true }
68+
# for SessionStateBuilderSpark tests
69+
datafusion = { workspace = true, default-features = false }
6270

6371
[[bench]]
6472
harness = false

‎datafusion/spark/src/lib.rs‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,10 +111,31 @@
111111
//! ```
112112
//!
113113
//![`Expr`]: datafusion_expr::Expr
114+
//!
115+
//! # Example: enabling Apache Spark features with SessionStateBuilder
116+
//!
117+
//! The recommended way to enable Apache Spark compatibility is to use the
118+
//! `SessionStateBuilderSpark` extension trait. This registers all
119+
//! Apache Spark functions (scalar, aggregate, window, and table) as well as the Apache Spark
120+
//! expression planner.
121+
//!
122+
//! Enable the `core` feature in your `Cargo.toml`:
123+
//! ```toml
124+
//! datafusion-spark = { version = "X", features = ["core"] }
125+
//! ```
126+
//!
127+
//! Then use the extension trait - see [`SessionStateBuilderSpark::with_spark_features`]
128+
//! for an example.
114129
115130
pub mod function;
116131
pub mod planner;
117132

133+
#[cfg(feature = "core")]
134+
mod session_state;
135+
136+
#[cfg(feature = "core")]
137+
pub use session_state::SessionStateBuilderSpark;
138+
118139
use datafusion_catalog::TableFunction;
119140
use datafusion_common::Result;
120141
use datafusion_execution::FunctionRegistry;
Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
use std::collections::HashMap;
19+
use std::sync::Arc;
20+
21+
use datafusion::execution::SessionStateBuilder;
22+
23+
use crate::planner::SparkFunctionPlanner;
24+
use crate::{
25+
all_default_aggregate_functions, all_default_scalar_functions,
26+
all_default_table_functions, all_default_window_functions,
27+
};
28+
29+
/// Extension trait for adding Apache Spark features to [`SessionStateBuilder`].
30+
///
31+
/// This trait provides a convenient way to register all Apache Spark-compatible
32+
/// functions and planners with a DataFusion session.
33+
///
34+
/// # Example
35+
///
36+
/// ```rust
37+
/// use datafusion::execution::SessionStateBuilder;
38+
/// use datafusion_spark::SessionStateBuilderSpark;
39+
///
40+
/// // Create a SessionState with Apache Spark features enabled
41+
/// // note: the order matters here, `with_spark_features` should be
42+
/// // called after `with_default_features` to overwrite any existing functions
43+
/// let state = SessionStateBuilder::new()
44+
/// .with_default_features()
45+
/// .with_spark_features()
46+
/// .build();
47+
/// ```
48+
pub trait SessionStateBuilderSpark {
49+
/// Adds all expr_planners, scalar, aggregate, window and table functions
50+
/// compatible with Apache Spark.
51+
///
52+
/// Note: This overwrites any previously registered items with the same name.
53+
fn with_spark_features(self) -> Self;
54+
}
55+
56+
impl SessionStateBuilderSpark for SessionStateBuilder {
57+
fn with_spark_features(mut self) -> Self {
58+
self.expr_planners()
59+
.get_or_insert_with(Vec::new)
60+
// planners are evaluated in order of insertion. Push Apache Spark function planner to the front
61+
// to take precedence over others
62+
.insert(0, Arc::new(SparkFunctionPlanner));
63+
64+
self.scalar_functions()
65+
.get_or_insert_with(Vec::new)
66+
.extend(all_default_scalar_functions());
67+
68+
self.aggregate_functions()
69+
.get_or_insert_with(Vec::new)
70+
.extend(all_default_aggregate_functions());
71+
72+
self.window_functions()
73+
.get_or_insert_with(Vec::new)
74+
.extend(all_default_window_functions());
75+
76+
self.table_functions()
77+
.get_or_insert_with(HashMap::new)
78+
.extend(
79+
all_default_table_functions()
80+
.into_iter()
81+
.map(|f| (f.name().to_string(), f)),
82+
);
83+
84+
self
85+
}
86+
}
87+
88+
#[cfg(test)]
89+
mod tests {
90+
use super::*;
91+
92+
#[test]
93+
fn test_session_state_with_spark_features() {
94+
let state = SessionStateBuilder::new().with_spark_features().build();
95+
96+
assert!(
97+
state.scalar_functions().contains_key("sha2"),
98+
"Apache Spark scalar function 'sha2' should be registered"
99+
);
100+
101+
assert!(
102+
state.aggregate_functions().contains_key("try_sum"),
103+
"Apache Spark aggregate function 'try_sum' should be registered"
104+
);
105+
106+
assert!(
107+
!state.expr_planners().is_empty(),
108+
"Apache Spark expr planners should be registered"
109+
);
110+
}
111+
}

‎datafusion/sqllogictest/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ bytes = { workspace = true, optional = true }
4747
chrono = { workspace = true, optional = true }
4848
clap = { version = "4.5.53", features = ["derive", "env"] }
4949
datafusion = { workspace = true, default-features = true, features = ["avro"] }
50-
datafusion-spark = { workspace = true, default-features = true }
50+
datafusion-spark = { workspace = true, features = ["core"] }
5151
datafusion-substrait = { workspace = true, default-features = true }
5252
futures = { workspace = true }
5353
half = { workspace = true, default-features = true }

‎datafusion/sqllogictest/src/test_context.rs‎

Lines changed: 5 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ use datafusion::{
4646
datasource::{MemTable, TableProvider, TableType},
4747
prelude::{CsvReadOptions, SessionContext},
4848
};
49+
use datafusion_spark::SessionStateBuilderSpark;
4950

5051
use crate::is_spark_path;
5152
use async_trait::async_trait;
@@ -84,21 +85,14 @@ impl TestContext {
8485

8586
let mut state_builder = SessionStateBuilder::new()
8687
.with_config(config)
87-
.with_runtime_env(runtime);
88+
.with_runtime_env(runtime)
89+
.with_default_features();
8890

8991
if is_spark_path(relative_path) {
90-
state_builder = state_builder.with_expr_planners(vec![Arc::new(
91-
datafusion_spark::planner::SparkFunctionPlanner,
92-
)]);
92+
state_builder = state_builder.with_spark_features();
9393
}
9494

95-
let mut state = state_builder.with_default_features().build();
96-
97-
if is_spark_path(relative_path) {
98-
info!("Registering Spark functions");
99-
datafusion_spark::register_all(&mut state)
100-
.expect("Can not register Spark functions");
101-
}
95+
let state = state_builder.build();
10296

10397
let mut test_ctx = TestContext::new(SessionContext::new_with_state(state));
10498

0 commit comments

Comments
 (0)