Skip to content

Commit 8e5c789

Browse files
FFI: plumb with_updated_config for FFI_ScalarUDF (apache#22797)
## Which issue does this PR close? Part of apache#22330. ## Rationale for this change `ForeignScalarUDF` inherits the default `with_updated_config`, so producer overrides are lost across the FFI boundary. ## What changes are included in this PR? - Forward `with_updated_config` through `FFI_ScalarUDF`. - Reuse the existing placement and timezone UDFs for unit and dynamic-library coverage. - Log transport failures before the infallible trait method returns `None`. - Update the FFI skill to cover error handling, owned returns, and fixture reuse. ## Are these changes tested? - `cargo test -p datafusion-ffi --features integration-tests` - `cargo clippy --all-targets --all-features -- -D warnings` - `./ci/scripts/doc_prettier_check.sh --write --allow-dirty` ## Are there any user-facing changes? The FFI ABI changes. Foreign libraries must rebuild against the new DataFusion version. --------- Signed-off-by: Amogh Ramesh <[email protected]> Co-authored-by: Tim Saucer <[email protected]>
1 parent 570d2e1 commit 8e5c789

4 files changed

Lines changed: 128 additions & 15 deletions

File tree

‎.ai/skills/datafusion-ffi/SKILL.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
name: datafusion-ffi
3-
description: Patterns and review checklist for the `datafusion-ffi` crate. Use whenever the user adds, edits, or reviews code under `datafusion/ffi/` — new `FFI_X` wrappers, `Foreign<X>` impls, codec changes, or expanding an existing wrapper to cover more of a trait's surface. Also use when reviewing PRs that touch this crate.
3+
description: Patterns and review checklist for the `datafusion-ffi` crate. Use whenever the user adds, edits, or reviews code under `datafusion/ffi/` — new `FFI_X` wrappers, `ForeignX` implementations, codec changes, or expanding an existing wrapper to cover more of a trait's surface. Also use when reviewing PRs that touch this crate.
44
---
55

66
# DataFusion FFI Skill
@@ -210,9 +210,11 @@ cargo test -p datafusion-ffi --features integration-tests
210210
To add coverage for a new wrapper:
211211

212212
1. **Add a constructor** in `src/tests/<area>.rs` (or a new file there). Return a populated `FFI_X` from a known-good native type.
213-
2. **Wire it into `ForeignLibraryModule`** in `src/tests/mod.rs`: add a field of type `extern "C" fn(...) -> FFI_X` and populate it in `datafusion_ffi_get_module`. This struct is the cross-library contract — adding a field is itself an ABI change for the test module; integration tests will rebuild the cdylib automatically.
213+
2. **Wire it into `ForeignLibraryModule`** in `src/tests/mod.rs`: add a field of type `extern "C" fn(...) -> FFI_X` and populate it in `datafusion_ffi_get_module`. This struct is the cross-library contract. Adding a field is itself an ABI change for the test module; integration tests will rebuild the cdylib automatically.
214214
3. **Add the test** in `tests/ffi_<area>.rs` under `#[cfg(feature = "integration-tests")] mod tests { … }`. Call `datafusion_ffi::tests::utils::get_module()` to load the cdylib, invoke your constructor through the returned `ForeignLibraryModule`, convert into `Arc<dyn X>`, and exercise every method.
215215

216+
When adding a method to an existing wrapper, reuse an existing fixture and constructor when a small trait-method override can cover it. This applies both to omitted default methods and methods newly added to the trait. Add a dedicated test type or `ForeignLibraryModule` field only when the existing fixtures cannot cover the method.
217+
216218
What integration tests catch that unit tests cannot:
217219

218220
- **Real ABI layout bugs.** Two builds means the consumer's view of `FFI_X` is reconstructed from declaration, not aliased to the producer's memory. Mismatched alignment, padding, niche optimization, or accidentally non-`#[repr(C)]` types surface here.
@@ -235,7 +237,7 @@ What integration tests catch that unit tests cannot:
235237
- Primitives (`u8`/`u64`/`bool`/`usize`, etc.) and `#[repr(u8)]` FFI enums (`FFI_TableType`, `Volatility`, `InsertOp`, `TableProviderFilterPushDown`).
236238
- A `stabby::string::String` (`SString`) returned by value, with no other args or returns.
237239

238-
Concrete skippable example: `fn name(&self) -> SString` reading a field already validated by another method. Concrete *non*-skippable examples: anything returning `SVec<T>`, `FFI_Option<T>`, `FFI_Result<T>`, `WrappedSchema`, `WrappedArray`, an `FfiFuture`, an `FFI_*` sub-struct, or any `*mut`/`*const` pointer — those exercise alignment / padding / niche-opt across the ABI boundary and need the two-build coverage. When unsure, write the integration test; the cost is one constructor + ~20 lines.
240+
Concrete skippable example: `fn name(&self) -> SString` reading a field already validated by another method. Concrete *non*-skippable examples: anything returning `SVec<T>`, `FFI_Option<T>`, `FFI_Result<T>`, `WrappedSchema`, `WrappedArray`, an `FfiFuture`, an `FFI_*` sub-struct, or any `*mut`/`*const` pointer. These exercise alignment / padding / niche-opt across the ABI boundary and need the two-build coverage. When unsure, write the integration test; the cost is one constructor + ~20 lines.
239241

240242
If you skip the integration test for a layout change, you have effectively shipped untested ABI.
241243

@@ -313,6 +315,8 @@ If a method's body is non-trivial, the consumer-side default is non-trivial too.
313315
- **Logical `Expr` / `LogicalPlan`**: serialize via `datafusion-proto` using the embedded `FFI_LogicalExtensionCodec`. Same for physical plans → `FFI_PhysicalExtensionCodec`.
314316
- **Enums** (`Volatility`, `TableType`, `InsertOp`, `TableProviderFilterPushDown`): `#[repr(u8)]`, with `From<Native> for FFI_X` and `From<&FFI_X> for Native`. Always write a round-trip unit test that exercises every variant.
315317
- **Errors**: every `FFI_X` method that can fail returns `FFI_Result<T>`. Use the `sresult!`, `sresult_return!`, `df_result!` macros from `src/util.rs` — do not roll your own.
318+
- **Infallible trait methods**: if an FFI call can fail but the native trait cannot return the error, log the transport error before returning the trait's fallback (`None`, `false`, or a default). Never discard it with `.ok()`, `.unwrap_or_default()`, or equivalent.
319+
- **Owned FFI returns**: consume an owned `FFI_X` with `From<FFI_X>` instead of converting through `&FFI_X` and cloning across the boundary. Keep the borrowed conversion's local marker fast path.
316320

317321
## Async, sessions, and task context
318322

‎datafusion/ffi/src/tests/udf_udaf_udwf.rs‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ use std::sync::Arc;
2020
use arrow_schema::DataType;
2121
use datafusion_catalog::TableFunctionImpl;
2222
use datafusion_common::ScalarValue;
23+
use datafusion_common::config::ConfigOptions;
2324
use datafusion_expr::sort_properties::ExprProperties;
2425
use datafusion_expr::{
2526
AggregateUDF, ColumnarValue, ExpressionPlacement, ScalarFunctionArgs, ScalarUDF,
@@ -104,6 +105,13 @@ impl ScalarUDFImpl for TimeZoneUDF {
104105
let tz = args.config_options.execution.time_zone.clone();
105106
Ok(ColumnarValue::Scalar(ScalarValue::from(tz)))
106107
}
108+
109+
fn with_updated_config(&self, config: &ConfigOptions) -> Option<ScalarUDF> {
110+
config.execution.time_zone.as_ref()?;
111+
Some(ScalarUDF::from(Self {
112+
signature: self.signature.clone(),
113+
}))
114+
}
107115
}
108116

109117
pub(crate) extern "C" fn create_timezone_func() -> FFI_ScalarUDF {

‎datafusion/ffi/src/udf/mod.rs‎

Lines changed: 86 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ use crate::expr::columnar_value::FFI_ColumnarValue;
4545
use crate::expr::expr_properties::FFI_ExprProperties;
4646
use crate::placement::FFI_ExpressionPlacement;
4747
use crate::util::{
48-
FFI_Result, rvec_wrapped_to_vec_datatype, vec_datatype_to_rvec_wrapped,
48+
FFI_Option, FFI_Result, rvec_wrapped_to_vec_datatype, vec_datatype_to_rvec_wrapped,
4949
};
5050
use crate::volatility::FFI_Volatility;
5151
use crate::{df_result, sresult, sresult_return};
@@ -123,6 +123,13 @@ pub struct FFI_ScalarUDF {
123123
udf: &Self,
124124
inputs: SVec<FFI_ExprProperties>,
125125
) -> FFI_Result<bool>,
126+
127+
/// FFI equivalent to [`ScalarUDFImpl::with_updated_config`].
128+
pub with_updated_config:
129+
unsafe extern "C" fn(
130+
udf: &Self,
131+
config: FFI_ConfigOptions,
132+
) -> FFI_Result<FFI_Option<FFI_ScalarUDF>>,
126133
}
127134

128135
unsafe impl Send for FFI_ScalarUDF {}
@@ -199,6 +206,21 @@ unsafe extern "C" fn preserves_lex_ordering_fn_wrapper(
199206
sresult!(result)
200207
}
201208

209+
unsafe extern "C" fn with_updated_config_fn_wrapper(
210+
udf: &FFI_ScalarUDF,
211+
config: FFI_ConfigOptions,
212+
) -> FFI_Result<FFI_Option<FFI_ScalarUDF>> {
213+
let config = sresult_return!(ConfigOptions::try_from(config));
214+
215+
let updated: Option<FFI_ScalarUDF> = udf
216+
.inner()
217+
.inner()
218+
.with_updated_config(&config)
219+
.map(|updated| Arc::new(updated).into());
220+
221+
FFI_Result::Ok(updated.into())
222+
}
223+
202224
unsafe extern "C" fn invoke_with_args_fn_wrapper(
203225
udf: &FFI_ScalarUDF,
204226
args: SVec<WrappedArray>,
@@ -298,6 +320,7 @@ impl From<Arc<ScalarUDF>> for FFI_ScalarUDF {
298320
private_data: Box::into_raw(private_data) as *mut c_void,
299321
library_marker_id: crate::get_library_marker_id,
300322
preserves_lex_ordering: preserves_lex_ordering_fn_wrapper,
323+
with_updated_config: with_updated_config_fn_wrapper,
301324
}
302325
}
303326
}
@@ -325,6 +348,21 @@ pub struct ForeignScalarUDF {
325348
unsafe impl Send for ForeignScalarUDF {}
326349
unsafe impl Sync for ForeignScalarUDF {}
327350

351+
impl ForeignScalarUDF {
352+
fn new(udf: FFI_ScalarUDF) -> Self {
353+
let name = udf.name.to_string();
354+
let signature = Signature::user_defined((&udf.volatility).into());
355+
let aliases = udf.aliases.iter().map(|s| s.to_string()).collect();
356+
357+
Self {
358+
name,
359+
aliases,
360+
udf,
361+
signature,
362+
}
363+
}
364+
}
365+
328366
impl PartialEq for ForeignScalarUDF {
329367
fn eq(&self, other: &Self) -> bool {
330368
let Self {
@@ -356,22 +394,22 @@ impl Hash for ForeignScalarUDF {
356394
}
357395
}
358396

397+
impl From<FFI_ScalarUDF> for Arc<dyn ScalarUDFImpl> {
398+
fn from(udf: FFI_ScalarUDF) -> Self {
399+
if (udf.library_marker_id)() == crate::get_library_marker_id() {
400+
Arc::clone(udf.inner().inner())
401+
} else {
402+
Arc::new(ForeignScalarUDF::new(udf))
403+
}
404+
}
405+
}
406+
359407
impl From<&FFI_ScalarUDF> for Arc<dyn ScalarUDFImpl> {
360408
fn from(udf: &FFI_ScalarUDF) -> Self {
361409
if (udf.library_marker_id)() == crate::get_library_marker_id() {
362410
Arc::clone(udf.inner().inner())
363411
} else {
364-
let name = udf.name.to_string();
365-
let signature = Signature::user_defined((&udf.volatility).into());
366-
367-
let aliases = udf.aliases.iter().map(|s| s.to_string()).collect();
368-
369-
Arc::new(ForeignScalarUDF {
370-
name,
371-
udf: udf.clone(),
372-
aliases,
373-
signature,
374-
})
412+
Arc::new(ForeignScalarUDF::new(udf.clone()))
375413
}
376414
}
377415
}
@@ -494,6 +532,22 @@ impl ScalarUDFImpl for ForeignScalarUDF {
494532
df_result!(result)
495533
})
496534
}
535+
536+
fn with_updated_config(&self, config: &ConfigOptions) -> Option<ScalarUDF> {
537+
let config: FFI_ConfigOptions = config.into();
538+
539+
let result = unsafe { (self.udf.with_updated_config)(&self.udf, config) };
540+
541+
let updated = match df_result!(result) {
542+
Ok(updated) => updated.into_option()?,
543+
Err(error) => {
544+
log::warn!("Unable to update scalar UDF configuration over FFI: {error}");
545+
return None;
546+
}
547+
};
548+
549+
Some(ScalarUDF::new_from_shared_impl(updated.into()))
550+
}
497551
}
498552

499553
#[cfg(test)]
@@ -542,6 +596,12 @@ mod tests {
542596

543597
Ok(inputs.iter().all(|input| input.preserves_lex_ordering))
544598
}
599+
600+
fn with_updated_config(&self, _config: &ConfigOptions) -> Option<ScalarUDF> {
601+
Some(ScalarUDF::from(Self {
602+
signature: self.signature.clone(),
603+
}))
604+
}
545605
}
546606

547607
#[test]
@@ -555,6 +615,11 @@ mod tests {
555615
let foreign_udf: Arc<dyn ScalarUDFImpl> = (&local_udf).into();
556616

557617
assert_eq!(original_udf.name(), foreign_udf.name());
618+
assert!(
619+
foreign_udf
620+
.with_updated_config(&ConfigOptions::default())
621+
.is_none()
622+
);
558623

559624
Ok(())
560625
}
@@ -629,6 +694,15 @@ mod tests {
629694
);
630695
assert!(foreign_udf.preserves_lex_ordering(&[]).is_err());
631696

697+
let updated = foreign_udf
698+
.with_updated_config(&ConfigOptions::default())
699+
.expect("provider should return an updated UDF");
700+
assert_eq!(
701+
updated
702+
.placement(&[ExpressionPlacement::Column, ExpressionPlacement::Literal]),
703+
ExpressionPlacement::MoveTowardsLeafNodes
704+
);
705+
632706
Ok(())
633707
}
634708
}

‎datafusion/ffi/tests/ffi_udf.rs‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
mod tests {
2222
use arrow::array::{Array, AsArray, record_batch};
2323
use arrow::datatypes::DataType;
24+
use datafusion::common::config::ConfigOptions;
2425
use datafusion::error::Result;
2526
use datafusion::logical_expr::{ExpressionPlacement, ScalarUDF, ScalarUDFImpl};
2627
use datafusion::prelude::{SessionContext, col};
@@ -157,4 +158,30 @@ mod tests {
157158

158159
Ok(())
159160
}
161+
162+
/// Validates that a provider's `with_updated_config` override survives the
163+
/// FFI boundary (the trait default returns `None`).
164+
#[test]
165+
fn test_with_updated_config_on_scalar_udf() -> Result<()> {
166+
let module = get_module()?;
167+
168+
let ffi_udf = (module.create_timezone_udf)();
169+
let foreign_udf: Arc<dyn ScalarUDFImpl> = (&ffi_udf).into();
170+
171+
assert!(
172+
foreign_udf
173+
.with_updated_config(&ConfigOptions::default())
174+
.is_none()
175+
);
176+
177+
let mut options = ConfigOptions::default();
178+
options.execution.time_zone = Some("AEST".into());
179+
180+
let updated = foreign_udf
181+
.with_updated_config(&options)
182+
.expect("provider should return an updated UDF");
183+
assert_eq!(updated.name(), "TimeZoneUDF");
184+
185+
Ok(())
186+
}
160187
}

0 commit comments

Comments
 (0)