Skip to content

Commit 4d3e79e

Browse files
timsaucerclaude
andauthored
fix: preserve provided arguments during FFI object construction (apache#24723)
## Which issue does this PR close? - Closes apache#24722. ## Rationale for this change Three `datafusion-ffi` constructors unwrap an already-foreign input and return its original handle, dropping the arguments passed alongside without an error or a warning: - `FFI_LogicalExtensionCodec::new` — discards `task_ctx_provider` - `FFI_PhysicalExtensionCodec::new` — discards `task_ctx_provider` - `FFI_TableProvider::new_with_ffi_codec` — discards `logical_codec` The consequence is that a consumer which imports a foreign codec can never rebind it. Re-wrapping with a different provider compiles, runs, and has no effect, so the handle keeps resolving against whatever session it was first built with. In `datafusion-python` that shows up as decode callbacks resolving names against a pre-fork session: a UDF registered after the fork is invisible to them, and the config they see is a stale snapshot. There is a second failure mode with the same root cause. The provider is held as a `Weak`, so a consumer that cannot rebind must keep the original session alive artificially or the capsule starts failing with `TaskContextProvider went out of scope over FFI boundary`. The two sibling constructors that hit the same case already do the opposite — `FFI_QueryPlanner::new_with_ffi_codecs` and `FFI_SessionRef::new_with_ffi_codecs` both adopt the supplied codecs on the unwrap path, and the former documents that guarantee explicitly. This PR makes the other three consistent with them. ## What changes are included in this PR? On the already-foreign path, each of the three constructors now clones the original handle and overwrites the relevant `#[repr(C)]` field before returning it, matching `FFI_QueryPlanner::new_with_ffi_codecs`: ```rust if let Some(codec) = (Arc::clone(&codec) as Arc<dyn Any>) .downcast_ref::<ForeignLogicalExtensionCodec>() { let mut codec = codec.0.clone(); codec.task_ctx_provider = task_ctx_provider.into(); return codec; } ``` The `runtime` argument is a deliberate exception. Unlike the codecs and the task context provider, `runtime` lives in `private_data`, which belongs to the library that owns the handle — this side cannot write it without an ABI change. `FFI_SessionRef::new_with_ffi_codecs` already takes the same position ("retaining its original private data and runtime"). Rather than leave that silent, all three constructors now document it, alongside the new adopt-on-unwrap guarantee. No public signatures change, and no behavior changes on the non-foreign path. ## Are these changes tested? Yes — five new unit tests, one per behavior, in each affected module's own test module. All five fail on `main` and pass here. - `ffi_logical_extension_codec_rebind_adopts_task_ctx_provider` - `ffi_logical_extension_codec_rebind_releases_original_session` — covers the dangling-`Weak` failure mode: session A is dropped after the rebind, and the handle stays usable - `ffi_physical_extension_codec_rebind_adopts_task_ctx_provider` - `test_rebind_foreign_table_provider_adopts_logical_codec` - `test_rebind_foreign_query_planner_adopts_codecs` — a control over the already-correct sibling, so the two paths stay in agreement Worth flagging for reviewers, since it is easy to write a test here that silently proves nothing: `impl From<&FFI_LogicalExtensionCodec> for Arc<dyn LogicalExtensionCodec>` compares `library_marker_id` first and returns the original local `Arc` on a match, so within one library the foreign branch is never reached. Each test overrides `library_marker_id` with `crate::mock_foreign_marker_id` and asserts the import really did produce a `Foreign*` wrapper before exercising the rebind. `cargo test -p datafusion-ffi --all-features` passes (152 tests). ## Are there any user-facing changes? Yes, a behavior change, though it replaces a silent no-op with the documented intent. Callers that pass a `task_ctx_provider` or `logical_codec` to these constructors alongside an already-foreign input previously had that argument ignored; it now takes effect. Anything relying on the old handle being returned untouched would see the change — but since the old path gave no way to observe or opt into that, it is hard to depend on deliberately. Downstream, this lets `datafusion-python` drop the workaround in apache/datafusion-python#1677, which retains the pre-fork `SessionContext` purely to keep the `Weak` valid. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
1 parent c56a7ab commit 4d3e79e

4 files changed

Lines changed: 202 additions & 5 deletions

File tree

‎datafusion/ffi/src/proto/logical_extension_codec.rs‎

Lines changed: 72 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -294,6 +294,15 @@ impl Drop for FFI_LogicalExtensionCodec {
294294

295295
impl FFI_LogicalExtensionCodec {
296296
/// Creates a new [`FFI_LogicalExtensionCodec`].
297+
///
298+
/// If `codec` is already foreign, this re-exports its original FFI handle
299+
/// rather than adding another wrapper layer. The handle still adopts the
300+
/// `task_ctx_provider` supplied here, so it is never silently discarded and
301+
/// an imported codec can be rebound to a different session.
302+
///
303+
/// `runtime` is only honored when a new wrapper is created. An
304+
/// already-foreign handle keeps the runtime of the library that owns it,
305+
/// because that value lives in private data this side cannot reach.
297306
pub fn new(
298307
codec: Arc<dyn LogicalExtensionCodec>,
299308
runtime: Option<Handle>,
@@ -302,7 +311,9 @@ impl FFI_LogicalExtensionCodec {
302311
if let Some(codec) = (Arc::clone(&codec) as Arc<dyn Any>)
303312
.downcast_ref::<ForeignLogicalExtensionCodec>()
304313
{
305-
return codec.0.clone();
314+
let mut codec = codec.0.clone();
315+
codec.task_ctx_provider = task_ctx_provider.into();
316+
return codec;
306317
}
307318

308319
let task_ctx_provider = task_ctx_provider.into();
@@ -507,7 +518,9 @@ mod tests {
507518
use datafusion_proto::logical_plan::LogicalExtensionCodec;
508519
use datafusion_proto::physical_plan::PhysicalExtensionCodec;
509520

510-
use crate::proto::logical_extension_codec::FFI_LogicalExtensionCodec;
521+
use crate::proto::logical_extension_codec::{
522+
FFI_LogicalExtensionCodec, ForeignLogicalExtensionCodec,
523+
};
511524
use crate::proto::physical_extension_codec::tests::TestExtensionCodec;
512525

513526
fn create_test_table() -> MemTable {
@@ -727,4 +740,61 @@ mod tests {
727740
let foreign_codec: Arc<dyn LogicalExtensionCodec> = (&ffi_codec).into();
728741
assert!(!arc_ptr_eq(&foreign_codec, &codec));
729742
}
743+
744+
/// Importing a codec and re-wrapping it with a different task context
745+
/// provider must rebind the handle. See
746+
/// <https://github.com/apache/datafusion/issues/24722>.
747+
#[test]
748+
fn ffi_logical_extension_codec_rebind_adopts_task_ctx_provider() {
749+
let (_ctx_a, provider_a) = crate::util::tests::test_session_and_ctx();
750+
let (ctx_b, provider_b) = crate::util::tests::test_session_and_ctx();
751+
752+
let mut ffi_codec = FFI_LogicalExtensionCodec::new(
753+
Arc::new(TestExtensionCodec {}) as Arc<dyn LogicalExtensionCodec>,
754+
None,
755+
provider_a,
756+
);
757+
ffi_codec.library_marker_id = crate::mock_foreign_marker_id;
758+
759+
let imported: Arc<dyn LogicalExtensionCodec> = (&ffi_codec).into();
760+
assert!(
761+
(Arc::clone(&imported) as Arc<dyn std::any::Any>)
762+
.downcast_ref::<ForeignLogicalExtensionCodec>()
763+
.is_some()
764+
);
765+
766+
let rebound = FFI_LogicalExtensionCodec::new(imported, None, provider_b);
767+
768+
let task_ctx: Arc<TaskContext> = (&rebound.task_ctx_provider)
769+
.try_into()
770+
.expect("rebound codec resolves");
771+
assert_eq!(task_ctx.session_id(), ctx_b.task_ctx().session_id());
772+
}
773+
774+
/// Because the provider is held as a `Weak`, a codec that cannot be rebound
775+
/// forces callers to keep the original session alive. Once rebinding works,
776+
/// dropping it must not invalidate the handle.
777+
#[test]
778+
fn ffi_logical_extension_codec_rebind_releases_original_session() {
779+
let (ctx_b, provider_b) = crate::util::tests::test_session_and_ctx();
780+
781+
let rebound = {
782+
let (ctx_a, provider_a) = crate::util::tests::test_session_and_ctx();
783+
let mut ffi_codec = FFI_LogicalExtensionCodec::new(
784+
Arc::new(TestExtensionCodec {}) as Arc<dyn LogicalExtensionCodec>,
785+
None,
786+
provider_a,
787+
);
788+
ffi_codec.library_marker_id = crate::mock_foreign_marker_id;
789+
let imported: Arc<dyn LogicalExtensionCodec> = (&ffi_codec).into();
790+
drop(ctx_a);
791+
792+
FFI_LogicalExtensionCodec::new(imported, None, provider_b)
793+
};
794+
795+
let task_ctx: Arc<TaskContext> = (&rebound.task_ctx_provider)
796+
.try_into()
797+
.expect("rebound codec must not depend on the original session");
798+
assert_eq!(task_ctx.session_id(), ctx_b.task_ctx().session_id());
799+
}
730800
}

‎datafusion/ffi/src/proto/physical_extension_codec.rs‎

Lines changed: 45 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -280,6 +280,15 @@ impl Drop for FFI_PhysicalExtensionCodec {
280280

281281
impl FFI_PhysicalExtensionCodec {
282282
/// Creates a new [`FFI_PhysicalExtensionCodec`].
283+
///
284+
/// If `codec` is already foreign, this re-exports its original FFI handle
285+
/// rather than adding another wrapper layer. The handle still adopts the
286+
/// `task_ctx_provider` supplied here, so it is never silently discarded and
287+
/// an imported codec can be rebound to a different session.
288+
///
289+
/// `runtime` is only honored when a new wrapper is created. An
290+
/// already-foreign handle keeps the runtime of the library that owns it,
291+
/// because that value lives in private data this side cannot reach.
283292
pub fn new(
284293
codec: Arc<dyn PhysicalExtensionCodec>,
285294
runtime: Option<Handle>,
@@ -288,7 +297,9 @@ impl FFI_PhysicalExtensionCodec {
288297
if let Some(codec) = (Arc::clone(&codec) as Arc<dyn Any>)
289298
.downcast_ref::<ForeignPhysicalExtensionCodec>()
290299
{
291-
return codec.0.clone();
300+
let mut codec = codec.0.clone();
301+
codec.task_ctx_provider = task_ctx_provider.into();
302+
return codec;
292303
}
293304

294305
let task_ctx_provider = task_ctx_provider.into();
@@ -449,7 +460,9 @@ pub(crate) mod tests {
449460
};
450461

451462
use crate::execution_plan::tests::EmptyExec;
452-
use crate::proto::physical_extension_codec::FFI_PhysicalExtensionCodec;
463+
use crate::proto::physical_extension_codec::{
464+
FFI_PhysicalExtensionCodec, ForeignPhysicalExtensionCodec,
465+
};
453466

454467
#[derive(Debug)]
455468
pub(crate) struct TestExtensionCodec;
@@ -710,4 +723,34 @@ pub(crate) mod tests {
710723
let foreign_codec: Arc<dyn PhysicalExtensionCodec> = (&ffi_codec).into();
711724
assert!(!arc_ptr_eq(&foreign_codec, &codec));
712725
}
726+
727+
/// Importing a codec and re-wrapping it with a different task context
728+
/// provider must rebind the handle. See
729+
/// <https://github.com/apache/datafusion/issues/24722>.
730+
#[test]
731+
fn ffi_physical_extension_codec_rebind_adopts_task_ctx_provider() {
732+
let (_ctx_a, provider_a) = crate::util::tests::test_session_and_ctx();
733+
let (ctx_b, provider_b) = crate::util::tests::test_session_and_ctx();
734+
735+
let mut ffi_codec = FFI_PhysicalExtensionCodec::new(
736+
Arc::new(TestExtensionCodec {}) as Arc<dyn PhysicalExtensionCodec>,
737+
None,
738+
provider_a,
739+
);
740+
ffi_codec.library_marker_id = crate::mock_foreign_marker_id;
741+
742+
let imported: Arc<dyn PhysicalExtensionCodec> = (&ffi_codec).into();
743+
assert!(
744+
(Arc::clone(&imported) as Arc<dyn std::any::Any>)
745+
.downcast_ref::<ForeignPhysicalExtensionCodec>()
746+
.is_some()
747+
);
748+
749+
let rebound = FFI_PhysicalExtensionCodec::new(imported, None, provider_b);
750+
751+
let task_ctx: Arc<TaskContext> = (&rebound.task_ctx_provider)
752+
.try_into()
753+
.expect("rebound codec resolves");
754+
assert_eq!(task_ctx.session_id(), ctx_b.task_ctx().session_id());
755+
}
713756
}

‎datafusion/ffi/src/query_planner.rs‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -442,4 +442,40 @@ mod tests {
442442

443443
Ok(())
444444
}
445+
446+
// Control for https://github.com/apache/datafusion/issues/24722: this
447+
// constructor adopts the supplied codecs on the already-foreign path.
448+
#[test]
449+
fn test_rebind_foreign_query_planner_adopts_codecs() {
450+
use datafusion_execution::TaskContext;
451+
452+
let ctx_a = Arc::new(SessionContext::new());
453+
let ctx_b = Arc::new(SessionContext::new());
454+
let provider_b = Arc::clone(&ctx_b) as Arc<dyn TaskContextProvider>;
455+
456+
let mut ffi_a = create_ffi_query_planner(Arc::clone(&ctx_a));
457+
ffi_a.library_marker_id = crate::mock_foreign_marker_id;
458+
let imported: Arc<dyn QueryPlanner + Send + Sync> = (&ffi_a).into();
459+
let any_ref: &dyn std::any::Any = imported.as_ref();
460+
assert!(any_ref.downcast_ref::<ForeignQueryPlanner>().is_some());
461+
462+
let rebound = FFI_QueryPlanner::new_with_ffi_codecs(
463+
imported,
464+
FFI_LogicalExtensionCodec::new(
465+
Arc::new(DefaultLogicalExtensionCodec {}),
466+
None,
467+
&provider_b,
468+
),
469+
FFI_PhysicalExtensionCodec::new(
470+
Arc::new(DefaultPhysicalExtensionCodec {}),
471+
None,
472+
&provider_b,
473+
),
474+
);
475+
476+
let bound_to: Arc<TaskContext> = (&rebound.logical_codec.task_ctx_provider)
477+
.try_into()
478+
.unwrap();
479+
assert_eq!(bound_to.session_id(), ctx_b.task_ctx().session_id());
480+
}
445481
}

‎datafusion/ffi/src/table_provider.rs‎

Lines changed: 49 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -564,14 +564,26 @@ impl FFI_TableProvider {
564564
)
565565
}
566566

567+
/// Creates an [`FFI_TableProvider`] using a prebuilt FFI logical codec.
568+
///
569+
/// If `provider` is already foreign, this re-exports its original FFI
570+
/// handle rather than adding another wrapper layer. The handle still adopts
571+
/// the `logical_codec` supplied here, so it is never silently discarded and
572+
/// an imported provider can be rebound to a different session.
573+
///
574+
/// `runtime` is only honored when a new wrapper is created. An
575+
/// already-foreign handle keeps the runtime of the library that owns it,
576+
/// because that value lives in private data this side cannot reach.
567577
pub fn new_with_ffi_codec(
568578
provider: Arc<dyn TableProvider>,
569579
can_support_pushdown_filters: bool,
570580
runtime: Option<Handle>,
571581
logical_codec: FFI_LogicalExtensionCodec,
572582
) -> Self {
573583
if let Some(provider) = provider.downcast_ref::<ForeignTableProvider>() {
574-
return provider.0.clone();
584+
let mut provider = provider.0.clone();
585+
provider.logical_codec = logical_codec;
586+
return provider;
575587
}
576588
let private_data = Box::new(ProviderPrivateData { provider, runtime });
577589

@@ -1271,4 +1283,40 @@ mod tests {
12711283

12721284
Ok(())
12731285
}
1286+
1287+
/// Re-wrapping an imported provider with a rebuilt logical codec must adopt
1288+
/// that codec. See <https://github.com/apache/datafusion/issues/24722>.
1289+
#[test]
1290+
fn test_rebind_foreign_table_provider_adopts_logical_codec() -> Result<()> {
1291+
let (_ctx_a, provider_a) = crate::util::tests::test_session_and_ctx();
1292+
let (ctx_b, provider_b) = crate::util::tests::test_session_and_ctx();
1293+
1294+
let mut ffi_provider = FFI_TableProvider::new(
1295+
create_test_table_provider()?,
1296+
true,
1297+
None,
1298+
provider_a,
1299+
None,
1300+
);
1301+
ffi_provider.library_marker_id = crate::mock_foreign_marker_id;
1302+
1303+
let imported: Arc<dyn TableProvider> = (&ffi_provider).into();
1304+
assert!(imported.downcast_ref::<ForeignTableProvider>().is_some());
1305+
1306+
// Rebuild the codec against session B and re-wrap.
1307+
let codec_b = FFI_LogicalExtensionCodec::new(
1308+
Arc::new(DefaultLogicalExtensionCodec {}),
1309+
None,
1310+
provider_b,
1311+
);
1312+
let rebound =
1313+
FFI_TableProvider::new_with_ffi_codec(imported, true, None, codec_b);
1314+
1315+
let task_ctx: Arc<TaskContext> = (&rebound.logical_codec.task_ctx_provider)
1316+
.try_into()
1317+
.expect("rebound provider's codec resolves");
1318+
assert_eq!(task_ctx.session_id(), ctx_b.task_ctx().session_id());
1319+
1320+
Ok(())
1321+
}
12741322
}

0 commit comments

Comments
 (0)