Four gaps for a query planner in datafusion-python's process, through its FFI — draft for upstream #58

🪭 Claude (einfold)

Spike S29 (report) runs einfold inside xarray-sql, an XQL Systems project for querying xarray Datasets with SQL. It installs einfold as a query planner through datafusion-python's FFI, with SessionContext.with_extensions. datafusion-python is Apache DataFusion's Python bindings, and its FFI is DataFusion's stable C interface between separately built libraries. Versions: datafusion-python 55.0.0, built with DataFusion and datafusion-ffi 55.2.0.

It works, but only through four workarounds in the planner's library. Each is a gap a planner living in the same process as the session will hit. Drafted here for the maintainer to decide what goes upstream, and where.

Reproduction. The spike's run.py, gradient.py and overhead.py. Remove the matching workaround from src/lib.rs to see each failure. These need the Python host, so einfold's Rust tests can't pin them.

1. A table can't reach the planner without a codec

datafusion-python hands the planner its logical plan as protobuf. A scan of an in-memory table, as register_record_batches and xarray-sql's from_dataset create, fails before the planner runs:

ValueError: Error serializing custom table at …/datafusion-proto-55.2.0/src/logical_plan/mod.rs:1534
caused by
This feature is not implemented: LogicalExtensionCodec is not provided

The spike's codec encodes any table as an index into a registry in this process, and decodes it back to the same table. Ask: a built-in by-reference codec, or a hook that hands the planner the plan as a handle, for planners in the host's process.

2. Built-in functions arrive as FFI wrappers

Decoded through the host's registry, sum is an FFI wrapper of the host's AggregateUDF, not datafusion::functions_aggregate::sum::Sum. Rules that match built-ins by type, as DataFusion's docs advise, stop matching. The spike relinks a foreign aggregate named sum to its own copy. That's unsound if a session registered another sum. Ask: decode DataFusion's built-in functions as the receiving library's own.

3. The physical plan returns as protobuf, tables included

The planner's physical plan returns to the host as protobuf. Its standard nodes come back as the host's own, and DataFusion's protobuf encodes an in-memory scan with its data. So a plan copies every table it reads on the way back: 800,000 rows, about 9 ms per query here. Nodes protobuf doesn't know need a physical codec. The spike returns each plan inside one node, with no visible children, passed by reference. Ask: an option for the result to come back as an FFI_ExecutionPlan handle.

4. A library's plans run without its Tokio runtime, and a spawn aborts Python

The planner's library holds its own copy of Tokio, and the host polls its plans from the host's threads. Any operator in the planner's library that spawns, as collect_partitioned, RepartitionExec and SpawnedTask do, then panics:

there is no reactor running, must be called from the context of a Tokio 1.x runtime
panic in a function that cannot unwind … aborting.

That happened even with a runtime handle passed to FFI_LogicalExtensionCodec::new and FFI_PhysicalExtensionCodec::new. The spike enters a runtime of its own whenever the host polls its plan, and einfold now runs without one (#57). Ask:

• datafusion-ffi enters the runtime a library supplied whenever it executes or polls that library's plans;
• at the least, it catches the panic instead of aborting.

Also measured

Installing the planner costs every query 1.4–3.4 ms, folds or not: SELECT 1 takes 1.65 ms rather than 0.24. Most of that is the logical plan's protobuf round trip and the session's calls across the FFI. That matters for xarray-sql's demos, which plan thousands of statements.

🤖 Generated with Claude Code

https://claude.ai/code/session_01M96xSez6PZwTmfaGh2eJgS