Plug into DataFusion Python natively through a custom FFI query planner  #2618

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

The Ballista Python client wraps DataFusion Python: it subclasses SessionContext and DataFrame. Every method that executes a plan is intercepted, the logical plan is serialized, and the plan is resubmitted through a separate Ballista-backed session. As a result:

• methods the wrapper doesn't know about run locally instead of on the cluster;
• every execution creates a new Ballista session;
• the wrapper has to be kept in line with DataFusion Python's API, which is an ongoing maintenance cost;
• users get a DistributedDataFrame instead of a regular datafusion.DataFrame.

Describe the solution you'd like

DataFusion Python 55 can install a custom query planner and extension codecs through FFI. Ballista can use this to plug into a standard datafusion.SessionContext with no wrapper: DataFusion plans each query as usual, and the Ballista planner sends the plan to the scheduler. Any operation that executes a plan (collect, show, to_pandas, write_parquet, EXPLAIN ANALYZE, ...) then runs on the cluster.

Tasks:

• Replace the wrapper-based client with a native FFI query planner and extension codecs (BallistaSessionContext returns a plain datafusion.SessionContext, DistributedDataFrame is removed, Ballista settings are registered as a DataFusion config extension, upgrade guide added) feat(python): Plug into DataFusion Python natively through a custom FFI query planner  #2588
• Python UDF support (scalar, aggregate, window). Python UDFs can't be serialized yet, because no extension codec handles them. feat(python): Add Python UDF support to Ballista Python bindings #2619
• Refactor configuration so it is more ergonomic for Ballista use cases. Settings are currently split between config (local planning only) and cluster_config (sent to the scheduler), and the table of DataFusion options Ballista restricts is duplicated between Rust and Python. The goal is one clear way to configure a Ballista context, with a single source of truth for restricted options. feat(python): Improve ballista session context configuration  #2637
• capture in tests that df.cache is broken at the moment

Describe alternatives you've considered

Keep the wrapper-based client. That leaves the problems above in place, and every DataFusion Python release needs matching wrapper changes.

Additional context

This is a breaking change for the Python client:

• DistributedDataFrame is removed;
• BallistaSessionContext becomes a factory function, not a class;
• explain_visual and collect_with_progress move to ballista.jupyter.

The upgrade guide covers these in detail.

Known limitations, to be tracked separately:

• In-memory table providers (from_pydict, register_record_batches, ...) can't be serialized.
• information_schema scans can't cross the FFI planner boundary.
• Executors don't apply file_compression_type when reading compressed CSV.
• The CSV writer quotes every field when writing a header.
• COPY ... OPTIONS are dropped by datafusion-proto (upstream).
• datafusion-ffi 55.2.0 doesn't support file formats in ForeignLogicalExtensionCodec. The client works around this for DataFrame.write_*, and the workaround should be removed once it's fixed upstream.