Skip to content

Commit 846fe6e

Browse files
timsaucerclaude
andcommitted
Link the upstream FFI issue, stop teaching execute by counter-example
The `ForeignExecutionPlan` arm's comment explained the symptom but left the reader no way to find out whether the workaround is still needed. Name the upstream umbrella issue, apache/datafusion#25152, and the cascade behind it: `FFI_PlanProperties` carries no `scheduling_type`, so `EnsureCooperative` reads every foreign leaf as non-cooperative and wraps it, and the resulting `ForeignExecutionPlan` then cannot serialize itself. Fixing either half retires the arm. Drop the out-of-range call from `SessionContext.execute`'s doctest. A docstring example shows a reader how to use the method, and this one put a wrong call in front of them; the message it asserted is already pinned by `test_execute_rejects_an_out_of_range_partition`. The `Raises:` section is the right home for that behaviour, so complete it: a negative or oversized index raises `OverflowError` from the `usize` conversion, not the `ValueError` the bounds check produces. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent 4c097c7 commit 846fe6e

2 files changed

Lines changed: 9 additions & 3 deletions

File tree

‎examples/datafusion-ffi-example/src/physical_extension_codec.rs‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,13 @@ impl PhysicalExtensionCodec for CountingPhysicalExtensionCodec {
140140
//
141141
// A library whose planner controls its own physical optimizer rules
142142
// never sees a foreign node and needs no such arm.
143+
//
144+
// Both halves are upstream defects, tracked together in
145+
// https://github.com/apache/datafusion/issues/25152: `FFI_PlanProperties`
146+
// carries no `scheduling_type`, so `EnsureCooperative` reads every
147+
// foreign leaf as non-cooperative and wraps it, and the resulting
148+
// `ForeignExecutionPlan` then has no way to serialize itself. Fixing
149+
// either one retires this arm.
143150
if node.is::<DataSourceExec>() || node.is::<ForeignExecutionPlan>() {
144151
self.counters
145152
.encode_execution_plan

‎python/datafusion/context.py‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2364,6 +2364,8 @@ def execute(self, plan: ExecutionPlan, partition: int) -> RecordBatchStream:
23642364
23652365
Raises:
23662366
ValueError: If ``partition`` is not a valid index for ``plan``.
2367+
OverflowError: If ``partition`` is negative, or too large to fit a
2368+
platform-sized unsigned integer.
23672369
23682370
Example usage:
23692371
@@ -2382,9 +2384,6 @@ def execute(self, plan: ExecutionPlan, partition: int) -> RecordBatchStream:
23822384
... for batch in ctx.execute(plan, p)
23832385
... )
23842386
3
2385-
>>> ctx.execute(plan, 2)
2386-
Traceback (most recent call last):
2387-
ValueError: Partition index 2 is out of range for a plan with 2 partition(s)
23882387
"""
23892388
return RecordBatchStream(self.ctx.execute(plan._raw_plan, partition))
23902389

0 commit comments

Comments
 (0)