You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add an example of an in-process model of distributed execution #23303
Is your feature request related to a problem or challenge?
DataFusion's physical plans are written and tested assuming everything runs in one process: one Arc<dyn ExecutionPlan> instance, state shared via Arc, and all partitions of an operator polled together by cooperative threads on a shared runtime.
Distributed engines built on DataFusion (Ballista, datafusion-distributed, comet, and proprietary engine like at Coralogix) violate those assumptions.
Describe the solution you'd like
Add a small educational example that runs queries in a distributed mode:
A example** of how distributed execution works on top of DataFusion: breaking a query into stages, serializing plans, and executing tasks in isolation.
A regression guard: once a subset of its tests runs in CI, it catches the class of change that breaks isolated execution before it ships to downstream projects.
The crate would be excluded from the workspace so it has zero impact on normal builds and published crates; it builds in CI.
Activity