You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit d03d393
Browse filesBrowse the repository at this point in the historyBrowse files
feat: protobuf serialization for the materialized CTE plan nodes
Add protobuf encodings for LogicalPlan::MaterializedCte, LogicalPlan::MaterializedCteScan, MaterializedCteExec and MaterializedCteScanExec.
The physical encoding carries only the id of the CTE. The decode context holds one buffer for each id, so a decoded scan and its MaterializedCteExec share the buffer wherever they are in the plan, also when the scan is in a scalar subquery. Decoded buffers get new ids, because the ids of another process can be equal to local ids.
A decoded plan fragment that has a scan without its MaterializedCteExec fails with a clear error when the scan is executed.
Closesapache#26231.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
@@ -53,10 +54,11 @@ use arrow::datatypes::SchemaRef;
53
54
use arrow::record_batch::RecordBatch;
54
55
use datafusion_common::config::ConfigOptions;
55
56
use datafusion_common::tree_node::TreeNodeRecursion;
56
-
use datafusion_common::{Result,Statistics, internal_err};
57
+
use datafusion_common::{Result,Statistics,exec_err,internal_err};
57
58
use datafusion_common_runtime::JoinSet;
58
59
use datafusion_execution::memory_pool::{MemoryConsumer,MemoryReservation};
59
60
use datafusion_execution::{SendableRecordBatchStream,SpillFile,TaskContext};
61
+
use datafusion_expr::MaterializedCteId;
60
62
use datafusion_physical_expr::{EquivalenceProperties,Partitioning,PhysicalExpr};
61
63
use futures::stream::BoxStream;
62
64
use futures::{Stream,StreamExt,TryStreamExt};
@@ -304,6 +306,11 @@ impl MaterializedCteBuffer {
304
306
self.id
305
307
}
306
308
309
+
/// The name of the CTE.
310
+
pubfnname(&self) -> &str{
311
+
&self.name
312
+
}
313
+
307
314
fnset_body(&self,body:Arc<dynExecutionPlan>){
308
315
*self.body.lock() = Some(body);
309
316
}
@@ -318,8 +325,13 @@ impl MaterializedCteBuffer {
318
325
self:&Arc<Self>,
319
326
context:Arc<TaskContext>,
320
327
) -> Result<OnceFut<MaterializedOutput>>{
328
+
// A plan from the planner always has the body. A decoded plan does
329
+
// not have it when the plan was split between the body and a scan.
321
330
letSome(body) = self.body.lock().clone()else{
322
-
returninternal_err!("MaterializedCte {} has no body",self.name);
331
+
returnexec_err!(
332
+
"The materialized CTE {} has a MaterializedCteScanExec but no MaterializedCteExec in this plan. The body and the scans of a materialized CTE must be in one plan and run in one process",
333
+
self.name
334
+
);
323
335
};
324
336
let once = Arc::clone(&self.output.lock());
325
337
let name = self.name.clone();
@@ -331,6 +343,36 @@ impl MaterializedCteBuffer {
331
343
}
332
344
}
333
345
346
+
/// The buffers of the materialized CTEs in one decoded plan.
347
+
///
348
+
/// A [`MaterializedCteExec`] and its [`MaterializedCteScanExec`]s are encoded
349
+
/// as separate nodes that have the same id. The decoder of each node asks for
350
+
/// the buffer of that id, so all of them share one buffer, in any decode
351
+
/// order. A scan in a scalar subquery is not below its `MaterializedCteExec`,
352
+
/// so one registry must be used for the whole plan.
353
+
///
354
+
/// The ids in an encoded plan come from another process and can be equal to
355
+
/// the ids of plans made in this process. Each buffer therefore gets a new id
0 commit comments