Skip to content

Commit 8c099dc

Browse files
Dandandanclaude
andauthored
perf(session): cut datafusion-session compile time ~1.7x (#24326)
## Which issue does this PR close? Adresses: #13814 ## Rationale for this change `datafusion-session` is 1,619 lines of source and spends **18.8s** in the frontend during a cold `cargo build -p datafusion` (`cargo build --timings`) — 11.6ms per line, the worst ratio in the workspace, against 0.09–0.55ms/line for crates like `datafusion-datasource` or `datafusion-physical-plan`. `-Zself-profile` puts ~72% of the crate's compile time in `evaluate_obligation`, and 98% of that in `Send`/`Sync`. ## What changes are included in this PR? `TableProvider::{delete_from, update, merge_into}` are one-line `not_impl_err!` stubs that nevertheless build a coroutine capturing `Vec<Expr>` / `Expr` — proving that coroutine `Send` walks the whole `Expr`/`LogicalPlan` type graph. They are now written as the desugaring of `async fn` returning `ready(..)`, so no coroutine is created and there is nothing expensive to prove. The signatures are exactly what `#[async_trait]` generates — verified against `-Zunpretty=expanded` output of this crate — so implementors are unaffected. I originally converted seven bodies, but measuring each one's marginal contribution (reverting one at a time; `evaluate_obligation` self time, all-converted baseline 659ms) showed only the ones taking `Expr` matter: | method left as `async fn` | trait solving | marginal cost | |---|---|---| | `delete_from` | 1.29s | +631ms | | `update` | 1.30s | +641ms | | `merge_into` | 1.29s | +631ms | | `truncate` | 697ms | +38ms | | `insert_into` | 665ms | ~0 | ## Are these changes tested? Interleaved A/B of `cargo rustc -p datafusion-session --lib` ``` base: 1.582s 1.576s 1.590s fix: 0.933s 0.938s 0.932s ``` `evaluate_obligation` drops from 1.28s to 0.646s. Those are standalone-build numbers. In the feature-unified build this crate's frontend is 18.8s rather than ~1.6s (each obligation is several times dearer there), so the absolute saving on a real build should be larger — I have not measured that directly, since isolating one crate's unit in a full build is hard to do without confounding it with machine drift. ## Are there any user-facing changes? No public signature changes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 <[email protected]>
1 parent 519687c commit 8c099dc

3 files changed

Lines changed: 43 additions & 15 deletions

File tree

‎Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎datafusion/session/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ datafusion-common = { workspace = true }
3737
datafusion-execution = { workspace = true }
3838
datafusion-expr = { workspace = true }
3939
datafusion-physical-plan = { workspace = true }
40+
futures = { workspace = true }
4041
parking_lot = { workspace = true }
4142

4243
# Note: add additional linter rules in lib.rs.

‎datafusion/session/src/table.rs‎

Lines changed: 41 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
use std::any::Any;
1919
use std::borrow::Cow;
2020
use std::fmt::Debug;
21+
use std::future::ready;
2122
use std::sync::Arc;
2223

2324
use crate::session::Session;
@@ -27,6 +28,7 @@ use datafusion_common::{Constraints, Statistics, not_impl_err};
2728
use datafusion_common::{DFSchemaRef, Result, internal_err};
2829
use datafusion_expr::Expr;
2930
use datafusion_expr::statistics::StatisticsRequest;
31+
use futures::future::BoxFuture;
3032

3133
use datafusion_expr::dml::{InsertOp, MergeIntoClause};
3234
use datafusion_expr::{
@@ -351,25 +353,41 @@ pub trait TableProvider: Any + Debug + Sync + Send {
351353
///
352354
/// Returns an [`ExecutionPlan`] producing a single row with `count` (UInt64).
353355
/// Empty `filters` deletes all rows.
354-
async fn delete_from(
355-
&self,
356-
_state: &dyn Session,
356+
fn delete_from<'life0, 'life1, 'async_trait>(
357+
&'life0 self,
358+
_state: &'life1 dyn Session,
357359
_filters: Vec<Expr>,
358-
) -> Result<Arc<dyn ExecutionPlan>> {
359-
not_impl_err!("DELETE not supported for {} table", self.table_type())
360+
) -> BoxFuture<'async_trait, Result<Arc<dyn ExecutionPlan>>>
361+
where
362+
'life0: 'async_trait,
363+
'life1: 'async_trait,
364+
Self: 'async_trait,
365+
{
366+
Box::pin(ready(not_impl_err!(
367+
"DELETE not supported for {} table",
368+
self.table_type()
369+
)))
360370
}
361371

362372
/// Update rows matching the filter predicates.
363373
///
364374
/// Returns an [`ExecutionPlan`] producing a single row with `count` (UInt64).
365375
/// Empty `filters` updates all rows.
366-
async fn update(
367-
&self,
368-
_state: &dyn Session,
376+
fn update<'life0, 'life1, 'async_trait>(
377+
&'life0 self,
378+
_state: &'life1 dyn Session,
369379
_assignments: Vec<(String, Expr)>,
370380
_filters: Vec<Expr>,
371-
) -> Result<Arc<dyn ExecutionPlan>> {
372-
not_impl_err!("UPDATE not supported for {} table", self.table_type())
381+
) -> BoxFuture<'async_trait, Result<Arc<dyn ExecutionPlan>>>
382+
where
383+
'life0: 'async_trait,
384+
'life1: 'async_trait,
385+
Self: 'async_trait,
386+
{
387+
Box::pin(ready(not_impl_err!(
388+
"UPDATE not supported for {} table",
389+
self.table_type()
390+
)))
373391
}
374392

375393
/// Remove all rows from the table.
@@ -391,15 +409,23 @@ pub trait TableProvider: Any + Debug + Sync + Send {
391409
/// The `clauses` describe the WHEN MATCHED / WHEN NOT MATCHED actions.
392410
///
393411
/// Returns an [`ExecutionPlan`] producing a single row with `count` (UInt64).
394-
async fn merge_into(
395-
&self,
396-
_state: &dyn Session,
412+
fn merge_into<'life0, 'life1, 'async_trait>(
413+
&'life0 self,
414+
_state: &'life1 dyn Session,
397415
_source: Arc<dyn ExecutionPlan>,
398416
_merge_schema: DFSchemaRef,
399417
_on: Expr,
400418
_clauses: Vec<MergeIntoClause>,
401-
) -> Result<Arc<dyn ExecutionPlan>> {
402-
not_impl_err!("MERGE INTO not supported for {} table", self.table_type())
419+
) -> BoxFuture<'async_trait, Result<Arc<dyn ExecutionPlan>>>
420+
where
421+
'life0: 'async_trait,
422+
'life1: 'async_trait,
423+
Self: 'async_trait,
424+
{
425+
Box::pin(ready(not_impl_err!(
426+
"MERGE INTO not supported for {} table",
427+
self.table_type()
428+
)))
403429
}
404430
}
405431

0 commit comments

Comments
 (0)