You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Substrait roundtrip loses pushed-down TableScan offsets and limits #26034
Substrait roundtripping can change query results by dropping TableScan::skip and TableScan::fetch.
The producer converts a table scan into a ReadRel, which cannot represent either field. from_limit compensates for the scan offset only when its immediate input is a TableScan. A standalone scan, or a scan separated from its parent limit by a projection, loses the pushed-down offset.
To Reproduce
Register a single CSV file as t, with one integer column x containing 0, 1, 2, 3, 4, 5. Build and roundtrip a plan using the public APIs:
let source = provider_as_source(ctx.table_provider("t").await?);let scan = TableScanBuilder::new("t", source).with_skip(Some(2)).with_fetch(Some(2)).build()?;let plan = LogicalPlan::TableScan(scan);let proto = to_substrait_plan(&plan,&ctx.state())?;let restored = from_substrait_plan(&ctx.state(),&proto).await?;let before = DataFrame::new(ctx.state(), plan).collect().await?;let after = DataFrame::new(ctx.state(), restored).collect().await?;
The original plan returns [2, 3]; the restored standalone scan returns all six rows. With Limit(0, 2) -> Projection(x) -> TableScan(skip=2, fetch=2), the restored plan returns [0, 1]. The direct Limit(0, 2) -> TableScan case happens to retain [2, 3] because of the parent-limit compensation.
An independently verified offset-only reproduction on 60477917bd9eadd1a299185db6a8a08ccae7a4c7 returns [2, 3, 4, 5] before roundtripping and [0, 1, 2, 3, 4, 5] afterward.
Expected behavior
Roundtripping preserves the scan's offset and row limit regardless of whether a parent limit exists or a projection separates the nodes.
Additional context
A fix can encode the scan's skip/fetch as a FetchRel wrapping its ReadRel, and remove the parent-limit offset compensation to avoid applying the scan offset twice. Regression coverage should compare executed results, including offset-only scans, offset plus fetch, projections, nested limits, and scans without an offset.
Activity
fix(substrait): Preserve pushed-down table scan offsets and limits (a…