Repository navigation
Description
Describe the bug
The Substrait consumer maps standard decimal arithmetic to DataFusion operators whose derived precision and scale can differ from the referenced Substrait function definition. It accepts the plan and exposes the different type without reporting the mismatch.
Reproduced on main at 1b6dc92a221b41088afa2753fe5526be6d057b7b.
Each plan references extension:io.substrait:functions_arithmetic_decimal and declares the return type given by its YAML formula. The add, multiply and divide definitions are identical in spec v0.87.0, v0.102.0 and v0.103.0.
| Expression | Spec / declared output_type | DataFusion logical schema |
|---|---|---|
| dec(10,2) + dec(5,1) | dec(11,2) | dec(11,2) |
| dec(38,10) + dec(38,10) | dec(38,9) | dec(38,10) |
| dec(10,2) * dec(5,1) | dec(16,3) | dec(16,3) |
| dec(38,10) * dec(38,10) | dec(38,6) | dec(38,20) |
| dec(10,2) / dec(5,1) | dec(21,8) | dec(15,6) |
Inputs and declared outputs are required. The table focuses on precision and scale, with the two small add/multiply cases serving as controls.
To reproduce
From a DataFusion checkout:
git clone https://github.com/alexandrefimov/substrait-conformance-cases conformance-cases
git -C conformance-cases checkout f12ba6103217fbd50b95c407b5dc3e8fae1e05a0
mkdir -p conformance-inputs datafusion/substrait/examples
cp conformance-cases/derived-schema/decimal_*.json conformance-inputs/
cp conformance-cases/probe/datafusion_corpus_probe.rs datafusion/substrait/examples/corpus_probe.rs
SUBSTRAIT_CORPUS_DIR="$PWD/conformance-inputs" cargo run --locked -p datafusion-substrait --example corpus_probeThe probe registers a table with the exact declared input types. Its DATAFUSION ACCEPTED line reports the logical schema before execution. For example, decimal_divide declares decimal(21,8) and reports Decimal128(15, 6).
Expected behavior
The consumed expression should follow the return-type rule of the referenced decimal function, or reject a function contract it cannot implement. For division here, the YAML gives scale = max(6, 2 + 5 + 1) = 8 and precision = 10 - 2 + 5 + 8 = 21.
from_scalar_function resolves the function name to a native operator and constructs a BinaryExpr without checking its return type against the declaration. This is a Substrait conversion issue, not a request to change the semantics of ordinary DataFusion SQL arithmetic.
The wide-add fixture also reports an execution overflow at scale 10. The core reproducer is the schema mismatch: these plans omit overflow options, so this report does not prescribe a rounding or overflow policy. In particular, the wide-multiply input value can overflow even with the correct declared type.
Activity
Satyr09 commentedon Sep 8, 2026
take
Put the relation corpus through DataFusion
alexandrefimov commentedon Oct 6, 2026
Additional execution case from a SQL producer: DuckDB 1.5.5 with the extension at DuckDB PR 302 head
a454a7c(unreleased) exports this query as standard decimaladdwith outputdecimal(38,6)andoverflow: [ERROR]:DataFusion main
bd8619015accepts the exported plan and returns100000000000000000000000000000000.000000asDecimal128(38,6). Arrow'svalidate_decimal_precision(38)rejects that value. Negative overflow behaves the same; the adjacent in-range value and precision-18 overflow control pass. Native DataFusion SQL also returns the out-of-range value.The original report omitted overflow options. This case explicitly selects ERROR, which the option contract requires the consumer to honor or reject. The default
from_scalar_functionmapsaddto aBinaryExprwithout handlingf.options.Can this case be included in the current fix, or should explicit-option handling be tracked separately?