Substrait intersection schemas retain primary-input nullability #25042

On the INTERSECT plans DuckDB and Isthmus write, the nullability DataFusion derives is the one the spec's table gives; the case that differs needs a nullable primary input against a required secondary one, which neither of them writes.

Describe the bug

The Substrait consumer derives all three supported intersection schemas from the primary input's nullability. A field that the intersection makes required can therefore remain nullable in the logical output schema.

Reproduced on main at 1b6dc92a221b41088afa2753fe5526be6d057b7b.

The Substrait Set Operation rules define different nullability rules for each operation. With the spec's three input patterns (R = required, N = nullable), all fields having type i64:

primary:     R R R R N N N N
secondary 1: R R N N R R N N
secondary 2: R N R N R N R N

the consumer returns:

To reproduce

The plans and the table-registration probe are pinned in the conformance corpus. From a DataFusion checkout:

git clone https://github.com/alexandrefimov/substrait-conformance-cases conformance-cases
git -C conformance-cases checkout f12ba6103217fbd50b95c407b5dc3e8fae1e05a0
mkdir -p conformance-inputs datafusion/substrait/examples
cp conformance-cases/derived-schema/setop_*.json conformance-inputs/
cp conformance-cases/probe/datafusion_corpus_probe.rs datafusion/substrait/examples/corpus_probe.rs
SUBSTRAIT_CORPUS_DIR="$PWD/conformance-inputs" cargo run --locked -p datafusion-substrait --example corpus_probe

The probe registers empty named tables with exactly those input schemas and prints the logical schema immediately after from_substrait_plan, before optimization or execution. The setop_intersection_* lines show the three mismatches. Union and primary-minus controls match their respective spec rules; MINUS_MULTISET is explicitly unsupported.

Expected behavior

For multiset intersections, a field is required if any input requires it. For primary intersection, it is nullable only when it is nullable in the primary input and in at least one secondary input.

from_set_rel delegates to LogicalPlanBuilder::intersect, which uses a left semi join. Keeping the left field nullability there explains the observed schema. This report concerns the schema of consumed Substrait plans; it does not assert incorrect rows. The multiplicity bugs #12955 and #12956 are separate.