All struct-aware optimizations are hardcoded to GetFieldFunc #21306

Activity

1. friendlymatthew

   added the

   Feature

   issue type on Apr 1, 2026
2. friendlymatthew

   added

   enhancementNew feature or requestNew feature or request

   on Apr 1, 2026
3. adriangb commented on Apr 1, 2026

   

   adriangb

   on Apr 1, 2026

   Contributor

   More actions

   There's no trait or capability API for a UDF to declare "I'm a struct field access."

   There is:

   datafusion/datafusion/expr/src/udf.rs

   Lines 972 to 984 in 1416ed4

   	/// Returns placement information for this function.
   	///
   	/// This is used by optimizers to make decisions about expression placement,
   	/// such as whether to push expressions down through projections.
   	///
   	/// The default implementation returns [`ExpressionPlacement::KeepInPlace`],
   	/// meaning the expression should be kept where it is in the plan.
   	///
   	/// Override this method to indicate that the function can be pushed down
   	/// closer to the data source.
   	fn placement(&self, _args: &[ExpressionPlacement]) -> ExpressionPlacement {
   	ExpressionPlacement::KeepInPlace
   	}
4. liningpan commented on Jul 16, 2026

   

   liningpan

   on Jul 16, 2026

   Contributor

   More actions

   I have a query calling a UDF that returns a struct (e.g., fields a and b in the struct). If I manually unnest the struct (e.g., func(x)['a'] and func(x)['b']), datafusion will call my UDF twice and can not optimize it into one call (if I'm reading the execution plan and metrics correctly). It will do the same even if I take the UDF call to a CTE or separate view, so it seems more like an issue with optimization.

   I'm not sure if this is the root cause of this behavior.
5. adriangb commented on Jul 16, 2026

   

   adriangb

   on Jul 16, 2026

   Contributor

   More actions

   @liningpan I don't think that's related to this issue, but it is a very valid point. Could you maybe open an issue to track that? Off the top of my head it feels like something the CSE (Common Sub-expression Elimination) optimizer might be able to handle. A good first step would be an MRE / benchmark showing the perf issue (you could write a UDF that has a sleep in it or something).
6. raghuvanshraj commented on Nov 4, 2026

   

   raghuvanshraj

   on Nov 4, 2026

   More actions

   A measured data point for the Variant case, from #27007 (closing that as a duplicate of this).

   One 100k-row Parquet file, two columns with the same 20 fields: s as a plain struct, v as a Variant shredded on all 20. Reading one small field:

   query	scan projection	bytes_scanned
   SELECT s['f0'] FROM t	get_field(s, f0)	11.2 KB
   SELECT variant_get(v, 'f0', 'BIGINT') FROM t	variant_get(v, f0)	126.4 MB

   The f0 leaf exists on the Variant side at the same size (1,147 bytes per row group) and is never read. With a WHERE on the field, the struct predicate reaches the scan; the Variant one runs in a FilterExec after decoding 126 MB. Standalone reproducer and raw EXPLAIN ANALYZE output in #27007.

   For variant_get the capability needs two things beyond get_field: one argument may require two leaves (the typed leaf and the value fallback), and pruning must keep any row group that holds a fallback row. Of the two open PRs, #26282 does both; #25013 does not cover pruning. Happy to test either against shredded and unshredded fixtures. This is the first workstream in #26517.
7. adriangb commented on Nov 4, 2026

   

   adriangb

   on Nov 4, 2026

   Contributor

   More actions

   In my mind what needs to happen here is:

   1. Variant provides an impl of PhysicalExprAdapter to be hooked up to the parquet scan that reads from shredded columns when present in that particular file.
   2. variant_get et al need to be marked to be pushed down into scans.
8. raghuvanshraj commented on Nov 4, 2026

   

   raghuvanshraj

   on Nov 4, 2026

   Last edited by raghuvanshraj

   More actions

   In my mind what needs to happen here is:

   1. Variant provides an impl of `PhysicalExprAdapter` to be hooked up to the parquet scan that reads from shredded columns when present in that particular file.
   
   2. `variant_get` et al need to be marked to be pushed down into scans.
   

   @adriangb makes sense, thanks. I'll start with the adapter in datafusion-variant since the factory already gets the file schema, so no core change needed for that part. The one thing I want to get right is a plain SELECT v on an unshredded file: it should rebuild the row from value rather than null-fill typed_value.

   For the pushdown side, do you lean towards #25013 or #26282? I'd rather build variant_get against whichever you expect to land.