Skip to content

Commit f2b4835

Browse files
authored
feat: Add support for explode_outer function for arrays (apache#22100)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes apache#19053. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> DataFusion's 'unnest' had no way to express Spark 'explode_outer' semantics, empty input lists were silently dropped, even with 'preserve_nulls' = true. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Adds a third unnest behavior that produces a `NULL` row for empty input lists, by replacing `UnnestOptions.preserve_nulls: bool` with a `NullHandling { Drop, Preserve, PreserveAndExpandEmpty }` enum. The `with_preserve_nulls(bool)` builder is kept as a backward-compat shim. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes, new unit test for the empty-list case, extended longest-length and DataFrame `unnest_column_nulls` tests, and existing proto round-trip coverage. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> The `preserve_nulls` field on `UnnestOptions` is renamed to `null_handling`. The `with_preserve_nulls(bool)` builder is preserved, so most callers are unaffected. Add the `api change` label for the field rename.
1 parent 062cbcb commit f2b4835

22 files changed

Lines changed: 1247 additions & 85 deletions

File tree

‎datafusion/common/src/lib.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,7 @@ pub use schema_reference::SchemaReference;
9999
pub use spans::{Location, Span, Spans};
100100
pub use stats::{ColumnStatistics, Statistics};
101101
pub use table_reference::{ResolvedTableReference, TableReference};
102-
pub use unnest::{RecursionUnnestOption, UnnestOptions};
102+
pub use unnest::{NullHandling, RecursionUnnestOption, UnnestOptions};
103103
pub use utils::project_schema;
104104

105105
// These are hidden from docs purely to avoid polluting the public view of what this crate exports.

‎datafusion/common/src/unnest.rs‎

Lines changed: 76 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -19,23 +19,38 @@
1919
2020
use crate::Column;
2121

22+
/// How [`UnnestOptions`] handles `NULL` and empty list values in the input column.
23+
///
24+
/// The variants enumerate the three observable behaviors so that callers do
25+
/// not have to compose multiple boolean flags to express what they want.
26+
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, PartialOrd, Hash)]
27+
pub enum NullHandling {
28+
/// Drop rows where the input list is `NULL` or empty. Matches the
29+
/// default behavior of systems such as DuckDB and ClickHouse.
30+
Drop,
31+
/// Preserve `NULL` input rows as a single output row containing `NULL`.
32+
/// Empty lists still produce zero output rows. This is the default and
33+
/// matches DataFusion's historical `preserve_nulls = true` behavior.
34+
#[default]
35+
Preserve,
36+
/// Like [`Self::Preserve`], and additionally treat an empty list
37+
/// identically to a `NULL` list, producing a single output row
38+
/// containing `NULL`.
39+
PreserveAndExpandEmpty,
40+
}
41+
2242
/// Options for unnesting a column that contains a list type,
2343
/// replicating values in the other, non nested rows.
2444
///
2545
/// Conceptually this operation is like joining each row with all the
2646
/// values in the list column.
2747
///
28-
/// If `preserve_nulls` is false, nulls and empty lists
29-
/// from the input column are not carried through to the output. This
30-
/// is the default behavior for other systems such as ClickHouse and
31-
/// DuckDB
32-
///
33-
/// If `preserve_nulls` is true (the default), nulls from the input
34-
/// column are carried through to the output.
48+
/// The behavior with `NULL` and empty input lists is controlled by
49+
/// [`NullHandling`]. See its variants for full details.
3550
///
3651
/// # Examples
3752
///
38-
/// ## `Unnest(c1)`, preserve_nulls: false
53+
/// ## `Unnest(c1)`, null_handling: NullHandling::Drop
3954
/// ```text
4055
/// ┌─────────┐ ┌─────┐ ┌─────────┐ ┌─────┐
4156
/// │ {1, 2} │ │ A │ Unnest │ 1 │ │ A │
@@ -49,7 +64,7 @@ use crate::Column;
4964
/// c1 c2
5065
/// ```
5166
///
52-
/// ## `Unnest(c1)`, preserve_nulls: true
67+
/// ## `Unnest(c1)`, null_handling: NullHandling::Preserve
5368
/// ```text
5469
/// ┌─────────┐ ┌─────┐ ┌─────────┐ ┌─────┐
5570
/// │ {1, 2} │ │ A │ Unnest │ 1 │ │ A │
@@ -63,13 +78,30 @@ use crate::Column;
6378
/// c1 c2 c1 c2
6479
/// ```
6580
///
81+
/// ## `Unnest(c1)`, null_handling: NullHandling::PreserveAndExpandEmpty
82+
/// ```text
83+
/// ┌─────────┐ ┌─────┐ ┌─────────┐ ┌─────┐
84+
/// │ {1, 2} │ │ A │ Unnest │ 1 │ │ A │
85+
/// ├─────────┤ ├─────┤ ├─────────┤ ├─────┤
86+
/// │ null │ │ B │ │ 2 │ │ A │
87+
/// ├─────────┤ ├─────┤ ────────────▶ ├─────────┤ ├─────┤
88+
/// │ {} │ │ D │ │ null │ │ B │
89+
/// ├─────────┤ ├─────┤ ├─────────┤ ├─────┤
90+
/// │ {3} │ │ E │ │ null │ │ D │
91+
/// └─────────┘ └─────┘ ├─────────┤ ├─────┤
92+
/// c1 c2 │ 3 │ │ E │
93+
/// └─────────┘ └─────┘
94+
/// c1 c2
95+
/// ```
96+
///
6697
/// `recursions` instruct how a column should be unnested (e.g unnesting a column multiple
6798
/// time, with depth = 1 and depth = 2). Any unnested column not being mentioned inside this
6899
/// options is inferred to be unnested with depth = 1
69100
#[derive(Debug, Clone, PartialEq, PartialOrd, Hash, Eq)]
70101
pub struct UnnestOptions {
71-
/// Should nulls in the input be preserved? Defaults to true
72-
pub preserve_nulls: bool,
102+
/// How to handle `NULL` and empty list values in the input column.
103+
/// Defaults to [`NullHandling::Preserve`].
104+
pub null_handling: NullHandling,
73105
/// If specific columns need to be unnested multiple times (e.g at different depth),
74106
/// declare them here. Any unnested columns not being mentioned inside this option
75107
/// will be unnested with depth = 1
@@ -88,8 +120,7 @@ pub struct RecursionUnnestOption {
88120
impl Default for UnnestOptions {
89121
fn default() -> Self {
90122
Self {
91-
// default to true to maintain backwards compatible behavior
92-
preserve_nulls: true,
123+
null_handling: NullHandling::Preserve,
93124
recursions: vec![],
94125
}
95126
}
@@ -101,13 +132,41 @@ impl UnnestOptions {
101132
Default::default()
102133
}
103134

104-
/// Set the behavior with nulls in the input as described on
105-
/// [`Self`]
106-
pub fn with_preserve_nulls(mut self, preserve_nulls: bool) -> Self {
107-
self.preserve_nulls = preserve_nulls;
135+
/// Set the [`NullHandling`] mode used when unnesting `NULL` or empty
136+
/// input lists.
137+
pub fn with_null_handling(mut self, null_handling: NullHandling) -> Self {
138+
self.null_handling = null_handling;
108139
self
109140
}
110141

142+
/// Backward-compatible setter that maps the previous boolean
143+
/// `preserve_nulls` flag onto [`NullHandling`].
144+
///
145+
/// `true` maps to [`NullHandling::Preserve`]; `false` maps to
146+
/// [`NullHandling::Drop`]. To opt into the new empty-list-preserving
147+
/// mode, call [`Self::with_null_handling`] directly with
148+
/// [`NullHandling::PreserveAndExpandEmpty`].
149+
pub fn with_preserve_nulls(self, preserve_nulls: bool) -> Self {
150+
let null_handling = if preserve_nulls {
151+
NullHandling::Preserve
152+
} else {
153+
NullHandling::Drop
154+
};
155+
self.with_null_handling(null_handling)
156+
}
157+
158+
/// Returns true if `NULL` input rows produce a single output row
159+
/// containing `NULL`.
160+
pub fn preserve_nulls(&self) -> bool {
161+
!matches!(self.null_handling, NullHandling::Drop)
162+
}
163+
164+
/// Returns true if empty input lists should produce a single
165+
/// output row containing `NULL`.
166+
pub fn expand_empty_as_null(&self) -> bool {
167+
matches!(self.null_handling, NullHandling::PreserveAndExpandEmpty)
168+
}
169+
111170
/// Set the recursions for the unnest operation
112171
pub fn with_recursions(mut self, recursion: RecursionUnnestOption) -> Self {
113172
self.recursions.push(recursion);

‎datafusion/core/tests/dataframe/mod.rs‎

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4370,6 +4370,28 @@ async fn unnest_column_nulls() -> Result<()> {
43704370
);
43714371

43724372
let options = UnnestOptions::new().with_preserve_nulls(false);
4373+
let results = df
4374+
.clone()
4375+
.unnest_columns_with_options(&["list"], options)?
4376+
.collect()
4377+
.await?;
4378+
assert_snapshot!(
4379+
batches_to_string(&results),
4380+
@r"
4381+
+------+----+
4382+
| list | id |
4383+
+------+----+
4384+
| 1 | A |
4385+
| 2 | A |
4386+
| 3 | D |
4387+
+------+----+
4388+
"
4389+
);
4390+
4391+
// Outer-unnest semantics: NULL and empty lists both produce a single
4392+
// output row containing NULL.
4393+
let options = UnnestOptions::new()
4394+
.with_null_handling(datafusion_common::NullHandling::PreserveAndExpandEmpty);
43734395
let results = df
43744396
.unnest_columns_with_options(&["list"], options)?
43754397
.collect()
@@ -4382,6 +4404,8 @@ async fn unnest_column_nulls() -> Result<()> {
43824404
+------+----+
43834405
| 1 | A |
43844406
| 2 | A |
4407+
| | B |
4408+
| | C |
43854409
| 3 | D |
43864410
+------+----+
43874411
"
@@ -4390,6 +4414,133 @@ async fn unnest_column_nulls() -> Result<()> {
43904414
Ok(())
43914415
}
43924416

4417+
/// Outer-unnest on a list-of-struct column. Verifies that
4418+
/// (a) struct elements unnest into flattened sub-columns and
4419+
/// (b) NULL and empty lists both still produce a single output row whose
4420+
/// struct sub-columns are all NULL.
4421+
#[tokio::test]
4422+
async fn unnest_outer_list_of_struct() -> Result<()> {
4423+
use arrow::array::{Int32Array, StructArray};
4424+
4425+
// Per-row sub-list lengths: 2, 1, 0 (empty), 0 (null)
4426+
let names = StringArray::from(vec!["alice", "bob", "carol"]);
4427+
let ages = Int32Array::from(vec![30, 40, 50]);
4428+
let struct_values = StructArray::from(vec![
4429+
(
4430+
Arc::new(Field::new("name", DataType::Utf8, true)),
4431+
Arc::new(names) as ArrayRef,
4432+
),
4433+
(
4434+
Arc::new(Field::new("age", DataType::Int32, true)),
4435+
Arc::new(ages) as ArrayRef,
4436+
),
4437+
]);
4438+
let struct_field =
4439+
Arc::new(Field::new("item", struct_values.data_type().clone(), true));
4440+
let offsets = arrow::buffer::OffsetBuffer::<i32>::from_lengths([2, 1, 0, 0]);
4441+
let validity = arrow::buffer::NullBuffer::from(vec![true, true, true, false]);
4442+
let people = ListArray::new(
4443+
struct_field,
4444+
offsets,
4445+
Arc::new(struct_values),
4446+
Some(validity),
4447+
);
4448+
let group = Int32Array::from(vec![1, 2, 3, 4]);
4449+
4450+
let batch = RecordBatch::try_from_iter(vec![
4451+
("people", Arc::new(people) as ArrayRef),
4452+
("group", Arc::new(group) as ArrayRef),
4453+
])?;
4454+
4455+
let ctx = SessionContext::new();
4456+
ctx.register_batch("teams", batch)?;
4457+
let df = ctx.table("teams").await?;
4458+
4459+
let options = UnnestOptions::new()
4460+
.with_null_handling(datafusion_common::NullHandling::PreserveAndExpandEmpty);
4461+
let results = df
4462+
// Unnest the list, then expand the resulting struct rows into columns.
4463+
.unnest_columns_with_options(&["people"], options.clone())?
4464+
.unnest_columns_with_options(&["people"], options)?
4465+
.collect()
4466+
.await?;
4467+
assert_snapshot!(
4468+
batches_to_string(&results),
4469+
@r"
4470+
+-------------+------------+-------+
4471+
| people.name | people.age | group |
4472+
+-------------+------------+-------+
4473+
| alice | 30 | 1 |
4474+
| bob | 40 | 1 |
4475+
| carol | 50 | 2 |
4476+
| | | 3 |
4477+
| | | 4 |
4478+
+-------------+------------+-------+
4479+
"
4480+
);
4481+
4482+
Ok(())
4483+
}
4484+
4485+
/// Outer-unnest applied to a `FixedSizeList` column. For fixed-size lists,
4486+
/// every non-null row has the fixed length, so "empty" never occurs —
4487+
/// `PreserveAndExpandEmpty` should behave identically to `Preserve` here.
4488+
/// The test pins that equivalence so we notice if it ever diverges.
4489+
#[tokio::test]
4490+
async fn unnest_outer_fixed_size_list() -> Result<()> {
4491+
let batch = get_fixed_list_batch()?;
4492+
let ctx = SessionContext::new();
4493+
ctx.register_batch("shapes", batch)?;
4494+
let df = ctx.table("shapes").await?;
4495+
4496+
let preserve_results = df
4497+
.clone()
4498+
.unnest_columns_with_options(
4499+
&["tags"],
4500+
UnnestOptions::new().with_preserve_nulls(true),
4501+
)?
4502+
.collect()
4503+
.await?;
4504+
let outer_results = df
4505+
.unnest_columns_with_options(
4506+
&["tags"],
4507+
UnnestOptions::new().with_null_handling(
4508+
datafusion_common::NullHandling::PreserveAndExpandEmpty,
4509+
),
4510+
)?
4511+
.collect()
4512+
.await?;
4513+
assert_eq!(
4514+
batches_to_sort_string(&preserve_results),
4515+
batches_to_sort_string(&outer_results),
4516+
"FixedSizeList has no empty case, so PreserveAndExpandEmpty must \
4517+
match Preserve exactly"
4518+
);
4519+
4520+
// And the snapshot itself, to make the expected shape explicit.
4521+
assert_snapshot!(
4522+
batches_to_sort_string(&outer_results),
4523+
@r"
4524+
+----------+-------+
4525+
| shape_id | tags |
4526+
+----------+-------+
4527+
| 1 | |
4528+
| 2 | tag21 |
4529+
| 2 | tag22 |
4530+
| 3 | tag31 |
4531+
| 3 | tag32 |
4532+
| 4 | |
4533+
| 5 | tag51 |
4534+
| 5 | tag52 |
4535+
| 6 | tag61 |
4536+
| 6 | tag62 |
4537+
+----------+-------+
4538+
"
4539+
);
4540+
4541+
Ok(())
4542+
}
4543+
43934544
#[tokio::test]
43944545
async fn unnest_fixed_list() -> Result<()> {
43954546
let batch = get_fixed_list_batch()?;

0 commit comments

Comments
 (0)