test: Add a test for RowFilter with nested type#5600
Conversation
| // Filter on the second element of the struct. | ||
| let struct_array = batch | ||
| .column(0) | ||
| .as_any() | ||
| .downcast_ref::<StructArray>() | ||
| .unwrap(); | ||
| eq(struct_array.column(0), &Scalar::new(&b_scalar)) |
There was a problem hiding this comment.
Btw, the row filter needs to know what the schema is so it can get correct (nested) column to do filtering. For the general filter implementation like apache/iceberg-rust#295 proposes to be, is any utility we can use to "flatten" nested columns from the batch?
In other words, is any existing way to flatten projected (nested) columns in the batch? So if we know a leaf column's index, we can know its position in projection mask and the flatten batch. Then we can simply get the column by flatten_batch.column.
There was a problem hiding this comment.
I think the challenge here is when the nested arrays are either repeated or nullable, in such a case trying to interpret the leaves in isolation isn't necessarily meaningful
2 participants
Which issue does this PR close?
Closes #.
Rationale for this change
While working on apache/iceberg-rust#295, I'm confused by the nested column behavior of
RowFilter. So writing the test to clarify its usage.What changes are included in this PR?
Are there any user-facing changes?