Repository navigation
[R] read_parquet(col_select =) silently returns the wrong columns when the file has nested (list) columns #55578
Description
read_parquet(col_select = ...) returns incorrect columns, with no error or warning, that
come after a nested column with more than one leaf (e.g. list<struct<...>>). Appears to be an
indexing error introduced by the nested list column.
reprex
library(arrow)
df <- data.frame(a = 1:2)
df$nested <- list(data.frame(x = 1, y = "p"), data.frame(x = 2, y = "q"))
df$b <- c("b1", "b2")
df$c <- c(TRUE, FALSE)
tf <- tempfile(fileext = ".parquet")
write_parquet(df, tf)
names(read_parquet(tf, col_select = "b"))
#> [1] "nested"
names(read_parquet(tf, col_select = "c"))
#> [1] "b"
names(read_parquet(tf, col_select = c("a", "c")))
#> [1] "a" "b"
# Expected: "b", "c", c("a", "c")
# Selecting after reading as a Table gives the right column
names(read_parquet(tf, as_data_frame = FALSE)[, "c"])
#> [1] "c"
# 4 top-level fields, 5 parquet leaf columns
reader <- ParquetFileReader$create(tf)
length(reader$GetSchema()$names)
#> [1] 4
reader$num_columns
#> [1] 5Likely cause
read_parquet() computes indices with tidyselect::eval_select() on
as.data.frame(reader$GetSchema()), i.e. top-level field positions, and
passes them to reader$ReadTable(indices).
ReadTable() appears to treat them as parquet leaf column indices.
Each nested column with more than one leaf shifts every later column, so
the result is off by one leaf per extra leaf.
In the reprex, field index 2 (b) maps to leaf 2 (nested.y), so nested is
returned.
Possible fix
translate the selected field indices to leaf
indices before calling ReadTable(), or read by field.
Reproduced with
arrow23.0.1.2 and 26.0.1 (CRAN binaries)- R 4.6.0, aarch64-apple-darwin23 (macOS)
Component(s)
R
Activity
[-]`read_parquet(col_select =)` silently returns the wrong columns when the file has nested (list) columns[/-][+][R] `read_parquet(col_select =)` silently returns the wrong columns when the file has nested (list) columns[/+]thisisnic commentedon Nov 12, 2026
Thanks for the clear reproducible example, can confirm this is an Arrow bug that we should fix.
stufield commentedon Nov 12, 2026
FWIW, I've been using this workaround locally:
Tho, admittedly this is just a workaround ... sorry it took me a while to post an issue about it 🙏🏽
stufield commentedon Nov 12, 2026
Thanks for taking a look so quickly 👍🏽