[R] Scanning a dataset whose parts have R extension types with different metadata aborts the R process #55605

Environment

• arrow (R package) 26.0.1 (apache.r-universe.dev binary), also seen on 26.0.0 (CRAN binary)
• R 4.6.2 (2026-10-29)
• macOS 15 (Darwin 24.6.0), x86_64

Description

Related: #37302 has the same "from a non-R thread" failure at a different call site (SafeRecordBatchReader::ReadNext()).

A dataset can have two files whose columns use the same R-registered extension type (same extension_name) with different extension_metadata. Collecting a threaded scan of that dataset terminates the R process. arrow does not raise an R error.

The cause is in RExtensionType::ExtensionEquals() (r/src/extension-impl.cpp):

1. It compares the extension names in C++ and returns false if they differ.
2. It compares Serialize() in C++ and returns true if they are equal.
3. Otherwise it calls the R6 ExtensionEquals() method through SafeCallIntoR().

During a dataset scan, step 3 runs on a worker thread outside RunWithCapturedR(), so SafeCallIntoR() returns an error status. ExtensionEquals() turns that status into throw std::runtime_error(...). Nothing catches the exception, so libc++abi calls std::terminate().

This happens with the default ExtensionType$ExtensionEquals() as well as with an overridden one; the R method is never reached.

Minimal reproducible example

library(arrow, warn.conflicts = FALSE)

# An extension type over int32 whose metadata records a "unit".
UnitType <- R6::R6Class("UnitType", inherit = ExtensionType)
unit_type <- function(unit) {
  new_extension_type(
    int32(),
    extension_name = "example.unit",
    extension_metadata = unit,
    type_class = UnitType
  )
}
register_extension_type(unit_type("m"))

dir <- tempfile()
part <- function(unit, name) {
  tab <- arrow_table(
    x = new_extension_array(Array$create(1:2), unit_type(unit))
  )
  write_dataset(tab, dir, basename_template = name)
}
part("m", "a-{i}.parquet")
part("km", "b-{i}.parquet")

# The two parts' types differ only in metadata, so ExtensionEquals() cannot
# decide in C++ and calls into R from a scan thread.
dplyr::collect(open_dataset(dir))

Observed

The R session terminates:

libc++abi: terminating due to uncaught exception of type std::runtime_error: Call to R (RExtensionType$ExtensionEquals()) from a non-R thread from an unsupported context

Scanner$create(open_dataset(dir))$ToTable() terminates the same way.

Paths where the comparison runs on the R thread give an ordinary R error instead:

• concat_arrays() reports "arrays to be concatenated must be identically typed".
• Scanner$create(..., use_threads = FALSE) and dplyr::arrange(...) |> dplyr::collect() report "Casting from 'X' to different extension type 'Y' not permitted".
• open_dataset(dir, unify_schemas = TRUE) reports "Unable to merge".

Expected

The two types are unequal, so the scan should fail with an ordinary R error, as the other paths above do. It should not call back into R off the R thread and should not terminate the process.

Two possible fixes:

• When SafeCallIntoR() cannot run, have ExtensionEquals() return false, since the serialized forms differ, rather than throwing.
• Or keep the comparison entirely in C++, treating types with differing Serialize() output as unequal by default, and call into R only when the R6 class overrides ExtensionEquals().

Why it matters

Extension types are the documented way for an R package to carry a parameterised type through Arrow. In our case that is a decimal type whose metadata names the storage precision and scale. Because the scan aborts, the package can't use its metadata to tell instances apart. If it leaves the metadata identical, arrow treats types with different storage as equal and can reinterpret one part's stored integers at another part's scale.