Expose a way to process a pa.RecordBatchFileReader without pa.dataset.Dataset #20435

Is your feature request related to a problem or challenge?

I am trying to apply an SQL query on a .feather file. This file cannot be materialized into RAM (too big) but each pa.RecordBatch can be.
I would like to be able to register the .feather file directly, or preferably via a pa.ipc.RecordBatchFileReader, then iter on the filtered pa.RecordBatch via df.execute_stream().

I haven't found any way to do that right now that does not involve a pa.dataset.Dataset.
It is linked to the fact pa.dataset.Dataset cannot be instantiated from a pa.ipc.RecordBatchFileReader...

Describe the solution you'd like

import datafusion
import pyarrow.dataset as pdata

query = "SELECT * FROM archive WHERE foo = 'bar'"

handle = pa.memory_map(path, "rb")
reader = pa.ipc.RecordBatchFileReader(handle)

ctx = datafusion.SessionContext()
ctx.register_batch_reader("archive", reader)
df = ctx.sql(query)

for batch in df.execute_stream():
    # do something
    pass

Describe alternatives you've considered

Right now, I am doing:

import datafusion
import pyarrow.dataset as pdata

query = "SELECT * FROM archive WHERE foo = 'bar'"
dataset = pdata.dataset(path, format="ipc")

ctx = datafusion.SessionContext()
ctx.register_dataset("archive", dataset)
df = ctx.sql(query)

for batch in df.execute_stream():
    # do something
    pass

This works but it forces me to read the .feather file via pa.dataset.Dataset.

Additional context

No response