Skip to content

Bring back to_arrow_table and to_arrow_reader on DB-API results over a re-feed of the rows - #702

Merged
evertlammerts merged 1 commit into
duckdb:v2.0-cyanopterafrom
evertlammerts:arrow-dbapi
Oct 20, 2026
Merged

evertlammerts merged 1 commit into
duckdb:v2.0-cyanopterafrom
evertlammerts:arrow-dbapi

Conversation

@evertlammerts

Copy link
Copy Markdown
Member

Brings back to_arrow_table(batch_size) and to_arrow_reader(batch_size) on DB-API results, on the connection and as module functions, over a re-feed of the rows the result holds.

The engine fixes a result's format at submission and execute() submits in the chunk format, so a result can no longer become an Arrow one. Instead, the rows the result still holds are moved into one ColumnDataCollection and submitted again in the Arrow format on the same connection, through the Session::Submit over a collection that 1dd1ab5 deleted. The conversion is the one the relational API's Arrow terminals use, so the two agree exactly: column names (duplicates included), batch sizes, types and settings. Each call emits a UserWarning pointing at con.sql(query).to_arrow_table(); deprecating or promoting them is for after 2.0.0.

Behavior against 1.5.x:

  • to_arrow_reader no longer streams: the query runs to its end before the first batch. A table takes about 1.5x the relational call's peak memory, same time.
  • After a partial row fetch, Arrow returns the remainder instead of raising "already fetched".
  • With preserve_insertion_order = false the rows come back in a different block order per call.
  • An aborted transaction, another open result and CONNECT mode refuse before anything is taken, rows still fetchable.

@evertlammerts
evertlammerts added this pull request to the merge queue Oct 20, 2026
@evertlammerts
evertlammerts removed this pull request from the merge queue due to a manual request Oct 20, 2026
@evertlammerts
evertlammerts merged commit 2f2a0dd into duckdb:v2.0-cyanoptera Oct 20, 2026
30 checks passed
@evertlammerts
evertlammerts deleted the arrow-dbapi branch October 20, 2026 11:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant