Skip to content

[C++] Scanner -> ScanNode #31787

Closed as not planned
Closed as not planned

Description

@asfimport

This is going to be a parent issue for a number of changes I'd like to make to the scanner.

  • I'd like to remove the concept of the scanner from the public API
  • Related to that, ScanOptions will probably break into two classes. QueryOptions will be the public facing half while ScanOptions will be options sent to the ScanNode. Most users won't see ScanOptions as it will be internal.
  • For example, QueryOptions will have batch_readahead which represents how many "query engine batches" to readahead. Since files & the query engine have different ideas of what constitutes a "batch" the related property in ScanOptions will be rows_to_readahead.
  • Another example is projection, in QueryOptions projection is column selection as well as custom projection expressions that a user wants to run. In ScanOptions "projection" is the desired list of columns and the output type for each column, which controls casting and inference.
  • Partially related (and partially unrelated) to the above two items I would like to move the scanner away from AsyncGenerator and recast it as an execution engine node.
  • The Scanner class will become deprecated and eventually go away. Some methods like Scanner::ToTable may move into a new QueryBuilder or Query object.

Reporter: Weston Pace / @westonpace

Related issues:

Note: This issue was originally created as ARROW-16410. Please see the migration documentation for further details.

Activity

  1. asfimport commented on Aug 26, 2022

    @asfimport
    Author

    Todd Farmer / @toddfarmer:
    This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned per project policy. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.

  2. coady commented on Aug 6, 2024

    @coady

    Is it still the plan to remove the Scanner, particularly in the Python API? Thanks.

    I'm using the Acero engine for out-of-core grouping, but it doesn't support scanners as a source, and its API is still documented as experimental.

  3. github-actions commented on Mar 30, 2026

    @github-actions

    This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.

  4. added
    Status: stale-warningIssues and PRs flagged as stale which are due to be closed if no indication otherwise
    on Mar 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions