Repository navigation
Closed as not planned
Labels
Component: C++Status: stale-warningIssues and PRs flagged as stale which are due to be closed if no indication otherwiseIssues and PRs flagged as stale which are due to be closed if no indication otherwiseType: enhancement
Description
This is going to be a parent issue for a number of changes I'd like to make to the scanner.
- I'd like to remove the concept of the scanner from the public API
- Related to that, ScanOptions will probably break into two classes. QueryOptions will be the public facing half while ScanOptions will be options sent to the ScanNode. Most users won't see ScanOptions as it will be internal.
- For example, QueryOptions will have
batch_readaheadwhich represents how many "query engine batches" to readahead. Since files & the query engine have different ideas of what constitutes a "batch" the related property in ScanOptions will berows_to_readahead. - Another example is projection, in QueryOptions projection is column selection as well as custom projection expressions that a user wants to run. In ScanOptions "projection" is the desired list of columns and the output type for each column, which controls casting and inference.
- Partially related (and partially unrelated) to the above two items I would like to move the scanner away from AsyncGenerator and recast it as an execution engine node.
- The
Scannerclass will become deprecated and eventually go away. Some methods likeScanner::ToTablemay move into a newQueryBuilderorQueryobject.
Reporter: Weston Pace / @westonpace
Related issues:
- [Python][R] Expose (hide?) ScanOptions (depends upon)
- [C++] Add support for Future (depends upon)
- [C++] Migrate scanner logic to ExecPlan, remove merged generator (depends upon)
- [C++][Python][R] Deprecate "scanner" (but keep "scan node") from public API (depends upon)
Note: This issue was originally created as ARROW-16410. Please see the migration documentation for further details.
Metadata
Metadata
Assignees
Labels
Component: C++Status: stale-warningIssues and PRs flagged as stale which are due to be closed if no indication otherwiseIssues and PRs flagged as stale which are due to be closed if no indication otherwiseType: enhancement
Activity
asfimport commentedon Aug 26, 2022
Todd Farmer / @toddfarmer:
This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned per project policy. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.
coady commentedon Aug 6, 2024
Is it still the plan to remove the
Scanner, particularly in the Python API? Thanks.I'm using the Acero engine for out-of-core grouping, but it doesn't support scanners as a source, and its API is still documented as experimental.
github-actions commentedon Mar 30, 2026
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.