[Python][R] Expose (hide?) ScanOptions

@westonpace

Currently R completely hides the ScanOptions class.

In python the class is exposed but the documentation prefers dataset.scan (which hides both the scanner and the scan options).

However, there is some useful information in the ScanOptions. Specifically, the projected schema (which is a product of the dataset schema and the projection expression and not easily recreated) and the materialized fields (the list of fields referenced by either the filter or the projection) which might be useful for reporting purposes.

Currently R uses the projected schema to convert a list of column names into a partition schema. Python does not rely on either field.

Options:

- Keep the status quo

- Expose the ScanOptions object (which itself is exposed via the Scanner)

- Expose the interesting fields via the Scanner

Currently the C++ design is halfway between the latter two (projected schema is exposed and options). My preference would be the third option. It raises a further question about how to expose the scanner itself in Python? Should the user be using ScannerBuilder? Should they use NewScan? Should they use the scanner directly at all or should it be hidden?

Reporter: Weston Pace / @westonpace

Related issues:

[C++] Scanner -> ScanNode (is depended upon by)

_{Note: This issue was originally created as ARROW-12311. Please see the migration documentation for further details.}

@lidavidm

David Li / @lidavidm:
Python is a bit inconsistent IMO: there are methods on Dataset that create a scanner for you and call to_table/scan_batches (this is fine). The scanner itself is also exposed (but not the builder), getting a scanner itself is less obvious (have to manually construct a scanner). The better thing would be to have dataset.scan(**kwargs) exposed. And then I would also lean towards #3. Particularly the projected schema doesn't really feel like an 'option' anyways.

@jorisvandenbossche

Joris Van den Bossche / @jorisvandenbossche:
Agreed on option 3, if we can more easily expose the Scanner object in Python, that seems the appropriate place to expose interesting fields (like the projected schema).

It's a bit a pity that Dataset.scan() already creates a Scanner ànd starts the scan, otherwise that method could have returned a Scanner instead. Basically it's Dataset._scanner(**kwargs) that we want to expose from the Dataset, right? (without the user having to call Scanner.from_dataset(dataset, **kwargs)

@nealrichardson

Neal Richardson / @nealrichardson:
If there are useful things on ScanOptions and we can get them from Scanner, that would work from my perspective in R. Users generally don't create a Scanner themselves but Scanner$create(dataset, filter_expr, projection, ...) exists and is used inside functions that do the scanning. If Scanner had useful information about the scan-to-be and were cheap to construct, that could be useful.

@lidavidm

David Li / @lidavidm:
From @westonpace on this PR, there are some other things we can clean up on ScanOptions:

Not for this PR but I feel like there are a number of places where we have to do something like this to maintain implicit invariants. It might be nice to have a ScanOptions cleanup at some point. Change the name so it isn't Options, hide it from the public API, give it more methods (like SetFilter) and private state.

@paleolimbot

Dewey Dunnington / @paleolimbot:
If this is still a go, here's the Scanner object: https://github.com/apache/arrow/blob/master/r/R/dataset-scan.R#L63-L66

And we expose options()->projected_schema here: https://github.com/apache/arrow/blob/master/r/src/dataset.cpp#L495-L498

Definition of the ScanOptions: https://github.com/apache/arrow/blob/master/cpp/src/arrow/dataset/scanner.h#L60-L144

@westonpace

Weston Pace / @westonpace:

If this is still a go

It's still very much a valid cleanup in my mind (though I probably won't be working on it in the very near future).

@paleolimbot

Dewey Dunnington / @paleolimbot:
What I should have asked is...should I work on a PR to implement access to any more of the options members from the R side for the 7.0.0 release? It's not particularly difficult to do but if the API is going to change then perhaps the 7.0.0 release isn't the right fit?

@westonpace

Weston Pace / @westonpace:
I don't think this is going to fit into 7.0.0 (sorry for the delay). Also, as I'm working on ARROW-13554 I'm wondering if there might be a simpler approach to take here. If we combine the ScanOptions fields projected_schema and projection into a single type ProjectionOptions which has utility methods for creation (e.g. create from column names, create from expressions, create a default project-everything) then I think we can get rid of ScannerBuilder and just have ScanOptions. This will also simplify ARROW-15257 since we can easily create scan options for use in scan nodes.

@westonpace

Weston Pace / @westonpace:
I spent some time today thinking about scan options. I'd like to propose a simplification. Note that this simplification requires changes to the existing behavior of the scanner itself in some substantial ways.

Here are my proposed options (with a bullet list of changes below)


/// Scan-specific options, which can be changed between scans of the same dataset.
struct ARROW_DS_EXPORT ScanOptions {
  /// A row filter
  ///
  /// This is an opportunistic pushdown filter.  Filtering capabilities will
  /// vary between fragments.
  ///
  /// Each fragment will do its best to filter the data based on the information
  /// (partitioning guarantees, statistics) available to it.  If it is able to
  /// apply some filtering then it will indicate what filtering it was able to
  /// apply by attaching a guarantee to the batch.
  ///
  /// For example, if a filter is x < 50 && y > 40 then a batch may be able to
  /// apply a guarantee x < 50.  Post-scan filtering would then only need to
  /// consider y > 40 (for this specific batch).  The next batch may not be able
  /// to attach any guarantee and both clauses would need to be applied to that batch.
  ///
  /// The filter expression may not be compatible with the schema of the underlying
  /// file / statistics.  For example, a filter expression may be x > literal(-50,
  /// int32()) The underlying data type might be uint64.  If possible, the reader may
  /// attempt to apply the filter (in this case the gaurantee x > -50 can be attached to
  /// the batch since we know all values will be >= 0 and thus x > -50 is true.
  ///
  /// If a reader is not capable of handling mismatched types in a filter then it should
  /// ignore the filter.
  ///
  /// A single guarantee-aware filtering operation should generally be applied to all
  /// resulting batches.  The scan node consumer is responsible for doing this.
  ///
  /// Note: If you are using a Scanner (as opposed to creating a scan node) there
  /// are a number of convenience methods such as Scanner::Head, Scanner::ScanBatches,
  /// etc.  These "convenience" methods will create an exec plan that includes a
  /// scan node, a filter node, and a projection node.  So if you are using these
  /// methods you do not need to worry about this final filtering step.
  compute::Expression filter = compute::literal(true);

  /// A mapping describing the desired output schema
  ///
  /// If a fragment contains a field that is not included in this mapping the field
  /// should not be returned.  If possible the column should not be loaded from disk.
  ///
  /// If a fragment does not contain a field specified by the mapping then this is
  /// not an error and the fragment should not return anything.  In particular, the
  /// reader should not attempt to create an all-null placeholder.
  ///
  /// If a reference is a nested reference then readers should only return the
  /// desired subset of the column.  For example, if the physical schema has a
  /// field with the type "points": struct<{"real": double, "imaginary": double}>
  /// and the ref is "points" then the entire column should be loaded and the data
  /// type of the column will be struct<...>.  If the ref is "points.real" then
  /// only the "real" portion of the column should be loaded and the data type of
  /// the column will be "real"
  ///
  /// If the reader does not support partial loading of nested columns (e.g. CSV) then
  /// the reader should load the entire column and then select the relevant nested
  /// portion.  Note that a single source nested column many end up populating several
  /// output columns.  Readers should take care not to load the source column from disk
  /// multiple times.
  ///
  /// The output field type is included for opportunistic purposes.  If the underlying
  /// column type does not match then this is not an error and the reader should return
  /// the underlying column as is.  In particular, the reader should not cast.  However,
  /// if the column type is unknown then the reader should attempt to infer the column
  /// based on the desired output type.
  ///
  /// For example, consider the mapping: [ "items": {"values": int64 } ].  If the fragment
  /// is a parquet file with a column named "items" that has a float64 data type then the
  /// reader should emit the column "values":float64.
  ///
  /// If the fragment is a CSV file with a column named "items" it should attempt to parse
  /// the column as int64, inserting null/na or erroring (depending on the reader options)
  /// if it cannot parse a particular cell.
  ///
  /// The scan node itself is responsible for a final casting of data from whatever
  /// physical type the reader was able to produce to the desired output type.  This means
  /// that a scan node will always emit batches according to the output mapping's schema.
  /// If a cast is impossible then the error_on_incompatible_types option will take
  /// effect.
  ///
  /// If empty then no columns will be loaded from the file.  This can be useful
  /// when you need row-count information only.  File readers that support a
  /// metadata-only scan should utilize such a scan when an empty vector is specified.
  std::vector<std::pair<FieldRef, Field>> output_mapping;

  /// If true the scanner will error when a fragment cannot satisfy a type
  /// requested by the output mapping
  ///
  /// If false the scanner will emit an all-null column (of the requested type) instead
  bool error_on_incompatible_types = true;

  /// If true the scanner will error when a batch does not include one of the fields
  /// requested in the output_mapping.
  ///
  /// If false the scanner will emit an all-null column (of the requested type) instead
  bool error_on_missing_fields = false;

  /// Maximum row count for scanned batches.
  int64_t batch_size = kDefaultBatchSize;

  /// How many batches to read ahead within a file
  ///
  /// Set to 0 to disable batch readahead
  ///
  /// Note: May not be supported by all formats
  int32_t batch_readahead = kDefaultBatchReadahead;

  /// How many files to read ahead
  ///
  /// Set to 0 to disable fragment readahead
  int32_t fragment_readahead = kDefaultFragmentReadahead;

  /// A pool from which materialized and scanned arrays will be allocated.
  MemoryPool* pool = arrow::default_memory_pool();

  /// If true the scanner will scan in parallel
  ///
  /// Note: If true, this will use threads from both the cpu_executor and the
  /// io_context.executor
  bool use_threads = false;
};

The filtering behavior is not really different from what we have today but hopefully more explicit. The only thing we don't handle great today is the case when the data type of the filter expression does not match the data type of the column. See: ARROW-15146
The projection behavior we have today is very confusing to the point that I don't even know if what I'm proposing is that much of a change. The actual syntax is different. The old syntax (projection_schema + projection + dataset_schema) was rather confusing so I think it's worth the cost of introducing a new almost-schema vector of field ref/field pairs. Unlike the filtering I don't think it is acceptable to leave some projection for the exec plan. I've learned more of relational algebra and it seems that nodes should really have a consistent output type known at plan creation time. Therefore, I think the scan node should be responsible for the final projection. However, I think the scan node can take care of the casting to keep the reader logic simple. So basically, readers will do the best they can, and the scan node will finish the rest.
Users often have clunky datasets that are full of inconsistencies. As such, I introduced error_on_incompatible_types and error_on_missing_fields to allow the user to get nulls instead of errors.
Similar to the above, we should be setup to handle datasets that have different files in different formats. I've removed fragment_scan_options. This should be a property of the fragment. If we want to make it easier for users then I think they can provide a default set of options during dataset discovery. This would also mean that FileSystemDataset eventually loses it's "format" property but that can maybe be handled in a follow-up.
I removed the io_context option as that should come from the filesystem.

@westonpace

Weston Pace / @westonpace:
CC @lidavidm @bkietz

@lidavidm

David Li / @lidavidm:

I think the projection behavior is the same, but this is a "user friendly" way of putting it. I suppose before you could have the projection be something completely arbitrary, if you implemented your own kernel returning a Struct type, but I don't think that's useful to support. However it seems we only support selecting/renaming/casting fields, not anything more complex? (Oh wait - "scan node" will become something distinct from the "scanner"? I'm a little confused here now.)
Do we want to error on reordered fields? That could matter if we allow indices for projection.
For multiple formats, we should perhaps consider the proposal in ARROW-11981: "Dataset could be simplified to a concrete class containing a set of compatibly typed/formatted Fragments".

@westonpace

Weston Pace / @westonpace:

I suppose before you could have the projection be something completely arbitrary, if you implemented your own kernel returning a Struct type, but I don't think that's useful to support.

Agreed, this is a little less flexible but pretty soon users will be able to issue full fledged queries through Ibis, etc. so they can still go that route and use an actual project node.

Oh wait - "scan node" will become something distinct from the "scanner"? I'm a little confused here now.

It's already confusing. ScanNode takes a ScanOptions and creates a Scanner yet a Scanner takes a ScanOptions and creates a ScanNode (?!). Scanner is both a high level "lightweight query plan producer" and a low level "file and dataset reading" utility. The low level utility can probably become more and more internal. Users will either interact with "scan node" via Ibis, dplyr, etc. or, if they don't want to use a 3rd party query plan producer, they can use Scanner, which is more of a substitute for those libraries with minimal functionality.

If we don't want to lose behavior in the high-level "convenience methods" (the ones that belong in the "lightweight query plan producer") like "ToTable", "ScanBatches", "Head", "Count" then we could have two "scan options" type classes. A lower level ScanOptions that is consumed by the scan node and takes the limited projection and a higher level ScanOptions that is consumed by the high-level methods and creates a project node as part of the plan. The convenience methods would have to extract the lower level projection out of the higher level dictionary of expressions but we already have logic to do that kind of thing today.

Do we want to error on reordered fields? That could matter if we allow indices for projection.

Good point but I think that would mean the schema has to be identical (or maybe a subset from left to right) since there isn't any good way to have a "hole" in the schema. I think we want to allow users some way to specify the dataset schema (e.g. the one the plan is built against and the output of the scan operator) and some kind of "schema column resolution behavior". For example:

Error if non-matching - If the schema doesn't match exactly then error
Error if not-subset - If the schema doesn't match but allow missing columns at the end
Resolve by name - Requires unique names. Find the output column position by looking for a column in the master schema with the same name
Resolve by id - Same as resolve by name but use the parquet ID field

For multiple formats, we should perhaps consider the proposal in ARROW-11981

That's a great idea. I agree completely and it helps keep things simple.

@toddfarmer

Todd Farmer / @toddfarmer:
This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned per project policy. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.

Navigation Menu

Repository navigation

[Python][R] Expose (hide?) ScanOptions #28113

Related issues:

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Python][R] Expose (hide?) ScanOptions #28113

Description

Related issues:

Activity

asfimport commented on Apr 9, 2021

asfimport commented on Apr 9, 2021

asfimport commented on Apr 9, 2021

asfimport commented on Apr 22, 2021

asfimport commented on Dec 17, 2021

asfimport commented on Dec 17, 2021

asfimport commented on Dec 21, 2021

asfimport commented on Jan 5, 2022

asfimport commented on Feb 11, 2022

asfimport commented on Feb 11, 2022

asfimport commented on Feb 11, 2022

asfimport commented on Feb 14, 2022

asfimport commented on Sep 28, 2022

github-actions commented on Jan 18, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions