UnionDataset allows Fragments of multiple schemas and differing file formats to be scanned together as a single Dataset. This is useful functionality but makes the Dataset interface somewhat difficult to reason about since it must be general enough to accommodate UnionDataset.

After ARROW-11928 it will probably be more natural to support unioning of datasets through a subclass of ExecNode. Reconciliation of differing schemas can then be trivially handled by a full ProjectNode.

Note this would obviate both ARROW-11001 and ARROW-11749. In addition, Dataset could be simplified to a concrete class containing a set of compatibly typed/formatted Fragments.

Reporter: Ben Kietzman / @bkietz
Assignee: Alexander Ocsa / @aocsa

Related issues:

• [C++] Query engine umbrella issue (is a child of)
• [C++][Dataset] Remove UnionDataset in favor of UnionExecNode (is related to)

PRs and other links:

• GitHub Pull Request #10927

_{Note: This issue was originally created as ARROW-11981. Please see the migration documentation for further details.}