Skip to content

[CORE] Mixed backend(s) #6920

@zhztheplayer

Description

@zhztheplayer

Note: It's not a goal to make CH backend + VL backend work together as of now (reason). If you got any requirement like this please let me know.

Currently only one single backend can be used at the same time in Gluten. This topic suggests a plug-able system to allow enabling multiple backends at compile time (at runtime in future, maybe). For example, eventually one could build Gluten with -P backends-velox,backends-another to enable both Velox backend and another backend that could possibly be added to Gluten code base in future. In the case when a query is being run, Gluten's query planner could choose the best path to implement the physical plan that may be a mixing of Velox sub-plans and another sub-plans.

The feature could drive Gluten to evolve from a hand-written middle layer to a development platform to offload Spark SQL with different data formats, acceleration libraries, or hardware architectures.

Prior to doing this, a bunch of preparation work should be done including essential code refactors.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions