-
Notifications
You must be signed in to change notification settings - Fork 589
[CORE] Mixed backend(s) #6920
Description
Note: It's not a goal to make CH backend + VL backend work together as of now (reason). If you got any requirement like this please let me know.
Currently only one single backend can be used at the same time in Gluten. This topic suggests a plug-able system to allow enabling multiple backends at compile time (at runtime in future, maybe). For example, eventually one could build Gluten with -P backends-velox,backends-another to enable both Velox backend and another backend that could possibly be added to Gluten code base in future. In the case when a query is being run, Gluten's query planner could choose the best path to implement the physical plan that may be a mixing of Velox sub-plans and another sub-plans.
The feature could drive Gluten to evolve from a hand-written middle layer to a development platform to offload Spark SQL with different data formats, acceleration libraries, or hardware architectures.
Prior to doing this, a bunch of preparation work should be done including essential code refactors.