Repository navigation
Is there any demo that use pandas and Flight SQL to access database? #35664
Description
Describe the usage question you have. Please include as many useful details as possible.
Currently, I am investigating a high performance framework for Apache Doris https://github.com/apache/doris, because many Data Science users are using python to access Doris and the mysql protocol is very slow. I find flight sql is a very perfect candidate. But I did not find many Pandas or Numpy usecases that using Flight SQL.
Anyone could provide some usecases?
Component(s)
FlightRPC, Python
Activity
lidavidm commentedon May 18, 2023
Flight SQL will give you Arrow data, which you can cheaply (in some cases zero-copy) convert to and from Pandas DataFrames and NumPy arrays.
For Pandas specifically, I would like to eventually let Pandas' SQL APIs leverage ADBC (ADBC being a client-side API access layer, to complement Flight SQL being the client-server protocol): https://arrow.apache.org/adbc/0.4.0/python/quickstart.html
Polars does this: https://pola-rs.github.io/polars-book/user-guide/io/database/
yiguolei commentedon May 19, 2023
@lidavidm thanks