Skip to content

Add Arrow Flight frontend #477

@gruuya

Description

@gruuya

Currently Seafowl provides two interfaces, the HTTP frontend and the PG endpoint. While they're fine for summarized (i.e. aggregated) data output for end users directly or via a web app, they fall short when it comes to transferring large result sets.

Putting aside the fact that not all Arrow types are supported on the currently existing frontends (e.g. see #393), returning large data sets will most likely be highly inefficient due to the overhead of converting the internal columnar representation into row-based JSON response (for the HTTP frontend) for non-trivial row counts. The particular scenario there would involve an external DB system which uses Seafowl for analytical workloads, but can't push down the entire query in some cases so it must fetch the underlying data to perform the original query itself.

Introducing the Arrow Flight (Arrow Fligt SQL in particular[1]) frontend here would solve this problem, since it provides a protocol for sending Arrow data, and would thus avoid unnecessary serialization.

[1] https://voltrondata.com/resources/apache-arrow-flight-sql-arrow-for-every-database-developer

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions