feat(csharp/src/Drivers): Add Databricks driver #2672
Conversation
|
@birschick-bq verified the new set of Databricks tests and the separated Spark tests. Tests are behaving as expected. |
|
I will incorporate the most recent updates and fix the merge issue. |
|
@eric-wang-1990 , @jadewang-db - can you review/test as well? |
davidhcoe
changed the title
davidhcoe
changed the title
CurtHagenlocher
left a comment
CurtHagenlocher
left a comment
There was a problem hiding this comment.
Thanks, especially for the usings cleanups! I think we should just go ahead and delete the old SparkDatabricksConnection class for the reason I gave.
CurtHagenlocher
left a comment
CurtHagenlocher
left a comment
There was a problem hiding this comment.
Thanks -- looks good to me! @eric-wang-1990, @jadewang-db - I'll wait to hear from one of you before committing.
| | STRUCT* | String | string | | ||
| | TIMESTAMP | Timestamp | DateTimeOffset | | ||
| | TINYINT | Int8 | sbyte | | ||
| | UNION | String | string | |
There was a problem hiding this comment.
Not related to this PR, but I do not recognize UNION and USER_DEFINED type.
And the list is missing TIMESTAMP_NTZ and VARIANT type.
There was a problem hiding this comment.
I think this probably comes from either the set of Arrow types or the TTypeId enum in the Thrift file.
After the initial Spark work, the Databricks-specific features were becoming more entwined in the code. There have been requests for additional Databricks-specific features such as CloudFetch, Entra authentication, and LZ4 compression to name a few. This PR moves the Databricks-specific code out of the Spark driver and into its own driver. It leverages the Thrift and Spark libraries that exist in the Apache driver but teases out the Databricks capabilities from the Spark capabilities. It does so by exposing the internals of the Apache driver so that the Databricks driver can leverage the same underlying libraries while still being able to offer differentiating features and raises the branding of the driver from under the Apache/Spark umbrella to the same level as BigQuery, FlightSQL, and Snowflake. --------- Co-authored-by: David Coe <> Co-authored-by: Bruce Irschick <[email protected]>
4 participants
After the initial Spark work, the Databricks-specific features were becoming more entwined in the code. There have been requests for additional Databricks-specific features such as CloudFetch, Entra authentication, and LZ4 compression to name a few.
This PR moves the Databricks-specific code out of the Spark driver and into its own driver. It leverages the Thrift and Spark libraries that exist in the Apache driver but teases out the Databricks capabilities from the Spark capabilities. It does so by exposing the internals of the Apache driver so that the Databricks driver can leverage the same underlying libraries while still being able to offer differentiating features and raises the branding of the driver from under the Apache/Spark umbrella to the same level as BigQuery, FlightSQL, and Snowflake.