feat(csharp/src/Drivers/Databricks): Fix for older DBR versions incorrect ResultFormat #3020
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
toddmeng-db
force-pushed
the
toddmeng-db/hybrid-read
branch
14 times, most recently
from
886f1db to
41ebff4
Compare
toddmeng-db
changed the title
toddmeng-db
force-pushed
the
toddmeng-db/hybrid-read
branch
3 times, most recently
from
e85ddab to
65584da
Compare
toddmeng-db
commented
Jun 26, 2025
toddmeng-db
force-pushed
the
toddmeng-db/hybrid-read
branch
2 times, most recently
from
4984bcb to
5544441
Compare
CurtHagenlocher
requested changes
Jun 30, 2025
Contributor
CurtHagenlocher
left a comment
CurtHagenlocher
left a comment
There was a problem hiding this comment.
Thanks! Just a few small things.
| // Process direct results first, if available | ||
| if (_statement.HasDirectResults && _statement.DirectResults?.ResultSet?.Results?.ResultLinks?.Count > 0) | ||
| if (_statement.HasDirectResults && _statement.DirectResults?.ResultSet?.Results?.ResultLinks?.Count > 0 || | ||
| (_initialResults?.Results?.ResultLinks?.Count > 0)) |
Contributor
There was a problem hiding this comment.
nit: fix indentation
| /// </summary> | ||
| public sealed class DatabricksCompositeReader : TracingReader | ||
| { | ||
|
|
Contributor
There was a problem hiding this comment.
nit: remove extra blank line
|
|
||
| public override Schema Schema { get { return _schema; } } | ||
|
|
||
|
|
Contributor
There was a problem hiding this comment.
nit: remove extra blank line
| /// A composite reader for Databricks that delegates to either CloudFetchReader or DatabricksReader | ||
| /// based on CloudFetch configuration and result set characteristics. | ||
| /// </summary> | ||
| public sealed class DatabricksCompositeReader : TracingReader |
Contributor
There was a problem hiding this comment.
Suggested change
| public sealed class DatabricksCompositeReader : TracingReader | |
| internal sealed class DatabricksCompositeReader : TracingReader |
toddmeng-db
force-pushed
the
toddmeng-db/hybrid-read
branch
from
5544441 to
ce1d237
Compare
CurtHagenlocher
approved these changes
Jul 1, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is because there is a bug on runtime-side:
This PR gets around that by directly inspecting the results. Since we do not always have DirectResults, we make an additional FetchResult call to determine whether to use
CloudFetchReaderor arrow-basedDatabricksReader. Only if the result contains links does it decide to useCloudFetchReader.This test does not pass in 11.3, 10.4 DBR until this commit.
dotnet test --filter "FullyQualifiedName~LZ4DecompressionCapabilityTest"
Remaining work:
Remaining work: need to update status polling to start earlier, ideally should be managed by new DatabricksCompositeReader