feat(csharp/src/Drivers/Apache): Add prefetch functionality to CloudFetch in Spark ADBC driver #2678
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jadewang-db
force-pushed
the
cloudfetch-pipeline
branch
2 times, most recently
from
01daf70 to
a388213
Compare
CurtHagenlocher
requested changes
Apr 16, 2025
Contributor
CurtHagenlocher
left a comment
CurtHagenlocher
left a comment
There was a problem hiding this comment.
Thanks! I'm still reviewing this logic but thought I'd give some initial feedback.
Also, please take a look at the linter output and make changes accordingly.
csharp/src/Drivers/Databricks/CloudFetch/cloudfetch-pipeline-design.md
Outdated
Show resolved
Hide resolved
csharp/src/Drivers/Databricks/CloudFetch/IHiveServer2Statement.cs
Outdated
Show resolved
Hide resolved
csharp/src/Drivers/Databricks/CloudFetch/ICloudFetchInterfaces.cs
Outdated
Show resolved
Hide resolved
csharp/src/Drivers/Databricks/CloudFetch/CloudFetchDownloadManager.cs
Outdated
Show resolved
Hide resolved
csharp/src/Drivers/Databricks/CloudFetch/CloudFetchDownloadManager.cs
Outdated
Show resolved
Hide resolved
jadewang-db
force-pushed
the
cloudfetch-pipeline
branch
2 times, most recently
from
6412369 to
6b733bc
Compare
Update DatabricksParameters.cs address comments fix linter rebase to master refactor to fix unit test refactor some code refactoring refactor Delete CloudFetchDownloadManagerTest.cs Initital changes
jadewang-db
force-pushed
the
cloudfetch-pipeline
branch
from
6b733bc to
c50c73d
Compare
CurtHagenlocher
approved these changes
Apr 24, 2025
Contributor
CurtHagenlocher
left a comment
CurtHagenlocher
left a comment
There was a problem hiding this comment.
Thanks! Looks great!
colin-rogers-dbt
pushed a commit
to dbt-labs/arrow-adbc
that referenced
this pull request
Jun 10, 2025
…etch in Spark ADBC driver (apache#2678) # Add Prefetch Functionality to CloudFetch in Spark ADBC Driver This PR enhances the CloudFetch feature in the Spark ADBC driver by implementing prefetch functionality, which improves performance by fetching multiple batches of results ahead of time. ## Changes ### CloudFetchResultFetcher Enhancements - **Initial Prefetch**: Added code to perform an initial prefetch of multiple batches when the fetcher starts, ensuring data is available immediately when needed. - **State Management**: Added tracking for current batch offset and size, with proper state reset when starting the fetcher. ### Interface Updates - Added new methods to `ICloudFetchResultFetcher` interface: ### Testing Infrastructure - Created `ITestableHiveServer2Statement` interface to facilitate testing - Updated tests to account for prefetch behavior - Ensured all tests pass with the new prefetch functionality ## Benefits - **Improved Performance**: By prefetching multiple batches, data is available sooner, reducing wait times. - **Better Reliability**: Enhanced error handling and state management make the system more robust. - **More Efficient Resource Usage**: Link caching reduces unnecessary server requests. This implementation maintains backward compatibility while providing significant performance improvements for CloudFetch operations.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add Prefetch Functionality to CloudFetch in Spark ADBC Driver
This PR enhances the CloudFetch feature in the Spark ADBC driver by implementing prefetch functionality, which improves performance by fetching multiple batches of results ahead of time.
Changes
CloudFetchResultFetcher Enhancements
Interface Updates
ICloudFetchResultFetcherinterface:Testing Infrastructure
ITestableHiveServer2Statementinterface to facilitate testingBenefits
This implementation maintains backward compatibility while providing significant performance improvements for CloudFetch operations.