feat(csharp): fix powerbi hang when reading cloudfetch result in Databricks driver #2747
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
csharp/src/Drivers/Databricks/CloudFetch/CloudFetchDownloader.cs
Outdated
Show resolved
Hide resolved
Contributor
CurtHagenlocher
left a comment
CurtHagenlocher
left a comment
There was a problem hiding this comment.
Looks fine overall, but I have a question about the tracing.
csharp/src/Drivers/Databricks/CloudFetch/CloudFetchDownloader.cs
Outdated
Show resolved
Hide resolved
jadewang-db
requested review from
CurtHagenlocher,
davidhcoe and
eric-wang-1990
CurtHagenlocher
approved these changes
Apr 29, 2025
colin-rogers-dbt
pushed a commit
to dbt-labs/arrow-adbc
that referenced
this pull request
Jun 10, 2025
…bricks driver (apache#2747) ### Summary This PR fixes an issue where PowerBI would hang when reading CloudFetch results and significantly improves the logging capabilities in the CloudFetch downloader component. ### Problem 1. The CloudFetchReader was not properly disposing of the download manager after completing downloads, causing resource leaks that led to PowerBI hanging. 2. The CloudFetchDownloader was using Debug.WriteLine for logging, which is inadequate for production scenarios and doesn't provide sufficient diagnostic information. ### Solution - Fixed resource management in CloudFetchReader by properly disposing the download manager after all files are processed - Replaced Debug.WriteLine calls with more comprehensive Trace logging - Added detailed performance metrics and diagnostics: - Download start/completion timestamps - File sizes and throughput calculations - Decompression metrics - Overall download statistics (total files, success/failure counts) - Added URL sanitization for secure logging - Added proper error tracking and reporting ### Testing - Enhanced CloudFetchE2ETest to verify that the reader properly completes after all data is consumed - Verified that PowerBI no longer hangs when reading CloudFetch results
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes an issue where PowerBI would hang when reading CloudFetch results and significantly improves the logging capabilities in the CloudFetch downloader component.
Problem
Solution
Testing