Skip to content

feat(csharp/src/Drivers/Databricks): Improve memory utilization of cloud downloads #3652

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
CurtHagenlocher merged 6 commits into from
Oct 30, 2025
Merged

feat(csharp/src/Drivers/Databricks): Improve memory utilization of cloud downloads #3652

CurtHagenlocher merged 6 commits into from
Oct 30, 2025

Conversation

@CurtHagenlocher
Copy link
Contributor

@CurtHagenlocher CurtHagenlocher commented Oct 30, 2025

Improves memory utilization of cloud downloads by casting the downloaded and/or decompressed cloud data sets directly into Arrow data rather than having to deserialize them through a stream.

NOTE: I have not benchmarked this change.

@github-actions github-actions bot added this to the ADBC Libraries 21 milestone Oct 30, 2025
CurtHagenlocher
CurtHagenlocher commented Oct 30, 2025
View reviewed changes
csharp/test/Drivers/Databricks/E2E/CloudFetch/CloudFetchDownloaderTest.cs
_resultQueue = new BlockingCollection<IDownloadResult>(new ConcurrentQueue<IDownloadResult>(), 10);
_mockMemoryManager = new Mock<ICloudFetchMemoryBufferManager>();
_mockStatement = new Mock<IHiveServer2Statement>();
_mockStatement.SetupGet(x => x.Trace).Returns(new ActivityTrace());
Copy link
Contributor Author

@CurtHagenlocher CurtHagenlocher Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A null value for Trace was preventing these tests from running correctly.

@CurtHagenlocher CurtHagenlocher mentioned this pull request Oct 30, 2025
Closed
@CurtHagenlocher
Copy link
Contributor Author

CurtHagenlocher commented Oct 30, 2025

This turns out not to make much of a change other than reducing the number of garbage collections -- but that's worthwhile, and I think we can find other ways to further reduce that number.

@CurtHagenlocher CurtHagenlocher merged commit eae39dd into apache:main Oct 30, 2025
6 checks passed
@CurtHagenlocher CurtHagenlocher deleted the MemoryResult branch October 30, 2025 13:53
@eric-wang-1990
Copy link
Contributor

eric-wang-1990 commented Oct 31, 2025

@CurtHagenlocher It seems this PR introduce a regression, I see this in the middle of a cloudfetch import:
Frown (Error)

Timestamp:
2025-10-30T15:19:38.4649362Z

Local Time:
2025-10-30T08:19:38.4649362-07:00

Session ID:
10423512-35fe-4714-af6b-d05dddbf2023

Release:
December 2025

Product Version:
2.150.0.0 (Main)+7c5552edc3b7e30ca4810aa7fea77f851cb4c03c (x64)

Error Message:
Corrupted IPC message. Received a continuation token at the end of the message.

Stack Trace:
Microsoft.Mashup.Host.Document.SerializedException
at Microsoft.Mashup.Evaluator.EvaluationHost.OnException(IEngineHost engineHost, IMessageChannel channel, ExceptionMessage message)
at Microsoft.Mashup.Evaluator.RemoteEvaluationContainerFactory.Container.OnException(IMessageChannel channel, ExceptionMessage message)
at Microsoft.Mashup.Evaluator.MessageHandlers.<>c__DisplayClass3_0`1.b__0(IMessageChannel channel, Message message)
at Microsoft.Mashup.Evaluator.MessageHandlers.TryDispatch(IMessageChannel channel, Message message)
at Microsoft.Mashup.Evaluator.MessageHandlers.Dispatch(IMessageChannel channel, Message message)
at Microsoft.Mashup.Evaluator.ChannelMessenger.ChannelMessageHandlers.TryDispatch(IMessageChannel channel, Message message)
at Microsoft.Mashup.Evaluator.MessageHandlers.Dispatch(IMessageChannel channel, Message message)

@CurtHagenlocher
Copy link
Contributor Author

CurtHagenlocher commented Oct 31, 2025

@CurtHagenlocher It seems this PR introduce a regression,

That's odd, but I'll revert it.

CurtHagenlocher added a commit to CurtHagenlocher/arrow-adbc that referenced this pull request Oct 31, 2025
@CurtHagenlocher
Revert "feat(csharp/src/Drivers/Databricks): Improve memory utilizati…
3556b46 
…on of cloud downloads (apache#3652)"

This reverts commit eae39dd.
@CurtHagenlocher CurtHagenlocher mentioned this pull request Oct 31, 2025
Merged
CurtHagenlocher added a commit that referenced this pull request Oct 31, 2025
@CurtHagenlocher
chore(csharp/src/Drivers/Databricks): Revert previous "memory utiliza…
0b01c6e 
…tion of cloud downloads" change (#3655)

Reverts #3652 as it seems to have caused a regression.
eric-wang-1990 added a commit to eric-wang-1990/arrow-adbc that referenced this pull request Oct 31, 2025
@eric-wang-1990
Revert "feat(csharp/src/Drivers/Databricks): Improve memory utilizati…
dae9c27 
…on of cloud downloads (apache#3652)"

This reverts commit eae39dd.
eric-wang-1990 added a commit to eric-wang-1990/arrow-adbc that referenced this pull request Oct 31, 2025
@eric-wang-1990
Reapply "feat(csharp/src/Drivers/Databricks): Improve memory utilizat…
4749feb 
…ion of cloud downloads (apache#3652)"

This reverts commit dae9c27.
@eric-wang-1990 eric-wang-1990 mentioned this pull request Oct 31, 2025
Merged
2 tasks
eric-wang-1990 added a commit to eric-wang-1990/arrow-adbc that referenced this pull request Nov 5, 2025
@eric-wang-1990
Revert "feat(csharp/src/Drivers/Databricks): Improve memory utilizati…
c811860 
…on of cloud downloads (apache#3652)"

This reverts commit eae39dd.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

ADBC Libraries 21

Development

Successfully merging this pull request may close these issues.

2 participants

@CurtHagenlocher @eric-wang-1990