ARROW-12161: [C++] Async streaming CSV reader deadlocking when being run synchronously from datasets#9868
Closed
westonpace wants to merge 8 commits intoapache:masterfrom
Closed
ARROW-12161: [C++] Async streaming CSV reader deadlocking when being run synchronously from datasets#9868westonpace wants to merge 8 commits intoapache:masterfrom
westonpace wants to merge 8 commits intoapache:masterfrom
Conversation
…n was a form of nested parallelism and causing nested deadlocks. This commit brings over some of the work in ARROW-7001 and allows the CSV scan task to be called in an async fashion. In addition, an async path is put in the scanner and dataset write so that all internal uses of ScanTask()->Execute happen in an async-friendly way. External uses of ScanTask()->Execute should already be outside the CPU thread pool and should not cause deadlock
Member
Author
|
The JNI failure looks to be a timeout. The integration failure is unrelated (Rust error). The R Windows 3.5 error seems legitimate. I am investigating. |
lidavidm
approved these changes
Apr 2, 2021
pachadotdev
pushed a commit
to pachadotdev/arrow
that referenced
this pull request
Apr 5, 2021
…run synchronously from datasets Calling the async streaming CSV reader from the synchronous Scanner::Scan was causing a form of nested parallelism and causing nested deadlocks. This commit brings over some of the work in ARROW-7001 and allows the CSV scan task to be called in an async fashion. In addition, an async path is put in the scanner and dataset write so that all internal uses of ScanTask()->Execute happen in an async-friendly way. External uses of ScanTask()->Execute should already be outside the CPU thread pool and should not cause deadlock. Some of this PR will be obsoleted by ARROW-7001 but the work in file_csv and the test cases should remain fairly intact. Closes apache#9868 from westonpace/bugfix/arrow-12161 Lead-authored-by: Weston Pace <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Signed-off-by: David Li <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Calling the async streaming CSV reader from the synchronous Scanner::Scan was causing a form of nested parallelism and causing nested deadlocks. This commit brings over some of the work in ARROW-7001 and allows the CSV scan task to be called in an async fashion. In addition, an async path is put in the scanner and dataset write so that all internal uses of ScanTask()->Execute happen in an async-friendly way. External uses of ScanTask()->Execute should already be outside the CPU thread pool and should not cause deadlock.
Some of this PR will be obsoleted by ARROW-7001 but the work in file_csv and the test cases should remain fairly intact.