Skip to content

Comments

Upgrade arrow parquet and arrow-flight to 24.0.0#3691

Merged
andygrove merged 3 commits intoapache:masterfrom
alamb:alamb/update_arrow_24.0.0
Oct 3, 2022
Merged

Upgrade arrow parquet and arrow-flight to 24.0.0#3691
andygrove merged 3 commits intoapache:masterfrom
alamb:alamb/update_arrow_24.0.0

Conversation

@alamb
Copy link
Contributor

@alamb alamb commented Oct 3, 2022 •
edited
Loading

Draft until

Which issue does this PR close?

Closes #3689

Rationale for this change

Keep up with upstream changes

What changes are included in this PR?

  1. Update to arrow
  2. Update for API changes

Are there any user-facing changes?

@alamb alamb marked this pull request as draft October 3, 2022 15:41
@github-actions github-actions bot added core Core DataFusion crate logical-expr Logical plan and expressions optimizer Optimizer rules physical-expr Changes to the physical-expr crates sql SQL Planner labels Oct 3, 2022
alamb
alamb commented Oct 3, 2022
View reviewed changes
datafusion/common/src/pyarrow.rs
// convert the pyarrow array to rust array using C data interface
let array = array.extract::<ArrayData>()?;
let scalar = ScalarValue::try_from_array(&array.into(), 0)?;
let array = arrow::array::make_array(ArrayData::from_pyarrow(array)?);
Copy link
Contributor Author

@alamb alamb Oct 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needed due to changes in apache/arrow-rs#2769

@alamb alamb mentioned this pull request Oct 3, 2022
Closed
5 tasks
@alamb alamb force-pushed the alamb/update_arrow_24.0.0 branch from 9c84637 to 6c05862 Compare October 3, 2022 21:02
@alamb alamb marked this pull request as ready for review October 3, 2022 21:05
andygrove
andygrove approved these changes Oct 3, 2022
View reviewed changes
Copy link
Member

@andygrove andygrove left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @alamb

@andygrove andygrove merged commit 011e6f5 into apache:master Oct 3, 2022
@ursabot
Copy link

ursabot commented Oct 3, 2022

Benchmark runs are scheduled for baseline = 010b352 and contender = 011e6f5. 011e6f5 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

@alamb alamb deleted the alamb/update_arrow_24.0.0 branch August 8, 2023 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andygrove andygrove andygrove approved these changes

Assignees

No one assigned

Labels

core Core DataFusion crate logical-expr Logical plan and expressions optimizer Optimizer rules physical-expr Changes to the physical-expr crates sql SQL Planner

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Upgrade to Arrow 24.0.0

3 participants

@alamb @ursabot @andygrove