Skip to content

Comments

feat: Upgrade to DataFusion 46.0.0-rc2#1423

Merged
andygrove merged 12 commits intoapache:mainfrom
andygrove:df-46
Mar 5, 2025
Merged

feat: Upgrade to DataFusion 46.0.0-rc2#1423
andygrove merged 12 commits intoapache:mainfrom
andygrove:df-46

Conversation

@andygrove
Copy link
Member

@andygrove andygrove commented Feb 19, 2025 •
edited
Loading

Which issue does this PR close?

Closes #1414

Rationale for this change

Upgrade to latest DataFusion.

What changes are included in this PR?

How are these changes tested?

@andygrove andygrove mentioned this pull request Feb 19, 2025
Closed
26 tasks
@codecov-commenter
Copy link

codecov-commenter commented Feb 19, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 58.55%. Comparing base (f09f8af) to head (f7eb13e).
Report is 57 commits behind head on main.

Additional details and impacted files 
@@             Coverage Diff              @@
##               main    #1423      +/-   ##
============================================
+ Coverage     56.12%   58.55%   +2.43%     
- Complexity      976     1015      +39     
============================================
  Files           119      122       +3     
  Lines         11743    12249     +506     
  Branches       2251     2304      +53     
============================================
+ Hits           6591     7173     +582     
+ Misses         4012     3919      -93     
- Partials       1140     1157      +17

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@andygrove
Copy link
Member Author

andygrove commented Feb 25, 2025

There are some failures with hash functions.

org.apache.spark.SparkException: Job aborted due to stage failure: Task 4 in stage 6204.0 failed 1 times, most recent failure: Lost task 4.0 in stage 6204.0 (TID 14921) (c8336c0abbf7 executor driver): org.apache.comet.CometNativeException: Function sha256 does not implement invoke but called

@andygrove andygrove changed the title chore: WIP prepare for upgrading to DataFusion 46 feat: Upgrade to DataFusion 46.0.0 Mar 3, 2025
andygrove
andygrove commented Mar 3, 2025
View reviewed changes
@andygrove andygrove changed the title feat: Upgrade to DataFusion 46.0.0 feat: Upgrade to DataFusion 46.0.0-rc2 Mar 4, 2025
@andygrove andygrove marked this pull request as ready for review March 4, 2025 22:35
@andygrove
Copy link
Member Author

andygrove commented Mar 4, 2025 •
edited
Loading

@kazuyukitanimura I'd like to go ahead and merge this now to give us a few days to test with DF 46 before the final release. WDYT?

kazuyukitanimura
kazuyukitanimura reviewed Mar 5, 2025
View reviewed changes
native/core/src/parquet/mod.rs Outdated
let parquet_source = ParquetSource::new(table_parquet_options).with_schema_adapter_factory(
Arc::new(SparkSchemaAdapterFactory::new(spark_parquet_options)),
);
//.with_schema(Arc::new(required_schema_arrow));
Copy link
Contributor

@kazuyukitanimura kazuyukitanimura Mar 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this comment a dev code?

Copy link
Member Author

@andygrove andygrove Mar 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have removed this.

native/spark-expr/src/agg_funcs/stddev.rs
name: name.into(),
signature: Signature::coercible(
vec![
vec![Coercion::new_exact(
Copy link
Contributor

@kazuyukitanimura kazuyukitanimura Mar 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this Coercion do?

Copy link
Member Author

@andygrove andygrove Mar 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This use of Coercion::new_exact means that we only accept the specific type (Float64) and will not perform any coercion.

From DataFusion docs:

/// There are two variants:
///
/// * `Exact` - Only accepts arguments that exactly match the desired type
/// * `Implicit` - Accepts the desired type and can coerce from specified source types

@andygrove andygrove merged commit 23dccce into apache:main Mar 5, 2025
74 checks passed
@andygrove andygrove deleted the df-46 branch March 5, 2025 18:43
coderfender pushed a commit to coderfender/datafusion-comet that referenced this pull request Dec 13, 2025
@andygrove
feat: Upgrade to DataFusion 46.0.0-rc2 (apache#1423)
f77e73b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kazuyukitanimura kazuyukitanimura kazuyukitanimura approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Upgrade to DataFusion 46

3 participants

@andygrove @codecov-commenter @kazuyukitanimura