Skip to content

Comments

chore: bump DataFusion to rev c6f0d3c#782

Closed
andygrove wants to merge 1 commit intoapache:mainfrom
andygrove:bump-df-c6f0d3c
Closed

chore: bump DataFusion to rev c6f0d3c#782
andygrove wants to merge 1 commit intoapache:mainfrom
andygrove:bump-df-c6f0d3c

Conversation

@andygrove
Copy link
Member

@andygrove andygrove commented Aug 5, 2024

Which issue does this PR close?

N/A

Rationale for this change

Weekly DataFusion revision bump.

What changes are included in this PR?

This bump includes the following DataFusion changes (generated with git log 35c2e7e..c6f0d3c --oneline):

c6f0d3cac (HEAD -> main, apache/main) Pass scalar to `eq` inside `nullif`  (#11697)
011a3f3e3 Improve readme page in crates.io (#11809)
f56a2ef28 doc: Add support for `map` and `make_map` functions (#11799)
0417e543e Minor: Update exected output due to logical conflict (#11824)
f19d30d52 chore(deps): update rstest requirement from 0.21.0 to 0.22.0 (#11811)
5c4254aeb Don't implement create_sliding_accumulator repeatedly (#11813)
45d85b1d1 Change name of MAX/MIN udaf to lowercase max/min  (#11795)
6aad19fad add valid distinct case for aggregate.slt (#11814)
682bc2eff Improve log func stability (#11808)
c340b6ab7 Skipping partial aggregation when it is not helping for high cardinality aggregates (#11627)
336c15e75 Improve MSRV CI check to print out problems to log (#11789)
1d3bdbe14 Minor: add ticket reference and fmt (#11805)
b4069a65a Remove `AggregateFunctionDefinition` (#11803)
c8e5996c4 Remove redundant Aggregate when `DISTINCT` & `GROUP BY` are in the same query (#11781)
a4d41d6a6 Support `LogicalPlan` `Debug` differently than `Display` (#11774)
f4e519f9d Move min and max to user defined aggregate function, remove `AggregateFunction` / `AggregateFunctionDefinition::BuiltIn` (#11013)
9e90e17a6 fix: Add additional required expression for natural join (#11713)
81668f3b2 Support planning `Map` literal (#11780)
6c4c24612 Doc: Add Sail to known users list (#11791)
de2da34cc Add docs and rename param (#11778)
b89037f0d Add references to github issue (#11784)
0332eb569 refactor: move ExecutionPlan and related structs into dedicated mod (#11759)
5ca4ec3b5 Extract CoalesceBatchesStream to a struct (#11610)
80848f2a0 Fix #11692: Improve doc comments within macros (#11694)
df4e6cc4e [Minor] Short circuit `ApplyFunctionRewrites` if there are no function rewrites (#11765)
d010ce90f refactor(11523): update OOM message provided for a single failed reservation (#11771)
70aba2bd6 minor: always time batch_filter even when the result is an empty batch (#11775)
a0ad37684 [Minor] Refactor approx_percentile (#11769)
f044bc837 Fix bug that `COUNT(DISTINCT)` on StringView panics  (#11768)
0d98b9974 Minor: Add comment explaining rationale for hash check (#11750)
45b40c711 Minor: add "clickbench extended" queries to unit tests (#11763)
6e2ff2955 Derive Debug for logical plan nodes (#11757)
921c3b6b1 Add `TrackedMemoryPool` with better error messages on exhaustion (#11665)
3fe18604e Add null test (#11760)
a4ac0829e Fix documentation warnings, make CsvExecBuilder and Unparsed pub (#11729)
cf98d94c9 Use upstream `DataType::from_str` in arrow-cast (#11254)
1ce546168 Fix `plan_to_sql`: Add wildcard projection to SELECT statement if no projection was set (#11744)
0b8da6d6e Rename RepartitionExec metric `repart_time` to `repartition_time` (#11703)
4884c08ce Use upstream StatisticsConveter (#11479)
9dd2cfc80 Do not push down Sorts if it violates the sort requirements (#11678)
0f554fa12 Minor: Improve documentation for AggregateUDFImpl::state_fields (#11740)
ae2ca6a0e Support  cross-timezone `timestamp` comparison via coercsion (#11711)
fa50636c6 Implement physical plan serialization for parquet Copy plans (#11735)
2887491fc fix: set `null_equals_null` to false when `convert_cross_join_to_inner_join` (#11738)
89677ae66 Check hashes first during probing the aggr hash table (#11718)
abeb8b4f8 Make DefaultSchemaAdapterFactory public (#11709)
6508fa2dc expose some fields on session state (#11716)
929568d60 Use `cargo release` in benchmarks (#11722)
cc6416e74 Minor: Add example for `ScalarUDF::call` (#11727)
8ac50e2db Reduce repetition in try_process_group_by_unnest and try_process_unnest (#11714)
7ca7456d3 Handle nulls in approx_percentile_cont (#11721)
cd786e275 fix: regr_count now returns Uint64 (#11731)
66a85706f Rename `input_type` --> `input_types` on AggregateFunctionExpr / AccumulatorArgs / StateFieldsArgs (#11666)

How are these changes tested?

@andygrove
Copy link
Member Author

andygrove commented Aug 5, 2024

Build fails with:

error[E0432]: unresolved imports `datafusion::physical_expr::expressions::Max`, `datafusion::physical_expr::expressions::Min`
  --> core/src/execution/datafusion/planner.rs:36:43
   |
36 |             Literal as DataFusionLiteral, Max, Min, NotExpr,
   |                                           ^^^  ^^^ no `Min` in `expressions`
   |                                           |
   |                                           no `Max` in `expressions`

Looks like we need to update Min and Max as we have done recently for Sum and Count (@huaxingao just fyi)

@huaxingao
Copy link
Contributor

huaxingao commented Aug 5, 2024

Looks like we need to update Min and Max as we have done recently for Sum and Count

Thanks for pinging me. I will fix this.

@huaxingao huaxingao mentioned this pull request Aug 5, 2024
Merged
@huaxingao
Copy link
Contributor

huaxingao commented Aug 5, 2024

@andygrove I am not so sure how to push changes to this PR so I have created a new PR instead.

@andygrove andygrove closed this Aug 6, 2024
@andygrove andygrove deleted the bump-df-c6f0d3c branch August 6, 2024 13:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@andygrove @huaxingao