Conversation
16 tasks
2010YOUY01
reviewed
Nov 20, 2025
2010YOUY01
reviewed
Nov 20, 2025
Co-authored-by: Yongting You <[email protected]>
Contributor
Author
|
Thank you @2010YOUY01 |
alamb
changed the title
comphead
reviewed
Nov 20, 2025
|
|
||
| ### Decimal32/Decimal64 support | ||
|
|
||
| The new Arrow types `Decimal32` and `Decimal64` are now supported in DataFusion |
comphead
approved these changes
Nov 20, 2025
Contributor
comphead
left a comment
comphead
left a comment
There was a problem hiding this comment.
Thanks @alamb and @2010YOUY01 IMO it is LGTM
Btw I found that release notes are now concise, easy to read and follow!
2010YOUY01
approved these changes
Nov 21, 2025
Contributor
2010YOUY01
left a comment
2010YOUY01
left a comment
There was a problem hiding this comment.
Great blog post, thank you!
Comment on lines
70
to
80
| **Fewer object store round-trips for Parquet by default** | ||
|
|
||
| DataFusion now sets a default `metadata_size_hint` for [Apache Parquet] scans | ||
| ([#18118]), avoiding the extra | ||
| “last 8‑byte” request many clouds require to read file footers. Remote scans | ||
| typically drop from five requests to four per file, cutting latency and transfer | ||
| costs without any application changes. Thanks to [zhuqi-lucas] for leading this | ||
| effort. | ||
|
|
||
| [apache parquet]: https://parquet.apache.org/ | ||
|
|
Contributor
There was a problem hiding this comment.
Suggested change
| **Fewer object store round-trips for Parquet by default** | |
| DataFusion now sets a default `metadata_size_hint` for [Apache Parquet] scans | |
| ([#18118]), avoiding the extra | |
| “last 8‑byte” request many clouds require to read file footers. Remote scans | |
| typically drop from five requests to four per file, cutting latency and transfer | |
| costs without any application changes. Thanks to [zhuqi-lucas] for leading this | |
| effort. | |
| [apache parquet]: https://parquet.apache.org/ |
I think this is a duplicate to the below 'Better Defaults for Remote Parquet Reads' section.
Contributor
Author
There was a problem hiding this comment.
That is a great catch -- I consolidated them in 33e4375
|
|
||
| We are proud to announce the release of [DataFusion 51.0.0]. This post highlights | ||
| some of the major improvements since [DataFusion 50.0.0]. The complete list of | ||
| changes is available in the [changelog]. Thanks to the [128 contributors] for |
Contributor
Author
There was a problem hiding this comment.
Indeed -- I think this is the part of this blog post I am most proud of
xudong963
approved these changes
Nov 21, 2025
Contributor
Author
|
I plan to publish this tomorrow, 2025-11-25. Please let me know if anyone wants more time to review or has any additional commetns |
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
51.0.0release datafusion#18548See rendered preview: https://datafusion.staged.apache.org/blog/2025/11/25/datafusion-51.0.0/
For anyone curious, I asked
codexto draft this PR with the following prompt. It did a pretty good job for the rough draftDetails
We are going to write a blog post for the DataFusion 51.0.0 release
We need to cover the major features in this release. If you are unsure of any content, please leave a "TODO" note in the text and we can fill it in later.
I have copied the old release post here as a starting point:
content/blog/2025-11-25-datafusion-51.0.0.mdHere are the PRs this release (approx based on dates) - https://github.com/apache/datafusion/pulls?q=is%3Apr+merged%3A2025-09-16..2025-11-08
The changelog is here: https://github.com/apache/datafusion/blob/branch-51/dev/changelog/51.0.0.md
The list of major features can be found here apache/datafusion#17558 under the section "Features to mention in the blog (if they make it)"
(please only include the ones that made it into the release, with a checkmark)