[WEBSITE] DataFusion 16.0.0 blog post by alamb · Pull Request #294 · apache/arrow-site

alamb · 2023-01-07T16:05:32Z

This blog post highlights some improvements and features in DataFusion the last 3 releases 😅

Rendered: https://arrow.apache.org/blog/2023/01/19/datafusion-16.0.0/

github-actions · 2023-01-07T16:05:46Z

Thanks for opening a pull request!

Could you open an issue for this pull request on JIRA?
https://issues.apache.org/jira/browse/ARROW

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

See also:

alamb · 2023-01-07T16:06:01Z

It is a work in progress, but I think it is no coherent enough to gather some community input

alamb

It would also be great to add a section to this document about planned feature work

alamb · 2023-01-07T16:07:58Z

_posts/2023-01-07-datafusion-16.0.0.md

+
+## Community Growth
+
+The three months since [our last update](https://arrow.apache.org/blog/2022/10/25/datafusion-13.0.0/) again saw significant growth in the DataFusion.


It would be great if someone could help clean this section up and clearly explain the growth of the community; There is a wonderful story there to tell

alamb · 2023-01-07T16:08:26Z

_posts/2023-01-07-datafusion-16.0.0.md

+closing the gap quickly. Performance highlights from the last three
+months:
+
+* XX% Faster Sorting and Merging using the new [Row Format](https://arrow.apache.org/blog/2022/11/07/multi-column-sorts-in-arrow-rust-part-1/)


@tustvold do you have any suggstions about what numbers to use here?

alamb · 2023-01-07T16:08:44Z

_posts/2023-01-07-datafusion-16.0.0.md

+* Basic filter selectivity analysis (#3868)
+
+
+In the coming few months, we plan work on:


Feedback from the rest of the community would be great

_posts/2023-01-07-datafusion-16.0.0.md

alamb · 2023-01-07T16:09:51Z

_posts/2023-01-07-datafusion-16.0.0.md

+- Implement current_date scalar function (#4022)
+- Compressed CSV/JSON support (#3642)
+
+The community has also been investing in sqllogic based tests to help keep DataFusion's quality high with less work (TODO add some more detail / lnks)


@xudong963 I wonder if you have any thoughts on how to word this better

alamb · 2023-01-07T16:10:04Z

_posts/2023-01-07-datafusion-16.0.0.md

+
+# Substrait
+
+TODO motivating introduction of substrait and why this is interesting


@andygrove perhaps you can help with content for the substrait area

Substrait isn't going to make it into 16, and maybe this should be a separate post? I started a google doc https://docs.google.com/document/d/1vK0AyDBhIibmKZ2scGN3jBypBvqMPuztbBzZ1eh0dKM/edit?usp=sharing

Well, maybe there is a chance it makes it in. Assuming the vote passes tomorrow, we could get the first PR merged 🤔

alamb · 2023-01-07T16:11:23Z

_posts/2023-01-07-datafusion-16.0.0.md

+DataFusion has basic python bindings which has the potential to expand datafusion  to more end users a major missing piece are the python bindings
+
+
+# python bindings and growing the community and ecosystem


I tried to work in a mention of the python bindings and encouraging a champion for them to step forward, however it felt more like it should be a separate post 🤔 -- @andygrove what do you think about a post describing the python bindings, why they are cool, and trying to find people to help drive that project?

I can't figure out how to work this in so I think we should write another post -- maybe on a different site

I took the content / notes I had and put them in a google doc: https://docs.google.com/document/d/1zNfK8pIOqgHURX2lHK0JhSKTaCH3t23tYDKpGLWdFRY/edit

Thanks. I agree. I can work on the Python post

_posts/2023-01-07-datafusion-16.0.0.md

Co-authored-by: Andy Grove <[email protected]>

ozankabak · 2023-01-08T20:05:34Z

_posts/2023-01-07-datafusion-16.0.0.md

+
+Growth of new systems based on as the engine in [many open source and commercial projects](https://github.com/apache/arrow-datafusion#known-uses) and was one of the early open source projects to provide this capability.
+
+Several new databases built on datafusion (synnada.ai, greptimedb, probably others)


Here is what I am aware of:

Databases: greptimedb (new), IOx (GA)
Data platform: Synnada (new)
Use case: Backend for PRQL (relatively new?)

Thanks -- added in ffe2e0a. Still needs polish

andygrove · 2023-01-10T16:41:18Z

I will start contributing to this tomorrow

alamb · 2023-01-10T21:42:50Z

Ok I think this one is now ready for some more review -- it is plausibly ready to publish

_posts/2023-01-07-datafusion-16.0.0.md

ozankabak

Read from start to finish. Apart from the small preposition comment I left inline, it looks great. Looking forward to seeing this published.

…date_16

…ite into alamb/datafusion_update_16

alamb · 2023-01-17T21:20:23Z

I plan to merge this tomorrow unless there are any other comments

[WEBSITE] Datafsuion 16.0.0 blog post

84073d3

alamb changed the title ~~[WEBSITE] DataFusion 16.0.0 blog post~~ [WEBSITE] DataFusion 16.0.0 blog post (WIP) Jan 7, 2023

alamb mentioned this pull request Jan 7, 2023

Blog post about datafusion 16 release apache/datafusion#4804

Closed

alamb commented Jan 7, 2023

View reviewed changes

andygrove reviewed Jan 7, 2023

View reviewed changes

_posts/2023-01-07-datafusion-16.0.0.md Outdated Show resolved Hide resolved

Update _posts/2023-01-07-datafusion-16.0.0.md

00b6ad6

Co-authored-by: Andy Grove <[email protected]>

ozankabak reviewed Jan 8, 2023

View reviewed changes

alamb added 2 commits January 10, 2023 14:49

Add note about streaming

4641b93

work on comunity growth page

ffe2e0a

alamb added 2 commits January 10, 2023 22:30

Updates

ee51391

update more

e4c330f

alamb changed the title ~~[WEBSITE] DataFusion 16.0.0 blog post (WIP)~~ [WEBSITE] DataFusion 16.0.0 blog post Jan 10, 2023

ozankabak reviewed Jan 10, 2023

View reviewed changes

_posts/2023-01-07-datafusion-16.0.0.md Outdated Show resolved Hide resolved

ozankabak reviewed Jan 10, 2023

View reviewed changes

alamb added 6 commits January 10, 2023 17:53

Update _posts/2023-01-07-datafusion-16.0.0.md

b5ca1c7

Merge remote-tracking branch 'origin/master' into alamb/datafusion_up…

76ff403

…date_16

Update publishing date

539b735

Update statistics

3223c81

copy edit

1b55bc5

Merge branch 'alamb/datafusion_update_16' of github.com:alamb/arrow-s…

4e1750f

…ite into alamb/datafusion_update_16

Update date

ad5427a

alamb merged commit 768be07 into apache:master Jan 19, 2023

alamb deleted the alamb/datafusion_update_16 branch January 19, 2023 12:38

alamb mentioned this pull request Mar 31, 2023

Blog post with DataFusion Jan - April 2023 apache/datafusion#5812

Closed


		## Community Growth

		The three months since [our last update](https://arrow.apache.org/blog/2022/10/25/datafusion-13.0.0/) again saw significant growth in the DataFusion.

		* Basic filter selectivity analysis (#3868)


		In the coming few months, we plan work on:


		# Substrait

		TODO motivating introduction of substrait and why this is interesting

		DataFusion has basic python bindings which has the potential to expand datafusion to more end users a major missing piece are the python bindings


		# python bindings and growing the community and ecosystem


		Growth of new systems based on as the engine in [many open source and commercial projects](https://github.com/apache/arrow-datafusion#known-uses) and was one of the early open source projects to provide this capability.

		Several new databases built on datafusion (synnada.ai, greptimedb, probably others)

Comments

Conversation

alamb commented Jan 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 7, 2023

Uh oh!

alamb commented Jan 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andygrove commented Jan 10, 2023

Uh oh!

alamb commented Jan 10, 2023

Uh oh!

Uh oh!

ozankabak left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Jan 17, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

alamb commented Jan 7, 2023 •

edited

Loading

alamb commented Jan 7, 2023 •

edited

Loading

ozankabak left a comment •

edited

Loading