You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 05296c7
Browse filesBrowse the repository at this point in the historyBrowse files
docs: Clarify Known Users criteria and clean up active project list (apache#26926)
## Which issue does this PR close?
Closesapache#26803.
## Rationale for this change
The "Known Users" section previously defined an active project using
only a six-month commit activity criterion, documented in an HTML
comment rather than visible to readers.
This made inclusion requirements unclear and allowed inactive or
minimally adopted projects to remain in the active list. Some entries
also contained outdated repository links.
This PR establishes visible, measurable inclusion criteria and
reorganizes the list to better distinguish active projects from other
projects using or having used DataFusion.
## What changes are included in this PR?
- **Clarify active-list criteria:** Public GitHub repositories must meet
all three requirements:
- At least one commit on the default branch within the last six calendar
months.
- At least 100 GitHub stars.
- At least five contributors, according to the GitHub repository
sidebar.
- **Document exceptions:** Projects without public repositories must
link to public evidence of DataFusion use, such as documentation or blog
posts.
- **Clarify scope:** These criteria apply only to Known Users, not
Integrations and Extensions.
- **Reorganize project listings:** Move datapress, EDB Postgres
Lakehouse, GlareDB, HoraeDB, IceGate, KalamDB, ROAPI, and VegaFusion
from the active list to the secondary list. Remove the Infino entry from
the active list.
- **Improve supporting references:** Add DataFusion usage links for
Funnel, hiop, Hotdata, Massive.com, Mycorr, Supermetal, and Telemetry.
- **Fix repository links:** Update Lance, Restate, and HoraeDB
repository URLs. Mark HoraeDB as archived.
- **Clarify the secondary list:** Rename its introduction to cover less
active projects and projects that do not meet the active-list criteria.
## What is the testing strategy for this PR?
Documentation-only change. No automated tests are added or modified in
this patch.
The changes can be reviewed by inspecting the Markdown, checking the
updated links, and verifying project placement against the documented
inclusion criteria.
## Are there any user-facing changes?
Yes. Readers of the DataFusion user guide will see explicit inclusion
criteria for the Known Users active list, an updated classification of
projects, and corrected or additional project references.
There are no changes to DataFusion runtime behavior or public APIs.
## LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated
content has been manually reviewed.
Copy file name to clipboardExpand all lines: docs/source/user-guide/introduction.md
+62-76Lines changed: 62 additions & 76 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -75,13 +75,14 @@ latency).
75
75
76
76
Here are some example systems built using DataFusion:
77
77
78
-
- Specialized Analytical Database systems such as [HoraeDB] and more general Apache Spark like system such as [Ballista]
79
-
- New query language engines such as [prql-query] and accelerators such as [VegaFusion]
80
-
- Research platform for new Database Systems, such as [Flock]
81
-
- SQL support to another library, such as [Vortex]
82
-
- Streaming data platforms such as [Synnada]
83
-
- Tools for reading / sorting / transcoding Parquet, CSV, AVRO, and JSON files such as [qv]
84
-
- Native Spark runtime replacement such as [Auron]
78
+
- Specialized Analytical Database systems such as [OpenObserve](https://github.com/openobserve/openobserve) and more general Apache Spark like systems such as [Ballista](https://github.com/apache/datafusion-ballista)
79
+
- Distributed SQL query engines for data in Iceberg catalogs, such as [Cloudflare R2 Query Engine](https://blog.cloudflare.com/r2-sql-deep-dive/)
80
+
- New query language engines such as [prql-query](https://github.com/prql/prql-query) and accelerators such as [VegaFusion](https://vegafusion.io/)
81
+
- Research platform for new Database Systems, such as [Flock](https://github.com/flock-lab/flock)
82
+
- SQL support to another library, such as [Vortex](https://vortex.dev/)
83
+
- Streaming data platforms such as [Synnada](https://synnada.ai/)
84
+
- Tools for reading / sorting / transcoding Parquet, CSV, AVRO, and JSON files such as [qv](https://github.com/timvw/qv)
85
+
- Native Spark runtime replacement such as [Apache Auron](https://github.com/apache/auron)
85
86
- Distributed data cache to boost GPU utilization of AI workloads with [Kubeflow Trainer](https://www.kubeflow.org/docs/components/trainer/user-guides/data-cache/)
86
87
87
88
By using DataFusion, projects are freed to focus on their specific
@@ -91,98 +92,83 @@ parallelized streaming execution plans, file format support, etc.
91
92
92
93
## Known Users
93
94
94
-
Here are some active projects using DataFusion:
95
+
Projects with a public GitHub repository must meet **all** of these criteria to
96
+
appear in the active list:
97
+
98
+
- At least one commit on the default branch in the last 6 months (calendar months).
99
+
- At least 100 stars.
100
+
- At least 5 contributors, as shown in the repository's GitHub sidebar.
95
101
96
-
<!-- "Active" means github repositories that had at least one commit in the last 6 months -->
102
+
Check these criteria when adding or updating an entry. Projects without a public
103
+
repository must instead link to public evidence of DataFusion use, such as a blog
104
+
post or documentation page. These criteria apply to Known Users, not the
105
+
Integrations and Extensions lists below.
106
+
107
+
Here are some active projects using DataFusion:
97
108
98
109
-[Arroyo](https://github.com/ArroyoSystems/arroyo) Distributed stream processing engine in Rust
-[Cube Store] Cube’s universal semantic layer platform is the next evolution of OLAP technology for AI, BI, spreadsheets, and embedded analytics
111
+
-[Apache Auron](https://github.com/apache/auron) The Auron accelerator for big data engine (e.g., Spark, Flink) leverages native vectorized execution to accelerate query processing
-[CnosDB](https://github.com/cnosdb/cnosdb) Open Source Distributed Time Series Database
-[Cube Store](https://github.com/cube-js/cube.js/tree/master/rust) Cube’s universal semantic layer platform is the next evolution of OLAP technology for AI, BI, spreadsheets, and embedded analytics
105
116
-[datafusion-dft](https://github.com/datafusion-contrib/datafusion-dft) Batteries included CLI, TUI, and server implementations for DataFusion.
106
-
-[datapress](https://docs.datap-rs.org) An opinionated small and fast data server on parquet and delta tables.
107
117
-[dbt Fusion engine](https://github.com/dbt-labs/dbt-fusion) The dbt Fusion engine, written in Rust, designed for speed and correctness with a native SQL understanding across DWH SQL dialects.
108
-
-[delta-rs] Native Rust implementation of Delta Lake
109
-
-[EDB Postgres Lakehouse] built with [Seafowl]
118
+
-[delta-rs](https://github.com/delta-io/delta-rs) Native Rust implementation of Delta Lake
110
119
-[Feldera](https://github.com/feldera/feldera) Fast query engine for incremental computation
111
-
-[Funnel](https://funnel.io/) Data Platform powering Marketing Intelligence applications.
112
-
-[GlareDB](https://github.com/GlareDB/glaredb) Fast SQL database for querying and analyzing distributed data.
113
-
-[GreptimeDB] Open Source & Cloud Native Distributed Time Series Database
114
-
-[hiop](https://hiop.io) Serverless Data Logistic Platform
115
-
-[HoraeDB] Distributed Time-Series Database
116
-
-[Hotdata](https://www.hotdata.dev) On-demand databases for AI agents with a unified query engine for vector, OLAP, and full-text search.
120
+
-[Funnel](https://funnel.io/) Data Platform powering Marketing Intelligence applications. ([DataFusion use](https://funnel.io/devblog/the-data-that-wont-sit-still-engineering-at-the-edge-of-marketing-data))
121
+
-[GreptimeDB](https://github.com/GreptimeTeam/greptimedb) Open Source & Cloud Native Distributed Time Series Database
122
+
-[hiop](https://hiop.io) Serverless Data Logistic Platform ([DataFusion use](https://github.com/apache/datafusion/pull/18114))
123
+
-[Hotdata](https://www.hotdata.dev) On-demand databases for AI agents with a unified query engine for vector, OLAP, and full-text search. ([DataFusion use](https://www.hotdata.dev/blog/bringing-ducklake-to-apache-datafusion))
117
124
-[Iceberg-rust](https://github.com/apache/iceberg-rust) Rust implementation of Apache Iceberg
118
-
-[IceGate](https://icegate.tech) Observability data lake engine for metrics, traces, logs, and events, built on Apache Iceberg with OpenTelemetry ingestion
119
-
-[InfluxDB] Time Series Database
120
-
-[Kamu] Planet-scale streaming data pipeline
125
+
-[InfluxDB](https://github.com/influxdata/influxdb) Time Series Database
126
+
-[Kamu](https://github.com/kamu-data/kamu-cli) Planet-scale streaming data pipeline
121
127
-[Kubeflow Trainer](https://github.com/kubeflow/trainer) Kubernetes-native project designed for
122
128
scalable LLMs fine-tuning and distributed AI model training.
123
129
-[LakeSoul](https://github.com/lakesoul-io/LakeSoul) Open source LakeHouse framework with native IO in Rust.
124
-
-[Lance](https://github.com/lancedb/lance) Modern columnar data format for ML
-[ParadeDB](https://github.com/paradedb/paradedb) PostgreSQL for Search & Analytics
127
-
-[Parseable] Open columnar observability data lake platform for logs, metrics, and traces.
128
-
-[Massive.com](https://massive.com/) Stock Market API
129
-
-[Mycorr](https://mycorr.app) Shared, versioned tables with units, links and SQL for humans and agents
130
-
-[qv] Quickly view your data
131
-
-[R2 Query Engine](https://blog.cloudflare.com/r2-sql-deep-dive/) Cloudflare's distributed engine for querying data in Iceberg Catalogs
133
+
-[Parseable](https://github.com/parseablehq/parseable) Open columnar observability data lake platform for logs, metrics, and traces.
134
+
-[Massive.com](https://github.com/massive-com) Stock Market API
135
+
-[Mycorr](https://mycorr.app) Shared, versioned tables with units, links and SQL for humans and agents ([DataFusion use](https://docs.mycorr.app/developers/mcp-server.html#querying-with-sql))
136
+
-[qv](https://github.com/timvw/qv) Quickly view your data
137
+
-[Cloudflare R2 Query Engine](https://blog.cloudflare.com/r2-sql-deep-dive/) Cloudflare's distributed engine for querying data in Iceberg Catalogs
132
138
-[rerun.io](https://rerun.io/) Visualize and query robotics logs and transform them into training data.
133
-
-[Restate](https://github.com/restatedev) Easily build resilient applications using distributed durable async/await
134
-
-[ROAPI] Create full-fledged APIs for slowly moving datasets without writing a single line of code
139
+
-[Restate](https://github.com/restatedev/restate) Easily build resilient applications using distributed durable async/await
135
140
-[Sail](https://github.com/lakehq/sail) Unifying stream, batch and AI workloads with Apache Spark compatibility
136
-
-[SedonaDB](https://github.com/apache/sedona-db) A single-node analytical database engine with geospatial as a first-class citizen
141
+
-[Apache SedonaDB](https://github.com/apache/sedona-db) A single-node analytical database engine with geospatial as a first-class citizen
137
142
-[Sleeper](https://github.com/gchq/sleeper) Serverless, cloud-native, log-structured merge tree based, scalable key-value store
138
-
-[Spice.ai] Building blocks for data-driven AI applications
139
-
-[Streamling](https://www.streamling.dev/) Data streaming runtime focused on performance, consistency, and extensibility
140
-
-[Supermetal](https://supermetal.io/) is a change data capture (CDC) platform that synchronizes data between databases, data warehouses, and lakehouses
141
-
-[Synnada] Streaming-first framework for data products
142
-
-[VegaFusion] Server-side acceleration for the [Vega](https://vega.github.io/) visualization grammar
143
-
-[Vortex] An extensible, state of the art columnar file format
144
-
-[Telemetry](https://telemetry.sh/) Structured logging made easy
143
+
-[Spice.ai](https://github.com/spiceai/spiceai) Building blocks for data-driven AI applications
144
+
-[Streamling](https://github.com/goldsky-io/streamling) Data streaming runtime focused on performance, consistency, and extensibility
145
+
-[Supermetal](https://supermetal.io/) is a change data capture (CDC) platform that synchronizes data between databases, data warehouses, and lakehouses ([DataFusion use](https://github.com/apache/datafusion/pull/23790))
146
+
-[Synnada](https://synnada.ai/) Streaming-first framework for data products
147
+
-[Vortex](https://vortex.dev/) An extensible, state of the art columnar file format
148
+
-[Telemetry](https://telemetry.sh/) Structured logging made easy ([DataFusion use](https://telemetry.sh/docs/discussion-topics/datafusion-sql-reference))
145
149
-[Xorq](https://github.com/xorq-labs/xorq/) Xorq is a multi-engine batch transformation framework built on Ibis, DataFusion and Arrow
146
-
-[KalamDB](https://github.com/jamals86/KalamDB) SQL-first realtime state database for AI agents, chat products, and multi-tenant SaaS.
147
150
-[Infino](https://github.com/infino-ai/infino) Fast retrieval engine for SQL, full-text search, and vector search over Parquet on object storage
148
151
149
-
Here are some less active projects that used DataFusion:
152
+
Here are other projects using or having used DataFusion, including less active
153
+
projects and projects that do not meet the active-list criteria:
150
154
151
155
-[bdt](https://github.com/datafusion-contrib/bdt) Boring Data Tool
152
-
-[Cloudfuse Buzz]
153
-
-[Dask SQL] Distributed SQL query engine in Python
154
-
-[Exon] Analysis toolkit for life-science applications
-[Dask SQL](https://github.com/dask-contrib/dask-sql) Distributed SQL query engine in Python
158
+
-[datapress](https://github.com/jeroenflvr/datapress) An opinionated small and fast data server on parquet and delta tables.
159
+
-[EDB Postgres Lakehouse](https://www.enterprisedb.com/products/analytics) built with [Seafowl](https://github.com/splitgraph/seafowl)
160
+
-[Exon](https://github.com/wheretrue/exon) Analysis toolkit for life-science applications
-[GlareDB](https://github.com/GlareDB/glaredb) Fast SQL database for querying and analyzing distributed data.
-[IceGate](https://github.com/icegatetech/icegate) Observability data lake engine for metrics, traces, logs, and events, built on Apache Iceberg with OpenTelemetry ingestion
165
+
-[KalamDB](https://github.com/jamals86/KalamDB) SQL-first realtime state database for AI agents, chat products, and multi-tenant SaaS.
166
+
-[ROAPI](https://github.com/roapi/roapi) Create full-fledged APIs for slowly moving datasets without writing a single line of code
-[VegaFusion](https://vegafusion.io/) Server-side acceleration for the [Vega](https://vega.github.io/) visualization grammar
157
169
158
170
If you know of another project, please submit a PR to add a link!
0 commit comments