Skip to content

Commit b8534c7

Browse files
2010YOUY012010YOUY01
authored
docs: clarify fuzz testing guidance (#27198)
> [!IMPORTANT] > > **Fork CI setup:** For pull requests from a fork to `main`, enable workflows in your fork's **Actions** tab before you push. > See the [fork CI setup guide](https://datafusion.apache.org/contributor-guide/#pull-requests-from-forks). ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> part of #27082 Follow up: #27048 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. Please explain the problem you are trying to solve in terms of the user-visible behavior, rather than the implementation. For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the implementation. "COUNT(DISTINCT) returns wrong results when the column contains nulls" is the user-visible problem. --> #27048 introduced a trait for implementing fuzzing UTs, this PR adds documentation for it. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR. --> ## What is the testing strategy for this PR? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code Briefly describe how this PR is tested, and point to the specific tests you added. For example: 'This new feature is covered by the `sqllogictest` cases added in `foo.slt`'. If this PR does not add tests, explain why. For example, if the change is already covered by existing tests, please mention it. You should also check the `codecov` bot reply on this PR to confirm the changed code is exercised. --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. -->
1 parent 32dd1d2 commit b8534c7

1 file changed

Lines changed: 36 additions & 22 deletions

File tree

‎docs/source/contributor-guide/testing.md‎

Lines changed: 36 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -170,6 +170,42 @@ For local SQLite test instructions, see the [instructions in the documentation].
170170
[instructions in the documentation]: https://github.com/apache/datafusion/tree/main/datafusion/sqllogictest#running-tests-sqlite
171171
[extended.yml]: https://github.com/apache/datafusion/blob/main/.github/workflows/extended.yml
172172

173+
## Rust Integration Tests
174+
175+
There are several public interface tests for the DataFusion library in the [tests](https://github.com/apache/datafusion/tree/main/datafusion/core/tests) directory.
176+
177+
You can run these tests individually using `cargo` as normal command such as
178+
179+
```shell
180+
cargo test -p datafusion --test parquet_integration
181+
```
182+
183+
## Fuzz testing
184+
185+
Fuzz testing generates random inputs and configurations, then checks that a property holds for every generated case. For example, an optimized operator should produce the same results as a reference implementation.
186+
187+
### Fuzzing unit tests
188+
189+
For fuzzing unit tests, such as tests for a single operator or an internal utility:
190+
191+
- If the test is small (easy to set up and takes a few seconds), keep it as a unit test.
192+
- If the test is large (requires a fuzzing driver and takes more than 10 seconds), it should implement the [`FuzzTest` trait][fuzz framework] and run through the [central fuzz-test entry point]. See the [spill-pool fuzzer] for a reference implementation.
193+
194+
[fuzz framework]: https://github.com/apache/datafusion/blob/main/test-utils/src/fuzz.rs
195+
[central fuzz-test entry point]: https://github.com/apache/datafusion/blob/main/datafusion/fuzz/tests/fuzz.rs
196+
[spill-pool fuzzer]: https://github.com/apache/datafusion/blob/main/datafusion/fuzz/tests/suites/spill_pool.rs
197+
198+
### SQL fuzz testing
199+
200+
For SQL fuzzing of more complex queries, use [datafusion-fuzzer], which is currently maintained outside the core DataFusion repository. See [its documentation][datafusion-fuzzer-docs] for details.
201+
202+
[datafusion-fuzzer]: https://github.com/datafusion-contrib/datafusion-fuzzer
203+
[datafusion-fuzzer-docs]: https://github.com/datafusion-contrib/datafusion-fuzzer/blob/main/README.md
204+
205+
---
206+
207+
The following sections cover **non-functional tests and checks** for CI workflows, documentation, and repository maintenance.
208+
173209
## CI Workflow Tests
174210

175211
The Rust workflow validator and fork status bridge have regression tests.
@@ -186,28 +222,6 @@ node --test ci/scripts/test_fork_ci_status.cjs
186222
The validator tests remove guards and add an invalid job dependency in memory.
187223
The bridge tests use stubbed GitHub API responses and make no network requests.
188224

189-
## Rust Integration Tests
190-
191-
There are several public interface tests for the DataFusion library in the [tests](https://github.com/apache/datafusion/tree/main/datafusion/core/tests) directory.
192-
193-
You can run these tests individually using `cargo` as normal command such as
194-
195-
```shell
196-
cargo test -p datafusion --test parquet_integration
197-
```
198-
199-
## SQL "Fuzz" testing
200-
201-
DataFusion uses the [SQLancer] for "fuzz" testing: it generates random SQL
202-
queries and execute them against DataFusion to find bugs.
203-
204-
The code is in the [datafusion-sqllancer] repository, and we welcome further
205-
contributions. Kudos to [@2010YOUY01] for the initial implementation.
206-
207-
[sqlancer]: https://github.com/sqlancer/sqlancer
208-
[datafusion-sqllancer]: https://github.com/datafusion-contrib/datafusion-sqllancer
209-
[@2010youy01]: https://github.com/2010YOUY01
210-
211225
## Documentation Examples
212226

213227
We use Rust [doctest] to verify examples from the documentation are correct and

0 commit comments

Comments
 (0)