test: Add a shared framework for reproducible fuzz tests by 2010YOUY01 · Pull Request #27048 · apache/datafusion

2010YOUY01 · 2026-11-03T12:36:24Z

Important

Fork CI setup: For pull requests from a fork to main, enable workflows in your fork's Actions tab before you push.
See the fork CI setup guide.

Which issue does this PR close?

Closes #.

Rationale for this change

Fuzz tests should follow these conventions:

Time budgets are configurable: each test case takes several milliseconds, and
the driver can run on a best-effort basis within a budget such as one minute.
Each test case can be reproduced from a seed.
The starting seed varies across runs to broaden coverage over time.

Most of the existing implementations don't follow all of these principles, or implemented it with unnecessary complexity:

Some individual fuzz unit tests take five minutes to run.
Some use environment variables for configuration.
Some use fixed seeds across runs.
Failures are hard to reproduce.

Refactoring to a shared trait can prevent these issues in the future.

This PR introduces a trait to enforce these conventions and migrates one fuzz unit test to use it. Other existing fuzzers remain available for later migration.

Demo

-- Runs all fuzzing UTs (now only 1 test refactored in this PR)
cargo xtask ci step test fuzz

What changes are included in this PR?

Read the key changes in the following order; the remaining changes are minor.

FuzzTest in test-utils/src/fuzz.rs: a trait that ensures all fuzz tests
follow the intended conventions.
Suite in datafusion/fuzz/tests/fuzz.rs: a single entry point for all fuzz
unit tests in DataFusion.
SpillPoolFuzzer in datafusion/fuzz/tests/suites/spill_pool.rs: the first unit
test migrated to the centralized FuzzTest trait and unified runner.
Update CI to run this fuzz test suite separately.

The key API is introduced in step 1. After implementing several fuzz tests in datafusion, I came up with this design to capture their common patterns. I also asked Codex to check whether all existing fuzz tests can be migrated to this trait.

Step 2: using a unified test entry point make it easier to configure fuzzing time in the future. For example it's possible to deploy a long-running background fuzzing job.

What is the testing strategy for this PR?

Are there any user-facing changes?

2010YOUY01 · 2026-11-03T12:37:46Z

+    let suites = [Suite::new(
+        &spill_pool,
+        RunConfig {
+            run_seed: Some(0),


This is the existing fuzz config, probably we can use rotating seed (timestamp), and time limit instead of 1000 cases.

alamb

The basic idea looks pretty good to me -- thank you @2010YOUY01

What I don't understand is the new crate structure and then the need to partially expose some of the existing testing code from the physical expr crate

alamb · 2026-11-03T21:12:25Z

-      - name: Run fuzzers with more iterations
+      - name: Run fuzz tests
        run: cargo xtask ci step test fuzz
+      # Migrate it to the above unified fuzz runner over time


Can you please file a ticket that explains the migration plan and has a checklist of tests to run and link it here? So we have the context in the future?

Good point, done in e205a1b

alamb · 2026-11-03T21:13:14Z

+
+mod suites;
+
+/// The single registration point for project-wide fuzz campaigns.


what is a fuzz campaign? I am not familar with the campaign terminology

Renamed it to run in dd6f892

alamb · 2026-11-03T21:14:58Z

+use datafusion_common::instant::Instant;
+use datafusion_common::{Result, exec_err};
+
+/// A component's fuzz experiment, run by the centralized driver.


it may help to add a link here to the centralized driver

2010YOUY01 · 2026-11-04T07:03:04Z

@alamb THanks for the review!

What I don't understand is the new crate structure and then the need to partially expose some of the existing testing code from the physical expr crate

Exposing some testing util was intended to keep the PR small, but this make the structure confusing.

This is fixed in dd6f892: the idea is the fuzzing implementation all lives in datafusion-fuzz crate, and since the testing target should be public interface (SpillPool in this case), so no extra public API change would be needed, only need to move the fuzzing implementation.

RE another major concern for this PR:

why did you make a new crate? Why not just a module in the existing datafusion crate?

I think this structure is clearer. Otherwise, integration tests would include regular, extended (slow), and fuzzing tests, making it awkward to maintain or run only a subset of them.

Since splitting them doesn't seem to introduce any other issues, I went with the clearer structure.

alamb

Thanks @2010YOUY01 -- looks good to me

I wonder if we should (as a follow on PR) add a note about fuzz testing to https://datafusion.apache.org/contributor-guide/testing.html

Specifically lay out the rationale / goal of in-project fuzz testing in an attempt to keep the implementations consistent

2010YOUY01 · 2026-11-07T03:52:19Z

Thanks @2010YOUY01 -- looks good to me

I wonder if we should (as a follow on PR) add a note about fuzz testing to https://datafusion.apache.org/contributor-guide/testing.html

Specifically lay out the rationale / goal of in-project fuzz testing in an attempt to keep the implementations consistent

I'll do it soon, thanks again.

> [!IMPORTANT] > > **Fork CI setup:** For pull requests from a fork to `main`, enable workflows in your fork's **Actions** tab before you push. > See the [fork CI setup guide](https://datafusion.apache.org/contributor-guide/#pull-requests-from-forks). ## Which issue does this PR close?  part of apache#27082 Follow up: apache#27048 ## Rationale for this change  apache#27048 introduced a trait for implementing fuzzing UTs, this PR adds documentation for it. ## What changes are included in this PR?  ## What is the testing strategy for this PR?  ## Are there any user-facing changes?

…ache#27258) ## Which issue does this PR close? Part of apache#27082. ## Rationale for this change The `union_fuzz` test in `key_range_bitmap.rs` previously ran as a standalone `#[test]` using environment variables (`DATAFUSION_KEY_RANGE_UNION_FUZZ_ITERATIONS` and `DATAFUSION_KEY_RANGE_UNION_FUZZ_SEED`) without standardized budget controls or centralized failure reporting. Migrating this suite to the `FuzzTest` framework introduced in apache#27048 makes failures reproducible via seed reporting, connects it to the unified runner in `datafusion-fuzz`, and aligns it with project-wide fuzz testing conventions. ## What changes are included in this PR? - **`datafusion-physical-plan`**: - Expose `KeyRangePart` and `KeyRangeBitmap::union` so `datafusion-fuzz` can construct partitions and test union behavior. - Expose `offset(&self)`, `span(&self)`, `shift(&self)`, and `bits(&self)` getter accessors on `KeyRangeBitmap` for internal invariant checking. - Remove legacy `union_fuzz` and `run_union_scenario` test functions from `key_range_bitmap.rs`. - **`datafusion-fuzz`**: - Implement `KeyRangeBitmapFuzzer` in `tests/suites/key_range_bitmap.rs` implementing the `FuzzTest` trait. - Register `KeyRangeBitmapFuzzer` in `tests/fuzz.rs` and `tests/suites/mod.rs`. - **`xtask`**: - Remove `union_fuzz` from `LEGACY_FUZZ_TESTS` in `ci_steps.rs` and clean up legacy runner configuration and snapshots. ## What is the testing strategy for this PR? - Validated `datafusion-physical-plan` unit tests: - `cargo test -p datafusion-physical-plan --lib -- joins::key_range_bitmap` (all 4 unit tests pass) - Validated unified fuzz runner: - `cargo test -p datafusion-fuzz --test fuzz --features extended_tests` (256 cases pass) - Validated `xtask` legacy fuzz test verification: - `cargo test -p datafusion-xtask -- legacy_fuzz_tests_exist` ## Are there any user-facing changes? No user-facing changes.

test: Add a shared framework for reproducible fuzz tests

cbacc83

2010YOUY01 commented Nov 3, 2026

View reviewed changes

2010YOUY01 mentioned this pull request Nov 3, 2026

Three slow tests set the length of the extended and hash-collision CI jobs #26759

Open

Merge remote-tracking branch 'upstream/main' into fuzz-ut-framework

9bec86e

github-actions Bot added development-process Related to development process of DataFusion physical-plan Changes to the physical-plan crate labels Nov 3, 2026

cleanup

2b56ad9

alamb reviewed Nov 3, 2026

View reviewed changes

review: move spill_pool_fuzz implementation to datafusion-fuzz crage

dd6f892

2010YOUY01 mentioned this pull request Nov 4, 2026

Tracking: Refactor existing fuzz unit tests to FuzzTest trait #27082

Open

1 of 31 tasks

2010YOUY01 added 5 commits November 4, 2026 15:08

review: add tracking issue in comments

e205a1b

Merge remote-tracking branch 'upstream/main' into fuzz-ut-framework

1d48f36

Merge remote-tracking branch 'upstream/main' into fuzz-ut-framework

620c25e

fix lint

00019d3

improve wording

a35deb5

alamb reviewed Nov 5, 2026

View reviewed changes

alamb approved these changes Nov 5, 2026

View reviewed changes

2010YOUY01 added this pull request to the merge queue Nov 7, 2026

Merged via the queue into apache:main with commit c3eeb47 Nov 7, 2026
45 checks passed

2010YOUY01 deleted the fuzz-ut-framework branch November 7, 2026 04:04

github-actions Bot added the v57.0.0 label Nov 7, 2026

2010YOUY01 mentioned this pull request Nov 7, 2026

docs: clarify fuzz testing guidance #27198

Merged

Awesome06 mentioned this pull request Nov 8, 2026

test: refactor key-range bitmap union fuzz test to FuzzTest trait #27245

Closed

Awesome06 mentioned this pull request Nov 9, 2026

test: refactor key-range bitmap union fuzz test to FuzzTest trait #27258

Merged

Navigation Menu

Repository navigation

test: Add a shared framework for reproducible fuzz tests - #27048

test: Add a shared framework for reproducible fuzz tests#27048
2010YOUY01 merged 9 commits into
apache:mainfrom
2010YOUY01:fuzz-ut-framework

2010YOUY01 commented Nov 3, 2026 •

edited

Loading

Uh oh!

2010YOUY01 Nov 3, 2026

Uh oh!

alamb left a comment

Uh oh!

alamb Nov 3, 2026

Uh oh!

2010YOUY01 Nov 4, 2026

Uh oh!

alamb Nov 3, 2026

Uh oh!

2010YOUY01 Nov 4, 2026

Uh oh!

Uh oh!

alamb Nov 3, 2026

Uh oh!

2010YOUY01 Nov 4, 2026

Uh oh!

2010YOUY01 commented Nov 4, 2026 •

edited

Loading

Uh oh!

alamb left a comment

Uh oh!

2010YOUY01 commented Nov 7, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		mod suites;

		/// The single registration point for project-wide fuzz campaigns.

Conversation

2010YOUY01 commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

Demo

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Uh oh!

2010YOUY01 Nov 3, 2026

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Nov 3, 2026

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 Nov 4, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Nov 3, 2026

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 Nov 4, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alamb Nov 3, 2026

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 Nov 4, 2026

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 commented Nov 7, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

2010YOUY01 commented Nov 3, 2026 •

edited

Loading

2010YOUY01 commented Nov 4, 2026 •

edited

Loading