Skip to content

Commit f943030

Browse files
committed
test: add a scenario fuzzer for the spill pool contract
The spill pool had three bugs since its unbuffered reads: apache#20683, apache#23447 and apache#24883. In each bug, two writers were in `push_batch` at the same time. No unit test could cause that overlap. This commit adds a test double and two tests: - `IoHookFactory` is a `TempFileFactory` that runs a hook before each disk write. A writer writes while it holds the lock of its file. Thus a hook that blocks can hold a writer at the point where it has a file checked out. The factory can also delay the reads of a file. - `test_reader_does_not_wait_on_drained_file_while_another_has_data` holds writer 1 inside its first write while writer 2 pushes a batch. Then the reader must give both batches while both writers are alive. - `spill_pool_scenario_fuzz` runs random scenarios from a seed: one to four writers, file rotation after each batch, after some batches or never, and one to six phases. In each phase, a random set of writers pushes one to three batches each. A pauser holds each writer of the phase inside its first write, so the pushes overlap. Then the reader reads a random number of the available batches. Some phases drop a writer. Each scenario checks the contract that `RepartitionExec` depends on: - When no push is in progress, the reader can read each pushed batch without more writer activity. - The reader does not signal EOF while a writer is alive. - After the last writer is dropped, the reader gives the backlog and then EOF. - The reader gives exactly the pushed batches. With one writer, it gives them in push order. - Disk usage is zero after the reader is dropped. The two tests are red on this commit. The fuzzer also fails on these versions of the pool: - The code before apache#23522. Seed 4 shows the apache#23447 stall after the drop of all writers. Seeds 0 and 1 show the apache#24883 stall. All 16 pool tests of that time pass on that code. - A draft of the fix in the next commit that skips a file with a read in progress. Seed 2 shows a FIFO violation with one writer. The default of 50 seeds takes approximately three seconds. The extended CI job runs 1000 seeds. `DATAFUSION_SPILL_POOL_FUZZ_SEED` replays one seed. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01RauD5PqPCp4y92RUZUD6C9
1 parent 15a8ebc commit f943030

2 files changed

Lines changed: 501 additions & 7 deletions

File tree

‎.github/workflows/extended.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,9 @@ jobs:
8888
- name: Run tests (excluding doctests)
8989
env:
9090
RUST_BACKTRACE: 1
91+
# Run more random scenarios of the spill pool fuzzer than the
92+
# default suite does (about a minute).
93+
DATAFUSION_SPILL_POOL_FUZZ_ITERATIONS: 1000
9194
run: |
9295
cargo test \
9396
--profile ci \

0 commit comments

Comments
 (0)