Skip to content

Commit cd1f82b

Browse files
committed
test: add a scenario fuzzer for the spill pool contract
The spill pool had three faults since its unbuffered reads: apache#20683, apache#23447 and apache#24883. In each fault, two writers were in `push_batch` at the same time. No unit test could cause that overlap. This commit adds a test double and three tests. `IoHookFactory` is a `TempFileFactory` that runs a hook before each disk write and consults a second hook before each read. A writer writes while it holds the lock of its file. Thus a hook that blocks can hold a writer at the point where it has a file checked out, a read delay makes the bytes of an older file arrive after those of a newer file, and a read hook can also fail a read. `test_reader_does_not_wait_on_drained_file_while_another_has_data` holds writer 1 inside its first write while writer 2 pushes a batch. The reader must then give both batches while both writers are alive. `test_read_errors_are_reported_to_the_reader` fails the read of a spill file in the two possible ways: the file does not open, and the file opens and then gives an error as the first item of its stream. Those are the two error paths of `SpillPoolFile::poll_file`, and the reader must give the error in both cases. `RepartitionExec` waits on this stream after each spilled marker, so an error that gets lost here stops the query in the same way as a deadlock. `spill_pool_scenario_fuzz` runs random scenarios from a seed: one to four writers, file rotation after each batch, after some batches or never, and one to six phases. In each phase a random set of writers pushes one to three batches each. A pauser holds each writer of the phase inside its first write, so the pushes overlap. Then the reader reads a random number of the available batches, which can leave a backlog for a later phase. Some phases drop a writer. Each scenario checks the contract that `RepartitionExec` depends on: - When no push is in progress, the reader can read each pushed batch without more writer activity. - The reader does not signal EOF while a writer is alive. - After the last writer is dropped, the reader gives the backlog and then EOF. - The reader gives exactly the pushed batches. With one writer, it gives them in push order. - Disk usage is zero after the reader is dropped. The first two tests are red on this commit. The fuzzer also fails on these versions of the pool: - The code before apache#23522. Seed 4 shows the apache#23447 stall after the drop of all writers, and seeds 0 and 1 show the apache#24883 stall. All 16 pool tests of that time pass on that code. - A draft of the fix in the next commit that skips a file with a read in progress. Seed 2 shows a FIFO violation with one writer. The default of 50 seeds takes approximately three seconds. The extended CI job runs 1000 seeds. `DATAFUSION_SPILL_POOL_FUZZ_SEED` replays one seed. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01RauD5PqPCp4y92RUZUD6C9
1 parent bd70a90 commit cd1f82b

2 files changed

Lines changed: 571 additions & 7 deletions

File tree

‎.github/workflows/extended.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,9 @@ jobs:
8888
- name: Run tests (excluding doctests)
8989
env:
9090
RUST_BACKTRACE: 1
91+
# Run more random scenarios of the spill pool fuzzer than the
92+
# default suite does (about a minute).
93+
DATAFUSION_SPILL_POOL_FUZZ_ITERATIONS: 1000
9194
run: |
9295
cargo test \
9396
--profile ci \

0 commit comments

Comments
 (0)