Repository navigation
Commit f943030
committed
test: add a scenario fuzzer for the spill pool contract
The spill pool had three bugs since its unbuffered reads: apache#20683, apache#23447
and apache#24883. In each bug, two writers were in `push_batch` at the same
time. No unit test could cause that overlap.
This commit adds a test double and two tests:
- `IoHookFactory` is a `TempFileFactory` that runs a hook before each
disk write. A writer writes while it holds the lock of its file. Thus
a hook that blocks can hold a writer at the point where it has a file
checked out. The factory can also delay the reads of a file.
- `test_reader_does_not_wait_on_drained_file_while_another_has_data`
holds writer 1 inside its first write while writer 2 pushes a batch.
Then the reader must give both batches while both writers are alive.
- `spill_pool_scenario_fuzz` runs random scenarios from a seed: one to
four writers, file rotation after each batch, after some batches or
never, and one to six phases. In each phase, a random set of writers
pushes one to three batches each. A pauser holds each writer of the
phase inside its first write, so the pushes overlap. Then the reader
reads a random number of the available batches. Some phases drop a
writer.
Each scenario checks the contract that `RepartitionExec` depends on:
- When no push is in progress, the reader can read each pushed batch
without more writer activity.
- The reader does not signal EOF while a writer is alive.
- After the last writer is dropped, the reader gives the backlog and
then EOF.
- The reader gives exactly the pushed batches. With one writer, it gives
them in push order.
- Disk usage is zero after the reader is dropped.
The two tests are red on this commit. The fuzzer also fails on these
versions of the pool:
- The code before apache#23522. Seed 4 shows the apache#23447 stall after the drop
of all writers. Seeds 0 and 1 show the apache#24883 stall. All 16 pool tests
of that time pass on that code.
- A draft of the fix in the next commit that skips a file with a read in
progress. Seed 2 shows a FIFO violation with one writer.
The default of 50 seeds takes approximately three seconds. The extended
CI job runs 1000 seeds. `DATAFUSION_SPILL_POOL_FUZZ_SEED` replays one
seed.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01RauD5PqPCp4y92RUZUD6C91 parent 15a8ebc commit f943030
2 files changed
Lines changed: 501 additions & 7 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
88 | 88 | | |
89 | 89 | | |
90 | 90 | | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
91 | 94 | | |
92 | 95 | | |
93 | 96 | | |
| |||
0 commit comments