Repository navigation
Commit de96b4a
committed
test: add a scenario fuzzer for the spill pool contract
The spill pool had three faults since its unbuffered reads: apache#20683, apache#23447
and apache#24883. In each fault, two writers were in `push_batch` at the same
time. No unit test could cause that overlap.
This commit adds a test double and three tests.
`IoHookFactory` is a `TempFileFactory` that runs a hook before each disk
write and consults a second hook before each read. A writer writes while it
holds the lock of its file. Thus a hook that blocks can hold a writer at the
point where it has a file checked out, a read delay makes the bytes of an
older file arrive after those of a newer file, and a read hook can also fail
a read.
`test_reader_does_not_wait_on_drained_file_while_another_has_data` holds
writer 1 inside its first write while writer 2 pushes a batch. The reader
must then give both batches while both writers are alive.
`test_read_error_is_reported_to_the_reader` fails the read of a spill file.
The reader must give the error. `RepartitionExec` waits on this stream
after each spilled marker, so an error that gets lost here stops the query
in the same way as a deadlock.
`spill_pool_scenario_fuzz` runs random scenarios from a seed: one to four
writers, file rotation after each batch, after some batches or never, and
one to six phases. In each phase a random set of writers pushes one to three
batches each. A pauser holds each writer of the phase inside its first
write, so the pushes overlap. Then the reader reads a random number of the
available batches, which can leave a backlog for a later phase. Some phases
drop a writer.
Each scenario checks the contract that `RepartitionExec` depends on:
- When no push is in progress, the reader can read each pushed batch
without more writer activity.
- The reader does not signal EOF while a writer is alive.
- After the last writer is dropped, the reader gives the backlog and then
EOF.
- The reader gives exactly the pushed batches. With one writer, it gives
them in push order.
- Disk usage is zero after the reader is dropped.
The first two tests are red on this commit. The fuzzer also fails on these
versions of the pool:
- The code before apache#23522. Seed 4 shows the apache#23447 stall after the drop of
all writers, and seeds 0 and 1 show the apache#24883 stall. All 16 pool tests
of that time pass on that code.
- A draft of the fix in the next commit that skips a file with a read in
progress. Seed 2 shows a FIFO violation with one writer.
The default of 50 seeds takes approximately three seconds. The extended CI
job runs 1000 seeds. `DATAFUSION_SPILL_POOL_FUZZ_SEED` replays one seed.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01RauD5PqPCp4y92RUZUD6C91 parent 311e670 commit de96b4a
2 files changed
Lines changed: 546 additions & 7 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
88 | 88 | | |
89 | 89 | | |
90 | 90 | | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
91 | 94 | | |
92 | 95 | | |
93 | 96 | | |
| |||
0 commit comments