Repository navigation
Commit 6ab3035
perf: create DiskManager Directories-mode working dirs lazily (apache#26583)
## Which issue does this PR close?
Closes apache#26446.
## Rationale for this change
`DiskManagerBuilder::build` in `DiskManagerMode::Directories` called
`create_local_dirs`, so a `datafusion-*` working directory was created
in every configured directory as soon as the `DiskManager` was built,
and torn down recursively when it was dropped — whether or not the query
ever spilled.
`DiskManagerMode::OsTmpDirectory` already defers this to the first
`create_tmp_file` (apache#1695). This makes `Directories` behave the same way.
The issue measures the cost: 11 syscalls per configured directory per
`DiskManager` on Linux, and in DataFusion Comet — which builds a
`RuntimeEnv` per native plan in a Spark task — a query of 501 short
tasks over 8 local directories ran 12% faster in wall time with 9% less
CPU once the directories were created lazily.
## What changes are included in this PR?
`DiskManager` now carries the configured paths in a `conf_dirs` field
and creates the working directories on the first `create_tmp_file`.
`build()` no longer touches the filesystem in any mode.
`conf_dirs` is empty for every other mode, so the existing "no
directories yet" branch keeps using an OS temporary directory and that
behaviour is unchanged.
Two visible differences, both called out on the issue:
- `temp_dir_paths()` returns nothing until the first spill. The one
in-tree caller, `RuntimeEnv::describe` (`runtime_env.rs:294`), already
treats an empty list as "not set", so its output is unchanged until a
spill happens.
- An unusable configured directory now fails the first spill rather than
`build()`.
## Are these changes tested?
Yes — `test_directories_mode_creates_dirs_lazily` builds a `DiskManager`
over a configured directory, asserts that directory is still empty and
that `temp_dir_paths()` is empty, then creates a temporary file and
asserts the working directory appears underneath the configured path.
I wrote it first and confirmed it fails on `main` with:
```
expected no directories before the first spill,
found [Ok(DirEntry(".../datafusion-wWF7BS"))]
```
Also run: the full `datafusion-execution` suite (122 + 20 passing), the
`datafusion-physical-plan` spill tests (159 passing), `cargo fmt --all
-- --check`, and `cargo clippy -p datafusion-execution --all-targets --
-D warnings`, all clean.
## Are there any user-facing changes?
No API change. The two behavioural differences above are user-visible:
`temp_dir_paths()` is empty until the first spill, and a bad configured
directory surfaces at first spill instead of at build time.
Co-authored-by: Andrew Lamb <[email protected]>1 parent ac279eb commit 6ab3035
1 file changed
Lines changed: 64 additions & 22 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
117 | 117 | | |
118 | 118 | | |
119 | 119 | | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
120 | 133 | | |
121 | 134 | | |
122 | 135 | | |
123 | 136 | | |
124 | 137 | | |
125 | 138 | | |
126 | | - | |
127 | | - | |
128 | | - | |
129 | | - | |
130 | | - | |
131 | | - | |
132 | | - | |
133 | | - | |
134 | | - | |
135 | | - | |
136 | | - | |
137 | | - | |
138 | | - | |
139 | | - | |
140 | 139 | | |
141 | 140 | | |
| 141 | + | |
142 | 142 | | |
143 | 143 | | |
144 | 144 | | |
| |||
147 | 147 | | |
148 | 148 | | |
149 | 149 | | |
| 150 | + | |
150 | 151 | | |
151 | 152 | | |
152 | 153 | | |
| |||
195 | 196 | | |
196 | 197 | | |
197 | 198 | | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
198 | 203 | | |
199 | 204 | | |
200 | 205 | | |
| |||
354 | 359 | | |
355 | 360 | | |
356 | 361 | | |
357 | | - | |
| 362 | + | |
358 | 363 | | |
359 | | - | |
| 364 | + | |
| 365 | + | |
360 | 366 | | |
361 | | - | |
362 | | - | |
363 | | - | |
364 | | - | |
365 | | - | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
366 | 372 | | |
367 | | - | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
368 | 379 | | |
369 | 380 | | |
370 | 381 | | |
| |||
719 | 730 | | |
720 | 731 | | |
721 | 732 | | |
| 733 | + | |
| 734 | + | |
| 735 | + | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
722 | 764 | | |
723 | 765 | | |
724 | 766 | | |
| |||
0 commit comments