DiskManager in Directories mode creates its temp directories when it is built, even if nothing spills #26446

Is your feature request related to a problem or challenge?

DiskManagerBuilder::build in DiskManagerMode::Directories calls create_local_dirs, which creates a datafusion-* directory in every configured directory, and each one is removed recursively when the DiskManager is dropped. DiskManagerMode::OsTmpDirectory creates its directory on the first create_tmp_file instead (#1695).

An engine that builds a RuntimeEnv per query or per task pays for this on every one, whether or not it spills. DataFusion Comet builds one per native plan in a Spark task, over the executor's local disks, and measured the directory syscalls at 4.7% of task samples for a small-task workload on macOS (apache/datafusion-comet#7060).

On Linux (ext4 on NVMe, kernel 6.8), each configured directory costs 11 syscalls per DiskManager (mkdir, unlinkat, two statx, two getdents64, two fcntl, openat, close and newfstatat), about 35 µs. The operations also serialize on the filesystem: with one directory, 8 or 16 threads building DiskManagers at once top out near 30k per second.

Describe the solution you'd like

Create the Directories mode directories on the first create_tmp_file, as OsTmpDirectory does. Two visible differences: temp_dir_paths() returns nothing until the first spill, and an unusable directory fails the first spill rather than build().

Describe alternatives you've considered

Comet now does this itself with a TempFileFactory (DiskManagerMode::Custom) that builds a Directories mode DiskManager on its first spill file (apache/datafusion-comet#7066). Every engine with a per-query RuntimeEnv would otherwise need the same wrapper.

Additional context

In Comet, with 8 local directories, a Spark query of 501 short tasks ran 12% faster in wall time, with 9% less CPU, once the directories were created lazily. With one directory, or with fewer and longer tasks, the difference was in the noise.