DuckDB delta extension install races across parallel tests #6743

Expected Behavior

The DuckDB offline integration suite should run reliably with 8 pytest-xdist workers on a fresh runner.

Current Behavior

DuckDBDelta tests intermittently fail while several workers auto-install the delta extension into the shared ~/.duckdb/extensions directory. The losing worker reports:

_duckdb.IOException: IO Error: Could not set lock on file ".../delta.duckdb_extension.tmp-...duckdb_extension.info": Conflicting lock is held in .../python3.10 (PID ...)

I found the same failure in 3 unrelated Linux CI runs:

• https://github.com/feast-dev/feast/actions/runs/31946521769
• https://github.com/feast-dev/feast/actions/runs/30604216643
• https://github.com/feast-dev/feast/actions/runs/30477658759

The failures affected 2 different DuckDBDelta tests, so this is not tied to one test's behavior.

Steps to reproduce

1. Start with an empty DuckDB extension cache on Linux.
2. Install the locked duckdb-tests Pixi environment.
3. Run pixi run -e duckdb-tests test, which executes the offline integration suite with pytest -n 8.
4. Several workers may observe that delta is not installed and attempt to install it into the same extension directory.

The race is intermittent. I could not reproduce the lock failure on Apple Silicon macOS, including 80 synchronized cold-cache extension loads, but the 3 GitHub-hosted Linux runs above provide independent positive reproductions.

Specifications

• Version: current master at f771ea4
• Platform: GitHub-hosted Linux, linux_amd64
• Subsystem: DuckDB offline integration tests
• Locked dependencies: Python 3.10.20, DuckDB 1.5.4, Ibis 12.0.0, pytest-xdist 3.8.0

Possible Solution

Install delta once before starting the parallel test workers. For example, add a serial Pixi task dependency that runs:

python -c "import duckdb; duckdb.connect().install_extension('delta')"

I validated this against a separate cold cache. A fresh Ibis connection reported delta as installed but not loaded, then load_extension("delta") changed only the loaded state. This avoids Ibis's per-worker INSTALL path. The full 8-worker Feast suite then passed with 84 tests passed and 5 skipped.

Worker-specific extension directories would also avoid contention, but would download and install the same extension once per worker.