A local fault-injection harness for the AI functions in Daft (prompt, embed_text, the vLLM provider) and its async Python sources. One command reproduces the findings table below against a built Daft checkout (tested on main 06c1fa2 and with the three fixes below).
Nothing leaves the machine: a fake vllm module, a fake OpenAI-compatible server on 127.0.0.1, a local CPU-only Ray cluster and a synthetic video made with PyAV. A sitecustomize guard makes Python's socket connects and DNS lookups refuse non-loopback addresses, in the cells and in Ray workers, and Daft/Ray telemetry is turned off (DO_NOT_TRACK, DAFT_ANALYTICS_ENABLED=0, SCARF_NO_ANALYTICS, RAY_USAGE_STATS_ENABLED=0) before Daft is imported. No GPU, no API keys.
This is an independent project; it is not affiliated with Eventual.
The tables below were produced against Daft main at 06c1fa2c7091d8449be8765e84817d45a63f196d (debug build, macOS arm64, Python 3.11). You need git, uv and rustup; Daft's make targets use both.
git clone https://github.com/TanishGudise/daft-ai-audit && cd daft-ai-audit
git clone https://github.com/Eventual-Inc/Daft && git -C Daft checkout 06c1fa2c7091d8449be8765e84817d45a63f196d
make -C Daft .venv build # Daft's dev environment (includes ray, openai, aiohttp, av) and native extension
python3 run_audit.py --daft Daftrun_audit.py uses <daft>/.venv/bin/python unless you pass --python. Options: --ray-runs N (runs per Ray cell, default 5), --only 1,7,C1 (a subset; it updates the results file for that commit), --ray-tmp DIR (Ray's temp dir, default ~/.daft-audit-ray; keep the path short, macOS caps socket paths at 103 bytes). Results go to results/<daft commit>.md and .json. With 5 Ray runs per cell, a full run took about 10 minutes against main (several cells wait for hangs to time out) and about 3.5 minutes with the fixes.
The 16 rows cover 11 bugs: rows 1-2, 3-6 and 12-13 are each one bug, measured per API or symptom. C1-C3 are controls that should pass on any build. Ray cells ran 5 times each.
| # | Finding | Observed on main | Status |
|---|---|---|---|
| 1 | Async row-wise UDFs ignore max_concurrency | 2000 of 2,000 calls in flight with max_concurrency=1 | present |
| 2 | prompt() on a large batch sends nothing for a long time | 0 requests reached the server in the first 15 s (3,000 rows) | present |
| 3 | on_error nulls the whole batch (prompt) | 200 of 200 rows null after 1 failed request | present |
| 4 | on_error nulls the whole batch (async @daft.func) | async @daft.func: 20000 of 20,000 rows null after 1 failing row | present |
| 5 | Retries re-run the whole batch | 8 calls for 4 rows with max_retries=1 and one transient failure | present |
| 6 | A failed row leaves the batch's other calls running | 199 requests completed after the query had already failed (0 cancelled) | present |
| 7 | vLLM: one failing request hangs the query | query still running after 20 s, no error | present |
| 8 | vLLM: periodic stats logging never runs | 1 do_log_stats calls while requests ran for 11 s | present |
| 9 | vLLM: each query leaves its engine alive | 3 engines still alive after 3 queries in one process | present |
| 10 | vLLM on Ray: multi-partition input loses rows or hangs | 4 input partitions, no faults: rows out of 40 per run: hang, 12, 12, 10, hang | present |
| 11 | vLLM on Ray: after an engine actor restart the query hangs or loses rows | engine actor killed once mid-request: rows out of 40 per run: hang, hang, hang, hang, hang | present |
| 12 | Python source keeps producing after an early stop | .limit(1): 2 batches produced when the query returned, 294 four seconds later | present |
| 13 | read_video_frames(...).limit(5) decodes the whole video | .limit(5) on a 600-frame 720p video: 600 frames decoded, peak RSS 1860.7 MB | present |
| 14 | embed_text answers a 429 with one request per input | one rate-limited 500-row batch became 503 requests, 500 in flight at once | present |
| 15 | Ctrl-C leaves HTTP requests open | Ctrl-C: KeyboardInterrupt after 3.0 s; 50 of 50 requests still open 3 s later | present |
| 16 | embed_text KeyError with base_url + dimensions | error: KeyError: 'my-local-embedder' | present |
| C1 | Control: 429s on prompt are retried by the client | 100 rows, 429 for 1 s from the first request (Retry-After 1 s): 100 rows, 0 null, 100 x 429 | ok |
| C2 | Control: .limit() is pushed below prompt | .limit(5) above prompt: 5 requests sent | ok |
| C3 | Control: a client timeout surfaces as an error | hung request with timeout=2: APITimeoutError: Request timed out. after 4.9 s | ok |
| Fix | Branch on TanishGudise/Daft | Commit |
|---|---|---|
| A: vLLM requests that fail raise instead of hanging; periodic stats logging runs | fix/vllm-executor-surface-errors (Eventual-Inc/Daft#7643) |
62140b603 |
| B: async row-wise UDFs cap calls in flight and retry/null per row | fix/async-udf-per-row-concurrency |
87b14419b |
| C: vLLM on Ray keeps each partition's results separate (built on A) | fix/vllm-ray-per-partition-results |
902271b02 |
results/786828205.* is B with A and C cherry-picked on top. To rebuild it: git -C Daft fetch https://github.com/TanishGudise/Daft fix/async-udf-per-row-concurrency fix/vllm-ray-per-partition-results && git -C Daft checkout 87b14419b && git -C Daft cherry-pick 0237fa5c9 902271b02 && make -C Daft build (C's branch contains A; the cherry-picked commit hash will differ). Rows 9 and 12-16 are not fixed yet.
| # | Finding | Observed with the fixes | Status | Fixed by |
|---|---|---|---|---|
| 1 | Async row-wise UDFs ignore max_concurrency | 1 of 2,000 calls in flight with max_concurrency=1 | not reproduced | B |
| 2 | prompt() on a large batch sends nothing for a long time | 3000 requests reached the server in the first 15 s (3,000 rows) | not reproduced | B |
| 3 | on_error nulls the whole batch (prompt) | 1 of 200 rows null after 1 failed request | not reproduced | B |
| 4 | on_error nulls the whole batch (async @daft.func) | async @daft.func: 1 of 20,000 rows null after 1 failing row | not reproduced | B |
| 5 | Retries re-run the whole batch | 5 calls for 4 rows with max_retries=1 and one transient failure | not reproduced | B |
| 6 | A failed row leaves the batch's other calls running | 0 requests completed after the query had already failed (63 cancelled) | not reproduced | B |
| 7 | vLLM: one failing request hangs the query | query ended: ValueError: The decoder prompt (request 39) is longer than the maximum model length (fake) | not reproduced | A |
| 8 | vLLM: periodic stats logging never runs | 2 do_log_stats calls while requests ran for 11 s | not reproduced | A |
| 9 | vLLM: each query leaves its engine alive | 3 engines still alive after 3 queries in one process | present | - |
| 10 | vLLM on Ray: multi-partition input loses rows or hangs | 4 input partitions, no faults: rows out of 40 per run: 40, 40, 40, 40, 40 | not reproduced | C |
| 11 | vLLM on Ray: after an engine actor restart the query hangs or loses rows | engine actor killed once mid-request: rows out of 40 per run: error, error, error, error, error | not reproduced | C |
| 12 | Python source keeps producing after an early stop | .limit(1): 2 batches produced when the query returned, 271 four seconds later | present | - |
| 13 | read_video_frames(...).limit(5) decodes the whole video | .limit(5) on a 600-frame 720p video: 600 frames decoded, peak RSS 1859.9 MB | present | - |
| 14 | embed_text answers a 429 with one request per input | one rate-limited 500-row batch became 503 requests, 500 in flight at once | present | - |
| 15 | Ctrl-C leaves HTTP requests open | Ctrl-C: KeyboardInterrupt after 3.01 s; 50 of 50 requests still open 3 s later | present | - |
| 16 | embed_text KeyError with base_url + dimensions | error: KeyError: 'my-local-embedder' | present | - |
| C1 | Control: 429s on prompt are retried by the client | 100 rows, 429 for 1 s from the first request (Retry-After 1 s): 100 rows, 0 null, 64 x 429 | ok | control |
| C2 | Control: .limit() is pushed below prompt | .limit(5) above prompt: 5 requests sent | ok | control |
| C3 | Control: a client timeout surfaces as an error | hung request with timeout=2: APITimeoutError: Request timed out. after 4.82 s | ok | control |
| File | What it does |
|---|---|
audit/openai_cell.py |
prompt / embed_text against audit/fake_openai.py (markers in the input pick a fault: 400, 500, 429 + Retry-After, hang, slow) |
audit/stall_cell.py |
requests reaching the server in the first N seconds of a large prompt |
audit/udf_on_error_cell.py, udf_concurrency_cell.py, retry_cell.py |
async vs sync @daft.func: nulls under on_error, calls in flight, calls per retry |
audit/vllm_one_fails_cell.py, vllm_cell.py |
vLLM provider with the fake engine (one failing request, periodic stats log, engine lifetime) |
audit/ray_vllm_cell.py |
vLLM provider on a local Ray cluster: rows in vs rows out, optional actor kill |
audit/source_cell.py, video_cell.py |
async Python sources after an early .limit() (counting source; read_video_frames) |
audit/sigint_cell.py |
Ctrl-C during prompt with hung requests |
audit/embed_keyerror_cell.py |
embed_text with a custom OpenAI-compatible model and dimensions= |
MIT, see LICENSE.