Skip to content

About

Fault-injection harness for Daft's AI functions: one command reproduces 11 bugs (fake vLLM/OpenAI backends, local Ray, no GPU or API keys)

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

daft-ai-audit

A local fault-injection harness for the AI functions in Daft (prompt, embed_text, the vLLM provider) and its async Python sources. One command reproduces the findings table below against a built Daft checkout (tested on main 06c1fa2 and with the three fixes below).

Nothing leaves the machine: a fake vllm module, a fake OpenAI-compatible server on 127.0.0.1, a local CPU-only Ray cluster and a synthetic video made with PyAV. A sitecustomize guard makes Python's socket connects and DNS lookups refuse non-loopback addresses, in the cells and in Ray workers, and Daft/Ray telemetry is turned off (DO_NOT_TRACK, DAFT_ANALYTICS_ENABLED=0, SCARF_NO_ANALYTICS, RAY_USAGE_STATS_ENABLED=0) before Daft is imported. No GPU, no API keys.

This is an independent project; it is not affiliated with Eventual.

Run

The tables below were produced against Daft main at 06c1fa2c7091d8449be8765e84817d45a63f196d (debug build, macOS arm64, Python 3.11). You need git, uv and rustup; Daft's make targets use both.

git clone https://github.com/TanishGudise/daft-ai-audit && cd daft-ai-audit
git clone https://github.com/Eventual-Inc/Daft && git -C Daft checkout 06c1fa2c7091d8449be8765e84817d45a63f196d
make -C Daft .venv build   # Daft's dev environment (includes ray, openai, aiohttp, av) and native extension
python3 run_audit.py --daft Daft

run_audit.py uses <daft>/.venv/bin/python unless you pass --python. Options: --ray-runs N (runs per Ray cell, default 5), --only 1,7,C1 (a subset; it updates the results file for that commit), --ray-tmp DIR (Ray's temp dir, default ~/.daft-audit-ray; keep the path short, macOS caps socket paths at 103 bytes). Results go to results/<daft commit>.md and .json. With 5 Ray runs per cell, a full run took about 10 minutes against main (several cells wait for hangs to time out) and about 3.5 minutes with the fixes.

Results (Daft main 06c1fa2)

The 16 rows cover 11 bugs: rows 1-2, 3-6 and 12-13 are each one bug, measured per API or symptom. C1-C3 are controls that should pass on any build. Ray cells ran 5 times each.

# Finding Observed on main Status
1 Async row-wise UDFs ignore max_concurrency 2000 of 2,000 calls in flight with max_concurrency=1 present
2 prompt() on a large batch sends nothing for a long time 0 requests reached the server in the first 15 s (3,000 rows) present
3 on_error nulls the whole batch (prompt) 200 of 200 rows null after 1 failed request present
4 on_error nulls the whole batch (async @daft.func) async @daft.func: 20000 of 20,000 rows null after 1 failing row present
5 Retries re-run the whole batch 8 calls for 4 rows with max_retries=1 and one transient failure present
6 A failed row leaves the batch's other calls running 199 requests completed after the query had already failed (0 cancelled) present
7 vLLM: one failing request hangs the query query still running after 20 s, no error present
8 vLLM: periodic stats logging never runs 1 do_log_stats calls while requests ran for 11 s present
9 vLLM: each query leaves its engine alive 3 engines still alive after 3 queries in one process present
10 vLLM on Ray: multi-partition input loses rows or hangs 4 input partitions, no faults: rows out of 40 per run: hang, 12, 12, 10, hang present
11 vLLM on Ray: after an engine actor restart the query hangs or loses rows engine actor killed once mid-request: rows out of 40 per run: hang, hang, hang, hang, hang present
12 Python source keeps producing after an early stop .limit(1): 2 batches produced when the query returned, 294 four seconds later present
13 read_video_frames(...).limit(5) decodes the whole video .limit(5) on a 600-frame 720p video: 600 frames decoded, peak RSS 1860.7 MB present
14 embed_text answers a 429 with one request per input one rate-limited 500-row batch became 503 requests, 500 in flight at once present
15 Ctrl-C leaves HTTP requests open Ctrl-C: KeyboardInterrupt after 3.0 s; 50 of 50 requests still open 3 s later present
16 embed_text KeyError with base_url + dimensions error: KeyError: 'my-local-embedder' present
C1 Control: 429s on prompt are retried by the client 100 rows, 429 for 1 s from the first request (Retry-After 1 s): 100 rows, 0 null, 100 x 429 ok
C2 Control: .limit() is pushed below prompt .limit(5) above prompt: 5 requests sent ok
C3 Control: a client timeout surfaces as an error hung request with timeout=2: APITimeoutError: Request timed out. after 4.9 s ok

With the three fixes applied

Fix Branch on TanishGudise/Daft Commit
A: vLLM requests that fail raise instead of hanging; periodic stats logging runs fix/vllm-executor-surface-errors (Eventual-Inc/Daft#7643) 62140b603
B: async row-wise UDFs cap calls in flight and retry/null per row fix/async-udf-per-row-concurrency 87b14419b
C: vLLM on Ray keeps each partition's results separate (built on A) fix/vllm-ray-per-partition-results 902271b02

results/786828205.* is B with A and C cherry-picked on top. To rebuild it: git -C Daft fetch https://github.com/TanishGudise/Daft fix/async-udf-per-row-concurrency fix/vllm-ray-per-partition-results && git -C Daft checkout 87b14419b && git -C Daft cherry-pick 0237fa5c9 902271b02 && make -C Daft build (C's branch contains A; the cherry-picked commit hash will differ). Rows 9 and 12-16 are not fixed yet.

# Finding Observed with the fixes Status Fixed by
1 Async row-wise UDFs ignore max_concurrency 1 of 2,000 calls in flight with max_concurrency=1 not reproduced B
2 prompt() on a large batch sends nothing for a long time 3000 requests reached the server in the first 15 s (3,000 rows) not reproduced B
3 on_error nulls the whole batch (prompt) 1 of 200 rows null after 1 failed request not reproduced B
4 on_error nulls the whole batch (async @daft.func) async @daft.func: 1 of 20,000 rows null after 1 failing row not reproduced B
5 Retries re-run the whole batch 5 calls for 4 rows with max_retries=1 and one transient failure not reproduced B
6 A failed row leaves the batch's other calls running 0 requests completed after the query had already failed (63 cancelled) not reproduced B
7 vLLM: one failing request hangs the query query ended: ValueError: The decoder prompt (request 39) is longer than the maximum model length (fake) not reproduced A
8 vLLM: periodic stats logging never runs 2 do_log_stats calls while requests ran for 11 s not reproduced A
9 vLLM: each query leaves its engine alive 3 engines still alive after 3 queries in one process present -
10 vLLM on Ray: multi-partition input loses rows or hangs 4 input partitions, no faults: rows out of 40 per run: 40, 40, 40, 40, 40 not reproduced C
11 vLLM on Ray: after an engine actor restart the query hangs or loses rows engine actor killed once mid-request: rows out of 40 per run: error, error, error, error, error not reproduced C
12 Python source keeps producing after an early stop .limit(1): 2 batches produced when the query returned, 271 four seconds later present -
13 read_video_frames(...).limit(5) decodes the whole video .limit(5) on a 600-frame 720p video: 600 frames decoded, peak RSS 1859.9 MB present -
14 embed_text answers a 429 with one request per input one rate-limited 500-row batch became 503 requests, 500 in flight at once present -
15 Ctrl-C leaves HTTP requests open Ctrl-C: KeyboardInterrupt after 3.01 s; 50 of 50 requests still open 3 s later present -
16 embed_text KeyError with base_url + dimensions error: KeyError: 'my-local-embedder' present -
C1 Control: 429s on prompt are retried by the client 100 rows, 429 for 1 s from the first request (Retry-After 1 s): 100 rows, 0 null, 64 x 429 ok control
C2 Control: .limit() is pushed below prompt .limit(5) above prompt: 5 requests sent ok control
C3 Control: a client timeout surfaces as an error hung request with timeout=2: APITimeoutError: Request timed out. after 4.82 s ok control

Cells

File What it does
audit/openai_cell.py prompt / embed_text against audit/fake_openai.py (markers in the input pick a fault: 400, 500, 429 + Retry-After, hang, slow)
audit/stall_cell.py requests reaching the server in the first N seconds of a large prompt
audit/udf_on_error_cell.py, udf_concurrency_cell.py, retry_cell.py async vs sync @daft.func: nulls under on_error, calls in flight, calls per retry
audit/vllm_one_fails_cell.py, vllm_cell.py vLLM provider with the fake engine (one failing request, periodic stats log, engine lifetime)
audit/ray_vllm_cell.py vLLM provider on a local Ray cluster: rows in vs rows out, optional actor kill
audit/source_cell.py, video_cell.py async Python sources after an early .limit() (counting source; read_video_frames)
audit/sigint_cell.py Ctrl-C during prompt with hung requests
audit/embed_keyerror_cell.py embed_text with a custom OpenAI-compatible model and dimensions=

License

MIT, see LICENSE.

About

Fault-injection harness for Daft's AI functions: one command reproduces 11 bugs (fake vLLM/OpenAI backends, local Ray, no GPU or API keys)

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages