You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit af8fc12
Browse filesBrowse the repository at this point in the historyBrowse files
description: Guidelines for designing or extending DataFusion benchmarks.
4
+
---
5
+
6
+
# Benchmark Design
7
+
8
+
Follow these guidelines when designing or adding a new benchmark.
9
+
10
+
## Design principles
11
+
12
+
### Use the highest-level interface
13
+
14
+
Use the highest-level interface possible, preferably SQL, while keeping the work
15
+
around the operator being measured as cheap as possible. When benchmarking a
16
+
function, exercise its evaluation path instead of benchmarking internal utility
17
+
functions in isolation.
18
+
19
+
This makes benchmarks easier to maintain and helps assess how much an optimization
20
+
matters to end-to-end runtime. It also helps avoid spending time optimizing code
21
+
that accounts for only a small fraction of the total runtime.
22
+
23
+
Practical criteria for choosing SQL or Rust benchmarks: Try implementing the benchmark in SQL first. If a Rust microbenchmark still seems like a better fit, use Criterion.
24
+
25
+
### Vary the key workload axes
26
+
27
+
First identify the key axes to vary. For example, for a join benchmark:
28
+
29
+
- Input size on each side.
30
+
- Join-filter selectivity.
31
+
- ...
32
+
33
+
Then choose benchmark cases that exercise representative variations. Full
34
+
combinatorial coverage is unnecessary; focus on typical workloads that reflect
35
+
real use cases.
36
+
37
+
When adding benchmark queries, simply tag each query with the decision made
38
+
for every axis. For example, for a join benchmark with input sizes/filter selectivity
39
+
to tune:
40
+
41
+
```sql
42
+
-- Q1: Small left input, large right input; 0.1% of pairs match.
0 commit comments