Skip to content

Commit 043a9e5

Browse files
Rachel IntRachel Int
Rachel Int
authored and
Rachel Int
committed
Merge origin/main into final-aggr-with-lazily-materialize-poc
2 parents 6481821 + 31a4ca0 commit 043a9e5

2,077 files changed

Lines changed: 260522 additions & 53087 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.ai/skills/add-benchmarks/SKILL.md‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
---
2+
name: add-benchmarks
3+
description: Guidelines for designing or extending DataFusion benchmarks.
4+
---
5+
6+
# Benchmark Design
7+
8+
Follow these guidelines when designing or adding a new benchmark.
9+
10+
## Design principles
11+
12+
### Use the highest-level interface
13+
14+
Use the highest-level interface possible, preferably SQL, while keeping the work
15+
around the operator being measured as cheap as possible. When benchmarking a
16+
function, exercise its evaluation path instead of benchmarking internal utility
17+
functions in isolation.
18+
19+
This makes benchmarks easier to maintain and helps assess how much an optimization
20+
matters to end-to-end runtime. It also helps avoid spending time optimizing code
21+
that accounts for only a small fraction of the total runtime.
22+
23+
Practical criteria for choosing SQL or Rust benchmarks: Try implementing the benchmark in SQL first. If a Rust microbenchmark still seems like a better fit, use Criterion.
24+
25+
### Vary the key workload axes
26+
27+
First identify the key axes to vary. For example, for a join benchmark:
28+
29+
- Input size on each side.
30+
- Join-filter selectivity.
31+
- ...
32+
33+
Then choose benchmark cases that exercise representative variations. Full
34+
combinatorial coverage is unnecessary; focus on typical workloads that reflect
35+
real use cases.
36+
37+
When adding benchmark queries, simply tag each query with the decision made
38+
for every axis. For example, for a join benchmark with input sizes/filter selectivity
39+
to tune:
40+
41+
```sql
42+
-- Q1: Small left input, large right input; 0.1% of pairs match.
43+
SELECT *
44+
FROM generate_series(1, 100) AS l
45+
JOIN generate_series(1, 100000) AS r
46+
ON (l.value + r.value) % 1000 = 0;
47+
48+
-- Q2: Medium inputs on both sides; no filter.
49+
SELECT *
50+
FROM generate_series(1, 1000) AS l
51+
CROSS JOIN generate_series(1, 1000) AS r;
52+
```
53+
54+
## SQL benchmarks
55+
56+
For implementation details, see the
57+
[SQL benchmark README](../../../benchmarks/sql_benchmarks/README.md).
58+
59+
1. **Keep other operators cheap.**
60+
61+
When a SQL benchmark targets a specific operator, keep the work done by
62+
other operators as lightweight as possible. For example, use a data source
63+
such as `generate_series()` instead of a Parquet scan so scan overhead does
64+
not dominate the measurement. See the [`nlj` benchmark](../../../benchmarks/sql_benchmarks/nlj/) for examples.
65+
66+
2. **Integrate with the top-level benchmark script.**
67+
68+
Ensure the benchmark can be prepared and run through `bench.sh`:
69+
70+
```bash
71+
# Run from the benchmarks directory.
72+
73+
# Generate any required dataset.
74+
./bench.sh data new_bench
75+
76+
# Run the benchmark.
77+
./bench.sh run new_bench
78+
```
79+
80+
3. **Keep query runtimes practical.**
81+
82+
Tune the workload so each query takes roughly a few seconds per execution.
83+
This helps reduce the relative impact of timing noise while keeping the
84+
suite practical to run.

0 commit comments

Comments
 (0)