Skip to content

Commit 4fdf68a

Browse files
joseph-isaacsclaude
andcommitted
docs: make the 1 ms benchmark iteration budget an explicit maximum
The guide asked for "less than 1ms" without saying whether that was a target or a limit, and without saying which number to measure. Benchmarks have landed at 23.8 ms and 123.4 ms per iteration, and the review conversation on those has no documented rule to point at. State that 1 ms is the maximum, say what to do when a benchmark does not fit, and point at CodSpeed's per-iteration number under HEAD as the value to check -- not the runtime of the whole benchmark binary, which is what "too long" usually gets confused with. Signed-off-by: Joe Isaacs <[email protected]> Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01FZJAp4wVgwTVhLyErCWFZn
1 parent fd43ebd commit 4fdf68a

1 file changed

Lines changed: 13 additions & 3 deletions

File tree

‎docs/developer-guide/benchmarking.md‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -119,10 +119,20 @@ const VECTOR_SIZE: &[usize] = &[16, 256, 2048, 8192];
119119
fn my_bench<const N: usize>(bencher: Bencher, num_indices: usize) { ... }
120120
```
121121

122-
### Keep per-iteration execution time under ~1 ms
122+
### Keep per-iteration execution time under 1 ms
123123

124-
Each individual iteration of the benchmarked closure should complete in
125-
**less than 1ms**. This is to keep benchmarks snappy, locally and on CI.
124+
**1 ms is the maximum, not a soft target.** Each individual iteration of the benchmarked
125+
closure must complete in less than 1 ms. This is to keep benchmarks snappy, locally and on
126+
CI.
127+
128+
A benchmark that needs longer than that is measuring too much work at once. Shrink the
129+
input size until a single iteration fits, split it into smaller parameterized cases, or
130+
gate it with `#[cfg(not(codspeed))]` if it genuinely cannot be made to fit.
131+
132+
The number to check against the budget is the per-iteration time, not the time the whole
133+
benchmark binary takes. CodSpeed reports exactly that: its performance report on a pull
134+
request lists the per-iteration time under `HEAD` for every benchmark the pull request adds
135+
or changes, so check any new benchmark there before merging.
126136

127137
### Gate CodSpeed-incompatible benchmarks
128138

0 commit comments

Comments
 (0)