Skip to content

Commit 70bc3ab

Browse files
jerome-benoitjerome-benoit
authored
feat: latency and throughput advanced statistics (#119)
Signed-off-by: Jérôme Benoit <[email protected]>
1 parent 1297a9b commit 70bc3ab

File tree

9 files changed

+1357
-176
lines changed

9 files changed

+1357
-176
lines changed

‎.size-limit.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
[
22
{
33
"path": "dist/index.js",
4-
"limit": "10 kB"
4+
"limit": "12 kB"
55
},
66
{
77
"path": "dist/index.cjs",
8-
"limit": "10 kB"
8+
"limit": "12 kB"
99
}
1010
]

‎README.md‎

Lines changed: 63 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,7 @@ _I'm transitioning to a full-time open source career. Your support would be grea
66
[![CI](https://github.com/tinylibs/tinybench/actions/workflows/test.yml/badge.svg?branch=main)](https://github.com/tinylibs/tinybench/actions/workflows/test.yml)
77
[![NPM version](https://img.shields.io/npm/v/tinybench.svg?style=flat)](https://www.npmjs.com/package/tinybench)
88

9-
Benchmark your code easily with Tinybench, a simple, tiny and light-weight `7KB` (`2KB` minified and gzipped)
10-
benchmarking library!
9+
Benchmark your code easily with Tinybench, a simple, tiny and light-weight `10KB` (`2KB` minified and gzipped) benchmarking library!
1110
You can run your benchmarks in multiple JavaScript runtimes, Tinybench is
1211
completely based on the Web APIs with proper timing using `process.hrtime` or
1312
`performance.now`.
@@ -23,7 +22,7 @@ _In case you need more tiny libraries like tinypool or tinyspy, please consider
2322

2423
## Installing
2524

26-
```bash
25+
```shell
2726
$ npm install -D tinybench
2827
```
2928

@@ -52,12 +51,12 @@ await bench.run();
5251
console.table(bench.table());
5352

5453
// Output:
55-
// ┌─────────┬───────────────┬──────────┬──────────────────────┬──────────┬──────────────────────────────────┬─────────┐
56-
// │ (index) │ Task name │ ops/sec │ Average time/op (ns) │ Margin │ Median time/op (ns) │ Samples │
57-
// ├─────────┼───────────────┼──────────┼──────────────────────┼──────────┼──────────────────────────────────┼─────────┤
58-
// │ 0 │ 'faster task' │ '38,832' │ 25751.297631307978 │ '±3.48%' │ '22016.49999997812±5.5000000145' │ 3884 │
59-
// │ 1 │ 'slower task' │ '669' │ 1493338.567164177 │ '±5.98%' │ '1445076.0000000286' │ 67 │
60-
// └─────────┴───────────────┴──────────┴──────────────────────┴──────────┴──────────────────────────────────┴─────────┘
54+
// ┌─────────┬───────────────┬────────────────────────────┬───────────────────────────┬──────────────────────┬─────────────────────┬─────────┐
55+
// │ (index) │ Task name │ Throughput average (ops/s) │ Throughput median (ops/s) │ Latency average (ns) │ Latency median (ns) │ Samples │
56+
// ├─────────┼───────────────┼────────────────────────────┼───────────────────────────┼──────────────────────┼─────────────────────┼─────────┤
57+
// │ 0 │ 'faster task' │ '102906 ± 0.89%' │ '82217 ± 14' │ '11909.14 ± 3.95%' │ '12163.00 ± 2.00' │ 8398 │
58+
// │ 1 │ 'slower task' │ '988 ± 26.26%' │ '710' │ '1379560.47 ± 6.72%' │ '1408552.00' │ 73 │
59+
// └─────────┴───────────────┴────────────────────────────┴───────────────────────────┴──────────────────────┴─────────────────────┴─────────┘
6160
```
6261

6362
The `add` method accepts a task name and a task function, so it can benchmark
@@ -164,6 +163,8 @@ function has been executed.
164163
- `setResult(result: Partial<TaskResult>)`: change the result object values
165164
- `reset()`: reset the task to make the `Task.runs` a zero-value and remove the `Task.result` object
166165

166+
FnOptions:
167+
167168
```ts
168169
export interface FnOptions {
169170
/**
@@ -188,118 +189,136 @@ export interface FnOptions {
188189
}
189190
```
190191

191-
## `TaskResult`
192+
### `TaskResult`
192193

193-
the benchmark task result object.
194+
The benchmark task result object:
194195

195196
```ts
196197
export interface TaskResult {
197198
/*
198-
* the last error that was thrown while running the task
199+
* the last task error that was thrown
199200
*/
200-
error?: unknown;
201+
error?: Error;
201202

202203
/**
203-
* The amount of time in milliseconds to run the benchmark task (cycle).
204+
* the time to run the task benchmark cycle (ms)
204205
*/
205206
totalTime: number;
206207

207208
/**
208-
* the minimum value in the samples
209+
* how long each operation takes (ms)
209210
*/
210-
min: number;
211+
period: number;
212+
211213
/**
212-
* the maximum value in the samples
214+
* the task latency statistics
213215
*/
214-
max: number;
216+
latency: Statistics;
217+
218+
/**
219+
* the task throughput statistics
220+
*/
221+
throughput: Statistics;
215222

216223
/**
217224
* the number of operations per second
225+
* @deprecated use `.throughput.mean` instead
218226
*/
219227
hz: number;
220228

221229
/**
222-
* how long each operation takes (ms)
230+
* latency samples (ms)
231+
* @deprecated use `.latency.samples` instead
223232
*/
224-
period: number;
233+
samples: number[];
225234

226235
/**
227-
* task samples of each task iteration time (ms)
236+
* the minimum latency samples value
237+
* @deprecated use `.latency.min` instead
228238
*/
229-
samples: number[];
239+
min: number;
240+
/**
241+
* the maximum latency samples value
242+
* @deprecated use `.latency.max` instead
243+
*/
244+
max: number;
230245

231246
/**
232-
* samples mean/average (estimate of the population mean)
247+
* the latency samples mean/average (estimate of the population mean/average)
248+
* @deprecated use `.latency.mean` instead
233249
*/
234250
mean: number;
235251

236252
/**
237-
* samples variance (estimate of the population variance)
253+
* the latency samples variance (estimate of the population variance)
254+
* @deprecated use `.latency.variance` instead
238255
*/
239256
variance: number;
240257

241258
/**
242-
* samples standard deviation (estimate of the population standard deviation)
259+
* the latency samples standard deviation (estimate of the population standard deviation)
260+
* @deprecated use `.latency.sd` instead
243261
*/
244262
sd: number;
245263

246264
/**
247-
* standard error of the mean (a.k.a. the standard deviation of the sampling distribution of the sample mean)
265+
* the latency standard error of the mean (a.k.a. the standard deviation of the sampling distribution of the sample mean/average)
266+
* @deprecated use `.latency.sem` instead
248267
*/
249268
sem: number;
250269

251270
/**
252-
* degrees of freedom
271+
* the latency samples degrees of freedom
272+
* @deprecated use `.latency.df` instead
253273
*/
254274
df: number;
255275

256276
/**
257-
* critical value of the samples
277+
* the latency samples critical value
278+
* @deprecated use `.latency.critical` instead
258279
*/
259280
critical: number;
260281

261282
/**
262-
* margin of error
283+
* the latency samples margin of error
284+
* @deprecated use `.latency.moe` instead
263285
*/
264286
moe: number;
265287

266288
/**
267-
* relative margin of error
289+
* the latency samples relative margin of error
290+
* @deprecated use `.latency.rme` instead
268291
*/
269292
rme: number;
270293

271294
/**
272-
* median absolute deviation
273-
*/
274-
mad: number;
275-
276-
/**
277-
* p50/median percentile
278-
*/
279-
p50: number;
280-
281-
/**
282-
* p75 percentile
295+
* the latency samples p75 percentile
296+
* @deprecated use `.latency.p75` instead
283297
*/
284298
p75: number;
285299

286300
/**
287-
* p99 percentile
301+
* the latency samples p99 percentile
302+
* @deprecated use `.latency.p99` instead
288303
*/
289304
p99: number;
290305

291306
/**
292-
* p995 percentile
307+
* the latency samples p995 percentile
308+
* @deprecated use `.latency.p995` instead
293309
*/
294310
p995: number;
295311

296312
/**
297-
* p999 percentile
313+
* the latency samples p999 percentile
314+
* @deprecated use `.latency.p999` instead
298315
*/
299316
p999: number;
300317
}
301318
```
302319

320+
[Statistics](https://github.com/tinylibs/tinybench/blob/main/src/types.ts#L30) type definition.
321+
303322
### `Events`
304323

305324
Both the `Task` and `Bench` objects extend the `EventTarget` object, so you can attach listeners to different types of events

‎examples/src/simple.ts‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,9 @@ await bench.run();
1717
console.table(bench.table());
1818

1919
// Output:
20-
// ┌─────────┬───────────────┬──────────┬──────────────────────┬──────────┬──────────────────────────────────┬─────────┐
21-
// │ (index) │ Task name │ ops/sec │ Average time/op (ns) │ Margin │ Median time/op (ns) │ Samples │
22-
// ├─────────┼───────────────┼──────────┼──────────────────────┼──────────┼──────────────────────────────────┼─────────┤
23-
// │ 0 │ 'faster task' │ '38,832' │ 25751.297631307978 │ '±3.48%' │ '22016.49999997812±5.5000000145' │ 3884 │
24-
// │ 1 │ 'slower task' │ '669' │ 1493338.567164177 │ '±5.98%' │ '1445076.0000000286' │ 67 │
25-
// └─────────┴───────────────┴──────────┴──────────────────────┴──────────┴──────────────────────────────────┴─────────┘
20+
// ┌─────────┬───────────────┬────────────────────────────┬───────────────────────────┬──────────────────────┬─────────────────────┬─────────┐
21+
// │ (index) │ Task name │ Throughput average (ops/s) │ Throughput median (ops/s) │ Latency average (ns) │ Latency median (ns) │ Samples │
22+
// ├─────────┼───────────────┼────────────────────────────┼───────────────────────────┼──────────────────────┼─────────────────────┼─────────┤
23+
// │ 0 │ 'faster task' │ '102906 ± 0.89%' │ '82217 ± 14' │ '11909.14 ± 3.95%' │ '12163.00 ± 2.00' │ 8398 │
24+
// │ 1 │ 'slower task' │ '988 ± 26.26%' │ '710' │ '1379560.47 ± 6.72%' │ '1408552.00' │ 73 │
25+
// └─────────┴───────────────┴────────────────────────────┴───────────────────────────┴──────────────────────┴─────────────────────┴─────────┘

‎src/bench.ts‎

Lines changed: 21 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,10 @@
11
import pLimit from 'p-limit';
2+
import {
3+
defaultMinimumIterations,
4+
defaultMinimumTime,
5+
defaultMinimumWarmupIterations,
6+
defaultMinimumWarmupTime,
7+
} from './constants';
28
import { createBenchEvent } from './event';
39
import Task from './task';
410
import type {
@@ -43,13 +49,13 @@ export default class Bench extends EventTarget {
4349

4450
throws: boolean;
4551

46-
warmupTime = 100;
52+
warmupTime = defaultMinimumWarmupTime;
4753

48-
warmupIterations = 5;
54+
warmupIterations = defaultMinimumWarmupIterations;
4955

50-
time = 500;
56+
time = defaultMinimumTime;
5157

52-
iterations = 10;
58+
iterations = defaultMinimumIterations;
5359

5460
now = now;
5561

@@ -188,19 +194,21 @@ export default class Bench extends EventTarget {
188194
return (
189195
convert?.(task) || {
190196
'Task name': task.name,
191-
'ops/sec': task.result.error
197+
'Throughput average (ops/s)': task.result.error
192198
? 'NaN'
193-
: Number.parseInt(task.result.hz.toString(), 10).toLocaleString(),
194-
'Average time/op (ns)': task.result.error
199+
: `${task.result.throughput.mean.toFixed(0)} \xb1 ${task.result.throughput.rme.toFixed(2)}%`,
200+
'Throughput median (ops/s)': task.result.error
195201
? 'NaN'
196-
: task.result.mean * 1e6,
197-
Margin: task.result.error
202+
: `${task.result.throughput.p50?.toFixed(0)}${Number.parseInt(task.result.throughput.mad!.toFixed(0), 10) > 0 ? ` \xb1 ${task.result.throughput.mad!.toFixed(0)}` : ''}`,
203+
'Latency average (ns)': task.result.error
198204
? 'NaN'
199-
: `\xb1${task.result.rme.toFixed(2)}%`,
200-
'Median time/op (ns)': task.result.error
205+
: `${(task.result.latency.mean * 1e6).toFixed(2)} \xb1 ${task.result.latency.rme.toFixed(2)}%`,
206+
'Latency median (ns)': task.result.error
201207
? 'NaN'
202-
: `${task.result.p50 * 1e6}${task.result.mad * 1e6 > 0 ? `\xb1${(task.result.mad * 1e6).toFixed(10)}` : ''}`,
203-
Samples: task.result.error ? 'NaN' : task.result.samples.length,
208+
: `${(task.result.latency.p50! * 1e6).toFixed(2)}${Number.parseFloat((task.result.latency.mad! * 1e6).toFixed(2)) > 0 ? ` \xb1 ${(task.result.latency.mad! * 1e6).toFixed(2)}` : ''}`,
209+
Samples: task.result.error
210+
? 'NaN'
211+
: task.result.latency.samples.length,
204212
}
205213
);
206214
}

0 commit comments

Comments
 (0)