Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: ionelmc/pytest-benchmark
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v5.3.0
Choose a base ref
...
head repository: ionelmc/pytest-benchmark
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v5.3.1
Choose a head ref
  • 9 commits
  • 19 files changed
  • 3 contributors

Commits on Oct 30, 2026

  1. Configuration menu
    Copy the full SHA
    9f669a3 View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2026

  1. Configuration menu
    Copy the full SHA
    f9b57d1 View commit details
    Browse the repository at this point in the history
  2. Fix hd15iqr returning the outlier it is meant to exclude

    Stats.hd15iqr is documented as the Tukey-style highest datum within 1.5 IQR
    over Q3, i.e. max{x : x <= Q3 + 1.5*IQR}. bisect_right returns the insertion
    point after every element at or below the fence, so the highest in-fence datum
    is at pos - 1; the code returned sorted_data[pos], which is the first datum
    strictly above the fence.
    
    So whenever a benchmark has a high outlier - a GC pause, a page fault, a
    scheduler preemption, exactly what the statistic exists to filter out - hd15iqr
    reported that outlier. It returned a value the same Stats object simultaneously
    counted in iqr_outliers.
    
    The pos == len(self.data) branch is what hid it: that branch is correct, and it
    fires precisely when there is no outlier, so the statistic looked right on clean
    data and was wrong on dirty data. That branch is now unnecessary, because pos is
    always at least 1: q3 is an interpolation between two elements of sorted_data so
    q3 >= sorted_data[0], and iqr >= 0, therefore sorted_data[0] is always at or
    below the fence.
    
    On ten rounds of [1..9, 100]: fence 15.5, hd15iqr reported 100.0 where the
    correct whisker is 9.0. On timings [10.0 .. 11.0 ms, 100.0 ms]: reported
    100.0 ms against a correct 11.0 ms.
    
    The symmetric ld15iqr uses bisect_left and was already correct, which is what
    makes this a plain inconsistency rather than a matter of convention.
    
    hd15iqr is in Stats.fields, so it reaches --benchmark-json, --benchmark-save
    and the Elasticsearch mapping, and histogram.py builds its box plot as
    ['min', 'ld15iqr', 'q1', 'median', 'q3', 'hd15iqr', 'max'] - the upper whisker
    was drawn at the outlier. The min(row['max'], row['hd15iqr']) cap on line 54 of
    histogram.py only makes sense if hd15iqr was expected to be an in-fence value.
    
    Adds four tests: the high-outlier case, the symmetric low-outlier case, the
    no-outlier case where the whiskers must equal min and max, and a randomised
    check over 500 samples asserting both whiskers stay inside their fences and
    equal the independently computed answer.
    WatchTree-19 authored and ionelmc committed Oct 31, 2026
    Configuration menu
    Copy the full SHA
    5b535e1 View commit details
    Browse the repository at this point in the history
  3. Update changelog.

    ionelmc committed Oct 31, 2026
    Configuration menu
    Copy the full SHA
    154cc0c View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    90204d9 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    defaee2 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    b9d2222 View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2026

  1. Ignore this too.

    ionelmc committed Nov 1, 2026
    Configuration menu
    Copy the full SHA
    1031904 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    97c52d8 View commit details
    Browse the repository at this point in the history
Loading