Skip to content

Switch to using numpy to calculate bin edges in histogram where possible and calculate bin edges in consistent way #8011

Open

Description

@mwcraig

numpy now has several methods for estimating bin widths. Some are the same as those in astropy, some are different. When possible, we should use the numpy methods (e.g. for 'scott' and 'freedman' (which numpy calls 'fd')).

There is also a difference between the way astropy calculates the bins for these cases and the way numpy does. In astropy, the bin width calculated from the estimator is used without modification, with the result that the last bin edge is often larger than the range or the last data point (see, e.g. https://github.com/astropy/astropy/blob/master/astropy/stats/histogram.py#L146) . In numpy, the upper bin edge is always either the last data point or the range; that results in a modestly smaller bin size than you get from the estimator (their calculation is at https://github.com/numpy/numpy/blob/v1.15.1/numpy/lib/histograms.py#L375).

I don't know that the numpy way is superior but since we are trying to offer a drop-in replacement I think we should follow their convention.

Sorry for the odd combination of labels -- I think we also ought to use numpy to make available the estimators that numpy has that astropy doesn't (numpy's list is at: https://docs.scipy.org/doc/numpy-1.15.1/reference/generated/numpy.histogram_bin_edges.html#numpy.histogram_bin_edges).

So the bug is our inconsistency regarding bin edges, the enhancement is adding new estimators.

Activity

  1. bsipocz commented on Oct 28, 2018

    @bsipocz
    Member

    These methods got added to numpy 1.11, so we're all in the clear to switch to those given they are available in all numpy versions we support.

  2. neutrinoceros commented on Nov 14, 2024

    @neutrinoceros
    Contributor

    I opened #17391 to address the bug part.
    Relying on numpy more (for freedman and scott) seems doable but I noted that it also gives slightly different bin edges (at least for the scott estimator). Given that tests use assert_allclose for bin edges I'd assume that we may not need to guarantee exact backward compatibility in this area but maybe I'm being overly optimistic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions