Skip to content

test_biweight_32bit_runtime_warnings fails on ppc64le #11027

Closed

Description

@astrofrog

I noticed the following test failure on ppc64le:

_____________________ test_biweight_32bit_runtime_warnings _____________________

    def test_biweight_32bit_runtime_warnings():
        """Regression test for #6905."""
        with NumpyRNGContext(12345):
            data = np.random.random(100).astype(np.float32)
            data[50] = 30000.
>           biweight_scale(data)

../../.tox/py37-test/lib/python3.7/site-packages/astropy/stats/tests/test_biweight.py:554: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../.tox/py37-test/lib/python3.7/site-packages/astropy/stats/biweight.py:270: in biweight_scale
    ignore_nan=ignore_nan))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

data = array([ 1.12123764e+00,  8.73053360e+00, -2.18580091e+10, -1.53760389e+37,
       -3.65725505e-11, -4.22522827e+02, -1...5961e-04, -2.99748559e-11, -1.03130915e-38,
       -1.94296332e-37,  1.04100448e+18, -2.68494430e-32, -1.35409688e+04])
c = 9.0, M = 1.4770741391265427e-33, axis = None, modify_sample_size = False

    def biweight_midvariance(data, c=9.0, M=None, axis=None,
                             modify_sample_size=False, *, ignore_nan=False):
        r"""
        Compute the biweight midvariance.
    
        The biweight midvariance is a robust statistic for determining the
        variance of a distribution.  Its square root is a robust estimator
        of scale (i.e. standard deviation).  It is given by:
    
        .. math::
    
            \zeta_{bivar} = n \ \frac{\sum_{|u_i| < 1} \
                (x_i - M)^2 (1 - u_i^2)^4} {(\sum_{|u_i| < 1} \
                (1 - u_i^2) (1 - 5u_i^2))^2}
    
        where :math:`x` is the input data, :math:`M` is the sample median
        (or the input location) and :math:`u_i` is given by:
    
        .. math::
    
            u_{i} = \frac{(x_i - M)}{c * MAD}
    
        where :math:`c` is the tuning constant and :math:`MAD` is the
        `median absolute deviation
        <https://en.wikipedia.org/wiki/Median_absolute_deviation>`_.  The
        biweight midvariance tuning constant ``c`` is typically 9.0 (the
        default).
    
        If :math:`MAD` is zero, then zero will be returned.
    
        For the standard definition of `biweight midvariance
        <https://en.wikipedia.org/wiki/Robust_measures_of_scale#The_biweight_midvariance>`_,
        :math:`n` is the total number of points in the array (or along the
        input ``axis``, if specified).  That definition is used if
        ``modify_sample_size`` is `False`, which is the default.
    
        However, if ``modify_sample_size = True``, then :math:`n` is the
        number of points for which :math:`|u_i| < 1` (i.e. the total number
        of non-rejected values), i.e.
    
        .. math::
    
            n = \sum_{|u_i| < 1} \ 1
    
        which results in a value closer to the true variance for small
        sample sizes or for a large number of rejected values.
    
        Parameters
        ----------
        data : array_like
            Input array or object that can be converted to an array.
            ``data`` can be a `~numpy.ma.MaskedArray`.
        c : float, optional
            Tuning constant for the biweight estimator (default = 9.0).
        M : float or array_like, optional
            The location estimate.  If ``M`` is a scalar value, then its
            value will be used for the entire array (or along each ``axis``,
            if specified).  If ``M`` is an array, then its must be an array
            containing the location estimate along each ``axis`` of the
            input array.  If `None` (default), then the median of the input
            array will be used (or along each ``axis``, if specified).
        axis : `None`, int, or tuple of ints, optional
            The axis or axes along which the biweight midvariances are
            computed.  If `None` (default), then the biweight midvariance of
            the flattened input array will be computed.
        modify_sample_size : bool, optional
            If `False` (default), then the sample size used is the total
            number of elements in the array (or along the input ``axis``, if
            specified), which follows the standard definition of biweight
            midvariance.  If `True`, then the sample size is reduced to
            correct for any rejected values (i.e. the sample size used
            includes only the non-rejected values), which results in a value
            closer to the true variance for small sample sizes or for a
            large number of rejected values.
        ignore_nan : bool, optional
            Whether to ignore NaN values in the input ``data``.
    
        Returns
        -------
        biweight_midvariance : float or `~numpy.ndarray`
            The biweight midvariance of the input data.  If ``axis`` is
            `None` then a scalar will be returned, otherwise a
            `~numpy.ndarray` will be returned.
    
        See Also
        --------
        biweight_midcovariance, biweight_midcorrelation, astropy.stats.mad_std, astropy.stats.median_absolute_deviation
    
        References
        ----------
        .. [1] https://en.wikipedia.org/wiki/Robust_measures_of_scale#The_biweight_midvariance
    
        .. [2] Beers, Flynn, and Gebhardt (1990; AJ 100, 32) (https://ui.adsabs.harvard.edu/abs/1990AJ....100...32B)
    
        Examples
        --------
        Generate random variates from a Gaussian distribution and return the
        biweight midvariance of the distribution:
    
        >>> import numpy as np
        >>> from astropy.stats import biweight_midvariance
        >>> rand = np.random.RandomState(12345)
        >>> bivar = biweight_midvariance(rand.randn(1000))
        >>> print(bivar)    # doctest: +FLOAT_CMP
        0.97362869104
        """
    
        median_func, sum_func = _stat_functions(data, ignore_nan=ignore_nan)
    
        if isinstance(data, np.ma.MaskedArray) and ignore_nan:
            data = np.ma.masked_where(np.isnan(data), data, copy=True)
    
        data = np.asanyarray(data).astype(np.float64)
    
        if M is None:
            M = median_func(data, axis=axis)
        if axis is not None:
            M = _expand_dims(M, axis=axis)  # NUMPY_LT_1_18
    
        # set up the differences
        d = data - M
    
        # set up the weighting
        mad = median_absolute_deviation(data, axis=axis, ignore_nan=ignore_nan)
    
        if axis is None:
            if mad == 0.:  # data is constant or mostly constant
                return 0.0
            if np.isnan(mad):  # data contains NaNs and ignore_nan=False
                return np.nan
        else:
            mad = _expand_dims(mad, axis=axis)  # NUMPY_LT_1_18
    
        with np.errstate(divide='ignore', invalid='ignore'):
            u = d / (c * mad)
    
        # now remove the outlier points
        # ignore RuntimeWarnings for comparisons with NaN data values
        with np.errstate(invalid='ignore'):
            mask = np.abs(u) < 1
        if isinstance(mask, np.ma.MaskedArray):
            mask = mask.filled(fill_value=False)  # exclude masked data values
    
        u = u ** 2
    
        if modify_sample_size:
            n = sum_func(mask, axis=axis)
        else:
            # set good values to 1, bad values to 0
            include_mask = np.ones(data.shape)
            if isinstance(data, np.ma.MaskedArray):
                include_mask[data.mask] = 0
            if ignore_nan:
                include_mask[np.isnan(data)] = 0
            n = np.sum(include_mask, axis=axis)
    
>       f1 = d * d * (1. - u)**4
E       RuntimeWarning: overflow encountered in multiply

../../.tox/py37-test/lib/python3.7/site-packages/astropy/stats/biweight.py:429: RuntimeWarning

Activity

  1. astrofrog commented on Nov 11, 2020

    @astrofrog
    MemberAuthor

    The data doesn't look right:

    
    data = array([ 1.12123764e+00,  8.73053360e+00, -2.18580091e+10, -1.53760389e+37,
           -3.65725505e-11, -4.22522827e+02, -1...5961e-04, -2.99748559e-11, -1.03130915e-38,
           -1.94296332e-37,  1.04100448e+18, -2.68494430e-32, -1.35409688e+04])
    

    I wonder whether this points to a Numpy bug.

  2. pllim commented on Nov 11, 2020

    @pllim
    Member
  3. larrybradley commented on Nov 11, 2020

    @larrybradley
    Member

    The input data definitely doesn't look like what you should get from np.random.random(100).astype(np.float32).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions