analytic_functions: Skip Tests on windows by MSeifert04 · Pull Request #4275 · astropy/astropy

MSeifert04 · 2015-10-27T01:40:26Z

I'm not really sure if that change makes sense. But the tests that are marked to xfail on AppVeyor are tried on windows - I'm not sure that's meant to be but I guess not.

At least if I start the tests with python setup.py test and import astropy; astropy.test() the tests in analytic_functions fail (F instead of x) on my windows computer.

closes #4171

pllim · 2015-10-27T14:26:13Z

Hmm, @embray said it didn't fail on his Windows machine.

pllim · 2015-10-27T14:43:40Z

Anyways, I looked at the tests on #4171 and posted my comment there. I am okay with skipping them on Windows.

astrofrog · 2015-11-09T13:27:29Z

I don't think we should skip these - it's not like the precision is off, some values are actually NaN instead of zero which doesn't seem ideal.

astrofrog · 2015-11-09T13:32:35Z

@MSeifert04 @lingxz - just to see if we can diagnose the issue, can you let me know the output of:

import numpy as np
print(np.expm1(14387.))
print(1 / np.expm1(14387.))
print(np.__version__)

On your Windows machines where these tests fail?

astrofrog · 2015-11-09T13:33:21Z

(by the way, I realize now that we are skipping these tests on AppVeyor, but before we extend to all Windows, let's see if we can figure out why these don't work)

MSeifert04 · 2015-12-05T16:40:40Z

Sorry I completely forgot about this.
@astrofrog Yes I just wanted to skip this test because a windows environment propably is at the root of the problem and I thought AppVeyor is just a windows environment.

I tested it with all my avaiably environments and the problem is windows together with python < 3.5. With python 3.5 the output is like expected and the tests do not fail.

Here the longish outputs:

Python 3.5

In [2]: print(np.expm1(14387.))
C:\Programming\Anaconda\Scripts\ipython-script.py:1: RuntimeWarning: overflow encountered in expm1
  if __name__ == '__main__':
inf

In [3]: print(1 / np.expm1(14387.))
C:\Programming\Anaconda\Scripts\ipython-script.py:1: RuntimeWarning: overflow encountered in expm1
  if __name__ == '__main__':
0.0

In [4]: print(np.__version__)
1.10.1

platform win32 -- Python 3.5.0 -- py-1.4.30 -- pytest-2.7.3
rootdir: C:\Programming\Anaconda\lib\site-packages\astropy\analytic_functions, inifile:

Running tests with Astropy version 1.0.6.
Running tests in C:\Programming\Anaconda\lib\site-packages\astropy\analytic_functions.

Platform: Windows-10.0.10240

Executable: C:\Programming\Anaconda\python.exe

Full Python Version:
3.5.0 |Anaconda 2.4.0 (64-bit)| (default, Dec  1 2015, 11:46:22) [MSC v.1900 64 bit (AMD64)]

encodings: sys: utf-8, locale: cp1252, filesystem: mbcs
byteorder: little
float info: dig: 15, mant_dig: 15

Numpy: 1.10.1
Scipy: 0.16.0
Matplotlib: 1.5.0
h5py: 2.5.0

collected 5 items

..\..\Programming\Anaconda\lib\site-packages\astropy\analytic_functions\tests\test_blackbody.py .....

========================== 5 passed in 6.71 seconds ===========================

But with python 2.7 and python 3.4 the test fails:

`Python 3.4.3 |Continuum Analytics, Inc.| (default, Dec  1 2015, 11:39:45) [MSC v.1600 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.

IPython 4.0.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import astropy

In [2]: astropy.test('analytic_functions')
============================= test session starts =============================
platform win32 -- Python 3.4.3 -- py-1.4.30 -- pytest-2.7.3
rootdir: C:\Programming\Anaconda\envs\py34\lib\site-packages\astropy\analytic_functions, inifile:

Running tests with Astropy version 1.0.6.
Running tests in C:\Programming\Anaconda\envs\py34\lib\site-packages\astropy\analytic_functions.

Platform: Windows-8-6.2.9200

Executable: C:\Programming\Anaconda\envs\py34\python.exe

Full Python Version:
3.4.3 |Continuum Analytics, Inc.| (default, Dec  1 2015, 11:39:45) [MSC v.1600 64 bit (AMD64)]

encodings: sys: utf-8, locale: cp1252, filesystem: mbcs
byteorder: little
float info: dig: 15, mant_dig: 15

Numpy: 1.10.1
Scipy: 0.16.0
Matplotlib: 1.5.0
h5py: 2.5.0

collected 5 items

..\..\Programming\Anaconda\envs\py34\lib\site-packages\astropy\analytic_functions\tests\test_blackbody.py FFF..

================================== FAILURES ===================================
____________________________ test_blackbody_scipy _____________________________

    @pytest.mark.skipif("os.environ.get('APPVEYOR')",  reason="fails on AppVeyor")
    @pytest.mark.skipif('not HAS_SCIPY')
    def test_blackbody_scipy():
        """Test Planck function.

        .. note:: Needs ``scipy`` to work.

        """
        flux_unit = u.Watt / (u.m ** 2 * u.um)
        wave = np.logspace(0, 8, 1e6) * u.AA
        temp = 100. * u.K
        with np.errstate(all='ignore'):
            bb_nu = blackbody_nu(wave, temp) * u.sr
        flux = bb_nu.to(flux_unit, u.spectral_density(wave)) / u.sr

        lum = wave.to(u.um)
        intflux = integrate.trapz(flux.value, x=lum.value)
        ans = const.sigma_sb * temp**4 / np.pi
>       np.testing.assert_allclose(intflux, ans.value, rtol=0.01)  # 1% accuracy

C:\Programming\Anaconda\envs\py34\lib\site-packages\astropy\analytic_functions\tests\test_blackbody.py:45:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\Programming\Anaconda\envs\py34\lib\site-packages\numpy\testing\utils.py:1347: in assert_allclose
    verbose=verbose, header=header)
C:\Programming\Anaconda\envs\py34\lib\site-packages\numpy\testing\utils.py:671: in assert_array_compare
    chk_same_position(x_isnan, y_isnan, hasval='nan')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

x_id = True, y_id = False, hasval = 'nan'

    def chk_same_position(x_id, y_id, hasval='nan'):
        """Handling nan/inf: check that x and y have the nan/inf at the same
            locations."""
        try:
            assert_array_equal(x_id, y_id)
        except AssertionError:
            msg = build_err_msg([x, y],
                                err_msg + '\nx and y %s location mismatch:'
                                % (hasval), verbose=verbose, header=header,
                                names=('x', 'y'), precision=precision)
>           raise AssertionError(msg)
E           AssertionError:
E           Not equal to tolerance rtol=0.01, atol=0
E
E           x and y nan location mismatch:
E            x: array(nan)
E            y: array(1.8049357842496399)

C:\Programming\Anaconda\envs\py34\lib\site-packages\numpy\testing\utils.py:651: AssertionError
___________________________ test_blackbody_overflow ___________________________

    @pytest.mark.skipif("os.environ.get('APPVEYOR')",  reason="fails on AppVeyor")
    def test_blackbody_overflow():
        """Test Planck function with overflow."""
        photlam = u.photon / (u.cm**2 * u.s * u.AA)
        wave = [0, 1000.0, 100000.0, 1e55]  # Angstrom
        temp = 10000.0  # Kelvin
        with np.errstate(all='ignore'):
            bb_lam = blackbody_lambda(wave, temp) * u.sr
        flux = bb_lam.to(photlam, u.spectral_density(wave * u.AA)) / u.sr

        # First element is NaN, last element is very small, others normal
        assert np.isnan(flux[0])
        assert np.log10(flux[-1].value) < -134
        np.testing.assert_allclose(
            flux.value[1:-1], [3.38131732e+16, 3.87451317e+15],
            rtol=1e-3)  # 0.1% accuracy in PHOTLAM/sr

        with np.errstate(all='ignore'):
            flux = blackbody_lambda(1, 1e4)
>       assert flux.value == 0
E       assert nan == 0
E        +  where nan = <Quantity nan erg / (Angstrom cm2 s sr)>.value

C:\Programming\Anaconda\envs\py34\lib\site-packages\astropy\analytic_functions\tests\test_blackbody.py:67: AssertionError
---------------------------- Captured stderr call -----------------------------
WARNING: Input contains invalid wavelength/frequency value(s) [astropy.analytic_functions.blackbody]
___________________________ test_blackbody_synphot ____________________________

    @pytest.mark.skipif("os.environ.get('APPVEYOR')",  reason="fails on AppVeyor")
    def test_blackbody_synphot():
        """Test that it is consistent with IRAF SYNPHOT BBFUNC."""
        # Solid angle of solar radius at 1 kpc
        fac = np.pi * (const.R_sun / const.kpc) ** 2 * u.sr

        with np.errstate(all='ignore'):
            flux = blackbody_nu([100, 1, 1000, 1e4, 1e5] * u.AA, 5000) * fac
        assert flux.unit == FNU

        # Special check for overflow value (SYNPHOT gives 0)
        assert np.log10(flux[0].value) < -143

        np.testing.assert_allclose(
            flux.value[1:], [0, 2.01950807e-34, 3.78584515e-26, 1.90431881e-27],
>           rtol=0.01)  # 1% accuracy

C:\Programming\Anaconda\envs\py34\lib\site-packages\astropy\analytic_functions\tests\test_blackbody.py:85:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\Programming\Anaconda\envs\py34\lib\site-packages\numpy\testing\utils.py:1347: in assert_allclose
    verbose=verbose, header=header)
C:\Programming\Anaconda\envs\py34\lib\site-packages\numpy\testing\utils.py:671: in assert_array_compare
    chk_same_position(x_isnan, y_isnan, hasval='nan')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

x_id = array([ True, False, False, False], dtype=bool)
y_id = array([False, False, False, False], dtype=bool), hasval = 'nan'

    def chk_same_position(x_id, y_id, hasval='nan'):
        """Handling nan/inf: check that x and y have the nan/inf at the same
            locations."""
        try:
            assert_array_equal(x_id, y_id)
        except AssertionError:
            msg = build_err_msg([x, y],
                                err_msg + '\nx and y %s location mismatch:'
                                % (hasval), verbose=verbose, header=header,
                                names=('x', 'y'), precision=precision)
>           raise AssertionError(msg)
E           AssertionError:
E           Not equal to tolerance rtol=0.01, atol=0
E
E           x and y nan location mismatch:
E            x: array([           nan,   2.018840e-34,   3.781005e-26,   1.901751e-27])
E            y: array([  0.000000e+00,   2.019508e-34,   3.785845e-26,   1.904319e-27])

C:\Programming\Anaconda\envs\py34\lib\site-packages\numpy\testing\utils.py:651: AssertionError
Some tests are known to fail when run from the IPython prompt; especially, but not limited to tests involving logging and warning handling.  Unless you are certain as to the cause of the failure, please check that the failure occurs outside IPython as well.  See http://docs.astropy.org/en/stable/known_issues.html#failing-logging-tests-when-running-the-tests-in-ipython for more information.
===================== 3 failed, 2 passed in 6.36 seconds ======================

and the output for you suggestion was:

In [4]: print(np.expm1(14387.))
C:\Programming\Anaconda\envs\py34\Scripts\ipython-script.py:1: RuntimeWarning: overflow encountered in expm1
  if __name__ == '__main__':
C:\Programming\Anaconda\envs\py34\Scripts\ipython-script.py:1: RuntimeWarning: invalid value encountered in expm1
  if __name__ == '__main__':
nan

In [5]: print(1 / np.expm1(14387.))
C:\Programming\Anaconda\envs\py34\Scripts\ipython-script.py:1: RuntimeWarning: overflow encountered in expm1
  if __name__ == '__main__':
C:\Programming\Anaconda\envs\py34\Scripts\ipython-script.py:1: RuntimeWarning: invalid value encountered in expm1
  if __name__ == '__main__':
nan

In [6]: print(np.__version__)
1.10.1

mwcraig · 2015-12-10T02:33:57Z

Note for bookkeeping purposes: this partially addresses the issues in #3242

pllim · 2015-12-10T14:52:07Z

Thanks, @MSeifert04 . I think you made it pretty clear that we can indeed skip the tests on Windows if Python version is < 3.5. @astrofrog , do you agree?

embray · 2015-12-11T17:34:26Z

I was able to reproduce these failures on Anaconda (thanks @pllim for pointing out this thread to me).
I don't want to merge this as-is, because it doesn't seem like anyone's demonstrated exactly why these tests are failing. If nothing else, instead of marking them skip I'd at least like to change the "skip" to "xfail"--this indicates that the tests are failing, but we don't know why or haven't prioritized fixing them yet. But it is still a failure that shouldn't be forgotten.

In the meantime let me see if I can actually debug what's going on here.

pllim · 2015-12-11T17:40:00Z

@embray , to reiterate @MSeifert04 above, I think he explained why, no?

In Python 2.7 and 3.4 (failed):

In [4]: print(np.expm1(14387.))
C:\Programming\Anaconda\envs\py34\Scripts\ipython-script.py:1: RuntimeWarning: overflow encountered in expm1
  if __name__ == '__main__':
C:\Programming\Anaconda\envs\py34\Scripts\ipython-script.py:1: RuntimeWarning: invalid value encountered in expm1
  if __name__ == '__main__':
nan

In [5]: print(1 / np.expm1(14387.))
C:\Programming\Anaconda\envs\py34\Scripts\ipython-script.py:1: RuntimeWarning: overflow encountered in expm1
  if __name__ == '__main__':
C:\Programming\Anaconda\envs\py34\Scripts\ipython-script.py:1: RuntimeWarning: invalid value encountered in expm1
  if __name__ == '__main__':
nan

In [6]: print(np.__version__)
1.10.1

In Python 3.5 (passed):

In [2]: print(np.expm1(14387.))
C:\Programming\Anaconda\Scripts\ipython-script.py:1: RuntimeWarning: overflow encountered in expm1
  if __name__ == '__main__':
inf

In [3]: print(1 / np.expm1(14387.))
C:\Programming\Anaconda\Scripts\ipython-script.py:1: RuntimeWarning: overflow encountered in expm1
  if __name__ == '__main__':
0.0

In [4]: print(np.__version__)
1.10.1

embray · 2015-12-11T17:59:50Z

Well, right, the difference is that in one case np.expm1 is returning nan, while in another it's returning inf. I think inf is the correct behavior. But I just wanted to investigate why, in the other case, it's returning nan and if that's a known upstream bug that can/has been fixed. In other words, I want to know what the specific bug is.

Then we might update the code to work around this bug. That is, compute expm1 separately, and if it returns nan for a large number replace that with np.inf manually...

embray · 2015-12-11T18:13:22Z

This seems to be the most relevant, but I don't understand why I'm encountering the issue in this case: numpy/numpy#4416

embray · 2015-12-11T18:45:01Z

Another thing I noticed is that on platforms where it's failing there's an additional

__main__:1: RuntimeWarning: invalid value encountered in expm1

that doesn't happen otherwise. So somewhere in the expm1 there's an invalid floating point operation, but I'm not sure where. np.exp() itself returns inf, correctly, on a large input.

The other wrinkle is that Numpy provides its own implementation of expm1 on platforms where it isn't available in libc, so I need to figure out which implementation it's using in this case. I don't think the Numpy implementation is buggy, so there could be a buggy libc implementation too. Not sure.

embray · 2015-12-11T19:15:03Z

Aha, this is the very bug: https://sourceware.org/bugzilla/show_bug.cgi?id=6814
This might also explain why I don't see this in my non-Anaconda installation of Numpy. ITSM the Numpy in Anaconda was compiled with MSVC, so its C runtime might have the same bug.

pllim · 2015-12-11T19:48:35Z

👍

embray · 2015-12-11T19:56:02Z

In #4393 I've proposed a workaround. I think we could still merge this PR for the v1.1.0 release, and delay this for v1.1.1. Or, since analytic_functions is new in v1.1.0 (I think?) I could merge #4171 prior to the release.

MSeifert04 · 2015-12-17T15:16:35Z

@embray: Definitly better than just to skip/xfail the tests. If you want you can close my PR. :-)

embray · 2015-12-23T16:43:04Z

Fixed (well, still a workaround) the underlying issue in #4393.

Skipped on Windows too.

44eca52

astrofrog added the zzz 💤 analytic_functions archived: this package no longer exists label Nov 9, 2015

pllim mentioned this pull request Dec 11, 2015

Failed tests when testing astropy in Anaconda #4171

Closed

embray mentioned this pull request Dec 11, 2015

Workaround to bug in #4171 #4393

Merged

embray mentioned this pull request Dec 11, 2015

expm1 buggy on some platforms numpy/numpy#6818

Open

embray closed this Dec 23, 2015

MSeifert04 deleted the fix-4171 branch April 29, 2016 08:45

Conversation

MSeifert04 commented Oct 27, 2015

Uh oh!

pllim commented Oct 27, 2015

Uh oh!

pllim commented Oct 27, 2015

Uh oh!

astrofrog commented Nov 9, 2015

Uh oh!

astrofrog commented Nov 9, 2015

Uh oh!

astrofrog commented Nov 9, 2015

Uh oh!

MSeifert04 commented Dec 5, 2015

Uh oh!

mwcraig commented Dec 10, 2015

Uh oh!

pllim commented Dec 10, 2015

Uh oh!

embray commented Dec 11, 2015

Uh oh!

pllim commented Dec 11, 2015

Uh oh!

embray commented Dec 11, 2015

Uh oh!

embray commented Dec 11, 2015

Uh oh!

embray commented Dec 11, 2015

Uh oh!

embray commented Dec 11, 2015

Uh oh!

pllim commented Dec 11, 2015

Uh oh!

embray commented Dec 11, 2015

Uh oh!

MSeifert04 commented Dec 17, 2015

Uh oh!

embray commented Dec 23, 2015

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants