Skip to content

MNT: Ignore divide by zero warning in np.log - #11042

Closed
pllim wants to merge 1 commit into
astropy:masterfrom
pllim:fix-numpy-dev
Closed

pllim wants to merge 1 commit into
astropy:masterfrom
pllim:fix-numpy-dev

Conversation

@pllim

@pllim pllim commented Nov 14, 2020

Copy link
Copy Markdown
Member

Description

This pull request is to fix warnings causing failure in numpy-dev job. np.log must be issuing extra warning now in numpy-dev or something.

This is a follow up of #10388 and address part of #11038 .

@pllim

pllim commented Nov 14, 2020

Copy link
Copy Markdown
Member Author

p.s. If you think we need to back port this all the way to 4.0.x, please change the milestone. Thanks.

@pllim pllim mentioned this pull request Nov 14, 2020
18 tasks done
@pllim
pllim requested review from larrybradley and mhvk November 14, 2020 15:03
values = _prepare(values, clip=clip, out=out)
replace_invalid = not clip and invalid is not None
with np.errstate(invalid='ignore'):
with np.errstate(divide='ignore', invalid='ignore'):

@dhomeier dhomeier Nov 14, 2020 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really have a good feeling about a blanket ignore on all divisions by zero as long as we don't understand the underlying failure. I cannot reproduce it locally (OS X) with the exact same numpy version 1.20.0.dev0+0ffaaf8, (same when still running devdeps on MacOS X), and looking e.g. at https://github.com/astropy/astropy/pull/11043/checks?check_run_id=1400152307
it is incomprehensible how LogStretch for input `DATA=[0., 0.25, 0.5, 0.75, 1.] can come up with

1289 ../../.tox/py37-test-devdeps/lib/python3.7/site-packages/astropy/visualization/tests/test_stretch.py:44: 
1290 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
1291
1292 self = <astropy.visualization.stretch.LogStretch object at 0x7f108716cfd0>
1293 values = array([      -inf, 5.52545294, 6.2166061 , 6.62140565, 6.90875478])
1294 clip = False, out = None, invalid = None

– it's as if DATA[0] somehow missed the np.add(values, 1.) step...
At this point I am more inclined to consider this a buggy numpy build (no idea what best to do about that, though).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think values refers to the input values, so first element is -inf - this will remain -inf after multiplication and addition. I think the problem is that it leads to an invalid division error instead of the invalid error in the test case. That does sounds like a numpy bug, but I cannot reproduce it on my Debian/linux system either: test-devdeps just passes (and has the same numpy version, though I have a much more recent python than 3.7; perhaps we should run devdeps on python 3.9?)

@dhomeier dhomeier Nov 15, 2020 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I first thought the warning was referring to the np.true_divide step by mistake, since I cannot see which prior step

np.multiply(values, self.exp, out=values)
np.add(values, 1., out=values)
np.log(values, out=values)
np.true_divide(values, np.log(self.exp + 1.), out=values)

could possibly produce -inf. Whereas

np.log([0, 251., 501., 751., 1001.])
array([      -inf, 5.52545294, 6.2166061 , 6.62140565, 6.90875478])

would produce exactly the reported output and warning, so I rather think out has already been copied back to values when the warning is emitted.

Running devdeps on a newer Python is certainly a good idea, although I suspect this issue is more related to build environment, possibly compilers, than the numpy + python versions as such.

@dhomeier dhomeier Nov 16, 2020 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. In #11048 I just tried to check if running a version with
values = np.add(values, 1.)
etc. would avoid the division by zero error. Running numpy-dev, which today has already advanced to 1.20.0.dev0+a645106, produced no failures. However for e.g. test_inplace the LogStretch call should not have been affected in any way by my modification, and apparently other builds with last night's numpy-dev have run successfully as well.
So this looks really like a one-time failure with 0ffaaf8. Don't know if there is anything more we could do to retrace this or avoid future failures; but nightly dev builds may of course come back with new bugs at any time.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is fine to see if the next "nightly" (really, weekly) still produces the problem... It is very odd I couldn't even reproduce it with exactly the same tox setup and same version.

@dhomeier dhomeier Nov 16, 2020 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's why I was also thinking of a compiler glitch or erratic memory failure (although on the build system it was obviously consistent over calls in LogScale). Among the changes between 0ffaaf8 and a645106 numpy/numpy#17755 looks like it might be related, without having looked any deeper inside; hopefully the next build should bring confirmation.
But it looks like this was a known bug fixed in numpy/numpy#17763 after all!

@pllim

pllim commented Nov 16, 2020

Copy link
Copy Markdown
Member Author

Hah. I like problem that disappears on its own. Thank you very much for checking!

@pllim pllim closed this Nov 16, 2020
@pllim
pllim deleted the fix-numpy-dev branch November 16, 2020 18:51
@bsipocz bsipocz removed this from the v4.2.1 milestone Nov 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants