👋 Thank you for your draft pull request! Do you know that you can use [ci skip] or [skip ci] in your commit messages to skip running continuous integration tests until you are ready?

Why not permanently add this to devdeps?

I guess we could... Will add it to infrastructure agenda.

Failure looks related. But what do we do, @lpsinger or @mhvk ?

______________________________ [doctest] arctan2 _______________________________
EXAMPLE LOCATION UNKNOWN, not showing all tests of that example
??? >>> np.arctan2([0., 0., np.inf], [+0., -0., np.inf])
Expected:
    array([ 0.        ,  3.14159265,  0.78539816])
Got:
    array([0.        , 3.14159265, 0.78539816])

.../astropy/units/quantity_helper/helpers.py:None: DocTestFailure

The failing doctest is in Numpy itself, here. This is proof that we are now discovering tests in ufuncs! 🤣

The question is, why is pytest discovering tests in Numpy?

#12858 should fix this.

What about these new errors?

_______________________________ [doctest] hyp2f1 _______________________________
EXAMPLE LOCATION UNKNOWN, not showing all tests of that example
??? >>> np.arctan(z) / z
Expected:
    0.9272952180016123
Got:
    0.9272952180016122

/home/runner/work/astropy/astropy/astropy/cosmology/flrw.py:None: DocTestFailure
_______________________________ [doctest] hyp2f1 _______________________________
EXAMPLE LOCATION UNKNOWN, not showing all tests of that example
??? >>> np.arctan(z) / z
Expected:
    0.9272952180016123
Got:
    0.9272952180016122

/home/runner/work/astropy/astropy/astropy/cosmology/tests/test_flrw.py:None: DocTestFailure

Hmm, there are several Astropy modules that have from scipy.special import ... in them.

But astropy.cosmology.flrw is the only module in which the import is at global rather than function scope.

So having this new pytest-doctestplus plugin means we have to be real careful of imports? I think this should be documented in the dev docs then. When to import globally and when not to.

@lpsinger - the fact that most imports are local is almost certainly because scipy is not (yet) a required dependency for most of astropy. But for cosmology it is, hence the imports on top.

I'm starting to wonder a bit if the gain is worth the pain... Maybe after all there should be a switch on doctestplus to turn off ufunc doc tests? Of course, really the ufuncs should expose the module they are defined in!

OK, try #12861 and scientific-python/pytest-doctestplus#175.

I'm starting to wonder a bit if the gain is worth the pain... Maybe after all there should be a switch on doctestplus to turn off ufunc doc tests? Of course, really the ufuncs should expose the module they are defined in!

Agreed. And instead of having to have __doctest_skip__ in every module where some ufunc is imported, we should probably do the opposite, i.e. define a __doctest_ufunc__ that would allow to activate doctesting ufuncs on a case by case basis ?

I'm starting to wonder a bit if the gain is worth the pain... Maybe after all there should be a switch on doctestplus to turn off ufunc doc tests? Of course, really the ufuncs should expose the module they are defined in!

Agreed. And instead of having to have __doctest_skip__ in every module where some ufunc is imported, we should probably do the opposite, i.e. define a __doctest_ufunc__ that would allow to activate doctesting ufuncs on a case by case basis ?

Or we could make the ufunc doctest discovery an opt-in using a --doctest-ufunc option to pytest-doctestplus, which was in my original draft.

Yes, so at least now we found the drawback of activating it by default ;). The problem with the global switch is that we would face the same issue if we want to use it in Astropy. So, since ufunc can be doctested only if imported in another module (I think ?), maybe a module level directive (e.g. __doctest_ufunc__) is a better approach ?

So... what is the path forward here?

I think we are all agreed that doctesting ufuncs is a good idea. The question is whether we want to opt in or opt out of testing specific files.

I actually think the optimal solution is a --doctest-ufunc flag that defaults to False. __doctest_ufunc__ turns on testing on a case-by-case basis, regardless of whether --doctest-ufunc is True or False, and __doctest_skip__ does the same, but for turning testing off on a case-by-case basis. This implementation nicely gets any scenario in the above table.
Unfortunately, this solution is the most work.

I agree; I like --doctest-ufunc. My original application is in a coordinated package with lots of ufuncs, so I'd rather not list them all out explicitly in __doctest_ufunc__. I'll add the --doctest-ufunc option back to this PR.

Rebased and dropped the cherry-picks. 🤞

21/21 = 💯 🎉

For those interested:

• TST: Test against dev infrastructure weekly, add scipy dev to devdeps #13080