Update the test runner to use the updated code in astropy 1.3 by Cadair · Pull Request #1983 · sunpy/sunpy

Cadair · 2017-02-01T14:35:09Z

This modifies our internal test runner (the thing that provides sunpy.self_test and python setup.py test) to be much closer to astropy's implementation. The differences between us and astropy now are the replacement of the astropy remote_data argument with online and offline and the addition of the figure argument.

This means we have many more options to self_test and setup.py test such as checking for open file handles, running pep8 checks etc.

pep8speaks · 2017-02-01T14:35:12Z

Hello @Cadair! Thanks for updating the PR.

In the file sunpy/data/test/__init__.py, following are the PEP8 issues :

Line 20:1: E303 too many blank lines (3)

In the file sunpy/instr/goes.py, following are the PEP8 issues :

Line 44:101: E501 line too long (130 > 100 characters)
Line 122:31: E128 continuation line under-indented for visual indent
Line 123:31: E128 continuation line under-indented for visual indent
Line 124:31: E128 continuation line under-indented for visual indent
Line 127:31: E128 continuation line under-indented for visual indent
Line 128:31: E128 continuation line under-indented for visual indent
Line 970:5: E122 continuation line missing indentation or outdented

In the file sunpy/net/helio/hec.py, following are the PEP8 issues :

Line 87:101: E501 line too long (144 > 100 characters)
Line 127:101: E501 line too long (107 > 100 characters)

In the file sunpy/tests/tests/test_self_test.py, following are the PEP8 issues :

Line 55:101: E501 line too long (110 > 100 characters)
Line 56:101: E501 line too long (111 > 100 characters)

Comment last updated on May 26, 2018 at 11:19 Hours UTC

Cadair · 2017-02-09T12:04:50Z

there is a lot more todo on this, for some reason the doctests are not being picked up by pytest. Also I am working on porting the extra plugins for pytest that astropy uses to provide things like open file checking into a form where we can use them (see astropy/astropy#5770). Oh and it just fails, so that needs fixing as well.

Cadair · 2017-05-17T16:55:36Z

I will close this for now, there may be a couple of different ways of achieving it, like using the new code I wrote in Astropy to expose our current pytest UI.

Cadair · 2018-05-10T11:17:14Z

This should now be resurrected as since this was closed we have adopted doctestplus and remote_data and all the astropy pytest plugins have been broken out into seperate packages, which should make this whole thing a lot easier.

I think the largest challenge here is now the setup command, but it should be doable.

Cadair · 2018-05-10T14:44:07Z

-from sunpy.tests import main as self_test
+from sunpy.tests.runner import SunPyTestRunner
+
+self_test = SunPyTestRunner.make_test_runner_in(__path__[0])


This needs to not be defined if ASTROPY_SETUP is true.

Cadair · 2018-05-10T14:45:13Z

+            return []
+
+    @keyword(False)
+    def figure(self, figure, kwargs):


need to add a figure_only

Cadair · 2018-05-10T14:45:59Z

+            cmd_pre += pre
+            cmd_post += post
+
        cmd = ('{cmd_pre}{0}; import {1.package_name}, sys; result = ('


need to make sure this list is upto date with the one in astropy.

Cadair · 2018-05-13T14:54:16Z

Online test fails looks to be because of a dead NOAA server.

@sunpy/sunpy-maintainers review please!

nabobalis · 2018-05-14T13:56:50Z

 .hypothesis/
 .pytest_cache/
 sunpydata.sqlite
+v/


What is this?

🤷‍♀️

nabobalis · 2018-05-14T13:57:50Z

         # Order of tests run in build stages
         - python: 3.6
           stage: Initial tests
-           env: SETUP_CMD='test --coverage'


So what does this run now?

the same thing, it just uses the default value

Oh that was the default value already. Nevermind.

nabobalis · 2018-05-14T13:58:14Z

    ...     if database_entry.observation_time_start.month in [3,4,5]:
    ...         database.tag(database_entry, 'spring')
-    >>> database.tags
+    >>> database.tags  # doctest: +SKIP


Why were these tests skipped?

nabobalis · 2018-05-14T13:58:25Z

    >>> import sunpy.data.sample  # doctest: +REMOTE_DATA
    >>> my_timeseries = ts.TimeSeries(sunpy.data.sample.GOES_XRS_TIMESERIES, source='XRS')  # doctest: +REMOTE_DATA

+.. doctest-skip-all


But why? :(

because they don't work!!

BUT HOW? I THOUGHT IT PASSED ON TRAVIS.

WHY DO WE HAVE AN ENTIRE FILE THAT HAS NO WORKING TESTS?

Yeah I have no idea how this was passing before! This file needs updating to fix the tests but this PR is not the time.

nabobalis · 2018-05-14T14:00:27Z

        >>> start = '2005/01/03'
        >>> end = '2005/12/03'
-        >>> temp = hc.time_query(start, end, max_records=10)   # doctest: +REMOTE_DATA
+        >>> temp = hc.time_query(start, end, max_records=10)   # doctest: +REMOTE_DATA +SKIP


this one also needs user input.

nabobalis · 2018-05-14T14:00:36Z

        >>> from sunpy.net.helio import hec
        >>> hc = hec.HECClient()  # doctest: +REMOTE_DATA
-        >>> hc.make_table_list()  # doctest: +REMOTE_DATA
+        >>> hc.make_table_list()  # doctest: +REMOTE_DATA +SKIP


But my skip?!

this one is real. It's just nonsense.

oh that's it, this function needs user input, so it can't be run as a doctest.

Cadair · 2018-05-17T10:43:20Z

oh balls: https://travis-ci.org/sunpy/sunpy/jobs/378357403#L2557

The -k remote_data in the online build skips all the damn doctests

fix online only test

Cadair · 2018-05-26T11:21:02Z

Assuming my rebase hasn't broken anything this is now good to go. Reviews please!

nabobalis

Looks like python code.

Cadair added the Tests Affects tests in some measure. label Feb 1, 2017

Cadair added Needs Review Needs review(s) before merge. [WIP] and removed Needs Review Needs review(s) before merge. labels Feb 1, 2017

Cadair force-pushed the test_runner branch from 321052b to 27c861c Compare February 3, 2017 12:03

dpshelio approved these changes Feb 5, 2017

View reviewed changes

Cadair mentioned this pull request Feb 9, 2017

Split pytest plugins out into individual modules astropy/astropy#5770

Closed

Cadair closed this May 17, 2017

Cadair removed the [WIP] label May 17, 2017

nabobalis added the Needs Adoption PRs that went stale but should be picked up again and completed. label Apr 12, 2018

Cadair reopened this May 10, 2018

Cadair force-pushed the test_runner branch from 149575e to 2a00351 Compare May 10, 2018 14:40

Cadair removed the Needs Adoption PRs that went stale but should be picked up again and completed. label May 10, 2018

Cadair added this to the 1.0 milestone May 10, 2018

Cadair commented May 10, 2018

View reviewed changes

Cadair force-pushed the test_runner branch from 464022a to 28bf727 Compare May 13, 2018 13:09

Cadair added the Needs Review Needs review(s) before merge. label May 13, 2018

nabobalis reviewed May 14, 2018

View reviewed changes

Cadair force-pushed the test_runner branch from 28bf727 to b9d3e06 Compare May 24, 2018 21:24

Cadair added 5 commits May 26, 2018 12:17

Update the test runner to use the updated code in astropy 1.3

156068c

Run --online for doctests

bdf8e9b

Fixup doctests

b42483b

Fix online only and also prep for astropy/astropy#7461

8617754

fix online only test

Fix doctest fails on Travis

689d7f3

Cadair force-pushed the test_runner branch from 3fa7499 to 689d7f3 Compare May 26, 2018 11:19

nabobalis approved these changes May 26, 2018

View reviewed changes

Cadair merged commit dc48228 into sunpy:master May 28, 2018

Cadair deleted the test_runner branch May 28, 2018 12:38

nabobalis removed the Needs Review Needs review(s) before merge. label Mar 21, 2024

Conversation

Cadair commented Feb 1, 2017

Uh oh!

pep8speaks commented Feb 1, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comment last updated on May 26, 2018 at 11:19 Hours UTC

Uh oh!

Cadair commented Feb 9, 2017

Uh oh!

Cadair commented May 17, 2017

Uh oh!

Cadair commented May 10, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Cadair commented May 13, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nabobalis May 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Cadair commented May 17, 2018

Uh oh!

Cadair commented May 26, 2018

Uh oh!

nabobalis left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pep8speaks commented Feb 1, 2017 •

edited

Loading

nabobalis May 14, 2018 •

edited

Loading