Repository navigation
Added a --repeat option for testing - #3019
Conversation
|
Will also be handy for debugging the random SAMP failures. |
|
I'm definitely +1 on the feature. Though I'm not sure this PR fully addresses all the changes needed to make a new test option work. I've been planning to refactor how we add test options so that it can be all in one place, rather than in like 3 or 4 different places like it is currently :( |
|
Ah, I see you acknowledged that in the text of your PR :) |
|
@embray - I've finalized this PR and have opened astropy/astropy-helpers#92 - can you give it a final review? |
|
@astrofrog - one question on this - presumably it would take a long time to actually run all of |
|
@eteq - yes the intent is you would actually use this with only specific tests or packages (with the |
|
Gotcha. 👍 from me, then. |
|
I'm going to wait a bit for this because I was to get astropy-helpers 0.4.3 out and this isn't urgent. |
|
This has been working smoothly (along with astropy/astropy-helpers#92) so I'm going to go ahead and merge. |
Added a --repeat option for testing
|
I have one concern about this now that I look at it again, which is that if an affiliated package has a I'll think about if I can find a decent workaround to this. |
Added a --repeat option for testing
… it actually takes. This provides at least a temporary workaround to the issue I raised in this comment: astropy/astropy#3019 (comment) (It would be nicer if we also warned when unsupported options were passed to the test command, but that would require a little more reworking of this--this is good enough to at least ensure a degree of compatibility.)
|
I added a temporary workaround to this issue in astropy/astropy-helpers@1068e26; will come up with something better another time. |
In some cases, we need to debug non-determininstic failures, and for this it can be handy to have a
--repeatoption. For example, for #2964 the failures only happen 10% of the time, so having this kind of option makes it easy to run the test say 100 times to determine how often it fails (and to force it to fail). This can also be useful for pre-empting issues like #3018 - there may be issues that only become apparent once we have enough tests. This change will need a corresponding change in astropy-helpers to add the option topython setup.py testbut in the mean time it works withpy.test --repeat=10for example. I'll do a PR for astropy-helpers if there is support for this PR.