Repository navigation
investigate and change a couple of -skips to -xfails - #20889
wyattscarpenter wants to merge 1 commit into
Conversation
There are surely more that should be dealt with. This addresses part of python#20881
hauntsaninja
left a comment
hauntsaninja
left a comment
There was a problem hiding this comment.
xfail tests in a data driven test setup like ours are pretty brittle, e.g. I recently fixed one that was only failing because a diagnostic had changed over the years to use X | Y union syntax. This makes me prefer running these long-term xfails as regular tests, with a detailed TODO comment describing what expected behaviour is, so you're guaranteed to notice if you change related behaviour.
|
I wonder if it'd be worth making up some new syntax like Edit: having written out the above and considered it, I doubt anyone will be motivated to take on the maintenance burden of such a feature. Although some such convention (not computer-enforced) could be used to improve greppability of this class of todo items specifically. |
|
I also view upgrading xfail tests to fake-xfail tests, or whatever one wants to call them, which I support because of the mentioned flaw of xfail, as an orthogonal improvement that should ideally be done after changing all of the wrong skips to xfails, although they could be done at the same time. (In making that effort, the readme about tests should also be changed to recommend fake-xfail over xfail.) Both of those efforts are quite tractable given the low numbers of each type of explicitly marked test. |
There are surely more that should be dealt with. This addresses part of #20881
I think these, and probably several more of the tests marked -skip in our corpus, are just mistakenly -skip when they should be -xfail. There are two benefits, both admittedly marginal, to making them xfail instead of skip: 1. It is semantically correct (at least according to my interpretation of https://docs.pytest.org/en/stable/how-to/skipping.html) 2. If they get fixed they will be reported as an unexpected xpass, instead of silently continuing to skip.