Repository navigation
Paper: Falsify your Software: validating scientific code with property-based testing - #549
Conversation
|
Hey! I took a quick look at the build errors, and it looks like bibtex tripping on your use of backslashes in journal names in the bibfile for this paper. If you can handle those, the rest of this should be okay. |
946cd9c to
fdb2392
Compare
b880820 to
39b22c5
Compare
| for the task has several advantages: | ||
|
|
||
| - a concise and expressive interface for describing inputs | ||
| - tests are never flaky - failing examples are cached and |
There was a problem hiding this comment.
this requires more explanation. how do libraries that do Property Based Testing prevent flakiness in tests? If anything, adding random data generators into tests, adds flakiness into the testing framework.
A more general comment - how can we reproduce test failures with PBT? With random number generators, we can seed the random number generator to reproduce results. Do the various PBT frameworks also do something similar under the hood?
There was a problem hiding this comment.
The basic trick is that we cache the output from the PRNG between runs, and automatically replay any previous failures on the next run.
There are a bunch of options to explicitly report and then force particular seeds or insert a buffer into the cache, but in normal development it's entirely automatic.
This means that we can get the benefit of a different set of examples each run, while also having any failures replayed every time. (or if you want the same set of examples each time, you can set the seed manually too)
anirudhacharya
left a comment
There was a problem hiding this comment.
Can property based testing replace the traditional example-based unit testing?
Mostly, yes. A good example might be my Traditional example-based tests are still nice to pin down specific or weird edge cases, where asserting the exact result is easy and a property would be fiddly or error-prone, but almost all of the tests I write are property-based. |
|
Hey @Zac-HD ! Thanks to some awesome behind the scenes work by @stargaser, we now have experimental support for publishing papers with ORCIDs. If you have one and would like to add it to your paper, it's as simple as adding an |
|
I do indeed have one, and took it out of my early draft when I noticed that it wasn't supported yet. Thanks for your persistence @stargaser! |
39b22c5 to
a454560
Compare
a454560 to
21d6a22
Compare
Wouldn't it also make sense to have unit tests for Test Driven Development to ensure basic sanity of the software as it is being developed. It would seem property-based tests are not so good for Test Driven Development. |
|
Hi @Zac-HD ! Do you have thoughts about using hypothesis with TDD? Or do you feel this is out of scope for the paper? |
|
(sorry I missed this before!) I think TDD is out of scope for this paper, but for what it's worth I also think property-based tests are just as applicable for TDD as for any other way testing fits into your development cycle. Personally, I'd rather do my basic sanity-checks with properties than specific examples - @anirudhacharya if you've found this problematic I'd be interested to hear what difficulties you ran into! |
|
Thanks for the response! @anirudhacharya do you feel that this paper is now ready for inclusion in the proceedings? |
@deniederhut Yes I do. |
PDF link: http://procbuild.scipy.org/download/Zac-HD-zac-hatfield-dodds