If execute_example messes up and causes an unexpected exception, during an example run that involved an assumption failure, the test output shows UnsatisfiedAssumption, rather than focusing on the unexpected exception that is the real source of the bug. This is a case of garbage in garbage out, but it would be nice if we saw the unexpected exception rather than the UnsatisfiedAssumption because the latter doesn't help us find and address the actual bug in our execute_example code.
In the original case of this, I got
...
hypothesis.errors.UnsatisfiedAssumption
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
...
So in at least some cases, the error reporting does acknowledge the actual problem that caused the crash. However, in the minimal cut that I show below, there is no "... another exception occurred:" part, so I'm not sure which cases lead to which output.
Repro:
from hypothesis import assume
import unittest
class UnsatisfiedAssumptionTest(unittest.TestCase):
def execute_example(self, fn):
try:
return fn()
except Exception as e:
print([][0]) # unexpected exception, due to invalid array indexing
raise e
def testUnsatisfiedAssumption(self):
assume(False)
if __name__ == "__main__":
unittest.main()
Result:
E
======================================================================
ERROR: testUnsatisfiedAssumption (__main__.UnsatisfiedAssumptionTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/example.py", line 14, in testUnsatisfiedAssumption
assume(False)
File "/home/nick/miniconda3/lib/python3.10/site-packages/hypothesis/control.py", line 38, in assume
raise UnsatisfiedAssumption
hypothesis.errors.UnsatisfiedAssumption
If
execute_examplemesses up and causes an unexpected exception, during an example run that involved an assumption failure, the test output showsUnsatisfiedAssumption, rather than focusing on the unexpected exception that is the real source of the bug. This is a case of garbage in garbage out, but it would be nice if we saw the unexpected exception rather than theUnsatisfiedAssumptionbecause the latter doesn't help us find and address the actual bug in ourexecute_examplecode.In the original case of this, I got
So in at least some cases, the error reporting does acknowledge the actual problem that caused the crash. However, in the minimal cut that I show below, there is no "... another exception occurred:" part, so I'm not sure which cases lead to which output.
Repro:
Result: