RuleBasedStateMachine finds Flaky data generation when using dictionary/Counter as class attribute #2338
Description
I'm trying to use Hypothesis' strategy infrastructure to generate some data for a sample data exchange of related files. I wanted to have a pool of at least 10 parent keys that I can reference in the child file, and ran into an issue where the RuleBasedStateMachine complains--intermittently, but frequent enough to easily reproduce--about flaky data generation. I've narrowed it down to using a dictionary or Counter to track the key count. Suspecting mutability to be a factor, I can confirm that plain integers do not throw the exception "hypothesis.errors.Flaky: Inconsistent data generation! Data generation behaved differently between different runs. Is your data generation depending on external state?"
I'm not sure whether this is truly a mutability issue, nor a Hypothesis problem, or even something else.
The code below is a reduced version to demonstrate the issue. The mutable version raises the exception within 5-10 consecutive runs, commonly already on the second run. The immutable version has always succeeded. Running both StateMachines each time increases the number of runs for the issue to show, so I commented the immutable one out.
I looked in the documentation for references on the use of mutable objects in a state machine, but could not find any. If there are, I would appreciate a link and I will educate myself some more. :)
Arnoud.
"""Generate test data files."""
from collections import Counter
from hypothesis import stateful as sf, strategies as st
class MutableMachine(sf.RuleBasedStateMachine):
record_count = Counter() # Both Counter and plain dict fail
# record_count = dict()
# record_count["parent"] = 0
# record_count["child"] = 0
Parents = sf.Bundle("parent")
Children = sf.Bundle("child")
@sf.rule(target=Parents, id=st.integers())
def create_parent(self, id):
p = dict()
p["id"] = id
self.record_count["parent"] += 1
return p
@sf.precondition(lambda self: self.record_count["parent"] >= 10)
@sf.rule(target=Children, id=st.integers(), parent=Parents)
def create_child(self, id, parent):
c = dict()
c["id"] = id
c["parent"] = parent
self.record_count["child"] += 1
return c
class ImmutableMachine(sf.RuleBasedStateMachine):
parent_count = 0
child_count = 0
Parents = sf.Bundle("parent")
Children = sf.Bundle("child")
@sf.rule(target=Parents, id=st.integers())
def create_parent(self, id):
p = dict()
p["id"] = id
self.parent_count += 1
return p
@sf.precondition(lambda self: self.parent_count >= 10)
@sf.rule(target=Children, id=st.integers(), parent=Parents)
def create_child(self, id, parent):
c = dict()
c["id"] = id
c["parent"] = parent
self.child_count += 1
return c
TestMutableMachine = MutableMachine.TestCase
# TestImmutableMachine = ImmutableMachine.TestCase