We hit a regression when updating to 6.150.0:
mozilla-releng/balrog#3603 (comment)
I don't understand the hypothesis code base at all, so apologies if the following LLM analysis is incorrect.. but figured it might at least lead you in the right direction :)
Details
Summary
After upgrading from Hypothesis 6.148.7 to 6.150.0, st.recursive() with max_leaves fails to generate recursive structures. Tests using assume() fail with Unsatisfiable after filtering out 1000/1000 examples.
Reproduction
import hypothesis.strategies as st
from hypothesis import assume, given, settings, HealthCheck
useful_values = st.none() | st.floats(allow_nan=False) | st.text()
def unique_items_only(i):
if isinstance(i, dict):
return tuple(i.keys())
if isinstance(i, list):
return tuple(i)
return i
useful_dict = st.dictionaries(st.text(), useful_values | st.lists(useful_values, max_size=10, unique_by=unique_items_only), max_size=10)
useful_list = st.lists(useful_values | useful_dict, max_size=10, unique_by=unique_items_only)
json = st.dictionaries(st.text(), st.recursive(useful_values, lambda x: useful_list | useful_dict, max_leaves=20), max_size=10)
@settings(max_examples=50, suppress_health_check=[HealthCheck.too_slow])
@given(json)
def test_example(base):
to_modify = None
for key in base:
if isinstance(base[key], list):
to_modify = key
break
assume(to_modify is not None)
assert to_modify is not NoneVersion 6.148.7 generates approximately 60% examples with lists. Version 6.150.0 generates 0% examples with lists and the test fails with Unsatisfiable.
Root Cause
This is a merge-induced regression between two changes:
PR #4616 (commit 72f5cb7) added min_leaves parameter. When empty collections are generated with leaves_drawn=0, retry events are written to data.events.
Commit 6c20067 made data.events realization unconditional (moved outside observability_enabled check for pytest statistics support).
The unconditional realization of retry events disrupts the data provider's internal state, breaking recursive structure generation.
Bisect Results
Commit 72f5cb7 alone works. Commit 074893e alone works. Merge commit 681ce55 is the first bad commit.
References
681ce55
#4616
6c20067
We hit a regression when updating to 6.150.0:
mozilla-releng/balrog#3603 (comment)
I don't understand the hypothesis code base at all, so apologies if the following LLM analysis is incorrect.. but figured it might at least lead you in the right direction :)
Details
Summary
After upgrading from Hypothesis 6.148.7 to 6.150.0, st.recursive() with max_leaves fails to generate recursive structures. Tests using assume() fail with Unsatisfiable after filtering out 1000/1000 examples.
Reproduction
Version 6.148.7 generates approximately 60% examples with lists. Version 6.150.0 generates 0% examples with lists and the test fails with Unsatisfiable.
Root Cause
This is a merge-induced regression between two changes:
PR #4616 (commit 72f5cb7) added min_leaves parameter. When empty collections are generated with leaves_drawn=0, retry events are written to data.events.
Commit 6c20067 made data.events realization unconditional (moved outside observability_enabled check for pytest statistics support).
The unconditional realization of retry events disrupts the data provider's internal state, breaking recursive structure generation.
Bisect Results
Commit 72f5cb7 alone works. Commit 074893e alone works. Merge commit 681ce55 is the first bad commit.
References
681ce55
#4616
6c20067