You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3fc38f1
Browse filesBrowse the repository at this point in the historyBrowse files
claude: teach /hypothesis about stateful tests and @example
The command's property catalogue had no entry for stateful properties, so
the model would not propose a RuleBasedStateMachine even for classes whose
methods obviously mutate them, and order-dependent bugs went unreached.
Step 4 also stopped at confirming a bug, leaving the falsifying input to be
rediscovered on each run rather than pinned with @example.
Both features are already documented on reference/api.html, which the
reference section links but described only as "@settings, @given, etc."
Co-Authored-By: Claude Opus 5 <[email protected]>
Copy file name to clipboardExpand all lines: .claude/commands/hypothesis.md
+5-1Lines changed: 5 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -42,6 +42,7 @@ Here are some examples of typical properties:
42
42
- Confluence: the order of function application doesn't matter (for example, in compiler optimization passes).
43
43
- Metamorphic property: some relationship between `f(x)` and `g(x)` holds for all x. For example, `sin(π − x) = sin(x)`.
44
44
- Single entry point. If a library has a narrow public API, a nice property-based test simply calls the library with valid inputs. Common in parsers.
45
+
- Stateful property: a sequence of operations preserves an invariant, or agrees with a simpler model. The signal is a class whose methods mutate it — `push`/`pop`, `add`/`remove`, `open`/`close`, incremental update. These bugs depend on the *order* of calls, so no single-call property reaches them.
45
46
46
47
While the following should generally not be tested:
47
48
@@ -72,18 +73,21 @@ When writing Hypothesis tests, follow these guidelines:
72
73
73
74
- Each Hypothesis test should be both sound (tests only inputs the code can actually be called with) and complete (tests all inputs the code can actually be called with). Sometimes this is difficult. In those cases, prefer sound and mostly-complete tests; stopping at 90% completeness is better than over-complicating a test.
74
75
- Only place constraints on Hypothesis strategies if required by the code. For example, prefer `st.lists(...)` (with no size bound) to `st.lists(..., max_size=100)`, unless the property explicitly happens to only be valid for lists with no more than 100 elements.
76
+
- For a stateful property, write a `RuleBasedStateMachine` with `@rule`, `@invariant`, and `@precondition`, and expose it to the test runner as `TestFoo = FooMachine.TestCase`. Keep the model as simple as possible — usually a plain `list` or `dict`. A model that reimplements the logic under test is not an independent oracle, and will agree with the code even when the code is wrong.
75
77
76
78
## 4. Run the new Hypothesis tests, and reflect on the result.
77
79
78
80
Run the new Hypothesis tests that you just added. If any fail, reflect on why. Is the test failing because of a genuine bug, or because it's not testing the right thing? Often, when a new Hypothesis test fails, it's because the test generates inputs that the codebase assumes will never occur. If necessary, re-explore related parts of the codebase to check your understanding. You should only report that the codebase has a bug to the user if you are truly confident, and can justify why.
79
81
82
+
Once you have confirmed a genuine bug, add the minimal falsifying input to the property test as an `@example(...)`, so that the specific regression is checked on every run in addition to the generated inputs.
83
+
80
84
# Hypothesis Reference
81
85
82
86
Documentation reference (fetch with the `WebFetch` tool if required):
83
87
84
88
-**Strategies API reference**: https://hypothesis.readthedocs.io/en/latest/reference/strategies.html
85
89
-**Other API reference**: https://hypothesis.readthedocs.io/en/latest/reference/api.html
86
-
- Documents `@settings`, `@given`, etc.
90
+
- Documents `@settings`, `@given`, `@example`, and stateful testing (`RuleBasedStateMachine`, `@rule`, `@invariant`, `@precondition`), among others.
87
91
88
92
These Hypothesis strategies are under-appreciated for how effective they are. Use them if they are a perfect or near-perfect fit for a property:
0 commit comments