Skip to content

Commit 3fc38f1

Browse files
dgutsonclaude
andcommitted
claude: teach /hypothesis about stateful tests and @example
The command's property catalogue had no entry for stateful properties, so the model would not propose a RuleBasedStateMachine even for classes whose methods obviously mutate them, and order-dependent bugs went unreached. Step 4 also stopped at confirming a bug, leaving the falsifying input to be rediscovered on each run rather than pinned with @example. Both features are already documented on reference/api.html, which the reference section links but described only as "@settings, @given, etc." Co-Authored-By: Claude Opus 5 <[email protected]>
1 parent 16f24b7 commit 3fc38f1

2 files changed

Lines changed: 6 additions & 1 deletion

File tree

‎.claude/commands/hypothesis.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ Here are some examples of typical properties:
4242
- Confluence: the order of function application doesn't matter (for example, in compiler optimization passes).
4343
- Metamorphic property: some relationship between `f(x)` and `g(x)` holds for all x. For example, `sin(π − x) = sin(x)`.
4444
- Single entry point. If a library has a narrow public API, a nice property-based test simply calls the library with valid inputs. Common in parsers.
45+
- Stateful property: a sequence of operations preserves an invariant, or agrees with a simpler model. The signal is a class whose methods mutate it — `push`/`pop`, `add`/`remove`, `open`/`close`, incremental update. These bugs depend on the *order* of calls, so no single-call property reaches them.
4546

4647
While the following should generally not be tested:
4748

@@ -72,18 +73,21 @@ When writing Hypothesis tests, follow these guidelines:
7273

7374
- Each Hypothesis test should be both sound (tests only inputs the code can actually be called with) and complete (tests all inputs the code can actually be called with). Sometimes this is difficult. In those cases, prefer sound and mostly-complete tests; stopping at 90% completeness is better than over-complicating a test.
7475
- Only place constraints on Hypothesis strategies if required by the code. For example, prefer `st.lists(...)` (with no size bound) to `st.lists(..., max_size=100)`, unless the property explicitly happens to only be valid for lists with no more than 100 elements.
76+
- For a stateful property, write a `RuleBasedStateMachine` with `@rule`, `@invariant`, and `@precondition`, and expose it to the test runner as `TestFoo = FooMachine.TestCase`. Keep the model as simple as possible — usually a plain `list` or `dict`. A model that reimplements the logic under test is not an independent oracle, and will agree with the code even when the code is wrong.
7577

7678
## 4. Run the new Hypothesis tests, and reflect on the result.
7779

7880
Run the new Hypothesis tests that you just added. If any fail, reflect on why. Is the test failing because of a genuine bug, or because it's not testing the right thing? Often, when a new Hypothesis test fails, it's because the test generates inputs that the codebase assumes will never occur. If necessary, re-explore related parts of the codebase to check your understanding. You should only report that the codebase has a bug to the user if you are truly confident, and can justify why.
7981

82+
Once you have confirmed a genuine bug, add the minimal falsifying input to the property test as an `@example(...)`, so that the specific regression is checked on every run in addition to the generated inputs.
83+
8084
# Hypothesis Reference
8185

8286
Documentation reference (fetch with the `WebFetch` tool if required):
8387

8488
- **Strategies API reference**: https://hypothesis.readthedocs.io/en/latest/reference/strategies.html
8589
- **Other API reference**: https://hypothesis.readthedocs.io/en/latest/reference/api.html
86-
- Documents `@settings`, `@given`, etc.
90+
- Documents `@settings`, `@given`, `@example`, and stateful testing (`RuleBasedStateMachine`, `@rule`, `@invariant`, `@precondition`), among others.
8791

8892
These Hypothesis strategies are under-appreciated for how effective they are. Use them if they are a perfect or near-perfect fit for a property:
8993

‎AUTHORS.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@ their individual contributions.
5151
* `Cory Benfield <https://www.github.com/Lukasa>`_
5252
* `Cristi Cobzarenco <https://github.com/cristicbz>`_ ([email protected])
5353
* `Damon Francisco <https://github.com/dtfrancisco>`_ ([email protected])
54+
* `Daniel Gutson <https://github.com/dgutson>`_
5455
* `Daniel J. West <https://github.com/danieljwest>`_
5556
* `Daniel Knell <https://github.com/danielknell>`_ ([email protected])
5657
* `David Bonner <https://github.com/rascalking>`_ ([email protected])

0 commit comments

Comments
 (0)