Add StateMachineTest framework for JUnit 5#2887
Conversation
Migrate the FX (Functional eXtensions) testing framework to a modern JUnit 5 Model-Based Testing (MBT) framework. New Features: - Core MBT Framework: - Model: Base class for state machine models with reflection-based discovery - ModelEngine: Executes state machine exploration with reproducible seeds - @ModelAction: Annotation for marking state transition methods - @ModelRequirement: Annotation for defining preconditions - @ModelVariable: Annotation for tracking state variables - ModelEngineOptions: Configuration for timeout, maxActions, weightSchemes - WeightScheme: Multiple action selection strategies (flat, weighted, etc.) - Connection Testing: - ConnectionModel: State machine for java.sql.Connection (equiv. to fxConnection) - ConnectionModelTest: JUnit 5 tests (equiv. to connection.java) - ConnectionProperty: Property definitions for connection strings - SqlServerDriverProperties: Driver property catalog (equiv. to fxSqlServerDriver) - JUnit 5 Integration: - @modeltest: Declarative test configuration annotation - ModelTestExtension: JUnit 5 extension for automatic seed handling Key Benefits: - Seed-based reproducibility: Failed tests can be reproduced exactly - Auto-exploration: Discovers edge cases through state machine traversal
Replace complex Model-Based Testing framework with simplified StateMachineTest approach for better maintainability. Removed (14 files): - mbt/core/: Model, ModelEngine, ModelAction, ModelRequirement, ModelVariable, ModelEngineOptions, WeightScheme, ModelTest, ModelTestExtension, ModelRequirements - mbt/connection/: ConnectionModel, ConnectionModelTest, ConnectionProperty, SqlServerDriverProperties Added (2 files): - statemachinetest/core/StateMachineTest.java - statemachinetest/resultset/ResultSetStateTest.java Key simplifications: - Single base class instead of multiple annotations - Built-in state tracking without complex reflection - Seed-based reproducibility preserved - Cleaner API for writing state machine tests
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2887 +/- ##
============================================
+ Coverage 60.55% 60.59% +0.04%
- Complexity 4887 4895 +8
============================================
Files 151 151
Lines 34827 34827
Branches 5834 5834
============================================
+ Hits 21088 21104 +16
+ Misses 10907 10899 -8
+ Partials 2832 2824 -8 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Add comprehensive Transaction State Machine tests covering FX framework scenarios: Transaction Operations: - Basic transaction control (setAutoCommit, commit, rollback) - Savepoint operations (setSavepoint, rollback to savepoint, releaseSavepoint) - Transaction isolation level testing (READ_UNCOMMITTED to SERIALIZABLE) - Real database transaction tests with commit/rollback verification FX Missing Test Coverage: - testCommitTransaction - Basic transaction commit - testRollbackTransaction - Basic transaction rollback - testRollbackTransactionOnCnClose - Implicit rollback on connection close - testRollbackTransactionOnPoolCnClose - Pool connection rollback simulation XA Transaction Tests (simulated): - testTransactionAfterPrepare - XA two-phase commit prepare - testCaseCleanMultipleSuspend_Join - XA suspend/resume (TMSUSPEND/TMJOIN) - testCaseClean - XA cleanup and recovery - testXATimeout - XA transaction timeout handling Connection Operations: - Statement creation (createStatement, prepareStatement, prepareCall) - Connection configuration (holdability, catalog, warnings) - DatabaseMetaData retrieval
There was a problem hiding this comment.
Pull request overview
Adds a lightweight model-based testing (MBT) harness for JUnit 5 and ports JDBC transaction + ResultSet scenarios into state-machine style tests as a replacement for the FX/KoKoMo approach.
Changes:
- Introduced a small
StateMachineTestframework with weighted random action execution and seed-based runs. - Added transaction-focused MBT tests (simulated + real DB) covering commit/rollback, savepoints, isolation, and related connection behaviors.
- Added ResultSet MBT tests (simulated + real DB) covering scrolling, updatable operations, forward-only behavior, and metadata/state checks.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 12 comments.
| File | Description |
|---|---|
| src/test/java/com/microsoft/sqlserver/jdbc/statemachinetest/core/StateMachineTest.java | New MBT engine/framework with weighted action selection, run result tracking, and logging. |
| src/test/java/com/microsoft/sqlserver/jdbc/statemachinetest/transaction/TransactionStateTest.java | New transaction/connection MBT scenarios, including real DB commit/rollback and simulated XA/pool flows. |
| src/test/java/com/microsoft/sqlserver/jdbc/statemachinetest/resultset/ResultSetStateTest.java | New ResultSet MBT scenarios for navigation, updates, forward-only constraints, and real DB cursor operations. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…bility - Remove all simulated test cases, keep only real JDBC database tests - Add seeded Random support via sm.getRandom() for full reproducibility - Engine now sets seeded Random on StateMachineTest before execution - Update RealExecuteUpdateAction and RealAbsoluteAction to use shared RNG - Fix getValidActions() to log exceptions instead of silently dropping actions - Add onValidRow state tracking for ResultSet cursor position - Tighten RealGetStringAction.canRun() to only allow when on valid row - Remove try-catch from RealGetStringAction to let exceptions propagate
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 8 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Make ResultSetStateTest and TransactionStateTest public for consistency - Add maxActions validation (n >= 1) with IllegalArgumentException - Handle total=0 case in weighted selection with uniform random fallback - Fix printf format issue for state map (avoid % interpretation) - Improve error message to show exception type, handle null message, print stack trace
…ate action classes - Add StateKey interface for type-safe state keys - Create TransactionState and ResultSetState enums replacing string constants - Extract action classes to TransactionActions.java and ResultSetActions.java - Rename state methods: setsetState, getgetStateValue, isisState - Add stateMachine tag constant to Constants.java - Combine Engine log output into single line This improves compile-time safety, reduces code duplication.
David-Engel
left a comment
There was a problem hiding this comment.
I really like this!
Co-authored-by: David Engel <[email protected]>
|
/azp run |
|
Azure Pipelines successfully started running 3 pipeline(s). |
Summary
Introduces a lightweight State Machine Testing (MBT) framework that randomly explores valid JDBC state transitions to discover edge-case bugs. Designed as a simpler, debuggable, seed-reproducible alternative to the existing MBT framework.
Motivation
Hand-written tests miss paths
We only test sequences we think of. Real applications produce unexpected combinations like insert, rollback, insert, commit, rollback, insert, commit -- nobody writes a test for that, yet it may crash the driver.
The existing MBT framework is hard to debug
It uses reflection, annotations, and an internal scheduler. When a test fails, stepping through in a debugger is impractical because execution jumps across framework internals.
Failures are not reproducible
The existing framework has no seed-based replay. A nightly test fails, but running it again produces a different random sequence -- the bug disappears. Engineers waste time trying to reproduce instead of fixing.
Key Design Decisions
Seed-based reproducibility -- Same seed produces the same action sequence on any machine. Failed CI? Paste the seed, set a breakpoint, reproduce on the first try.
Weighted random selection -- executeUpdate (weight 15) fires 3x more often than setAutoCommit (weight 5), simulating realistic workloads.
No third-party dependencies -- Pure JUnit 5 and standard library. No external libraries, no annotations, no reflection.
What This PR Adds
Core Framework (statemachinetest/core/) -- 5 classes, around 200 lines
Action -- Base class with canRun() (precondition), run() (execute), and weight (selection probability)
Engine -- Execution loop with weighted random selection, seed, timeout
StateMachineTest -- Shared state map, action registry, seeded Random
StateKey -- Interface for type-safe enum state keys (no raw strings)
Result -- Outcome containing success, actionCount, seed, log, error
Transaction Tests (statemachinetest/transaction/)
TransactionState enum with keys CONN, AUTO_COMMIT, CLOSED, TABLE_NAME
6 actions: setAutoCommit(true/false), commit, rollback, executeUpdate, executeQuery
TransactionStateTest -- 50 random actions against real SQL Server
ResultSet Tests (statemachinetest/resultset/)
ResultSetState enum with keys RS, CLOSED, ON_VALID_ROW
6 actions: next, previous, first, last, absolute, getString
ResultSetStateTest -- 50 random actions against real SQL Server
Testing
2 test classes, 50 actions each, run against real SQL Server