Skip to content

Add StateMachineTest framework for JUnit 5#2887

Merged
divang merged 11 commits intomainfrom
dev/divang/FX-migarate-to-Junit-MBT
Feb 18, 2026
Merged

Add StateMachineTest framework for JUnit 5#2887
divang merged 11 commits intomainfrom
dev/divang/FX-migarate-to-Junit-MBT

Conversation

@divang
Copy link
Copy Markdown
Contributor

@divang divang commented Jan 28, 2026

Summary

Introduces a lightweight State Machine Testing (MBT) framework that randomly explores valid JDBC state transitions to discover edge-case bugs. Designed as a simpler, debuggable, seed-reproducible alternative to the existing MBT framework.

Motivation

Hand-written tests miss paths
We only test sequences we think of. Real applications produce unexpected combinations like insert, rollback, insert, commit, rollback, insert, commit -- nobody writes a test for that, yet it may crash the driver.
The existing MBT framework is hard to debug
It uses reflection, annotations, and an internal scheduler. When a test fails, stepping through in a debugger is impractical because execution jumps across framework internals.
Failures are not reproducible
The existing framework has no seed-based replay. A nightly test fails, but running it again produces a different random sequence -- the bug disappears. Engineers waste time trying to reproduce instead of fixing.

Key Design Decisions

Seed-based reproducibility -- Same seed produces the same action sequence on any machine. Failed CI? Paste the seed, set a breakpoint, reproduce on the first try.
Weighted random selection -- executeUpdate (weight 15) fires 3x more often than setAutoCommit (weight 5), simulating realistic workloads.
No third-party dependencies -- Pure JUnit 5 and standard library. No external libraries, no annotations, no reflection.

What This PR Adds

Core Framework (statemachinetest/core/) -- 5 classes, around 200 lines
Action -- Base class with canRun() (precondition), run() (execute), and weight (selection probability)
Engine -- Execution loop with weighted random selection, seed, timeout
StateMachineTest -- Shared state map, action registry, seeded Random
StateKey -- Interface for type-safe enum state keys (no raw strings)
Result -- Outcome containing success, actionCount, seed, log, error

Transaction Tests (statemachinetest/transaction/)
TransactionState enum with keys CONN, AUTO_COMMIT, CLOSED, TABLE_NAME
6 actions: setAutoCommit(true/false), commit, rollback, executeUpdate, executeQuery
TransactionStateTest -- 50 random actions against real SQL Server

ResultSet Tests (statemachinetest/resultset/)
ResultSetState enum with keys RS, CLOSED, ON_VALID_ROW
6 actions: next, previous, first, last, absolute, getString
ResultSetStateTest -- 50 random actions against real SQL Server

Testing

2 test classes, 50 actions each, run against real SQL Server

Migrate the FX (Functional eXtensions) testing framework to a modern
JUnit 5 Model-Based Testing (MBT) framework.

New Features:
- Core MBT Framework:
  - Model: Base class for state machine models with reflection-based discovery
  - ModelEngine: Executes state machine exploration with reproducible seeds
  - @ModelAction: Annotation for marking state transition methods
  - @ModelRequirement: Annotation for defining preconditions
  - @ModelVariable: Annotation for tracking state variables
  - ModelEngineOptions: Configuration for timeout, maxActions, weightSchemes
  - WeightScheme: Multiple action selection strategies (flat, weighted, etc.)

- Connection Testing:
  - ConnectionModel: State machine for java.sql.Connection (equiv. to fxConnection)
  - ConnectionModelTest: JUnit 5 tests (equiv. to connection.java)
  - ConnectionProperty: Property definitions for connection strings
  - SqlServerDriverProperties: Driver property catalog (equiv. to fxSqlServerDriver)

- JUnit 5 Integration:
  - @modeltest: Declarative test configuration annotation
  - ModelTestExtension: JUnit 5 extension for automatic seed handling

Key Benefits:
- Seed-based reproducibility: Failed tests can be reproduced exactly
- Auto-exploration: Discovers edge cases through state machine traversal
@github-project-automation github-project-automation Bot moved this to In progress in MSSQL JDBC Jan 28, 2026
@divang divang self-assigned this Jan 28, 2026
@divang divang marked this pull request as draft January 28, 2026 11:24
Replace complex Model-Based Testing framework with simplified
StateMachineTest approach for better maintainability.

Removed (14 files):
- mbt/core/: Model, ModelEngine, ModelAction, ModelRequirement,
  ModelVariable, ModelEngineOptions, WeightScheme, ModelTest,
  ModelTestExtension, ModelRequirements
- mbt/connection/: ConnectionModel, ConnectionModelTest,
  ConnectionProperty, SqlServerDriverProperties

Added (2 files):
- statemachinetest/core/StateMachineTest.java
- statemachinetest/resultset/ResultSetStateTest.java

Key simplifications:
- Single base class instead of multiple annotations
- Built-in state tracking without complex reflection
- Seed-based reproducibility preserved
- Cleaner API for writing state machine tests
@divang divang changed the title Add Model-Based Testing (MBT) framework for JUnit 5 Add StateMachineTest framework for JUnit 5 Jan 30, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented Jan 30, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.59%. Comparing base (26c55e1) to head (d14fd0c).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #2887      +/-   ##
============================================
+ Coverage     60.55%   60.59%   +0.04%     
- Complexity     4887     4895       +8     
============================================
  Files           151      151              
  Lines         34827    34827              
  Branches       5834     5834              
============================================
+ Hits          21088    21104      +16     
+ Misses        10907    10899       -8     
+ Partials       2832     2824       -8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Add comprehensive Transaction State Machine tests covering FX framework scenarios:

Transaction Operations:
- Basic transaction control (setAutoCommit, commit, rollback)
- Savepoint operations (setSavepoint, rollback to savepoint, releaseSavepoint)
- Transaction isolation level testing (READ_UNCOMMITTED to SERIALIZABLE)
- Real database transaction tests with commit/rollback verification

FX Missing Test Coverage:
- testCommitTransaction - Basic transaction commit
- testRollbackTransaction - Basic transaction rollback
- testRollbackTransactionOnCnClose - Implicit rollback on connection close
- testRollbackTransactionOnPoolCnClose - Pool connection rollback simulation

XA Transaction Tests (simulated):
- testTransactionAfterPrepare - XA two-phase commit prepare
- testCaseCleanMultipleSuspend_Join - XA suspend/resume (TMSUSPEND/TMJOIN)
- testCaseClean - XA cleanup and recovery
- testXATimeout - XA transaction timeout handling

Connection Operations:
- Statement creation (createStatement, prepareStatement, prepareCall)
- Connection configuration (holdability, catalog, warnings)
- DatabaseMetaData retrieval
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a lightweight model-based testing (MBT) harness for JUnit 5 and ports JDBC transaction + ResultSet scenarios into state-machine style tests as a replacement for the FX/KoKoMo approach.

Changes:

  • Introduced a small StateMachineTest framework with weighted random action execution and seed-based runs.
  • Added transaction-focused MBT tests (simulated + real DB) covering commit/rollback, savepoints, isolation, and related connection behaviors.
  • Added ResultSet MBT tests (simulated + real DB) covering scrolling, updatable operations, forward-only behavior, and metadata/state checks.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 12 comments.

File Description
src/test/java/com/microsoft/sqlserver/jdbc/statemachinetest/core/StateMachineTest.java New MBT engine/framework with weighted action selection, run result tracking, and logging.
src/test/java/com/microsoft/sqlserver/jdbc/statemachinetest/transaction/TransactionStateTest.java New transaction/connection MBT scenarios, including real DB commit/rollback and simulated XA/pool flows.
src/test/java/com/microsoft/sqlserver/jdbc/statemachinetest/resultset/ResultSetStateTest.java New ResultSet MBT scenarios for navigation, updates, forward-only constraints, and real DB cursor operations.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…bility

- Remove all simulated test cases, keep only real JDBC database tests
- Add seeded Random support via sm.getRandom() for full reproducibility
- Engine now sets seeded Random on StateMachineTest before execution
- Update RealExecuteUpdateAction and RealAbsoluteAction to use shared RNG
- Fix getValidActions() to log exceptions instead of silently dropping actions
- Add onValidRow state tracking for ResultSet cursor position
- Tighten RealGetStringAction.canRun() to only allow when on valid row
- Remove try-catch from RealGetStringAction to let exceptions propagate
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 8 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/test/java/com/microsoft/sqlserver/jdbc/statemachinetest/core/Engine.java Outdated
Comment thread src/test/java/com/microsoft/sqlserver/jdbc/statemachinetest/core/Engine.java Outdated
Comment thread src/test/java/com/microsoft/sqlserver/jdbc/statemachinetest/core/Engine.java Outdated
- Make ResultSetStateTest and TransactionStateTest public for consistency
- Add maxActions validation (n >= 1) with IllegalArgumentException
- Handle total=0 case in weighted selection with uniform random fallback
- Fix printf format issue for state map (avoid % interpretation)
- Improve error message to show exception type, handle null message, print stack trace
…ate action classes

- Add StateKey interface for type-safe state keys
- Create TransactionState and ResultSetState enums replacing string constants
- Extract action classes to TransactionActions.java and ResultSetActions.java
- Rename state methods: setsetState, getgetStateValue, isisState
- Add stateMachine tag constant to Constants.java
- Combine Engine log output into single line

This improves compile-time safety, reduces code duplication.
Copy link
Copy Markdown
Collaborator

@David-Engel David-Engel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like this!

@divang divang marked this pull request as ready for review February 10, 2026 05:44
Comment thread src/test/java/com/microsoft/sqlserver/testframework/Constants.java Outdated
@divang
Copy link
Copy Markdown
Contributor Author

divang commented Feb 17, 2026

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 3 pipeline(s).

machavan
machavan previously approved these changes Feb 18, 2026
Ananya2
Ananya2 previously approved these changes Feb 18, 2026
@divang divang dismissed stale reviews from Ananya2 and machavan via d14fd0c February 18, 2026 07:02
@divang divang merged commit 20ed864 into main Feb 18, 2026
8 of 19 checks passed
@github-project-automation github-project-automation Bot moved this from In progress to Closed/Merged PRs in MSSQL JDBC Feb 18, 2026
@divang divang added this to the 13.3.2 milestone Feb 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Closed/Merged PRs

Development

Successfully merging this pull request may close these issues.

5 participants